cmu-delphi/delphi-epidata

Remove footgun from meta cache update

Open

#776 opened on Dec 2, 2021

 (2 comments) (0 reactions) (0 assignees)Python (67 forks)auto 404
acquisitionbugdata qualityhelp wanted

Repository metrics

Stars
 (104 stars)
PR merge metrics
 (PR metrics pending)

Description

So this happened:

Traceback (most recent call last):
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 216, in _open_connection
   self._cmysql.connect(**cnx_kwargs)
_mysql_connector.MySQLInterfaceError: Too many connections
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 932, in _bootstrap_inner
   self.run()
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 870, in run
   self._target(*self._args, **self._kwargs)
 File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 327, in worker
   worker_dbc.connect(connector_impl=self._connector_impl)
 File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 89, in connect
   self._connection = self._connector_impl.connect(
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/__init__.py", line 264, in connect
   return CMySQLConnection(*args, **kwargs)
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 80, in __init__
   self.connect(**kwargs)
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/abstracts.py", line 966, in connect
   self._open_connection()
 File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 218, in _open_connection
   raise errors.get_mysql_exception(msg=exc.msg, errno=exc.errno,
mysql.connector.errors.OperationalError: 1040 (08004): Too many connections

It turns out all 7 worker threads dropped out with this exception, and while in theory srcsigs.join() blocks until all items have been retrieved and completed, we do straight-up overwrite meta with the results no matter what they are, which makes us nervous.

Let's try:

  • finite retry to open connections
  • some basic checks to make sure we don't accidentally overwrite meta with a blank list

Contributor guide