acquisitionbugdata qualityhelp wanted
Repository metrics
- Stars
- (104 stars)
- PR merge metrics
- (PR metrics pending)
Description
So this happened:
Traceback (most recent call last):
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 216, in _open_connection
self._cmysql.connect(**cnx_kwargs)
_mysql_connector.MySQLInterfaceError: Too many connections
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 327, in worker
worker_dbc.connect(connector_impl=self._connector_impl)
File "/home/automation/driver/delphi/epidata/acquisition/covidcast/database.py", line 89, in connect
self._connection = self._connector_impl.connect(
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/__init__.py", line 264, in connect
return CMySQLConnection(*args, **kwargs)
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 80, in __init__
self.connect(**kwargs)
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/abstracts.py", line 966, in connect
self._open_connection()
File "/home/automation/.pyenv/versions/3.8.2/lib/python3.8/site-packages/mysql/connector/connection_cext.py", line 218, in _open_connection
raise errors.get_mysql_exception(msg=exc.msg, errno=exc.errno,
mysql.connector.errors.OperationalError: 1040 (08004): Too many connections
It turns out all 7 worker threads dropped out with this exception, and while in theory srcsigs.join() blocks until all items have been retrieved and completed, we do straight-up overwrite meta with the results no matter what they are, which makes us nervous.
Let's try:
- finite retry to open connections
- some basic checks to make sure we don't accidentally overwrite meta with a blank list