将ThreadPoolExecutor与Django一起使用时，数据库“正在被其他用户访问”错误

Question

我正在研究一个项目，在该项目中，我们分析了一个较大的文件并异步处理每一行（我们对每一行进行API调用）ThreadPoolExecutor。过去这是同步完成的，我们有一个通过测试套件。现在，但是，当运行测试时，Django的默认测试运行器错误出现在teardown_databases：

Traceback (most recent call last):
  File "manage.py", line 34, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/test.py", line 29, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/test.py", line 72, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/local/lib/python3.5/site-packages/django/test/runner.py", line 551, in run_tests
    self.teardown_databases(old_config)
  File "/usr/local/lib/python3.5/site-packages/django/test/runner.py", line 526, in teardown_databases
    connection.creation.destroy_test_db(old_name, self.verbosity, self.keepdb)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 264, in destroy_test_db
    self._destroy_test_db(test_database_name, verbosity)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 283, in _destroy_test_db
    % self.connection.ops.quote_name(test_database_name))
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.OperationalError: database "test_sftpm_db" is being accessed by other users
DETAIL:  There are 10 other sessions using the database.

（我们正在使用10名工人）

我尝试在代码中的许多地方手动关闭连接，但无济于事。是否有解决此问题的适当方法？

Answer 1

确保connection.close()在使用线程时调用代码。这为我解决了这个问题，而其他建议的解决方案（测试方法的装饰器、更改数据库设置、上面 Nick 所示的数据库函数、重新启动 postgres）则没有。有用的例子

Answer 2

与数据库的连接是线程本地的。我最终通过向执行器返回的每个 future 添加回调来解决这个问题。

from django.db import connections

def on_done(future):
    # Because each thread has a connection, so here you can call close_all() to close all connections under this thread name.
    connections.close_all()

def main():
    # ...
    with ThreadPoolExecutor() as executor:
        while True:
            future = executor.submit(do, get_a_job())
            future.add_done_callback(on_done)

更多信息请参见：https ://www.programmerought.com/article/3618244269/

另一件事可能是问题是您正在子类化，TestCase它对您的测试持有全局锁定。子类化TransactionTestCase将解决这个问题，并允许您的测试可能产生的任何线程与数据库进行通信

Answer 3

这看起来更像是 Django 的问题，而不是 Postgres 的问题——例如，请参阅此票证： https: //code.djangoproject.com/ticket/22420

从您提供的内容来看，我发现 Django 在尝试删除测试数据库之前并未关闭与测试数据库的所有连接。在这种情况下，Postges 会尝试保护其他会话免遭数据丢失，因此在它们全部断开连接之前它无法删除/重命名数据库。

如果需要，您可以使用已使用的pg_terminate_backend(..)函数和pg_stat_activity视图手动删除测试数据库：

select pg_terminate_backend(pid) 
from pg_stat_activity 
where
  datname = 'DATABASE_NAME'
;

drop database DATABASE_NAME;

如果由于某种原因，有人很快并设法在这两个命令之间进行连接，drop database则会再次失败。在这种情况下，您可以重复它，但在此之前撤销连接到该数据库的权限public——这将阻止与其连接：

revoke connect on database DATABASE_NAME from public;

...然后重复上述操作。