芹菜:远程工作者经常失去联系

Question

我有一个在云服务器(Django应用程序)上运行的Celery代理,我办公室的本地服务器上的两个工作人员连接在NAT后面.本地工作人员经常失去连接,必须重新启动以重新建立与代理的连接.通常celeryd restart在我第一次尝试时挂起,所以我必须ctr + C并重试一次或两次以使其备份并连接.工人记录两个最常见的错误:

[2014-08-03 00:08:45,398: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 796, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/loops.py", line 72, in asynloop
    next(loop)
  File "/usr/local/lib/python2.7/dist-packages/kombu/async/hub.py", line 320, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 159, in on_readable
    reader(loop)
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 142, in _read
    raise ConnectionError('Socket was disconnected')
ConnectionError: Socket was disconnected

[2014-03-07 20:15:41,963: CRITICAL/MainProcess] Couldn't ack 11, reason:RecoverableConnectionError(None, 'connection already closed', None, '')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kombu/message.py", line 93, in ack_log_error
    self.ack()
  File "/usr/local/lib/python2.7/dist-packages/kombu/message.py", line 88, in ack
    self.channel.basic_ack(self.delivery_tag)
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1583, in basic_ack
    self._send_method((60, 80), args)
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 50, in _send_method
    raise RecoverableConnectionError('connection already closed')

我该如何调试呢？工人是NAT背后的问题吗？是否有一个很好的工具来监控工人是否失去联系？至少有了这个,我可以通过手动重启工作人员让他们恢复在线状态.

Answer 1

不幸的是，Celery+Kombu 中的延迟确认存在问题 - 任务处理程序尝试使用关闭的连接。我像这样解决了这个问题：

CELERY_CONFIG = {
    'CELERYD_MAX_TASKS_PER_CHILD': 1,
    'CELERYD_PREFETCH_MULTIPLIER': 1,
    'CELERY_ACKS_LATE': True,
}

CELERYD_MAX_TASKS_PER_CHILD - 保证工作线程在完成任务后将重新启动。

对于已经失去连接的任务，你现在无能为力。也许它会在版本 4 中得到修复。我只是确保任务尽可能是幂等的。