发现差异,Celery任务随机失败,没有错误

Question

我的一些远程Celery任务似乎永远不会成为我的经纪人(RabbitMQ).这appears是随机发生的.我的日志中没有错误,它们永远不会发送给工作人员或失败.Flower/Rabbit从不报告任务失败.

我曾经tcpflow -p -c -i eth0 port 5672监控发送任务的API上的流量(client).

当API成功发送任务时,传出流量的记录如下:

(删除敏感数据)

192.018.000.002.42738-052.048.150.171.05672: AMQP
052.048.150.171.05672-192.018.000.002.42738:

capabilitiesFpublisher_confirmstexchange_exchange_bindingst
basic.nacktconsumer_cancel_notifytconnection.blockedtconsumer_prioritiestauthentication_failure_closetper_consumer_qostcluster_nameSrabbit@d8b85eb5ab91copyrightS.Copyright (C) 2007-2015 Pivotal Software, Inc.informationS5Licensed under the MPL.  See http://www.rabbitmq.com/platformS
Erlang/OTPproductSRabbitMQversionS3.6.0PLAIN AMQPLAINen_US
192.018.000.002.42738-052.048.150.171.05672:
nproductSpy-amqpproduct_versionS1.4.9capabilitiesF.connection.blockedtconsumer_cancel_notifytAMQPLAIN1LOGINSusernamePASSWORDSxxxxxxen_US
052.048.150.171.05672-192.018.000.002.42738:
<
192.018.000.002.42738-052.048.150.171.05672:

192.018.000.002.42738-052.048.150.171.05672:
(/
052.048.150.171.05672-192.018.000.002.42738:
)
192.018.000.002.42738-052.048.150.171.05672:

052.048.150.171.05672-192.018.000.002.42738:
192.018.000.002.42738-052.048.150.171.05672: $(
estimate_geometrydirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
estimate_geometry
052.048.150.171.05672-192.018.000.002.42738: 2estimate_geometry
192.018.000.002.42738-052.048.150.171.05672: G2estimate_geometryestimate_geometrytasks.estimate_geometry
052.048.150.171.05672-192.018.000.002.42738: 2
192.018.000.002.42738-052.048.150.171.05672: 1<(estimate_geometrytasks.estimate_geometry
192.018.000.002.42738-052.048.150.171.05672: <application/x-python-serializebinary$021e5308-e6ac-43eb-9a06-8473ba386802$bedeb08f-9614-38b1-9b60-9eded43c3c71
192.018.000.002.42738-052.048.150.171.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.estimate_geometryqUtimelimitqNNUetaqNUkwargsq}qu.
192.018.000.002.42738-052.048.150.171.05672:  (
segment_imagedirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
segment_image
segment_image71.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: ;2
052.048.150.171.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: )<(
192.018.000.002.42738-052.048.150.171.05672: <application/x-python-serializebinary$45280975-9611-41e1-bf99-388cdf1b7064$bedeb08f-9614-38b1-9b60-9eded43c3c71
192.018.000.002.42738-052.048.150.171.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.segment_imageqUtimelimitqNNUetaqNUkwargsq}qu.skq

这是一个从未向经纪人提出的任务,任何人都可以发现差异并告诉我什么是错的？

192.018.000.002.35908-052.017.119.221.05672: 1<(estimate_geometrytasks.estimate_geometry
192.018.000.002.35908-052.017.119.221.05672: <application/x-python-serializebinary$206c1ae0-43d0-4031-bac6-92d2df92b13c$f4c7420c-b9c2-3525-bd5a-d5955f884f43
192.018.000.002.35908-052.017.119.221.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.estimate_geometryqUtimelimitqNNUetaqNUkwargsq}qu.
segment_imagetasks.segment_image9.221.05672: )<(
192.018.000.002.35908-052.017.119.221.05672: <application/x-python-serializebinary$ce0da18a-6534-42d0-9919-cd2e85c8d5e9$f4c7420c-b9c2-3525-bd5a-d5955f884f43
192.018.000.002.35908-052.017.119.221.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.segment_imageqUtimelimitqNNUetaqNUkwargsq}qu.skq

附加信息:

celery_app = Celery('tasks')
celery_app.config_from_object('django.conf:settings')
celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

celery_app.send_task('tasks.estimate_geometry', args=[instance.id], kwargs={})

设置:

import os
from kombu import Exchange, Queue

BROKER_URL = 'amqp://xxxx:xxxx@broker-domain.com:5672//'

CELERY_RESULT_BACKEND = "cache"
CELERY_CACHE_BACKEND = 'memcached://xxxxxx:11211'

CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
CELERY_DEFAULT_ROUTING_KEY = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('estimate_geometry', Exchange('estimate_geometry'), routing_key='tasks.estimate_geometry'),
    Queue('segment_image', Exchange('segment_image'), routing_key='tasks.segment_image'),
    Queue('geometry_feature', Exchange('geometry_feature'), routing_key='tasks.geometry_feature'),
)
CELERY_ROUTES = {
    'tasks.estimate_geometry': {
        'queue': 'estimate_geometry',
        'routing_key': 'tasks.estimate_geometry',
    },
    'tasks.segment_image': {
        'queue': 'segment_image',
        'routing_key': 'tasks.segment_image',
    },
    'tasks.geometry_feature': {
        'queue': 'geometry_feature',
        'routing_key': 'tasks.geometry_feature',
    },
}


BROKER_HEARTBEAT = 10

Answer 1

看来您在调试问题时采取了错误的路径。我看到行：

segment_imagetasks.segment_image0.171.05672: )<(" 

在两个日志中。

如果它是代码的打印输出，那么它可以工作，但不正确。所以问题不在于传输层。

在第一种情况下，我可以看到以下几行：

192.018.000.002.42738-052.048.150.171.05672:  (
segment_imagedirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
segment_image
segment_image71.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: ;2
052.048.150.171.05672-192.018.000.002.42738: 2

所以，有时它运作良好。

您应该使用遇到麻烦的实例 ID 手动运行代码（tasks.estimate_geometry）。

如果是服务器日志：可能是某些 C 库未在服务器上编译，并且您的代码可以在本地运行，但不能在服务器上运行。因此，检查图像格式。