慢芹菜任务时间

Question

我正在使用Django,Celery和RabbitMQ.我有一个发送电子邮件的简单任务.这项任务有效,但速度很慢.

例如,我发送5000封电子邮件,所有5000封电子邮件正常直接进入RabbitMQ,但一旦进入消息代理,它就会花费大约30分钟来完成并清除所有任务.

如果没有Celery,这些相同的任务只需几分钟即可处理所有5000个任务.

我错过了配置的东西吗？如果有人能发现我的速度问题会非常有帮助.

task.py

class SendMessage(Task):
    name = "Sending SMS"
    max_retries = 10
    default_retry_delay = 3

    def run(self, message_id, gateway_id=None, **kwargs):
        logging.debug("About to send a message.")


        try:
            message = Message.objects.get(pk=message_id)
        except Exception as exc:
            raise SendMessage.retry(exc=exc)

        if not gateway_id:
            if hasattr(message.billee, 'sms_gateway'):
                gateway = message.billee.sms_gateway
            else:
                gateway = Gateway.objects.all()[0]
        else:
            gateway = Gateway.objects.get(pk=gateway_id)

        account = Account.objects.get(user=message.sender)
        if account._balance() >= message.length:
            response = gateway._send(message)

            if response.status == 'Sent':
                # Take credit from users account.
                transaction = Transaction(
                    account=account,
                    amount=- message.charge,

                )
                transaction.save()
                message.billed = True
                message.save()
        else:
            pass

settings.py

# Celery
BROKER_URL = 'amqp://admin:xxxxxx@xx.xxx.xxx.xxx:5672//'
CELERY_SEND_TASK_ERROR_EMAILS = True

Apache配置

<VirtualHost *:80>
ServerName www.domain.com

DocumentRoot /srv/project/domain


WSGIDaemonProcess domain.com processes=2 threads=15 display-name=%{GROUP}
WSGIProcessGroup domain.com

WSGIScriptAlias / /srv/project/domain/apache/django.wsgi
ErrorLog /srv/project/logs/error.log
</VirtualHost>

CONF

# Name of nodes to start, here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="w1 w2 w3"

# Where to chdir at start.
CELERYD_CHDIR="/srv/project/domain"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="$CELERYD_CHDIR/manage.py celeryctl"

# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=900 --concurrency=8"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/srv/project/logs/celery/%n.log"
CELERYD_PID_FILE="/srv/project/celery/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="root"
CELERYD_GROUP="root"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="domain.settings"

# Celery Beat Settings.

# Where to chdir at start.
CELERYBEAT_CHDIR="/srv/project/domain"

# Path to celerybeat
CELERYBEAT="$CELERYBEAT_CHDIR/manage.py celerybeat"

Answer 1

你正在处理~2.78个任务/秒(30分钟内5000个任务),我同意这个并不高.您有3个节点,每个节点以8的并发运行,因此您应该能够并行处理24个任务.

要检查的事项:

CELERYD_PREFETCH_MULTIPLIER - 默认设置为4,但如果你有很多短任务,那么增加它是值得的.它将减少从代理获取消息的时间影响,代价是任务不会在工作者之间平均分配.

数据库连接/查询 - 我为成功案例执行了5个以上的数据库查询.如果您使用django-celery的默认结果后端,则会有其他查询将任务结果存储在DB中.django-celery也将在每个任务之后关闭并重新打开数据库连接,这会增加一些开销.如果您有5个查询,每个查询需要100毫秒,那么无论是否使用芹菜,您的任务至少需要500毫秒.单独运行查询是一回事,但您还需要确保任务中没有任何内容锁定表/行,以防止其他任务并行高效运行.

网关响应时间 - 您的任务似乎调用远程服务,我假设它是一个SMS网关.如果该服务器响应缓慢,那么您的任务将很慢.同样,单次调用的响应时间可能与在峰值负载下执行此操作时的响应时间不同.在美国,长码短信只能以每秒1的速率发送,并且取决于网关在哪里进行速率限制,这可能会减慢您的任务速度.