芹菜 - 在一台服务器上运行不同的工作人员

Question

我有两种任务:Type1 - 一些高优先级的小任务.Type2 - 优先级较低的繁重任务.

最初我使用默认路由进行简单配置,但没有使用路由密钥.这还不够 - 有时候所有工作人员都在忙于Type2任务,所以Task1被推迟了.我添加了路由键:

CELERY_DEFAULT_QUEUE = "default"
CELERY_QUEUES = {
    "default": {
        "binding_key": "task.#",
    },
    "highs": {
        "binding_key": "starter.#",
    },
}
CELERY_DEFAULT_EXCHANGE = "tasks"
CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
CELERY_DEFAULT_ROUTING_KEY = "task.default"

CELERY_ROUTES = {
        "search.starter.start": {
            "queue": "highs",
            "routing_key": "starter.starter",
        },
}

所以现在我有2个队列 - 具有高优先级和低优先级的任务.

问题是 - 如何使用不同的并发设置启动2 celeryd？

以前celery用于守护进程模式(根据这个),所以只/etc/init.d/celeryd start 需要启动,但现在我必须运行具有不同队列和并发性的2个不同的芹菜.我该怎么做？

Answer 1

基于上面的答案,我制定了以下/ etc/default/celeryd文件(最初基于此处文档中描述的配置:http://ask.github.com/celery/cookbook/daemonizing.html ),该文件适用于在同一台机器上运行两个芹菜工人,每个工人为不同的队列服务(在这种情况下,队列名称是"默认"和"重要").

基本上这个答案只是前一个答案的扩展,因为它简单地说明了如何做同样的事情,但是对于ceemon模式中的celery.请注意,我们在这里使用django-celery:

CELERYD_NODES="w1 w2"

# Where to chdir at start.
CELERYD_CHDIR="/home/peedee/projects/myproject/myproject"

# Python interpreter from environment.
#ENV_PYTHON="$CELERYD_CHDIR/env/bin/python"
ENV_PYTHON="/home/peedee/projects/myproject/myproject-env/bin/python"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"

# Extra arguments to celeryd
# Longest task: 10 hrs (as of writing this, the UpdateQuanitites task takes 5.5 hrs)
CELERYD_OPTS="-Q:w1 default -c:w1 2 -Q:w2 important -c:w2 2 --time-limit=36000 -E"

# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/celeryd.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"

# celerycam configuration
CELERYEV_CAM="djcelery.snapshot.Camera"
CELERYEV="$ENV_PYTHON $CELERYD_CHDIR/manage.py celerycam"
CELERYEV_LOG_FILE="/var/log/celery/celerycam.log"

# Where to chdir at start.
CELERYBEAT_CHDIR="/home/peedee/projects/cottonon/cottonon"

# Path to celerybeat
CELERYBEAT="$ENV_PYTHON $CELERYBEAT_CHDIR/manage.py celerybeat"

# Extra arguments to celerybeat.  This is a file that will get
# created for scheduled tasks.  It's generated automatically
# when Celerybeat starts.
CELERYBEAT_OPTS="--schedule=/var/run/celerybeat-schedule"

# Log level. Can be one of DEBUG, INFO, WARNING, ERROR or CRITICAL.
CELERYBEAT_LOG_LEVEL="INFO"

# Log file locations
CELERYBEAT_LOGFILE="/var/log/celerybeat.log"
CELERYBEAT_PIDFILE="/var/run/celerybeat.pid"

Answer 2

似乎答案 - 芹菜多 - 目前没有很好地记录.

我需要的可以通过以下命令完成:

celeryd-multi start 2 -Q:1 default -Q:2 starters -c:1 5 -c:2 3 --loglevel=INFO --pidfile=/var/run/celery/${USER}%n.pid --logfile=/var/log/celeryd.${USER}%n.log

我们做的是启动2个工作人员,这些工作人员正在监听不同的队列(-Q:1是默认值,Q:2是启动者),具有不同的并发-c:1 5 -c:2 3

Answer 3

另一种方法是使用-n参数为工作进程提供唯一的名称.

我有两个Pyramid应用程序在同一个物理硬件上运行,每个应用程序都有自己的celery实例(在他们自己的virtualenvs中).

他们都有Supervisor控制他们两个,都有一个独特的supervisord.conf文件.

APP1:

[program:celery]                                            
autorestart=true                                            
command=%(here)s/../bin/celery worker -n ${HOST}.app1--app=app1.queue -l debug
directory=%(here)s     

[2013-12-27 10:36:24,084: WARNING/MainProcess] celery@maz.local.app1 ready.

APP 2:

[program:celery]                                 
autorestart=true                                 
command=%(here)s/../bin/celery worker -n ${HOST}.app2 --app=app2.queue -l debug
directory=%(here)s                               

[2013-12-27 10:35:20,037: WARNING/MainProcess] celery@maz.local.app2 ready.