Celery with Upstart - processes are dying unexpectedly

Question

When I'm running Celery with Upstart, after a while, the child processes or the main processes die without any trace.

The Upstart script I'm using (/etc/init/celery):

description "celery"

start on runlevel [2345]
stop on runlevel [!2345]

kill timeout 20
# console log
setuid ***
setgid ***

script
chdir /opt/***/project/
exec /opt/***/virtualenvs/***/bin/python manage.py celery worker --settings=settings.staging -B -c 4 -l DEBUG
end script

respawn

When running exectly the same command without upstart (manually running the exec part), everything works fine.

With the respawn stanza, the master process will die and get respawned while the lost child processes still exist, causing memory overflows. Without it the processes will just disappear until there are no workers left.

Celery spawns a master process and worker processes (4 of them in this case). I also tried running it with eventlet instead of multiprocessing (1 master, 1 child process) but the results are similar.

Did anyone encouter such behaviour before?

---

Update:

• Celery when run with -c N starts with N + 2 processes, N of which are workers (what are the other 2?).
• I'm beginning to think that this is related to the expect stanza, but not sure what the value should be. With eventlet expect fork makes sense. But what about multiprocessing?

---

Update2:

Using except fork seemed to stop the processing from dying, but when trying to stop or restart the job it just hangs.

Answer 1

解决方案是不要与 worker 一起运行 Celery beat（-B从 exec 命令中删除部分）。

显然，这是一个“额外”的过程，并且以某种方式把事情搞砸了。

这是我最终得到的最终脚本：

description "celery"

start on started postgresql
stop on runlevel [!2345]

kill timeout 20
setuid ***
setgid ***
respawn

chdir /opt/***/project/
exec /opt/***/virtualenvs/***/bin/python manage.py celery worker --settings=settings.staging -c 4 -l DEBUG

并celery beat单独运行：

description "celerybeat"

start on started celery
stop on stopped celery

setuid ***
setgid ***
respawn

chdir /opt/***/project/
exec /opt/***/virtualenvs/***/bin/python manage.py celery beat --settings=settings.staging -l DEBUG

Answer 2

chdir在script子句中使用显然是错误的，这意味着您无法理解 upstart 中的一个非常基本的概念（无意冒犯）。（作为旁注，exec关键字只是无用，但没有害处）。

有一个想法对于理解 upstart 的运作方式非常重要。Upstart 尝试确定由该script节产生的进程中的哪个进程是该服务的实际守护进程。然后，它使用这个过程来确定这个作业是正在运行、停止还是失败或其他什么。因此，确保流程正确至关重要。

确定过程的算法非常简单，它取决于expect节。expect fork意思是“在script节中取第二个叉子”，expect daemon--相同，但第三个。

现在，使用chdirinsidescript意味着它调用实际的/bin/chdir二进制文件，这算作一个单独的分支。你需要做的是把它移到script节外，然后玩expect节直到你做对了。您可以检查，如果你这样做是正确通过比较器的输出initctl status celery对ps。PID 应该匹配。