使用"PicklingError"无法在Windows 10上启动Celery worker

Question

我有一个在Linux上成功运行的简单测试代码,但它不能在我的Windows 10 x64计算机上运行.

当我试图开始一个芹菜工人时,它抱怨了不可恢复的错误:PicklingError.(芹菜版:3.1.20)

在我的celery配置中,我将序列化设置为'json',但它仍然没有任何帮助.

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

这是完整的错误消息:

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)   
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__
    self._create_worker_process(i)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process
    w.start()
  File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

Answer 1

我遇到了同样的问题.奇怪的是,这个问题只存在于Windows上,在Linux Celery运行时没有任何问题.结果我需要将配置模块作为名称而不是对象传递:

app.config_from_object('celeryconfig')

代替

app.config_from_object(celeryconfig)

来自Celery文档的解释:

小费

建议使用模块名称,因为这意味着在使用prefork池时不需要序列化模块.如果您遇到配置pickle错误,请尝试使用模块的名称.

显然,当Celery作为在Windows上失败的对象传递时,它需要挑选配置.如果作为模块名称传递它是有效的.谢谢你指点我正确的方向@JoyLy!