将文件实例作为参数传递给celery任务会引发"ValueError:对已关闭文件的I/O操作"

Question

我需要将文件作为参数传递给celery任务,但传递的文件以某种方式关闭了.它发生在我正在执行任务异步方式的情况下.这是预期的行为吗？

观点:

from engine.tasks import s3_upload_handler
def myfunc():
    f = open('/app/uploads/pic.jpg', 'rb')
    s3_file_handler.apply_async(kwargs={"uploaded_file" : f,"file_name" : "test.jpg"})

任务:

def s3_upload_handler(uploaded_file,file_name):
    ...
    #some code for uploading to s3

追溯:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/photohosting/engine/tasks.py", line 34, in s3_upload_handler
    key.set_contents_from_file(uploaded_file)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1217, in set_contents_from_file
    spos = fp.tell()
ValueError: I/O operation on closed file

花原木:

kwargs  {
         'file_name': 'test.jpg', 
         'uploaded_file': <closed file '<uninitialized file>', 
          mode '<uninitialized file>' at 0x7f6ab9e75e40>
         }

Answer 1

是的,当然,文件将关闭.异步芹菜任务在一个完全独立的进程中运行(此外,它们甚至可以在不同的机器上运行),并且无法将打开的文件传递给它.

您应该在调用任务的过程中关闭文件,然后将其名称传递给文件(如果需要),然后在任务中重新打开它.