将接受类成员函数作为变量的函数传递给python multiprocess pool.map()

Question

嗨,我一直在为这个早上的大部分时间而苦苦挣扎,希望有人能指出我正确的方向.

这是我目前的代码:

def f(tup):
    return some_complex_function(*tup)

def main():

    pool = Pool(processes=4) 
    #import and process data omitted 
    _args = [(x.some_func1, .05, x.some_func2) for x in list_of_some_class]
    results = pool.map(f, _args)
    print results

我得到的第一个错误是:

> Exception in thread Thread-2: Traceback (most recent call last):  
> File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
>     self.run()   File "/usr/lib/python2.7/threading.py", line 504, in run
>     self.__target(*self.__args, **self.__kwargs)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in
> _handle_tasks
>     put(task) PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

任何帮助将非常感激.

Answer 1

该multiprocess模块使用该pickle模块序列化传递给function(f)的参数,该参数在另一个进程中执行.

许多内置类型都可以进行pickle,但实例方法无法进行pickle.所以.05很好,但x.some_func1不是.看什么可以腌制和去腌制？更多细节.

没有简单的解决方案.您需要重新构建程序,因此实例方法不需要作为参数传递(或避免使用multiprocess).