Python多处理映射函数错误

Question

我有一个简单的多处理示例,我正在尝试创建.普通的map()函数版本有效,但是当改为Pool.map时,我收到一个奇怪的错误:

from multiprocessing import Pool
from functools import partial
x = [1,2,3]
y = 10
f = lambda x,y: x**2+y

# ordinary map works:
map(partial(f,y=y),x)
# [11, 14, 19]

# multiprocessing map does not
p = Pool(4)
p.map(partial(f, y=y), x)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

酸洗错误？这究竟是什么？

Answer 1

Pool.map必须可以选择的参数.模块级函数是可选择的,但未partial(f, y=y)在模块级定义,因此不可选.

有一个简单的解决方法:

def g(x, y=y):
    return f(x, y)

p.map(g, x)

functools.partial以前的功能是不可用的.但是,使用Python2.7或更高版本,您还可以g使用functools.partial 定义(在模块级别):

import multiprocessing as mp
import functools

def f(x, y):
    return x**2 + y

x = [1,2,3]
y = 10

g = functools.partial(f, y=y)

if __name__ == '__main__':
    p = mp.Pool()
    print(p.map(g, x))

收益率[11, 14, 19].但请注意,f必须定义此结果def而不是lambda.我认为这是因为pickle依赖"完全限定"的名称引用来查找函数对象值.