将多个参数传递给Python中的pool.map()函数

Question

我需要一些方法来在pool.map()中使用一个接受多个参数的函数.根据我的理解,pool.map()的目标函数只能有一个iterable作为参数,但有没有办法可以传递其他参数？在这种情况下,我需要传递一些配置变量,比如我的Lock()和记录信息到目标函数.

我试图做一些研究,我认为我可以使用部分功能来使其工作？但是我不完全理解这些是如何工作的.任何帮助将不胜感激!这是我想要做的一个简单示例:

def target(items, lock):
    for item in items:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    pool.map(target(PASS PARAMS HERE), iterable)
    pool.close()
    pool.join()

Answer 1

您可以使用functools.partial此(如您所怀疑):

from functools import partial

def target(lock, iterable_item):
    for item in iterable_item:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    l = multiprocessing.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

例:

def f(a, b, c):
    print("{} {} {}".format(a, b, c))

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    a = "hi"
    b = "there"
    func = partial(f, a, b)
    pool.map(func, iterable)
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

输出:

hi there 1
hi there 2
hi there 3
hi there 4
hi there 5

Answer 2

你可以使用地图功能，允许多个参数，一样的叉multiprocessing中发现的pathos。

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> 
>>> def add_and_subtract(x,y):
...   return x+y, x-y
... 
>>> res = Pool().map(add_and_subtract, range(0,20,2), range(-5,5,1))
>>> res
[(-5, 5), (-2, 6), (1, 7), (4, 8), (7, 9), (10, 10), (13, 11), (16, 12), (19, 13), (22, 14)]
>>> Pool().map(add_and_subtract, *zip(*res))
[(0, -10), (4, -8), (8, -6), (12, -4), (16, -2), (20, 0), (24, 2), (28, 4), (32, 6), (36, 8)]

pathos 使您能够轻松地嵌套具有多个输入的层次结构并行映射，因此我们可以扩展示例来演示这一点。

>>> from pathos.multiprocessing import ThreadingPool as TPool
>>> 
>>> res = TPool().amap(add_and_subtract, *zip(*Pool().map(add_and_subtract, range(0,20,2), range(-5,5,1))))
>>> res.get()
[(0, -10), (4, -8), (8, -6), (12, -4), (16, -2), (20, 0), (24, 2), (28, 4), (32, 6), (36, 8)]

更有趣的是，构建一个可以传递到Pool中的嵌套函数。这是可能的，因为pathosusing dill可以在python中序列化几乎所有内容。

>>> def build_fun_things(f, g):
...   def do_fun_things(x, y):
...     return f(x,y), g(x,y)
...   return do_fun_things
... 
>>> def add(x,y):
...   return x+y
... 
>>> def sub(x,y):
...   return x-y
... 
>>> neato = build_fun_things(add, sub)
>>> 
>>> res = TPool().imap(neato, *zip(*Pool().map(neato, range(0,20,2), range(-5,5,1))))
>>> list(res)
[(0, -10), (4, -8), (8, -6), (12, -4), (16, -2), (20, 0), (24, 2), (28, 4), (32, 6), (36, 8)]

但是，如果您不能超出标准库的范围，则必须采用另一种方法。在这种情况下，最好的选择是使用multiprocessing.starmap如下所示：Python multiprocessing pool.map用于多个参数（在OP帖子的评论中由@Roberto指出）

到达pathos这里：https : //github.com/uqfoundation