具有可迭代和多个参数的 Python 多处理

Question

使用多处理，我想传递一个可迭代的和多个参数：

a) 到在 n_core cpu 上运行的函数 b) 一次产生（或返回）n_core 结果 c) 以任何完成顺序

from multiprocessing import Pool 

def func(iterable, args):
    this, that, other = args[0], args[1], args[2]

    for s in iterable:
        return ' '.join([s, this, that, other])        

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, iterable, args):
        print(r)

if __name__ == '__main__':
    main()

预期的结果是：

"abc this that other"
"bcd this that other"
"cde this that other"
"def this that other" 
"efg this that other" 
"fgh this that other"
"ghi this that other" 
"hij this that other"

使这项工作的正确方法是什么？

其次， concurrent.futures.ProcessPoolExecutor 会是这个问题的更好选择吗？

Answer 1

您可以创建一个new_iterable将 中的值iterable与组合在一起的args：

from multiprocessing import Pool

def func(args):
    iterable, this, that, other = args[0], args[1][0], args[1][1], args[1][2]
    return ' '.join([iterable, this, that, other])

def main():
    iterable = ['abc', 'bcd', 'cde', 'def', 'efg', 'fgh', 'ghi', 'hij']
    args = ['this', 'that', 'other']
    new_iterable = ([x, args] for x in iterable)
    n_core = 2

    p = Pool(n_core)
    for r in p.imap_unordered(func, new_iterable):
        print(r)

if __name__ == '__main__':
    main()

输出

abc this that other
bcd this that other
cde this that other
def this that other
efg this that other
fgh this that other
ghi this that other
hij this that other

此解决方案使用生成器表达式来创建一个新的可迭代对象，该可迭代对象将 中的条目iterable与所需的args. 您也可以使用生成器函数来做同样的事情。

更新：我进行了修改func以产生您在评论中提到并添加到您的问题中的预期结果。