Python,使用初始化启动concurrent.futures.ProcessPoolExecutor？

Question

我打算用来concurrent.futures.ProcessPoolExecutor并行执行函数.根据文档,它的executor对象只能接受一个简单的函数map.我的实际情况涉及在执行'待并行化'功能之前初始化(加载数据).我该如何安排？

"待并行化"函数在迭代中被多次调用.我不希望每次都重新初始化它.

换句话说,有一个init函数可以为这个tbp函数产生一些输出.每个孩子都应该有自己的输出副本,因为功能取决于它.

Answer 1

从 开始Python 3.7， theThreadPoolExecutor和 theProcessPoolExecutor都有可选的initializer和initargs参数。每个线程/进程initializer(*initargs)在启动后都会调用。

请参阅https://docs.python.org/3.7/library/concurrent.futures.html。

Answer 2

听起来你正在寻找与所需的initializer/ initargs选项相当的东西multiprocessing.Pool.目前,该行为不存在concurrent.futures.ProcessPoolExecutor,但有一个补丁等待审查添加该行为.

所以,您可以使用multiprocessing.Pool(这可能适用于您的用例),等待该补丁合并和释放(您可能正在等待一段时间:)),或者推出自己的解决方案.事实证明,为地图编写一个包装函数并不太难initializer,但每个进程只调用一个:

from concurrent.futures import ProcessPoolExecutor
from functools import partial

inited = False
initresult = None

def initwrapper(initfunc, initargs, f, x):
    # This will be called in the child. inited
    # Will be False the first time its called, but then
    # remain True every other time its called in a given
    # worker process.
    global inited, initresult
    if not inited:
        inited = True
        initresult = initfunc(*initargs)
    return f(x)

def do_init(a,b):
    print('ran init {} {}'.format(a,b))
    return os.getpid() # Just to demonstrate it will be unique per process

def f(x):
    print("Hey there {}".format(x))
    print('initresult is {}'.format(initresult))
    return x+1

def initmap(executor, initializer, initargs, f, it):
    return executor.map(partial(initwrapper, initializer, initargs, f), it)


if __name__ == "__main__":
    with ProcessPoolExecutor(4) as executor:
        out = initmap(executor, do_init, (5,6), f, range(10))
    print(list(out))

输出:

ran init 5 6
Hey there 0
initresult is 4568
ran init 5 6
Hey there 1
initresult is 4569
ran init 5 6
Hey there 2
initresult is 4570
Hey there 3
initresult is 4569
Hey there 4
initresult is 4568
ran init 5 6
Hey there 5
initresult is 4571
Hey there 6
initresult is 4570
Hey there 7
initresult is 4569
Hey there 8
initresult is 4568
Hey there 9
initresult is 4570
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]