在python中并行运行带有位置和可选参数的函数(跟进)

Question

这是一个后续问题:Python:如何并行运行python函数？

最小工作范例:

'''
Created on 06.05.2015
https://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel
'''
from multiprocessing import Process
import time

def runInParallel(*fns):
    proc = []
    for fn in fns:
        p = Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

def func1():
    s=time.time()
    print 'func1: starting', s 
    for i in xrange(1000000000):
        if i==i:
            pass
    e = time.time()
    print 'func1: finishing', e
    print 'duration', e-s

if __name__ == '__main__':
    s =time.time()
    runInParallel(func1, func1, func1, func1, func1)
    print time.time()-s

哪个利用了(这正是我想要的):

func1:从1430920678.09开始

func1:起始于1430920678.53

func1:从1430920679.02开始

func1:起始于1430920679.57

func1:从1430920680.55开始

func1:完成1430920729.68

持续时间51.1449999809

func1:完成1430920729.78

持续时间51.6889998913

func1:完成1430920730.69

持续时间51.1239998341

func1:完成1430920748.64

持续时间69.6180000305

func1:完成1430920749.25

持续时间68.7009999752

71.5629999638

但是,我的函数有很多参数,所以我测试它是这样的:

- > func1(a)现在获得一个参数传递.

'''
Created on 06.05.2015
https://stackoverflow.com/questions/7207309/python-how-can-i-run-python-functions-in-parallel
'''
from multiprocessing import Process
import time

def runInParallel(*fns):
    proc = []
    for fn in fns:
        p = Process(target=fn)
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

def func1(a):
    s=time.time()
    print 'func1: starting', s 
    for i in xrange(a):
        if i==i:
            pass
    e = time.time()
    print 'func1: finishing', e
    print 'duration', e-s

if __name__ == '__main__':
    s =time.time()
    g=s
    runInParallel(func1(1000000000), func1(1000000000),
                  func1(1000000000), func1(1000000000),
                  func1(1000000000))
    print time.time()-s

所以现在这发生了:

func1:起始于1430921299.08

func1:完成1430921327.84

持续时间28.760999918

func1:起始1430921327.84

func1:完成1430921357.68

持续时间29.8410000801

func1:起始于1430921357.68

func1:完成1430921387.14

持续时间29.4619998932

func1:起始于1430921387.14

func1:完成1430921416.52

持续时间29.3849999905

func1:起始于1430921416.52

func1:完成1430921447.39

持续时间30.864000082

151.392999887

这个过程现在是顺序的,不再是平行的,我不明白为什么!我错过了什么,做错了什么？

编辑:另外,一个例子怎么样,只有一些参数是位置的,其他的是可选的？

Answer 1

您必须将参数传递给Process使用参数args.例如:

def runInParallel(*fns):
    proc = []
    for fn, arg in fns:
        p = Process(target=fn, args=(arg,))
        p.start()
        proc.append(p)
    for p in proc:
        p.join()

然后使用以下命令调用该函数:

runInParallel((func1, 10**9),
              (func1, 10**9),
              (func1, 10**9))

此外,您可以考虑使用池:

from multiprocessing import Pool

pool = Pool()
pool.apply_async(func1, (10**9,))
pool.apply_async(func1, (10**9,))
pool.apply_async(func1, (10**9,))

编辑:

Process并Pool.apply_asynch以同样的方式工作.他们采用两个可选参数args和kwargs.这些是python中位置参数和关键字参数的标准变量:

f(1, 2, a=3, b=4)  # is equivalent to
args, kwargs = (1, 2), {"a":3, "b":4}
f(*args, **kwargs)

同样的例子multiprocessing:

args, kwargs = (1, 2), {"a":3, "b":4}
Process(target=f, args=args, kwargs=kwargs).start()
# Or
pool = Pool()
args, kwargs = (1, 2), {"a":3, "b":4}
pool.apply_async(f, args, kwargs)