并行运行单独的进程 - Python

Question

我使用python'multiprocessing'模块在多个内核上运行单个进程,但我希望并行运行几个独立的进程.对于ex - 进程一解析大文件,进程二找不到文件中的模式,进程三进行一些计算; 可以并行运行具有不同参数集的所有三个不同的处理吗？

def Process1(largefile):
    Parse large file
    runtime 2hrs
    return parsed_file

def Process2(bigfile)
    Find pattern in big file
    runtime 2.5 hrs
    return pattern

def Process3(integer)
    Do astronomical calculation
    Run time 2.25 hrs
    return calculation_results

def FinalProcess(parsed,pattern,calc_results):
    Do analysis
    Runtime 10 min
    return final_results

def main():
parsed = Process1(largefile)
pattern = Process2(bigfile)
calc_res = Process3(integer)
Final = FinalProcess(parsed,pattern,calc_res)

if __name__ == __main__:
    main()
    sys.exit()

在上面的伪代码Process1中,Process2和Process3是单核过程,即它们不能在多个处理器上运行.这些过程按顺序进行,取2 + 2.5 + 2.25hrs = 6.75小时.是否可以并行运行这三个过程？这样它们可以在不同的处理器/内核上同时运行,并且当大部分时间(Process2)完成时,我们将转移到Final Process.

我将衷心感谢您的帮助.

AK

Answer 1

从16.6.1.5开始.使用工人池:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
    print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

因此,您可以对池进行apply_async,并在一切准备就绪后获得结果.

from multiprocessing import Pool

# all your methods declarations above go here
# (...)

def main():
    pool = Pool(processes=3)
    parsed = pool.apply_async(Process1, [largefile])
    pattern = pool.apply_async(Process2, [bigfile])
    calc_res = pool.apply_async(Process3, [integer])

    pool.close()
    pool.join()

    final = FinalProcess(parsed.get(), pattern.get(), calc_res.get())

# your __main__ handler goes here
# (...)