我编写了一个使用的小python脚本multiprocessing(参见/sf/answers/2931299801/).它在我测试它时起作用:
$ ./forkiter.py
0
1
2
3
4
sum of x+1: 15
sum of 2*x: 20
sum of x*x: 30
但是当我尝试对其进行分析时cProfile,我会得到以下结果:
$ python3.6 -m cProfile -o forkiter.prof ./forkiter.py
0
1
2
3
4
Traceback (most recent call last):
File "/home/bli/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/bli/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/bli/lib/python3.6/cProfile.py", line 160, in <module>
main()
File "/home/bli/lib/python3.6/cProfile.py", line 153, in main
runctx(code, globs, None, options.outfile, options.sort)
File "/home/bli/lib/python3.6/cProfile.py", …我有一个像这样工作的生成器函数(Python)
def Mygenerator(x, y, z, ...):
while True:
# code that makes two matrices based on sequences of input arrays
yield (matrix1, matrix2)
我想要做的是添加此生成器的输出.这条线完成了这项工作:
M1, M2 = reduce(lambda x, y: x[0] + y[0], x[1] + y[1], Mygenerator(x, y, z, ...))
我想将其并行化以加速计算.重要的是Mygenerator的输出会随着它的产生而减少,因为它list(Mygenerator(...))会占用太多内存.