我有一个正常运行时运行良好的 Python 脚本：
$ python script.py <options>
我正在尝试使用 cProfile 模块分析代码：
$ python -m cProfile -o script.prof script.py <options>
当我启动上面的命令时，我收到一个关于无法pickle函数的错误：
Traceback (most recent call last):
File "scripts/process_grid.py", line 1500, in <module>
_compute_write_index(kwrgs)
File "scripts/process_grid.py", line 626, in _compute_write_index
args,
File "scripts/process_grid.py", line 1034, in _parallel_process
pool.map(_apply_along_axis_palmers, chunk_params)
File "/home/james/miniconda3/envs/climate/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/james/miniconda3/envs/climate/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/home/james/miniconda3/envs/climate/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
put(task)
File "/home/james/miniconda3/envs/climate/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File …我multiprocessing正在使用模块,我正在使用UpdateMessage对象(我自己的类),通过multiprocessing.Queue对象发送,在进程之间进行通信.这是类:
class UpdateMessage:
def __init__(self, arrayref, rowslice, colslice, newval):
self.arrayref = arrayref
self.rowslice = rowslice
self.colslice = colslice
self.newval = newval
def do_update(self):
if self.arrayref == 'uL':
arr = uL
elif self.arrayref == 'uR':
arr = uR
else:
raise Exception('UpdateMessage.arrayref neither uL nor uR')
arr[self.rowslice, self.colslice] = self.newval
当我运行脚本时,它完全正常.但是,当我使用cProfile或运行它时profile,它会出现以下错误:
_pickle.PicklingError: Can't pickle <class '__main__.UpdateMessage'>: attribute lookup __main__.UpdateMessage failed
它似乎试图挑选课程,但我不明白为什么会发生这种情况.我的代码没有这样做,没有它就可以正常工作,所以它可能就是multiprocessing模块.但为什么需要腌制UpdateMessage,我该如何修复错误呢？
编辑:这是发送的代码的一部分UpdateMessage(脚本的多个部分执行此操作,但所有都以相同的方式):
msg = UpdateMessage(uLref, refer[0] …考虑以下脚本,我在其中测试两种方法,对通过以下方式获得的生成器执行某些计算itertools.tee:
#!/usr/bin/env python3
from sys import argv
from itertools import tee
from multiprocessing import Process
def my_generator():
for i in range(5):
print(i)
yield i
def double(x):
return 2 * x
def compute_double_sum(iterable):
s = sum(map(double, iterable))
print(s)
def square(x):
return x * x
def compute_square_sum(iterable):
s = sum(map(square, iterable))
print(s)
g1, g2 = tee(my_generator(), 2)
try:
processing_type = argv[1]
except IndexError:
processing_type = "no_multi"
if processing_type == "multi":
p1 = Process(target=compute_double_sum, args=(g1,))
p2 = Process(target=compute_square_sum, args=(g2,))
print("p1 starts") …