我目前正在尝试使用带有多处理后端的 joblib 库在 python 3.5 中运行并行进程。但是,每次运行时都会出现此错误:
Process ForkServerPoolWorker-5:
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/opt/anaconda3/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/opt/anaconda3/lib/python3.5/multiprocessing/pool.py", line 108, in worker
task = get()
File "/opt/anaconda3/lib/python3.5/site-packages/joblib/pool.py", line 362, in get
return recv()
File "/opt/anaconda3/lib/python3.5/multiprocessing/connection.py", line 251, in recv
return ForkingPickler.loads(buf.getbuffer())
TypeError: __new__() missing 1 required positional argument: 'path'
Run Code Online (Sandbox Code Playgroud)
这是我用来运行它的 joblib 代码:
from joblib import Parallel, delayed
results = Parallel(n_jobs=6) (
delayed(func)(i) for i in array)
Run Code Online (Sandbox Code Playgroud)
默认情况下,后端是多处理的。当我将后端更改为“线程”时,代码运行良好,但对于我的用例,与使用多处理相比,线程效率低下。
我还尝试使用以下代码直接使用多处理,但我仍然遇到相同的错误: …
python runtime-error python-multithreading joblib python-multiprocessing