无法腌制 <class 'MemoryError'>：它与 builtins.MemoryError 不是同一个对象

Question

我在 python 3.7 中使用多进程池映射，我在使用 docker 容器时遇到如下错误

  multiprocessing.pool.MaybeEncoding Error: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'

我正在使用如下代码，

def foo(track):

    track_url = track[1]
    url = requests.get(track_url)
    hashes = fingerprint.fingerprint_worker(
        BytesIO(url.content)
    )
    return True

def worker:
 tracks = [[1, "track_url_1"], [2, "track_url_2"]]

 with multiprocessing.Pool(const.MULTIPROCESSING_POOL_SIZE) as pool:
                pool.map(foo, tracks)
                pool.close()
                pool.terminate()
                pool.join()

MULTIPROCESSING_POOL_SIZE = 1

此错误的完整堆栈跟踪是，

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1836, in __call__
 return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in wsgi_app
 response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1403, in handle_exception
  reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
 raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1817, in wsgi_app
 response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
 rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise 
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/radio-streaming-fingerprint/app.py", line 37, in fingerprint_covers_tracks
resp = fp.fingerprint_tracks(track, generate_fingerprints_with_pool)
File "/radio-streaming-fingerprint/fingerprint.py", line 37, in fingerprint_tracks
pool.map(generate_fingerprints_with_pool, tracks)
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'

我不明白为什么在我的生产环境中出现这个错误，而不是在我的本地项目设置中，我的 docker 容器中有 4GB RAM。

请建议我解决方案。

提前致谢。

Answer 1

在多处理池中，资源不能共享。因为它在隔离的核心中运行。看来“fingerprint_worker”是进程之间的共享方法。使用 multiprocessing.pool.ThreadPool 代替：

with multiprocessing.pool.ThreadPool(const.MULTIPROCESSING_POOL_SIZE) as pool:
    pool.map(foo, tracks)