悲情不能腌制由 GDAL 模块创建的 SwigPyObject

Question

我有一个类，它使用 GDAL 模块（https://pypi.org/project/GDAL/）打开一个大的光栅图像，并在多个位置从中提取小图像，由传递的坐标元组列表定义。我想使用 Pathos 处理大量坐标，就像我的代码的这个简化版本：

import gdal
import pathos.pools as pp

class MyClass:

    def __init__(self, image):
        self.image_object = gdal.Open(image)

    def get_small_image(self, coord, size=100):
        small_image = self.image_object.ReadAsArray(coord[0], coord[1], size, size)

        return small_image

    def run_multi(self, coords_in):
        pool = pp.ProcessPool(2)
        output = pool.map(self.get_small_image, coords_in)

        return output

image = r'C:\path\to\image.JP2'

class_obj = MyClass(image)
coords = [(200, 200), (400, 400), (600, 600), (800, 800)]
results = class_obj.run_multi(coords)

print(results)

运行它会出现以下错误：

Traceback (most recent call last):
  File "C:/Users/lharris/PycharmProjects/road_condition_py3/error_recreate.py", line 26, in <module>
    results = class_obj.run_multi(coords)
  File "C:/Users/lharris/PycharmProjects/road_condition_py3/error_recreate.py", line 17, in run_multi
    output = pool.map(self.get_small_image, coords_in)
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\pathos\multiprocessing.py", line 137, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\multiprocess\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\multiprocess\pool.py", line 644, in get
    raise self._value
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\multiprocess\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\multiprocess\connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\multiprocess\reduction.py", line 54, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Python36\lib\pickle.py", line 409, in dump
    self.save(obj)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 751, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\dill\_dill.py", line 1377, in save_function
    obj.__dict__), obj=obj)
  File "C:\Python36\lib\pickle.py", line 610, in save_reduce
    save(args)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 751, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\dill\_dill.py", line 1120, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "C:\Python36\lib\pickle.py", line 610, in save_reduce
    save(args)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\dill\_dill.py", line 1069, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "C:\Python36\lib\pickle.py", line 610, in save_reduce
    save(args)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\Python36\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python36\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\dill\_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Python36\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Python36\lib\pickle.py", line 852, in _batch_setitems
    save(v)
  File "C:\Python36\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python36\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\lharris\PycharmProjects\road_condition_py3\venv\lib\site-packages\dill\_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Python36\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Python36\lib\pickle.py", line 852, in _batch_setitems
    save(v)
  File "C:\Python36\lib\pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle SwigPyObject objects

仔细看看，看起来 Pathos 确实正确地使用了 Dill 和 Multiprocess。那么有没有一种并行处理 SwigPyObjects 的方法，无论是使用 Patos 还是其他方式？

几周前有人问过类似的问题，但没有得到任何答案（Python pathos error can't pickle SwigPyObject objects），当我以前无法使用Pathos 时我问了一个相关的问题（How can type 'SwigPyObject beregistered在 Python 中使用 copy_reg.pickle？）。

我在 Windows 7 中使用 Python 3.6.5 遇到了这个错误，在 PyCharm 中开发，使用虚拟环境，我需要的任何包都从 PyPi 下载的 .whl 文件安装到该虚拟环境中（我无法正常使用 pip install我的网络）。

任何人都可以给我的任何帮助都会很棒！

Answer 1

因为get_small_image它是一个绑定方法，所以每个工作人员都需要一个 的副本class_obj。对于多处理，这是通过传递 pickled python 对象来完成的。然而，由于该对象包含不可选取的类成员SwigPyObject class_obj，因此整个事情都会失败。

最直接的修复方法是这样的：

class MyClass:

    def __init__(self, image):
        self.image_object = None
        self.image = image

    def get_small_image(self, coord, size=100):
        if self.image_object is None:
            self.image_object = gdal.Open(self.image)

        small_image = self.image_object.ReadAsArray(coord[0], coord[1], size, size)

        return small_image

    def run_multi(self, coords_in):
        pool = pp.ProcessPool(2)
        output = pool.map(self.get_small_image, coords_in)

        return output

这种图像的延迟加载会导致在子进程对对象进行 pickle/unpickled之后SwigPyObject创建，从而避免您遇到的问题。