Python NotImplementedError:池对象不能在进程之间传递

Question

当页面附加到页面列表时,我正在尝试传递工作,但我的代码输出返回NotImplementedError.这是我正在尝试做的代码:

码:

from multiprocessing import Pool, current_process
import time
import random
import copy_reg
import types
import threading


class PageControler(object):
    def __init__(self):
        self.nProcess = 3
        self.pages = [1,2,3,4,5,6,7,8,9,10]
        self.manageWork()


    def manageWork(self):

        self.pool = Pool(processes=self.nProcess)

        time.sleep(2)
        work_queue = threading.Thread(target=self.modifyQueue)
        work_queue.start()

        #pool.close()
        #pool.join()

    def deliverWork(self):    
        if self.pages != []:
            pag = self.pages.pop()
            self.pool.apply_async(self.myFun)


    def modifyQueue(self):
        t = time.time()
        while (time.time()-t) < 10:
            time.sleep(1)
            self.pages.append(99)
            print self.pages
            self.deliverWork()

    def myFun(self):
        time.sleep(2)


if __name__ == '__main__':
    def _pickle_method(m):
        if m.im_self is None:
            return getattr, (m.im_class, m.im_func.func_name)
        else:
            return getattr, (m.im_self, m.im_func.func_name)

    copy_reg.pickle(types.MethodType, _pickle_method)

    PageControler()

输出:

NotImplementedError: pool objects cannot be passed between processes or pickled

这是在进程之间传递池对象的任何方法吗？

编辑:

我正在使用Python 2.6

Answer 1

为了挑选你想要传递给的实例方法Pool,Python需要挑选整个PageControler对象,包括它的实例变量.其中一个实例变量是Pool对象本身,并且Pool对象无法被pickle,因此错误.您可以通过__getstate__在对象上实现来解决此问题,并使用它pool在pickle之前从实例中删除对象:

class PageControler(object):
    def __init__(self):
        self.nProcess = 3
        self.pages = [1,2,3,4,5,6,7,8,9,10]
        self.manageWork()


    def manageWork(self):

        self.pool = Pool(processes=self.nProcess)

        time.sleep(2)
        work_queue = threading.Thread(target=self.modifyQueue)
        work_queue.start()

        #pool.close()
        #pool.join()

    def deliverWork(self):    
        if self.pages != []:
            pag = self.pages.pop()
            self.pool.apply_async(self.myFun)


    def modifyQueue(self):
        t = time.time()
        while (time.time()-t) < 10:
            time.sleep(1)
            self.pages.append(99)
            print self.pages
            self.deliverWork()

    def myFun(self):
        time.sleep(2)

    def __getstate__(self):
        self_dict = self.__dict__.copy()
        del self_dict['pool']
        return self_dict

    def __setstate__(self, state):
        self.__dict__.update(state)

__getstate__在对一个对象进行酸洗之前总是会调用它,并允许您准确指定实际应该对哪些对象的状态进行pickle.然后在unpickling时,__setstate__(state)如果它被实现(在我们的情况下)将被调用,或者如果不是,则dict返回的__getstate__将被用作__dict__未修改的实例.在上面的例子中,我们明确地设置__dict__了dict我们返回的内容__getstate__,但是我们可能没有实现__setstate__并获得相同的效果.