Windows上的多处理中断

Question

我在Linux上用Python开发,从未真正看到过Windows的这类问题.我正在使用该multiprocessing库来加速计算,这在Linux上非常适合我.

但是,在Windows上,事情并不顺利:

 * [INFO] Parsing 1 file using 2 threads

Traceback (most recent call last):
  File "main.py", line 170, in <module>
    master = ParsingMaster(parser, list(input_file), output_list, threads=num_threads)
Traceback (most recent call last):
  File "main.py", line 39, in __init__
  File "<string>", line 1, in <module>
    self.input_process.start()
  File "C:\Python26\lib\multiprocessing\forking.py", line 342, in main
  File "C:\Python26\lib\multiprocessing\process.py", line 104, in start
        self._popen = Popen(self)
self = load(from_parent)
  File "C:\Python26\lib\multiprocessing\forking.py", line 239, in __init__
  File "C:\Python26\lib\pickle.py", line 1370, in load
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python26\lib\multiprocessing\forking.py", line 162, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python26\lib\pickle.py", line 224, in dump
    return Unpickler(file).load()
    self.save(obj)
 File "C:\Python26\lib\pickle.py", line 858, in load
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    dispatch[key](self)
  File "C:\Python26\lib\pickle.py", line 880, in load_eof
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    r aise EOFError
 File "C:\Python26\lib\pickle.py", line 649, in save_dict
EOFError
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\multiprocessing\forking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:\Python26\lib\multiprocessing\managers.py", line 458, in __reduce__
    return type(self).from_address, \
AttributeError: type object 'SyncManager' has no attribute 'from_address'

我正在Windows 7上测试Python 2.6和2.7并且一遍又一遍地得到同样的错误.有谁知道这意味着什么？

Answer 1

Windows有限制,以下是您看到的错误的相关部分:

由于Windows缺少os.fork(),因此它有一些额外的限制:

更多的可挑选性

确保所有参数Process.__init__()都是可选择的.这尤其意味着绑定或未绑定的方法不能直接用作Windows上的目标参数 - 只需定义一个函数并使用它.

此外,如果您是子类,Process则确保在Process.start()调用方法时可以选择实例.

这意味着作为参数传递的东西Process.__init__() 无法被pickle或unpickled(Python中的序列化).什么是SyncManager抱怨无法找到该对象的属性AttributeError: type object 'SyncManager' has no attribute 'from_address',这可能是你的根本原因.这个SyncManager对象实际上可以被腌制,它是否符合泡菜规则？

如果从Windows上的命令行运行此操作,则显然无法执行此操作.

不要那样做.将代码保存在文件中,然后使用以下命令从文件中运行它:

python myfile.py

这将解决您的问题.