PyInstaller构建的Windows EXE因多处理而失败

Question

在我的项目中,我使用Python的multiprocessing库在__main__中创建多个进程.使用PyInstaller 2.1.1将项目打包到单个Windows EXE中.

我创建了这样的新流程:

from multiprocessing import Process
from Queue import Empty

def _start():
    while True:
        try:
            command = queue.get_nowait()
        # ... and some more code to actually interpret commands
        except Empty:
            time.sleep(0.015)

def start():
    process = Process(target=_start, args=args)
    process.start()
    return process

并在__main__:

if __name__ == '__main__':
    freeze_support()

    start()

不幸的是,当将应用程序打包到EXE并启动它时,我WindowsError在这一行得到5或6(似乎是随机的):

command = queue.get_nowait()

PyInstaller主页上的一个食谱声称我必须修改我的代码,以便在将应用程序打包为单个文件时在Windows中启用多处理.

我在这里复制代码:

import multiprocessing.forking
import os
import sys


class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            # Last character is stripped in C-loader. We have to add
            # '/' or '\\' at the end.
            os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')


class Process(multiprocessing.Process):
    _Popen = _Popen


class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'


if __name__ == '__main__':
    # On Windows calling this function is necessary.
    if sys.platform.startswith('win'):
        multiprocessing.freeze_support()
    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

我对这个"解决方案"感到沮丧的是,其中一个,它绝对不清楚究竟是什么修补,而且,两个,它以如此复杂的方式编写,以至于无法推断哪些部分是解决方案,哪些只是一个插图.

任何人都可以就此问题分享一些看法,并提供在PyInstaller构建的单文件Windows可执行文件中实现多处理的项目中确切需要更改的内容吗？

Answer 1

添加到nikola的答案......

*nix(Linux,Mac OS X等)不需要对PyInstaller进行任何更改即可.(这包括两者--onedir和--onefile选项.)如果您只打算支持*nix系统,则无需担心任何此类问题.

但是,如果您计划支持Windows,则需要添加一些代码,具体取决于您选择的选项:--onedir或--onefile.

如果您打算使用--onedir,您需要添加的是一个特殊的方法调用:

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

根据文档,此调用必须在之后立即进行if __name__ == '__main__':,否则将无效.(强烈建议您在主模块中使用这两行.)

但实际上,您可以在电话会议前进行检查,事情仍然有效:

if __name__ == '__main__':
    if sys.platform.startswith('win'):
        # On Windows calling this function is necessary.
        multiprocessing.freeze_support()

但是,multiprocessing.freeze_support()也可以在其他平台和情况下进行调用- 运行它只会影响Windows上的冻结支持.如果你是一个字节码螺母,你会注意到if语句添加了一些字节码,并且使用if语句可以忽略不计.因此,您应该multiprocessing.freeze_support()立即坚持一个简单的电话if __name__ == '__main__':.

如果您打算使用--onefile,则需要添加nikola的代码:

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

您可以将上述内容与其余代码或以下内容结合使用:

class SendeventProcess(Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue

        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

我从这里得到了代码,PyInstaller的多处理配方的新站点.(他们似乎关闭了他们的Trac网站.)

请注意,它们的代码具有较小的错误,可用于--onefile多处理支持.他们将os.sep添加到他们的_MEIPASS2环境变量中.(行os.putenv('_MEIPASS2', sys._MEIPASS + os.sep):)这打破了事情:

  File "<string>", line 1
    sys.path.append(r"C:\Users\Albert\AppData\Local\Temp\_MEI14122\")
                                                                    ^
SyntaxError: EOL while scanning string literal

我上面提供的代码是相同的,没有os.sep.删除os.sep修复此问题并允许使用--onefile配置进行多处理.

综上所述:

--onedir在Windows上启用多处理支持(在Windows上不起作用--onefile,但在所有平台/配置上都是安全的):

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

--onefile在Windows上启用多处理支持(在所有平台/配置上都是安全的,兼容--onedir):

import multiprocessing.forking
import os
import sys

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            # We have to set original _MEIPASS2 value from sys._MEIPASS
            # to get --onefile mode working.
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                # available. In those cases we cannot delete the variable
                # but only set it to the empty string. The bootloader
                # can handle this case.
                if hasattr(os, 'unsetenv'):
                    os.unsetenv('_MEIPASS2')
                else:
                    os.putenv('_MEIPASS2', '')

class Process(multiprocessing.Process):
    _Popen = _Popen

# ...

if __name__ == '__main__':
    # On Windows calling this function is necessary.
    multiprocessing.freeze_support()

    # Use your new Process class instead of multiprocessing.Process

来源:PyInstaller Recipe,Python多处理文档

Answer 2

找到PyInstaller票后回答我自己的问题:

显然我们所要做的就是提供一个Process(和_Popen)类,如下所示,并使用它而不是multiprocessing.Process.我已经纠正并简化了仅适用于Windows的类,*ix系统可能需要不同的代码.

为了完整起见,以下是上述问题的改编样本:

import multiprocessing
from Queue import Empty

class _Popen(multiprocessing.forking.Popen):
    def __init__(self, *args, **kw):
        if hasattr(sys, 'frozen'):
            os.putenv('_MEIPASS2', sys._MEIPASS)
        try:
            super(_Popen, self).__init__(*args, **kw)
        finally:
            if hasattr(sys, 'frozen'):
                os.unsetenv('_MEIPASS2')


class Process(multiprocessing.Process):
    _Popen = _Popen


def _start():
    while True:
        try:
            command = queue.get_nowait()
        # ... and some more code to actually interpret commands
        except Empty:
            time.sleep(0.015)

def start():
    process = Process(target=_start, args=args)
    process.start()
    return process