ProcessPoolExecutor日志记录无法在Windows上记录内部函数,但在Unix/Mac上无法记录

Question

当我在Windows计算机上运行以下脚本时,我没有看到该log_pid函数的任何日志消息,但是当我在Unix/Mac上运行时,我会这样做.我之前已经读过,与Mac相比,Windows上的多处理方式不同,但我不清楚应该做些什么更改才能让这个脚本在Windows上运行.我正在运行Python 3.6.

import logging
import sys
from concurrent.futures import ProcessPoolExecutor
import os


def log_pid(x):
    logger.info('Executing on process: %s' % os.getpid())


def do_stuff():
    logger.info('this is the do stuff function.')
    with ProcessPoolExecutor(max_workers=4) as executor:
        executor.map(log_pid, range(0, 10))


def main():
    logger.info('this is the main function.')
    do_stuff()


if __name__ == '__main__':
    logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
    logger = logging.getLogger(__name__)

    logger.info('Start of script ...')

    main()

    logger.info('End of script ...')

Answer 1

Unix进程是通过fork策略创建的,其中子进程从父进程克隆并在父进行分叉时继续执行.

在Windows上是完全不同的:创建一个空白进程并启动一个新的Python解释器.然后,解释器将加载log_pid函数所在的模块并执行它.

这意味着该__main__部分不会由新生成的子进程执行.因此,logger不创建对象,并且log_pid函数相应地崩溃.您没有看到错误,因为您忽略了计算结果.尝试按如下方式修改逻辑.

def do_stuff():
    logger.info('this is the do stuff function.')
    with ProcessPoolExecutor(max_workers=4) as executor:
        iterator = executor.map(log_pid, range(0, 10))
        list(iterator)  # collect the results in a list

问题将变得明显.

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python36-32\lib\concurrent\futures\process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "C:\Program Files (x86)\Python36-32\lib\concurrent\futures\process.py", line 153, in _process_chunk
    return [fn(*args) for args in chunk]
  File "C:\Program Files (x86)\Python36-32\lib\concurrent\futures\process.py", line 153, in <listcomp>
    return [fn(*args) for args in chunk]
  File "C:\Users\cafama\Desktop\pool.py", line 8, in log_pid
    logger.info('Executing on process: %s' % os.getpid())
NameError: name 'logger' is not defined

当处理池处理(无论是concurrent.futures或multiprocessing的)总是收集计算的结果,以避免错误无声造成混乱.

要解决这个问题,只需logger在模块的顶层移动创建,一切都可以在所有平台上运行.

import logging
import sys
from concurrent.futures import ProcessPoolExecutor
import os

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logger = logging.getLogger(__name__)   

def log_pid(x):
    logger.info('Executing on process: %s' % os.getpid())

...