从另一个 python 脚本调用 python 脚本时，Python 日志记录挂起

Question

我在 python 日志类中观察到这个奇怪的问题，我有两个脚本，一个是从另一个脚本调用的。第一个脚本等待其他脚本结束，而其他脚本使用记录一个巨大的日志logging.info

这是代码片段

#!/usr/bin/env python

import subprocess
import time
import sys

chars = ["/","-","\\","|"]
i = 0
command = 'sudo python /home/tejto/test/writeIssue.py'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
while process.poll() is None:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
output = process.communicate()

另一个脚本是

#!/usr/bin/env python

import os
import logging as log_status

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
            log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
    self.upgradethreadstatus.status_change("Entered upgrade routine")
    procoutput = 'very huge logs, mine were 145091 characters'
    self.upgradethreadstatus.status_change(procoutput)
    self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:]

如果我运行第一个脚本，两个脚本都会挂起，问题似乎出在代码上while process.poll() is None，如果我注释此代码，则一切正常。（无法将其与我的问题联系起来！！）

PS我也尝试调试python日志记录类，其中我发现该进程被类的emit函数卡住StreamHandler，其中它卡在stream.write函数调用处并且在写入巨大日志后没有出来，但是我的退出日志没有出现。

那么这些脚本中可能出现什么问题而导致死锁情况呢？

编辑1（带线程的代码）

脚本.py

#!/usr/bin/env python 

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
    output = process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\\","|"]
i = 0

while t.is_alive:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
t.join()

脚本2.py

#!/usr/bin/env python  
import os
import sys
import logging as log_status
import time

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                        os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
        log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        procoutput = "Please put logs of 145091 characters over here otherwise the situtation wouldn't remain same or run any command whose output is larger then 145091 characters"
    self.upgradethreadstatus.status_change(procoutput)
        time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:])

在这种情况下 t.is_alive 函数不会返回 false （不知道，但启动函数已经返回，所以理想情况下它应该返回 false!!:()

Answer 1

标准输出缓冲区堵塞。

该process.communicate()调用不会被执行，因为它是在 while process.poll() is None:. 因此writeIssue.py，尝试写入太多字节stdout，并且所有字节都被缓冲在 PIPE 中，并且在调用subprocess.PIPE之前不会从 PIPE 中拉出。communicate

缓冲区的大小是有限的。当缓冲区满时，stream.write将阻塞，直到缓冲区有空间。如果缓冲区从未清空（正如您的代码中所发生的那样），则进程会死锁。

communicate()修复方法是在缓冲区完全填满之前调用。您可以通过writeIssue.py在线程中启动并在 while-thread-is-alive 循环在主线程中运行时communicate() 并发调用来实现这一点。

脚本.py：

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command)
    process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\\","|"]
i = 0
while t.is_alive():
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4

t.join()

脚本2.py：

import sys
import logging
import time

class UpgradeStatus():
    def __init__(self):
        logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.DEBUG)

        consoleHandler = logging.StreamHandler()
        consoleHandler.setLevel(logging.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        self.logger.addHandler(consoleHandler)

    def status_change(self, status):
        self.logger.info(str(status))

class UpgradeThread():
    def __init__(self, link):
        self.upgradethreadstatus = UpgradeStatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        for i in range(5):
            procoutput = 'very huge logs, mine were 145091 characters'
            self.upgradethreadstatus.status_change(procoutput)
            time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = UpgradeThread(sys.argv[1:])

请注意，如果有两个线程同时写入 stdout，则输出将出现乱码。如果您想避免这种情况，那么所有输出都应该由配备队列的单个线程处理 。希望写入输出的所有其他线程或进程应将字符串或日志记录推送到队列，以供专用输出线程处理。

然后，该输出线程可以使用 for 循环从队列中提取输出：

for message from iter(queue.get, None):
    print(message)