使用模块'subprocess'和超时

Question

这是运行任意命令返回其stdout数据的Python代码,或者在非零退出代码上引发异常:

proc = subprocess.Popen(
    cmd,
    stderr=subprocess.STDOUT,  # Merge stdout and stderr
    stdout=subprocess.PIPE,
    shell=True)

communicate 用于等待进程退出:

stdoutdata, stderrdata = proc.communicate()

该subprocess模块不支持超时 - 能够终止运行超过X秒的进程 - 因此,communicate可能需要永久运行.

在旨在在Windows和Linux上运行的Python程序中实现超时的最简单方法是什么？

Answer 1

我对低级细节知之甚少; 但是,鉴于在python 2.6中,API提供了等待线程和终止进程的能力,那么在单独的线程中运行进程呢？

import subprocess, threading

class Command(object):
    def __init__(self, cmd):
        self.cmd = cmd
        self.process = None

    def run(self, timeout):
        def target():
            print 'Thread started'
            self.process = subprocess.Popen(self.cmd, shell=True)
            self.process.communicate()
            print 'Thread finished'

        thread = threading.Thread(target=target)
        thread.start()

        thread.join(timeout)
        if thread.is_alive():
            print 'Terminating process'
            self.process.terminate()
            thread.join()
        print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'")
command.run(timeout=3)
command.run(timeout=1)

我机器中此代码段的输出是:

Thread started
Process started
Process finished
Thread finished
0
Thread started
Process started
Terminating process
Thread finished
-15

可以看出,在第一次执行中,进程正确完成(返回代码0),而在第二次执行中进程终止(返回代码-15).

我没有在Windows中测试过; 但是,除了更新示例命令之外,我认为它应该可以工作,因为我没有在文档中找到任何说明不支持thread.join或process.terminate的内容.

Answer 2

在Python 3.3+中:

from subprocess import STDOUT, check_output

output = check_output(cmd, stderr=STDOUT, timeout=seconds)

output 是一个包含命令的合并stdout,stderr数据的字节字符串.

与代码不同check_output,此代码会引发问题文本中指定的非零退出状态CalledProcessError.

我已删除,proc.communicate()因为它经常被不必要地使用.如果shell=True确实需要,您可以随时添加它.如果你添加cmdie,如果子进程产生自己的后代; shell=True可以在超时指示的时间之后返回,请参阅子进程超时失败.

通过check_output()3.2+子进程模块的后端,Python 2.x上提供了超时功能.

Answer 3

使用threading.Timer类可以简化jcollado的答案:

import shlex
from subprocess import Popen, PIPE
from threading import Timer

def run(cmd, timeout_sec):
    proc = Popen(shlex.split(cmd), stdout=PIPE, stderr=PIPE)
    timer = Timer(timeout_sec, proc.kill)
    try:
        timer.start()
        stdout, stderr = proc.communicate()
    finally:
        timer.cancel()

# Examples: both take 1 second
run("sleep 1", 5)  # process ends normally at 1 second
run("sleep 5", 1)  # timeout happens at 1 second

Answer 4

如果你在Unix上,

import signal
  ...
class Alarm(Exception):
    pass

def alarm_handler(signum, frame):
    raise Alarm

signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(5*60)  # 5 minutes
try:
    stdoutdata, stderrdata = proc.communicate()
    signal.alarm(0)  # reset the alarm
except Alarm:
    print "Oops, taking too long!"
    # whatever else

Answer 5

Alex Martelli的解决方案是一个具有适当进程终止的模块.其他方法不起作用,因为它们不使用proc.communicate().因此,如果您有一个产生大量输出的进程,它将填充其输出缓冲区然后阻塞,直到您从中读取内容.

from os import kill
from signal import alarm, signal, SIGALRM, SIGKILL
from subprocess import PIPE, Popen

def run(args, cwd = None, shell = False, kill_tree = True, timeout = -1, env = None):
    '''
    Run a command with a timeout after which it will be forcibly
    killed.
    '''
    class Alarm(Exception):
        pass
    def alarm_handler(signum, frame):
        raise Alarm
    p = Popen(args, shell = shell, cwd = cwd, stdout = PIPE, stderr = PIPE, env = env)
    if timeout != -1:
        signal(SIGALRM, alarm_handler)
        alarm(timeout)
    try:
        stdout, stderr = p.communicate()
        if timeout != -1:
            alarm(0)
    except Alarm:
        pids = [p.pid]
        if kill_tree:
            pids.extend(get_process_children(p.pid))
        for pid in pids:
            # process might have died before getting to this line
            # so wrap to avoid OSError: no such process
            try: 
                kill(pid, SIGKILL)
            except OSError:
                pass
        return -9, '', ''
    return p.returncode, stdout, stderr

def get_process_children(pid):
    p = Popen('ps --no-headers -o pid --ppid %d' % pid, shell = True,
              stdout = PIPE, stderr = PIPE)
    stdout, stderr = p.communicate()
    return [int(p) for p in stdout.split()]

if __name__ == '__main__':
    print run('find /', shell = True, timeout = 3)
    print run('find', shell = True)

Answer 6

我修改了sussudio的答案.现在函数返回:( ,returncode,stdout,stderr)timeout - stdout和stderr被解码为UTF-8字符串

def kill_proc(proc, timeout):
  timeout["value"] = True
  proc.kill()

def run(cmd, timeout_sec):
  proc = subprocess.Popen(shlex.split(cmd), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  timeout = {"value": False}
  timer = Timer(timeout_sec, kill_proc, [proc, timeout])
  timer.start()
  stdout, stderr = proc.communicate()
  timer.cancel()
  return proc.returncode, stdout.decode("utf-8"), stderr.decode("utf-8"), timeout["value"]

Answer 7

惊讶没人提到使用 timeout

timeout 5 ping -c 3 somehost

这显然不适用于每个用例,但如果你处理一个简单的脚本,这很难被击败.

也可以作为macu homebrew用户的coreutils中的gtimeout 使用.

Answer 8

另一种选择是写入临时文件以防止stdout阻塞而不需要使用communic()进行轮询.这对我有用,而其他答案没有; 例如在Windows上.

    outFile =  tempfile.SpooledTemporaryFile() 
    errFile =   tempfile.SpooledTemporaryFile() 
    proc = subprocess.Popen(args, stderr=errFile, stdout=outFile, universal_newlines=False)
    wait_remaining_sec = timeout

    while proc.poll() is None and wait_remaining_sec > 0:
        time.sleep(1)
        wait_remaining_sec -= 1

    if wait_remaining_sec <= 0:
        killProc(proc.pid)
        raise ProcessIncompleteError(proc, timeout)

    # read temp streams from start
    outFile.seek(0);
    errFile.seek(0);
    out = outFile.read()
    err = errFile.read()
    outFile.close()
    errFile.close()

Answer 9

timeout现在支持通过call()和communicate()子进程模块(如Python3.3的)中:

import subprocess

subprocess.call("command", timeout=20, shell=True)

这将调用该命令并引发异常

subprocess.TimeoutExpired

如果命令在20秒后没有完成.

然后,您可以处理异常以继续您的代码,例如:

try:
    subprocess.call("command", timeout=20, shell=True)
except subprocess.TimeoutExpired:
    # insert code here

希望这可以帮助.

Answer 10

我不知道为什么它不mentionned但是因为Python 3.5，有一个新的subprocess.run通用指令（即意味着取代check_call，check_output......），并且其具有timeout参数也是如此。

subprocess.run（args，*，stdin = None，input = None，stdout = None，stderr = None，shell = False，cwd = None，timeout = None，check = False，encoding = None，errors = None）

Run the command described by args. Wait for command to complete, then return a CompletedProcess instance.

subprocess.TimeoutExpired超时到期时会引发异常。

Answer 11

我将带有线程的解决方案添加jcollado到我的 Python 模块easyprocess 中。

安装：

pip install easyprocess

例子：

from easyprocess import Proc

# shell is not supported!
stdout=Proc('ping localhost').call(timeout=1.5).stdout
print stdout

Answer 12

预置 Linux 命令timeout并不是一个糟糕的解决方法，它对我有用。

cmd = "timeout 20 "+ cmd
subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()

Answer 13

我使用的解决方案是在 shell 命令前加上timelimit。如果命令花费的时间太长，timelimit 将停止它并且 Popen 将有一个由 timelimit 设置的返回码。如果大于 128，则表示 timelimit 杀死了进程。

另请参阅具有超时和大输出（> 64K）的python子进程

Answer 14

这是我的解决方案,我使用线程和事件:

import subprocess
from threading import Thread, Event

def kill_on_timeout(done, timeout, proc):
    if not done.wait(timeout):
        proc.kill()

def exec_command(command, timeout):

    done = Event()
    proc = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    watcher = Thread(target=kill_on_timeout, args=(done, timeout, proc))
    watcher.daemon = True
    watcher.start()

    data, stderr = proc.communicate()
    done.set()

    return data, stderr, proc.returncode

在行动:

In [2]: exec_command(['sleep', '10'], 5)
Out[2]: ('', '', -9)

In [3]: exec_command(['sleep', '10'], 11)
Out[3]: ('', '', 0)

Answer 15

如果您使用的是 python 2，请尝试一下

import subprocess32

try:
    output = subprocess32.check_output(command, shell=True, timeout=3)
except subprocess32.TimeoutExpired as e:
    print e