Python子进程:cmd退出时的回调

Question

我正在推出一个使用的程序 subprocess.Popen(cmd, shell=TRUE)

我对Python很新,但感觉就像应该有一些api让我做类似的事情:

subprocess.Popen(cmd, shell=TRUE,  postexec_fn=function_to_call_on_exit)

我这样做是为了function_to_call_on_exit能够在知道cmd退出的基础上做一些事情(例如保持当前正在运行的外部进程数的计数)

我假设我可以相当简单地将子进程包装在一个将线程与Popen.wait()方法结合起来的类中,但是因为我还没有在Python中进行线程化,而且看起来这可能是一个常见的API存在,我想我会先尝试找一个.

提前致谢 :)

Answer 1

你是对的 - 没有很好的API.你的第二点也是对的 - 设计一个使用线程为你做这个功能的功能非常简单.

import threading
import subprocess

def popenAndCall(onExit, popenArgs):
    """
    Runs the given args in a subprocess.Popen, and then calls the function
    onExit when the subprocess completes.
    onExit is a callable object, and popenArgs is a list/tuple of args that 
    would give to subprocess.Popen.
    """
    def runInThread(onExit, popenArgs):
        proc = subprocess.Popen(*popenArgs)
        proc.wait()
        onExit()
        return
    thread = threading.Thread(target=runInThread, args=(onExit, popenArgs))
    thread.start()
    # returns immediately after the thread starts
    return thread

在Python中,即使线程也很容易,但请注意,如果onExit()的计算成本很高,那么你需要将它放在一个单独的进程中,而不是使用多处理(这样GIL不会减慢你的程序速度).它实际上非常简单 - 你基本上只需要替换所有的调用threading.Thread,multiprocessing.Process因为它们遵循(几乎)相同的API.

Answer 2

concurrent.futuresPython 3.2中有模块(可pip install futures用于较旧的Python <3.2):

pool = Pool(max_workers=1)
f = pool.submit(subprocess.call, "sleep 2; echo done", shell=True)
f.add_done_callback(callback)

回调将在调用的同一进程中调用f.add_done_callback().

完整的计划

import logging
import subprocess
# to install run `pip install futures` on Python <3.2
from concurrent.futures import ThreadPoolExecutor as Pool

info = logging.getLogger(__name__).info

def callback(future):
    if future.exception() is not None:
        info("got exception: %s" % future.exception())
    else:
        info("process returned %d" % future.result())

def main():
    logging.basicConfig(
        level=logging.INFO,
        format=("%(relativeCreated)04d %(process)05d %(threadName)-10s "
                "%(levelname)-5s %(msg)s"))

    # wait for the process completion asynchronously
    info("begin waiting")
    pool = Pool(max_workers=1)
    f = pool.submit(subprocess.call, "sleep 2; echo done", shell=True)
    f.add_done_callback(callback)
    pool.shutdown(wait=False) # no .submit() calls after that point
    info("continue waiting asynchronously")

if __name__=="__main__":
    main()

产量

$ python . && python3 .
0013 05382 MainThread INFO  begin waiting
0021 05382 MainThread INFO  continue waiting asynchronously
done
2025 05382 Thread-1   INFO  process returned 0
0007 05402 MainThread INFO  begin waiting
0014 05402 MainThread INFO  continue waiting asynchronously
done
2018 05402 Thread-1   INFO  process returned 0

Answer 3

我修改了Daniel G的答案,简单地传递subprocess.Popen args和kwargs作为自己而不是单独的tupple/list,因为我想在subprocess.Popen中使用关键字参数.

在我的情况下,我有一个postExec()我想追求的方法subprocess.Popen('exe', cwd=WORKING_DIR)

使用下面的代码,它就变成了 popenAndCall(postExec, 'exe', cwd=WORKING_DIR)

import threading
import subprocess

def popenAndCall(onExit, *popenArgs, **popenKWArgs):
    """
    Runs a subprocess.Popen, and then calls the function onExit when the
    subprocess completes.

    Use it exactly the way you'd normally use subprocess.Popen, except include a
    callable to execute as the first argument. onExit is a callable object, and
    *popenArgs and **popenKWArgs are simply passed up to subprocess.Popen.
    """
    def runInThread(onExit, popenArgs, popenKWArgs):
        proc = subprocess.Popen(*popenArgs, **popenKWArgs)
        proc.wait()
        onExit()
        return

    thread = threading.Thread(target=runInThread,
                              args=(onExit, popenArgs, popenKWArgs))
    thread.start()

    return thread # returns immediately after the thread starts

Answer 4

我有同样的问题,并使用解决了它multiprocessing.Pool.涉及两个hacky技巧:

1. 制作游泳池1的大小
2. 在长度为1的可迭代内传递可迭代参数

result是在完成时使用回调执行的一个函数

def sub(arg):
    print arg             #prints [1,2,3,4,5]
    return "hello"

def cb(arg):
    print arg             # prints "hello"

pool = multiprocessing.Pool(1)
rval = pool.map_async(sub,([[1,2,3,4,5]]),callback =cb)
(do stuff) 
pool.close()

就我而言,我希望调用也是非阻塞的.工作得很漂亮