在后台运行 shell 命令，无需等待

Question

我想在 python 中同时运行多个命令，但程序似乎在前一个命令完成后执行一个命令，

这是我的代码：

import os
import glob
for i in (glob.glob("PATH/SUB/*.csv")):
    file = i.split('\\')[1]
os.system("scrapy crawl quotes -a file=%s -o /OUTPUT/%s.csv &" % (file,file)) 

Answer 1

简短回答：

#!/usr/bin/env python3.6
import glob
import asyncio


async def run(shell_command):
    p = await asyncio.create_subprocess_shell(shell_command)
    await p.communicate()


async def main(shell_commands):
    for f in asyncio.as_completed([run(c) for c in shell_commands]):
        await f


commands = []
for i in (glob.glob("PATH/SUB/*.csv")):
    file = i.split('\\')[1]
    commands.append("scrapy crawl quotes -a file=%s -o /OUTPUT/%s.csv &" % (file, file))

loop = asyncio.ProactorEventLoop() 
loop.run_until_complete(main(commands))
loop.close()

长答案：

您可以检查 asyncio 子进程。

我们来测试一下。我们将尝试运行 5 次“sleep 5”命令 - 我们预计总运行时间将类似于 5 秒（而不是 25 秒）。

#!/usr/bin/env python3.6
import time
import asyncio
from asyncio.subprocess import PIPE, STDOUT


async def run(shell_command):
    p = await asyncio.create_subprocess_shell(shell_command, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    return (await p.communicate())[0].splitlines()


async def main():
    commands = [run('sleep 5; echo {i}'.format(i=i)) for i in range(5)]
    for f in asyncio.as_completed(commands):
        print(await f)


start = time.time()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

total = time.time() - start

print("Total time {} s".format(total))

输出：

[b'2']
[b'3']
[b'4']
[b'1']
[b'0']
Total time 5.0065038204193115 s

一切都如预期。

根据您的需求进行调整后：

#!/usr/bin/env python3.6
import glob
import asyncio


async def run(shell_command):
    p = await asyncio.create_subprocess_shell(shell_command)
    await p.communicate()


async def main(shell_commands):
    for f in asyncio.as_completed([run(c) for c in shell_commands]):
        await f


commands = []
for i in (glob.glob("PATH/SUB/*.csv")):
    file = i.split('\\')[1]
    commands.append("scrapy crawl quotes -a file=%s -o /OUTPUT/%s.csv &" % (file, file))

loop = asyncio.get_event_loop()
loop.run_until_complete(main(commands))
loop.close()

最后，如果您使用的是 Windows，则需要进行更改

loop = asyncio.get_event_loop()

进入

loop = asyncio.ProactorEventLoop() 

我假设你使用的是Windows，因为你的代码

file = i.split('\\')[1].