相关疑难解决方法(0)

Python:并行执行cat子进程

cat | zgrep在远程服务器上运行几个命令并单独收集它们的输出以进行进一步处理:

class MainProcessor(mp.Process):
    def __init__(self, peaks_array):
        super(MainProcessor, self).__init__()
        self.peaks_array = peaks_array

    def run(self):
        for peak_arr in self.peaks_array:
            peak_processor = PeakProcessor(peak_arr)
            peak_processor.start()

class PeakProcessor(mp.Process):
    def __init__(self, peak_arr):
        super(PeakProcessor, self).__init__()
        self.peak_arr = peak_arr

    def run(self):
        command = 'ssh remote_host cat files_to_process | zgrep --mmap "regex" '
        log_lines = (subprocess.check_output(command, shell=True)).split('\n')
        process_data(log_lines)
Run Code Online (Sandbox Code Playgroud)

但是,这会导致子进程('ssh ... cat ...')命令的顺序执行.第二个峰值等待第一个完成,依此类推.

如何修改此代码以便子进程调用并行运行,同时仍能够单独收集每个的输出?

python shell subprocess python-multithreading

14
推荐指数
2
解决办法
1万
查看次数

标签 统计

python ×1

python-multithreading ×1

shell ×1

subprocess ×1