如何拆分文件并将它们并行处理然后再缝合？UNIX

Question

我有一个文本文件infile.txt:

abc what's the foo bar.
foobar hello world, hhaha cluster spatio something something.
xyz trying to do this in parallel
kmeans you're mean, who's mean?

文件中的每一行都将通过此perl命令处理到out.txt中

`cat infile.txt | perl dosomething > out.txt`

想象一下,如果文本文件是100,000,000行.我想并行化bash命令,所以我尝试了这样的事情:

$ mkdir splitfiles
$ mkdir splitfiles_processed
$ cd splitfiles
$ split -n3 ../infile.txt
$ for i in $(ls); do "cat $i | perl dosomething > ../splitfiles_processed/$i &"; done
$ wait
$ cd ../splitfiles_processed
$ cat * > ../infile_processed.txt

但是,有一种不那么冗长的方式来做同样的事情吗？

Answer 1

@Ulfalizer的答案为您提供了解决方案的良好提示,但缺乏细节.

你可以使用GNU parallel(apt-get install parallel在Debian上)

因此,使用以下命令可以解决您的问题:

parallel -a infile.txt -l 1000 -j 10 -k --spreadstdin perl dosomething > result.txt

这是论证的意义

-a: read input from file instead of stdin
-l 1000: send 1000 lines blocks to command
-j 10: launch 10 jobs in parallel
-k: keep sequence of output
--spreadstdin: sends the above 1000 line block to the stdin of the command

Answer 2

我自己从未尝试过,但GNU parallel可能值得一试.

这是摘自man page(parallel(1))的摘录,与你目前正在做的类似.它也可以用其他方式分割输入.

EXAMPLE: Processing a big file using more cores
       To process a big file or some output you can use --pipe to split up
       the data into blocks and pipe the blocks into the processing program.

       If the program is gzip -9 you can do:

       cat bigfile | parallel --pipe --recend '' -k gzip -9 >bigfile.gz

       This will split bigfile into blocks of 1 MB and pass that to gzip -9
       in parallel. One gzip will be run per CPU core. The output of gzip -9
       will be kept in order and saved to bigfile.gz

这是否值得取决于您的处理CPU密集程度.对于简单的脚本,您将花费大部分时间将数据移入和移出磁盘,并行化并不会让您受到太多影响.

您可以找到由GNU并行笔者部分介绍影片在这里.