bash gnu并行帮助

Question

它关于 http://en.wikipedia.org/wiki/Parallel_(software)和非常丰富的联机帮助页http://www.gnu.org/software/parallel/man.html

(for x in `cat list` ; do 
       do_something $x
   done) | process_output

被这取代

 cat list | parallel do_something | process_output

我正试图在此实现这一点

    while [ "$n" -gt 0 ]
        do          
        percentage=${"scale=2;(100-(($n / $end) * 100))"|bc -l}}
    #get url from line specified by n from file done1              
nextUrls=`sed -n "${n}p" < done1`
    echo -ne "${percentage}%  $n / $end urls saved going to line 1. current: $nextUrls\r"
#    function that gets links from the url
    getlinks $nextUrls
#save n
    echo $n > currentLine
    let "n--"
    let "end=`cat done1 |wc -l`"
    done

同时阅读gnu parallel文档

我发现不支持函数,所以getlinks不会并行使用

我到目前为止找到的最好的是

seq 30 | parallel -n 4 --colsep '  ' echo {1} {2} {3} {4}

输出

1 2 3 4 
5 6 7 8 
9 10 11 12 
13 14 15 16 
17 18 19 20 
21 22 23 24 
25 26 27 28 
29 30 

如果我是对的,上面提到的循环应该是这样的

end=`cat done1 |wc -l`
seq $end -1 1 |  parallel -j+4 -k
#(all exept getlinks function goes here, but idk how? )|
# everytime it finishes do
 getlinks $nextUrls

thx提前帮助

Answer 1

看起来你想要的是一个进度表.尝试:

cat done1 | parallel --eta wget

如果那不是您想要的,请查看sem(sem是parallel --semaphoreGNU Parallel 的别名,通常是安装的):

for i in `ls *.log` ; do
  echo $i
  sem -j+0 gzip $i ";" echo done
done
sem --wait

在你的情况下,它将是这样的:

while [ "$n" -gt 0 ]
    do          
    percentage=${"scale=2;(100-(($n / $end) * 100))"|bc -l}}
    #get url from line specified by n from file done1
    nextUrls=`sed -n "${n}p" < done1`
    echo -ne "${percentage}%  $n / $end urls saved going to line 1. current: $nextUrls\r"
    #    function that gets links from the url
    THE_URL=`getlinks $nextUrls`
    sem -j10 wget $THE_URL
    #save n
    echo $n > currentLine
    let "n--"
    let "end=`cat done1 |wc -l`"
done
sem --wait
echo All done