BASH脚本:使用wget下载连续编号的文件

Question

我有一个Web服务器,用于保存编号为Web应用程序的日志文件.这个文件名示例如下:

dbsclog01s001.log
dbsclog01s002.log
dbsclog01s003.log

最后3位是计数器,它们可以达到100的时间.

我经常打开一个Web浏览器,浏览到如下文件:

http://someaddress.com/logs/dbsclog01s001.log

并保存文件.当你获得50个日志时,这当然会有点烦人.我试图想出一个使用wget和传递的BASH脚本

http://someaddress.com/logs/dbsclog01s*.log

但我的剧本有问题.无论如何,任何人都有关于如何做到这一点的样本？

谢谢!

Answer 1

#!/bin/sh

if [ $# -lt 3 ]; then
        echo "Usage: $0 url_format seq_start seq_end [wget_args]"
        exit
fi

url_format=$1
seq_start=$2
seq_end=$3
shift 3

printf "$url_format\\n" `seq $seq_start $seq_end` | wget -i- "$@"

保存上面的seq_wget,给它执行权限(chmod +x seq_wget),然后运行,例如:

$ ./seq_wget http://someaddress.com/logs/dbsclog01s%03d.log 1 50

或者,如果你有Bash 4.0,你可以输入

$ wget http://someaddress.com/logs/dbsclog01s{001..050}.log

或者,如果你有,curl而不是wget,你可以跟随丹尼斯威廉姆森的答案.

Answer 2

curl似乎支持范围.从man页面:

URL  
       The URL syntax is protocol dependent. You’ll find a  detailed  descrip?
       tion in RFC 3986.

       You  can  specify  multiple  URLs or parts of URLs by writing part sets
       within braces as in:

        http://site.{one,two,three}.com

       or you can get sequences of alphanumeric series by using [] as in:

        ftp://ftp.numericals.com/file[1-100].txt
        ftp://ftp.numericals.com/file[001-100].txt    (with leading zeros)
        ftp://ftp.letters.com/file[a-z].txt

       No nesting of the sequences is supported at the moment, but you can use
       several ones next to each other:

        http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

       You  can  specify  any amount of URLs on the command line. They will be
       fetched in a sequential manner in the specified order.

       Since curl 7.15.1 you can also specify step counter for the ranges,  so
       that you can get every Nth number or letter:

        http://www.numericals.com/file[1-100:10].txt
        http://www.letters.com/file[a-z:2].txt

您可能已经注意到它"带有前导零"!

Answer 3

您可以在wget url中使用echo类型序列来下载一串数字......

wget http://someaddress.com/logs/dbsclog01s00{1..3}.log

这也适用于字母

{a..z} {A..Z}

Answer 4

不确定你遇到了什么问题,但听起来像bash中的简单for循环会为你做.

for i in {1..999}; do
wget -k http://someaddress.com/logs/dbsclog01s$i.log -O your_local_output_dir_$i;
done

Answer 5

可以使用的组合为循环I A与n的bash printf的命令(当然修改echo以wget根据需要):

$ for i in {1..10}; do echo "http://www.com/myurl`printf "%03d" $i`.html"; done
http://www.com/myurl001.html
http://www.com/myurl002.html
http://www.com/myurl003.html
http://www.com/myurl004.html
http://www.com/myurl005.html
http://www.com/myurl006.html
http://www.com/myurl007.html
http://www.com/myurl008.html
http://www.com/myurl009.html
http://www.com/myurl010.html