Sequence expressions in wget and curl #

Given a range of JPGs to batch download:
one approach using wget would be:
$ wget ''{00..99}'.jpg'
Bash 4 is required for brace expansion using a range. Unfortunately, the expression fails to expand at all when read from a file (e.g., $ wget -i urls.txt).

curl, on the other hand, does not depend on Bash for sequencing:
$ curl -O ''[00-99]'.jpg'
so it will correctly parse sequences in a properly-formatted text file (e.g., urls.txt):
url = "[00-99].jpg"
url = "[00-99].jpg"
url = "[00-99].jpg"
like so:
$ curl --remote-name-all -K urls.txt

UPDATE: For sites that require a login, cookie.txt export makes it easy to grab the necessary cookies.txt file from Google Chrome's cache. Then it's simply a matter of:

$ wget --user-agent="" --force-directories --load-cookies cookies.txt ''{1..10}


$ curl --header "User-Agent:" --cookie ./cookies.txt --remote-name ''[1-10]

If filenames need to be made unique (e.g., downloaded files have the same name and would otherwise be overwritten), curl has variable renaming built-in:

$ curl --output "index_#1.html" ''[2012-2015]'/index.html'


/nix | Sep 12, 2011

Subscribe or visit the archives.