Comments on: Parallel downloads with wget

By: Eduardo Bustamante

Eduardo Bustamante — Sun, 12 Feb 2012 22:31:32 +0000

I think it’s more efficient if you do something like:

split url.txt url.txt.; ls -1 url.txt.* | xargs -n 1 -P 10 wget -i

This solves the one-URL-one-Wget problem, since in this case you will get one Wget instance per file. Also, if the URLs are from the same domain, and if the server supports it, Wget will use persistent connections, further speeding up the downloads.

By: Ole Tange

Ole Tange — Fri, 11 Jun 2010 09:26:38 +0000

Using GNU Parallel http://www.gnu.org/software/parallel/ is it even shorter:

cat url.txt | parallel wget

By: Sciencia-Geek » Descargas en paralelo con Wget

Sciencia-Geek » Descargas en paralelo con Wget — Tue, 25 May 2010 16:41:35 +0000

[...] http://techblog.zabuchy.net/ Written by Jp Neira in: Computación / Internet | Etiquetas: descargas, wget, ¡¡¡Software [...]