Wget Command Line Tool
Wget is a powerful command-line utility for retrieving files from the web using various protocols including HTTP, HTTPS, and FTP. It's an essential tool for developers and system administrators for downloading content, automating file transfers, and creating local mirrors of websites.
Essential Wget Commands for File Retrieval
Below are common `wget` commands to download files efficiently:
# Download a file from a given URL and save it in the current directory
wget http://example.com/file.zip
# Download a file and save it with a different name using the -O flag
wget -O newname.zip http://example.com/file.zip
# Download files in the background using the -b flag
wget -b http://example.com/file.zip
# Limit the download speed to conserve bandwidth with --limit-rate
wget --limit-rate=200k http://example.com/file.zip
# Download files from a list of URLs provided in a text file using -i
wget -i urls.txt
# Resume an incomplete download using the -c flag
wget -c http://example.com/file.zip
# Show download progress in a more readable format with --show-progress
wget --show-progress http://example.com/file.zip
# Download a file without checking the server's SSL certificate using --no-check-certificate
wget --no-check-certificate https://example.com/file.zip
# Download an entire website for offline browsing using mirroring options
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent http://example.com
# Download a file with HTTP authentication using --user and --password
wget --user=username --password=password http://example.com/secure-file.zip
# Download a file with cookies, useful for sites that require logins
wget --load-cookies cookies.txt http://example.com/file.zip
# Download a directory from an FTP server using recursive download (-r)
wget -r ftp://example.com/pub/
# Download only files newer than the local version using the -N flag
wget -N http://example.com/file.zip
# Download a file only if it has been updated on the server using --spider
wget -S --spider http://example.com/file.zip
# Quietly download a file, continuing where it left off if the connection fails
wget -qc http://example.com/file.zip
# Specify a location to download the given file using -O [PATH]
wget -qcO /path/to/save/file.zip http://example.com/file.zip
# Download URL using a specific user agent string with the -U flag
wget -U 'Mozilla/5.0' http://example.com/file.zip
Understanding Wget Protocols and Options
Wget supports multiple protocols, making it versatile for various download scenarios. Understanding its options allows for fine-grained control over the download process.
HTTP and HTTPS Downloads
The most common use case for `wget` is downloading files over HTTP and HTTPS. The basic command `wget [URL]` is sufficient for simple downloads.
FTP File Transfer
For downloading files from FTP servers, `wget` can be used with `ftp://` URLs. The `-r` option enables recursive downloading, useful for entire directories.
Resuming Interrupted Downloads
The `-c` (continue) option is invaluable for large files or unstable connections. It allows `wget` to pick up where a previous download left off, saving time and bandwidth.
Mirroring Websites
Wget's mirroring capabilities (`--mirror`) are powerful for creating offline copies of websites. Combined with options like `--convert-links` and `--page-requisites`, it can download a site with all its assets and make it browsable locally.