Curl is a powerful command-line tool for transferring data with URLs. It supports various protocols, including HTTP, HTTPS, FTP, and more, making it an indispensable utility for developers and system administrators. This guide covers essential Curl commands and options for interacting with web services, APIs, and downloading files.

Curl Basics and Essential Commands

Understanding the fundamental commands of Curl is key to leveraging its capabilities. Here are some of the most frequently used options:

#!/bin/bash
##############################################################################
# BASICS
##############################################################################

# Display help information
curl -h                 # help
curl --help             # same as -h

# Show the full manual page
curl --manual           # whole man page

# Enable verbose output for detailed connection information
curl -v                 # verbose
curl -vv                # even more verbose

# Redirect output to a file
curl http://url/file > file
curl -o file http://url/file # write output to a specific file
curl --output file http://url/file # same as -o

# Save output to a file named as the remote file
curl -O http://url/remote-file.zip # uses the remote filename

# Execute a remote script securely
bash <(curl -s http://url/myscript.sh)

# Download only the HTTP headers
curl -I url             # display header

# Basic authentication
curl --user username:password http://example.com/
curl -u username:password http://example.com/

# Handling SSL certificates
# -k, --insecure: Allow insecure server connections when using SSL (use with caution)
curl -k https://server_with_self_signed_cert/endpoint
curl --insecure https://server_with_self_signed_cert/endpoint

# Specify HTTP request method
# -X, --request <command>: Specify the request command to use (e.g., GET, POST)
# Example:
curl -X GET http://url/endpoint

# Add custom HTTP headers
# -H, --header <header/@file>: Pass custom header(s) to the server
# Example:
curl -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36' http://url/endpoint

# Sending HTTP POST data
# -d, --data <data>: Send data in a POST request
# -d, --data @data: Send data from a file in a POST request
# Example for JSON POST:
curl -d '{json}' -H 'Content-Type: application/json' http://url/endpoint

# Using a configuration file
curl -K file            # Read configuration from a specified file
curl --config file      # Same as -K
# Default config file location on UNIX-like systems
$HOME/.curlrc           # Default config file

Advanced Curl Output and Timing

Curl offers detailed control over its output, including writing out specific information after a transfer completes and timing various stages of the request. This is invaluable for performance analysis and debugging.

##############################################################################
# WRITE OUT PARAMETERS
##############################################################################

# -w, --write-out <format>: Use output FORMAT after completion
# Example: Print the size of the header when accessing google.com
curl -w %{size_header} --silent -o /dev/null http://gogle.com

# Supported FORMAT specifiers:
# %{content_type}       # Content-Type of the document.
# %{filename_effective} # The ultimate filename Curl writes out to (with --remote-name or --output).
# %{ftp_entry_path}     # Initial path on the remote FTP server.
# %{response_code}      # Numerical HTTP response code.
# %{http_connect}       # Numerical code from a proxy CONNECT response.
# %{local_ip}           # IP address of the local end of the connection.
# %{local_port}         # Local port number of the connection.
# %{num_connects}       # Number of new connections made.
# %{num_redirects}      # Number of redirects followed.
# %{redirect_url}       # The URL a redirect would take you to (without -L).
# %{remote_ip}          # Remote IP address of the connection.
# %{remote_port}        # Remote port number of the connection.
# %{size_download}      # Total number of bytes downloaded.
# %{size_header}        # Total number of bytes of downloaded headers.
# %{size_request}       # Total number of bytes sent in the HTTP request.
# %{size_upload}        # Total number of bytes uploaded.
# %{speed_download}     # Average download speed in bytes per second.
# %{speed_upload}       # Average upload speed in bytes per second.
# %{ssl_verify_result}  # Result of SSL peer certificate verification (0 for success).
# %{time_appconnect}    # Time in seconds for SSL/SSH/etc connect/handshake.
# %{time_connect}       # Time in seconds for TCP connect to the remote host/proxy.
# %{time_namelookup}    # Time in seconds for name resolving.
# %{time_pretransfer}   # Time in seconds until the file transfer is about to begin.
# %{time_redirect}      # Time in seconds for all redirection steps.
# %{time_starttransfer} # Time in seconds until the first byte is about to be transferred.
# %{time_total}         # Total time in seconds for the full operation.
# %{url_effective}      # The URL that was fetched last.

External Resources for Curl

• Curl Manual - The official and comprehensive documentation for Curl.
• HTTP Request Methods - Understand the different HTTP methods Curl can use.
• HTTP Headers - Learn about common HTTP headers you might use with Curl.
• Curl on Stack Overflow - Find solutions and discussions for common Curl problems.