nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)

       Richard W.M. Jones

       Parts derived from Alexander Graf's "QEMU Block driver for CURL images".

       Copyright Red Hat

-Dcurl.scripts=1
           This prints out the headers and cookies generated by the "header-script" and "cookie-script" options,
           which can be useful when debugging these scripts.

       -Dcurl.times=1
           This prints out additional information about the total time taken to do name resolution,  connect  to
           the  remote  server,  etc.   The information is printed in the debug output before nbdkit exits.  The
           output will look like:

            nbdkit: debug: times (-D curl.times=1):
            nbdkit: debug: name resolution               :    0.128442 s
            nbdkit: debug: connection                    :    4.945213 s
            nbdkit: debug: SSL negotiation               :    4.291362 s
            nbdkit: debug: pretransfer                   :    0.104137 s
            nbdkit: debug: first byte received           :   56.115269 s
            nbdkit: debug: data transfer                 :  222.633831 s
            nbdkit: debug: redirection time              :    0.000000 s

           The cumulative time taken to perform each step is shown (summed across all  HTTP  connections).   The
           redirection  time  is  the  total  time  taken  doing HTTP redirections.  For further information see
           "TIMES" in curl_easy_getinfo(3).

       -Dcurl.verbose=1
           This enables very verbose curl debugging.  See CURLOPT_VERBOSE(3).  This  is  mainly  useful  if  you
           suspect there is a bug inside libcurl itself.

       -Dcurl.verbose.ids=1
           This enhances "-D curl.verbose=1" by printing connection and transfer IDs next to each debug message.
           As this has some overhead it is not enabled by default.

       "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content served over HTTP, FTP, and more, into
       a Network Block Device.  It uses a library called libcurl (also known as cURL) to read data from URLs.
       The exact list of protocols that libcurl can handle depends on how it was compiled, but most versions
       will handle HTTP, HTTPS, FTP, FTPS and more (see: "curl -V").

       Note: This plugin supports writes.  However for HTTP, you may not want nbdkit to issue PUT requests to
       the remote server (which probably doesn't understand them).  To force nbdkit to use a readonly
       connection, pass the -r flag.  Using the -r flag also enables NBD multi-conn, which usually performs much
       better (if the client supports it).

       Although this plugin can access SFTP (ie. SSH) servers, it is much better to use nbdkit-ssh-plugin(1).
       This plugin can be used to access "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.

        nbdkit -r curl http://example.com/disk.img

       serves the remote disk image as NBD on TCP port 10809 (to control ports and protocols used to serve NBD
       see nbdkit(1)).

$plugindir/nbdkit-curl-plugin.so
           The plugin.

           Use "nbdkit --dump-config" to find the location of $plugindir.

       While  the  "header"  and "cookie" parameters can be used to specify static headers and cookies which are
       used in every HTTP/HTTPS request, the alternate "header-script" and  "cookie-script"  parameters  can  be
       used  to  run  an  external  script  or program to generate headers and/or cookies.  This is particularly
       useful to access services which require an authorization token.  In  addition  the  "header-script-renew"
       and  "cookie-script-renew"  parameters allow you to renew the authorization token by rerunning the script
       periodically.

       "header-script" is incompatible with "header", and "cookie-script" is incompatible with "cookie".

   Headerscript
       The header script should print zero or more HTTP headers, each line of output in the same format  as  the
       "header" parameter.  The headers printed by the script are passed to CURLOPT_HTTPHEADER(3).

       In  the  following example, an imaginary web service requires authentication using a token fetched from a
       separate login server.  The token expires after 60 seconds, so we also tell the plugin that it must renew
       the token (by re-running the script) if more than 50 seconds have elapsed since the last request:

        nbdkit curl https://service.example.com/disk.img \
               header-script='
                 printf "Authorization: Bearer "
                 curl -s -X POST https://auth.example.com/login |
                      jq -r .token
               ' \
               header-script-renew=50

   Cookiescript
       The cookie script should print a single line in the same format  as  the  "cookie"  parameter.   This  is
       passed to CURLOPT_COOKIE(3).

   Headerandcookiescriptshellvariables
       Within the "header-script" and "cookie-script" the following shell variables are available:

       $iteration
           The  number  of  times  that  the  script  has been called.  The first time the script is called this
           contains 0.

       $url
           The URL as passed to the plugin.

   Example:VMwareESXicookies
       VMware ESXi’s web server can expose both VMDK and raw format disk images, but  requires  you  to  log  in
       using  HTTP  Basic  Authentication.   While you can use the "user" and "password" parameters to send HTTP
       Basic Authentication headers in every request, tests have shown that it is faster to  accept  the  cookie
       which  the  server  returns  and send that instead.  (It is not clear why it is faster, but one theory is
       that VMware has to do a more expensive username and password check each time.)

       The web server can be accessed as below.  Since the cookie expires after a certain period of time, we use
       "cookie-script-renew", and because the server uses a self-signed certificate we must use  --insecure  and
       "sslverify=false".

        SERVER=esx.example.com
        DCPATH=data
        DS=datastore1
        GUEST=guest-name
        URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"

        nbdkit curl "$URL" \
               cookie-script='
                   curl --head -s --insecure -u root:password "$url" |
                        sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
               ' \
               cookie-script-renew=500 \
               sslverify=false

   Example:DockerHubauthorizationtokens
       Accessing  objects  like  container layers from Docker Hub requires that you first fetch an authorization
       token, even for anonymous access.  These tokens expire after about 5 minutes (300  seconds)  so  must  be
       periodically renewed.

       You will need this authorization script (/tmp/auth.sh):

        #!/bin/sh -
        IMAGE=library/fedora
        curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
             jq -r .token

       You will also need this script to get the blobSum of the layer (/tmp/blobsum.sh):

        #!/bin/sh -
        TOKEN=`/tmp/auth.sh`
        IMAGE=library/fedora
        curl -s -X GET -H "Authorization: Bearer $TOKEN" \
             "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
             jq -r '.fsLayers[0].blobSum'

       Both  scripts  must  be  executable,  and both can be run on their own to check they are working.  To run
       nbdkit:

        IMAGE=library/fedora
        BLOBSUM=`/tmp/blobsum.sh`
        URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"

        nbdkit curl "$URL" \
               header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
               header-script-renew=200 \
               --filter=gzip

       Note that this exposes a tar file over NBD.  See also nbdkit-tar-filter(1).

       Redistribution and use in source and binary forms, with or without modification, are  permitted  provided
       that the following conditions are met:

       •   Redistributions  of  source  code must retain the above copyright notice, this list of conditions and
           the following disclaimer.

       •   Redistributions in binary form must reproduce the above copyright notice, this list of conditions and
           the following disclaimer in the documentation and/or other materials provided with the distribution.

       •   Neither the name of Red Hat nor the names of its contributors may  be  used  to  endorse  or  promote
           products derived from this software without specific prior written permission.

       THIS  SOFTWARE  IS  PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
       INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND  FITNESS  FOR  A  PARTICULAR
       PURPOSE  ARE  DISCLAIMED.  IN  NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
       INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,  PROCUREMENT  OF
       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
       ON  ANY  THEORY  OF  LIABILITY,  WHETHER  IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
       DAMAGE.

nbdkit-1.42.2                                      2025-04-02                              nbdkit-curl-plugin(1)

       nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)

       nbdkit ≤ 1.32 used a simple model where a new NBD connection would create a new libcurl handle.  Since  a
       libcurl  handle  maintains  a  small cache of connections, this meant that the number of HTTP connections
       would be a small multiple of the number of incoming NBD connections and the total would  not  be  limited
       (assuming "http:" or "https:" URL).

       nbdkit  1.34 changed to using a fixed pool of libcurl handles shared across all NBD connections.  You can
       control the maximum number of curl handles in the pool with  the  "connections"  parameter  (default  4).
       Since  each  curl  handle  maintains  a  small  cache  of connections, this meant that the number of HTTP
       connections would be a small multiple of the "connections" parameter.  If there are more than 4  incoming
       NBD connections, they will contend for the libcurl handles, unless you adjust "connections".

       nbdkit  ≥  1.36  changed  again  to  use  a  curl multi handle (libcurl-multi(3)).  Now the "connections"
       parameter  controls  the   maximum   number   of   HTTP   connections   made   to   the   remote   server
       (CURLMOPT_MAX_TOTAL_CONNECTIONS(3)).   This  is  more  efficient especially with HTTP/2 and HTTP/3, where
       each HTTP connection can contain a very large number of streams (typically up to  100)  multiplexed  over
       one connection.  The default for "connections" was raised to 16.

cainfo=FILENAME
           (nbdkit ≥ 1.18)

           Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.

           Pass empty string in order to not use the default certificate store that libcurl is compiled with.

       capath=PATH
           (nbdkit ≥ 1.18)

           Set CA certificates directory location for libcurl. See CURLOPT_CAPATH(3) for more information.

       connections=N
           (nbdkit ≥ 1.34)

           Open up to "N" connections to the web server.  The default is 16.  Connections are shared between all
           NBD  clients,  so  you  may  wish  to increase this if you expect many simultaneous NBD clients (or a
           single client using many multi-conn connections).

           See "NBD CONNECTIONS AND CURL HANDLES" below.

       cookie=COOKIE
       cookie=+FILENAME
       cookie=-cookie=-FD
           (nbdkit ≥ 1.12)

           Set a cookie in the request header when connecting to the remote server.

           A typical example is:

            cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'

           This option can be used at most once.  It only works for HTTP and HTTPS transports.  To set  multiple
           cookies you must concatenate them yourself, eg:

            cookie='name1=content1; name2=content2'

           See  CURLOPT_COOKIE(3)  for more information about this.  The format is quite strict and must consist
           of "key=value", each cookie separated by exactly "; " (semicolon and space).

           If the cookie is used for authentication then passing it on the command line is not secure on  shared
           machines.   Use  the  alternate  "+FILENAME"  syntax  to  pass  it  in a file, "-" to read the cookie
           interactively, or "-FD" to read it from a file descriptor.

       cookiefile=
           (nbdkit ≥ 1.26)

           Enable cookie processing (eg. when following redirects), starting with  an  empty  list  of  cookies.
           This is equivalent to calling CURLOPT_COOKIEFILE(3) with an empty string.

       cookiefile=FILENAME
           (nbdkit ≥ 1.26)

           Enable  cookie  processing (eg. when following redirects), prepopulating cookies from the given file.
           The file can contain cookies in any format supported by  curl,  see  CURLOPT_COOKIEFILE(3).   Cookies
           sent by the server are not saved when nbdkit exits.

       cookiejar=FILENAME
           (nbdkit ≥ 1.26)

           Enable  cookie  processing (eg. when following redirects), prepopulating cookies from the given file,
           and writing server cookies back to the file when the NBD handle is  closed.   The  file  can  contain
           cookies in any format supported by curl, see CURLOPT_COOKIEJAR(3).

           There is some advice on the internet telling you to set this to /dev/null, but you shouldnot do this
           because  it  can corrupt /dev/null.  If you don't want a cookiejar, omit this option.  If you want to
           enable cookie processing without  updating  a  permanent  cookiejar,  use  the  "cookiefile="  option
           instead.

       cookie-script=SCRIPT
       cookie-script-renew=SECS
           (nbdkit ≥ 1.22, not Windows)

           Run   "SCRIPT"   (a   command   or  shell  script  fragment)  to  generate  the  HTTP/HTTPS  cookies.
           "cookie-script" cannot be used with "cookie".  See "HEADER AND COOKIE SCRIPTS" below.

       followlocation=false
           (nbdkit ≥ 1.26)

           Do not follow redirects from the server.  The default is true (follow redirects).

           You can follow redirects but avoid redirecting to a less secure protocol (eg.  HTTPS  redirecting  to
           FTP) by using the "protocols" parameter instead.

       header=HEADER
           (nbdkit ≥ 1.22)

           For  HTTP/HTTPS,  send  a  custom  header,  or  remove a header that curl has added.  To add a custom
           header:

            header='X-My-Name: John Doe'

           To remove a header that curl has added, add the header followed by a colon and no value:

            header='User-Agent:'

           To add a custom header that has no value, you have to use a semicolon instead of colon.  This adds an
           "X-Empty:" header with no value:

            header='X-Empty;'

           See CURLOPT_HTTPHEADER(3).  You can use this option multiple times in order to add  several  headers.
           Note  this  sends the header in all requests, even when following a redirect, which can cause headers
           (eg. containing sensitive authorization  information)  to  be  sent  to  hosts  other  than  the  one
           originally requested.

       header-script=SCRIPT
       header-script-renew=SECS
           (nbdkit ≥ 1.22, not Windows)

           Run   "SCRIPT"   (a   command   or  shell  script  fragment)  to  generate  the  HTTP/HTTPS  headers.
           "header-script" cannot be used with "header".  See "HEADER AND COOKIE SCRIPTS" below.

       http-version=nonehttp-version=1.0http-version=1.1http-version=2.0http-version=2TLShttp-version=2-prior-knowledgehttp-version=3http-version=3only
           (nbdkit ≥ 1.34)

           Force curl to use a particular HTTP protocol.  The default is "none" meaning curl will negotiate  the
           best   protocol   with   the   server.    The   other   settings   are   mainly   for  testing.   See
           CURLOPT_HTTP_VERSION(3) for details.

       ipresolve=anyipresolve=v4ipresolve=v6
           (nbdkit ≥ 1.36)

           Force curl to use only IPv4 ("ipresolve=v4"), only IPv6 ("ipresolve=v6") or any IP version  supported
           by your system ("ipresolve=any").  The default is "any".  See CURLOPT_IPRESOLVE(3).

       password=PASSWORD
           Set the password to use when connecting to the remote server.

           Note that passing this on the command line is not secure on shared machines.

       password=-
           Ask for the password (interactively) when nbdkit starts up.

       password=+FILENAME
           Read  the password from the named file.  This is a secure method to supply a password, as long as you
           set the permissions on the file appropriately.

       password=-FD
           Read the password from file descriptor number "FD", inherited from the  parent  process  when  nbdkit
           starts up.  This is also a secure method to supply a password.

       protocols=PROTO,PROTO,...
           (nbdkit ≥ 1.12)

           Limit the protocols that are allowed in the URL.  Use this option for extra security if the URL comes
           from an untrusted source and you want to avoid security isues in the more obscure protocols that curl
           supports.   (See  qemu  CVE-2013-0249  for  an  example  of  a  security  bug  introduced by allowing
           unrestricted protocols).

           For example if  you  only  intend  HTTP  and  HTTPS  URLs  to  be  used,  then  add  this  parameter:
           "protocols=http,https"

           This  restriction  also  applies  if the plugin follows a redirect to another protocol (eg. you start
           with an "https://" URL which the server redirects to "ftp://").  To prevent redirects altogether  see
           the "followlocation" parameter.

           The value of this parameter is a comma-separated list of protocols, for example "protocols=https", or
           "protocols=http,https",  or  "protocols=file,ftp,gopher".   For  a  list  of known protocols, see the
           libcurl manual (CURLOPT_PROTOCOLS_STR(3)), and in nbdkit ≥ 1.40 the output of:

            nbdkit curl --dump-plugin

           The default is to allow any protocol.

       proxy=PROXY
           (nbdkit ≥ 1.20)

           Set the proxy.  See CURLOPT_PROXY(3).

       proxy-password=PASSWORD
       proxy-password=-proxy-password=+FILENAME
       proxy-password=-FD
       proxy-user=USERNAME
           (nbdkit ≥ 1.12)

           Set the proxy username and password.

       resolve=HOST:PORT:ADDRESS
           Provide custom host name to IP address resolution.  You can supply  this  option  as  many  times  as
           needed.  See CURLOPT_RESOLVE(3) for the full details of this option.

       sslverify=false
           Don't verify the SSL certificate of the remote host.

       ssl-cipher-list=CIPHER[:CIPHER...]
       ssl-version=defaultssl-version=tlsv1ssl-version=sslv2ssl-version=sslv3ssl-version=tlsv1.0ssl-version=tlsv1.1ssl-version=tlsv1.2ssl-version=tlsv1.3ssl-version=max-defaultssl-version=max-tlsv1.0ssl-version=max-tlsv1.1ssl-version=max-tlsv1.2ssl-version=max-tlsv1.3
           Set  the  SSL  ciphers  and  TLS version.  For further information see CURLOPT_SSL_CIPHER_LIST(3) and
           CURLOPT_SSLVERSION(3).

       tcp-keepalive=true
           (nbdkit ≥ 1.20)

           Enable TCP keepalives.

       tcp-nodelay=false
           (nbdkit ≥ 1.20)

           Enable Nagle’s algorithm.  Small writes on the network socket will not be sent immediately  but  will
           be  held in a local buffer and coalesced if possible.  This is more efficient for the network but can
           cause increased latency.

           The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is disabled for  better  latency  at  the
           expense of network efficiency.

           See CURLOPT_TCP_NODELAY(3).

       timeout=SECS
           Set the timeout for requests.

       timeout=0
           Use the default libcurl timeout for requests.

       tls13-ciphers=CIPHER[:CIPHER...]
           Select       TLSv1.3       ciphers       available.        See      CURLOPT_TLS13_CIPHERS(3)      and
           https://curl.se/docs/ssl-ciphers.html

       unix-socket-path=PATH
           (nbdkit ≥ 1.10)

           Instead of using a TCP connection, connect to the server over the  named  Unix  domain  socket.   See
           CURLOPT_UNIX_SOCKET_PATH(3).

       [url=]URL
           The URL of the remote disk image.  This is passed to libcurl directly via CURLOPT_URL(3).

           This parameter is required.

           "url=" is a magic config key and may be omitted in most cases.  See "Magic parameters" in nbdkit(1).

       user=USERNAME
           Set  the  username to use when connecting to the remote server.  This may also be set in the URL (eg.
           "http://foo@example.com/disk.img")

       user-agent=USER-AGENT
           (nbdkit ≥ 1.22)

           Send user-agent header when using HTTP or HTTPS.  The default is no user-agent header.

curl(1),  libcurl(3),  CURLOPT_CAINFO(3),  CURLOPT_CAPATH(3),  CURLOPT_COOKIE(3),  CURLOPT_COOKIEFILE(3),
       CURLOPT_COOKIEJAR(3),     CURLOPT_FOLLOWLOCATION(3),     CURLOPT_HTTPHEADER(3),     CURLOPT_IPRESOLVE(3),
       CURLOPT_PROXY(3),       CURLOPT_RESOLVE(3),       CURLOPT_SSL_CIPHER_LIST(3),      CURLOPT_SSLVERSION(3),
       CURLOPT_TCP_KEEPALIVE(3),     CURLOPT_TCP_NODELAY(3),      CURLOPT_TLS13_CIPHERS(3),      CURLOPT_URL(3),
       CURLOPT_UNIX_SOCKET_PATH(3),         CURLOPT_USERAGENT(3),         CURLOPT_VERBOSE(3),         nbdkit(1),
       nbdkit-extentlist-filter(1),                nbdkit-file-plugin(1),                nbdkit-retry-filter(1),
       nbdkit-retry-request-filter(1),    nbdkit-ssh-plugin(1),    nbdkit-torrent-plugin(1),   nbdkit-plugin(3),
       http://curl.haxx.se, https://curl.se/docs/ssl-ciphers.html

        nbdkit -r curl [url=]http://example.com/disk.img

       "nbdkit-curl-plugin" first appeared in nbdkit 1.2.