htcp, htmv, htrm, htls, htll, htmkdir, htfind, htping - file transfers and queries via

       Andrew McNab <Andrew.McNab@manchester.ac.uk>

       htcp is part of GridSite: http://www.gridsite.org/

htcp is a client to fetch files or directory listings from remote servers using HTTP or HTTPS, or to  put
       or  delete  files  or  directories  onto  remote servers using HTTPS. htcp is similar to scp(1), but uses
       HTTP/HTTPS rather than ssh as its transfer protocol. htcp can also use the HTCP protocol to query HTTP(S)
       fileservers via SiteCast.

       When talking to a fileserver with  HTTPS,  htcp  can  run  "anonymously",  with  a  standard  X.509  user
       certificate  and  key,  or  with a GSI Proxy. This makes htcp very useful in Grid environments where many
       users have certificates and where jobs and users have access to GSI proxies.

       X509_CERT_DIR
              Holds directory to search for Certification Authority  root  certificates  when  verifying  server
              certificates. (Tried if --capath is not given on the command line.)

       X509_USER_PROXY
              Holds file name of a GSI Proxy to use as user certificate. (Tried if --cert or --key are not given
              on the command line.)

       X509_USER_CERT and X509_USER_KEY
              Holds file name of X.509 user certificate and key. (Tried if X509_USER_PROXY is not valid.)

       0  is  returned  on  complete success. Curl error codes are returned when reported by the underlying curl
       library, and CURLE_HTTP_RETURNED_ERROR (22) is returned when the HTTP(S) server returns  a  code  outside
       the range 200-299.  The manpage libcurl-errors(3) lists all the curl error codes.

       /tmp/x509up_uID
              Default GSI Proxy file for Unix UID equal to ID.

       /etc/grid-security/certificates
              Default location for trusted Certification Authority root certificates to use when checking server
              certificates.

       /tmp/.ca-roots-XXXXXX
              Prior to 7.9.8, the underlying curl library did not support the CA  root  certificates  directory.
              If  built  with  an old version of libcurl, htcp will concatenate the certificates in the CA roots
              directory into a unique temporary file and use that.

htcp,htmv,htrm,htls,htll,htmkdir,htfind,htping  -  file  transfers  and  queries  via
       HTTP/HTTPS/SiteCast

       -v/--verbose
              Turn on debugging information. Used once, this option will enable htcp's messages to stderr.  Used
              twice, will also enable the underlying libcurl messages.

       --delete
              Instead  of  copying files, delete all the URLs given on the command line.  Calling the program as
              htrm has the same effect.

       --list Instead of copying files, output lists of files  located  in  the  URL-directories  given  on  the
              command line. Calling the program as htls has the same effect.

       --long-list
              Instead  of  copying  files, output long listings of files located in the URL-directories given on
              the command line. If available, the size in bytes and modification time of  each  file  is  given.
              Calling the program as htll has the same effect.

       --mkdir
              Instead  of  copying  files,  attempt  to create a directory on a remote server with HTTP PUT. The
              server must support the convention that PUT to  a  URL  with  a  trailing  slash  means  create  a
              directory. No file body is sent. Calling the program as htmkdir has the same effect.

       --move Move/rename  files  on  a  single  remote  server, given the two, absolute URLs of the remote file
              names. Server must support HTTP/WebDAV MOVE. Calling the program as htmv has the same effect.

       --ping Query specified multicast groups with the  HTCP  NOP  ("No  Operation")  code.   SiteCast  enabled
              servers  will  respond immediately with a NOP reply, and all of the responses will be listed, with
              the round trip time in milliseconds.  Any waiting times specified in the --groups option  will  be
              ignored.  Calling  the  program  as  htping  has the same effect.  (--groups must be used for this
              option to work.)

       --find Query specified multicast groups with the HTCP TST code. SiteCast  enabled  servers  will  respond
              with  TST replies if they have the files corresponding to the given SiteCast target URL(s). All of
              the transfer URLs returned will be listed. Waiting times specified in the --groups option will  be
              used  to  space  out  the  multicast  queries, but the program listens for responses continuously.
              Calling the program as htfind has the same effect.  (--groups must be  used  for  this  option  to
              work.)

       --groups <IP Groups>
              IP multicast groups to use for SiteCast queries. IP Groups is a comma separated list of groups, in
              the  format: nnn.nnn.nnn.nnn:port[:ttl[:seconds]] The IP number and port must be specified. The IP
              time-to-live, ttl, controls how many networks  the  multicast  packets  may  pass  through  -  the
              default,  1,  limits  packets to the local network. Multiple groups may be specified, separated by
              commas. If multiple groups are specified, then seconds is the time to wait before making the  next
              multicast - 1 second is the default.

       --timeout <seconds>
              A request timeout used for multicast ping.

       --anon Do  not  attempt to use X.509 user certificates or GSI proxies to authenticate to the remote HTTPS
              server. This means you are "anonymous", but the server's identity may still be  verified  and  the
              connection is still encrypted.

       --cert <X.509 cert path>  and  --key <X.509 key path>
              Path  to the PEM-encoded X.509 or GSI Proxy user certificate and key to use for HTTPS connections,
              instead of "anonymous mode." If only one of --key or --cert is given, then that will be tried  for
              both.  If  neither is given, then the following order of precedence is used: the file name held by
              the variable X509_USER_PROXY; the file /tmp/x509up_uID (with Unix UID equal to ID); the file names
              held by X509_USER_CERT / X509_USER_KEY; the files ~/.globus/usercert.pem and ~/.globus/userkey.pem
              (where ~/ is the home directory of the user.)

       --capath <X.509 CA root certs directory or file>
              Path to the  PEM-encoded  CA  root  certificates  to  use  when  verifying  remote  servers'  host
              certificates in HTTPS connections. Ideally this should be a directory of hash.0 files as described
              in  the  OpenSSL verify(1) man page, but a file may be used instead. If --capath is not given, the
              value of the environment variable X509_CERT_DIR will  be  tried.   If  this  is  not  valid,  then
              /etc/grid-security/certificates will be used.

       --no-verify
              Do  not  use CA root certificates to verify remote servers' host certificates.  This is useful for
              testing sites before their certificate is set up properly, but leaves you vulnerable  to  "man  in
              the middle" attacks by hostile servers masquerading as your target.

       --grid-http
              Try to use GridHTTP redirection for HTTPS URLs. Compatible servers will perform authentication and
              authorization  on the HTTPS connection and then redirect to HTTP for the GET or PUT file transfer.
              htcp makes the HTTP request using the GRID_AUTH_PASSCODE single-use passcode obtained  via  HTTPS.
              The  --grid-http  option  will  be  ignored for directory operations or HTTP URLs. If a redirected
              transfer isn't possible, a normal HTTPS data transfer will be attempted.

       --sitecast
              Try to use SiteCast to locate remote files which are to be copied (currently only for the fetching
              of remote files.) If no location is found via SiteCast, then a direct request for the given URL is
              tried. (--groups must be used for this option to work.)

       --domain <SiteCast domain>
              Try to use SiteCast to locate remote files which are to be copied (currently only for the fetching
              of remote files) ifthedomaincomponentoftheURLmatches the  SiteCast  domain  given.   If  no
              location  is  found via SiteCast, then a direct request for the given URL is tried. (--groups must
              be used for this option to work.)

scp(1),curl(1),wget(1),verify(1),libcurl-errors(3)

htcp                                              October 2005                                           HTCP(1)

htcp,htmv [options] Source-URL[s] Destination-URL

       htrm,htls,htll,htmkir,htfind [options] Target-URL[s]

       htping [options]

       Recursive copying. Server-side wildcards. Parallel streams. Better error recovery.

       htcp supports the file:, http: and https: URL schemes as sources and destinations. If no scheme is given,
       the URL scheme is assumed to be file: and relative to the current directory if not an absolute path.

       If multiple sources are given during a copy, they will be used in turn and  the  destination  must  be  a
       directory  (directories  are indicated by a trailing /) However, source and destination cannot both refer
       to remote servers.