1HTCP(1)                         GridSite Manual                        HTCP(1)
2
3
4

NAME

6       htcp,  htmv, htrm, htls, htll, htmkdir, htfind, htping - file transfers
7       and queries via HTTP/HTTPS/SiteCast
8

SYNOPSIS

10       htcp, htmv [options] Source-URL[s] Destination-URL
11
12       htrm, htls, htll, htmkir, htfind [options] Target-URL[s]
13
14       htping [options]
15

DESCRIPTION

17       htcp is a client to fetch  files  or  directory  listings  from  remote
18       servers  using  HTTP or HTTPS, or to put or delete files or directories
19       onto remote servers using HTTPS. htcp is similar to  scp(1),  but  uses
20       HTTP/HTTPS  rather than ssh as its transfer protocol. htcp can also use
21       the HTCP protocol to query HTTP(S) fileservers via SiteCast.
22
23       When talking to a fileserver with HTTPS, htcp  can  run  "anonymously",
24       with  a  standard  X.509 user certificate and key, or with a GSI Proxy.
25       This makes htcp very useful in Grid environments where many users  have
26       certificates and where jobs and users have access to GSI proxies.
27
28

URLs

30       htcp  supports  the  file:, http: and https: URL schemes as sources and
31       destinations. If no scheme is given, the URL scheme is  assumed  to  be
32       file: and relative to the current directory if not an absolute path.
33
34       If  multiple sources are given during a copy, they will be used in turn
35       and the destination must be a directory (directories are indicated by a
36       trailing /) However, source and destination cannot both refer to remote
37       servers.
38
39

OPTIONS

41       -v/--verbose
42              Turn on debugging  information.  Used  once,  this  option  will
43              enable  htcp's  messages to stderr. Used twice, will also enable
44              the underlying libcurl messages.
45
46
47       --delete
48              Instead of copying files, delete all the URLs given on the  com‐
49              mand line.  Calling the program as htrm has the same effect.
50
51
52       --list Instead  of  copying files, output lists of files located in the
53              URL-directories given on the command line. Calling  the  program
54              as htls has the same effect.
55
56
57       --long-list
58              Instead  of copying files, output long listings of files located
59              in the URL-directories given on the command line. If  available,
60              the  size  in bytes and modification time of each file is given.
61              Calling the program as htll has the same effect.
62
63
64       --mkdir
65              Instead of copying files, attempt to create  a  directory  on  a
66              remote server with HTTP PUT. The server must support the conven‐
67              tion that PUT to a URL with a  trailing  slash  means  create  a
68              directory.  No file body is sent. Calling the program as htmkdir
69              has the same effect.
70
71
72       --move Move/rename files on a single  remote  server,  given  the  two,
73              absolute  URLs  of  the  remote  file names. Server must support
74              HTTP/WebDAV MOVE. Calling the  program  as  htmv  has  the  same
75              effect.
76
77
78       --ping Query  specified  multicast groups with the HTCP NOP ("No Opera‐
79              tion") code.  SiteCast enabled servers will respond  immediately
80              with  a NOP reply, and all of the responses will be listed, with
81              the round trip time in milliseconds.  Any waiting  times  speci‐
82              fied in the --groups option will be ignored. Calling the program
83              as htping has the same effect.  (--groups must be used for  this
84              option to work.)
85
86
87       --find Query  specified  multicast groups with the HTCP TST code. Site‐
88              Cast enabled servers will respond with TST replies if they  have
89              the files corresponding to the given SiteCast target URL(s). All
90              of the transfer URLs returned  will  be  listed.  Waiting  times
91              specified  in  the --groups option will be used to space out the
92              multicast queries, but the program listens for responses contin‐
93              uously.  Calling  the  program  as  htfind  has the same effect.
94              (--groups must be used for this option to work.)
95
96
97       --groups <IP Groups>
98              IP multicast groups to use for SiteCast queries. IP Groups is  a
99              comma    separated    list    of    groups,   in   the   format:
100              nnn.nnn.nnn.nnn:port[:ttl[:seconds]] The IP number and port must
101              be  specified.  The IP time-to-live, ttl, controls how many net‐
102              works the multicast packets may pass through - the  default,  1,
103              limits  packets  to  the  local  network. Multiple groups may be
104              specified, separated by commas. If multiple  groups  are  speci‐
105              fied,  then  seconds  is the time to wait before making the next
106              multicast - 1 second is the default.
107
108
109       --timeout <seconds>
110              A request timeout used for multicast ping.
111
112
113       --anon Do not attempt to use X.509 user certificates or GSI proxies  to
114              authenticate  to  the  remote  HTTPS  server. This means you are
115              "anonymous", but the server's identity may still be verified and
116              the connection is still encrypted.
117
118
119       --cert <X.509 cert path>  and  --key <X.509 key path>
120              Path  to the PEM-encoded X.509 or GSI Proxy user certificate and
121              key to use for HTTPS connections, instead of  "anonymous  mode."
122              If only one of --key or --cert is given, then that will be tried
123              for both. If neither is  given,  then  the  following  order  of
124              precedence   is  used:  the  file  name  held  by  the  variable
125              X509_USER_PROXY; the file /tmp/x509up_uID (with Unix  UID  equal
126              to  ID);  the file names held by X509_USER_CERT / X509_USER_KEY;
127              the  files  ~/.globus/usercert.pem   and   ~/.globus/userkey.pem
128              (where ~/ is the home directory of the user.)
129
130
131       --capath <X.509 CA root certs directory or file>
132              Path to the PEM-encoded CA root certificates to use when verify‐
133              ing remote servers' host certificates in HTTPS connections. Ide‐
134              ally  this should be a directory of hash.0 files as described in
135              the OpenSSL verify(1) man page, but a file may be used  instead.
136              If  --capath is not given, the value of the environment variable
137              X509_CERT_DIR will  be  tried.   If  this  is  not  valid,  then
138              /etc/grid-security/certificates will be used.
139
140
141       --no-verify
142              Do  not  use CA root certificates to verify remote servers' host
143              certificates.  This is useful for  testing  sites  before  their
144              certificate  is  set  up  properly, but leaves you vulnerable to
145              "man in the middle" attacks by hostile servers  masquerading  as
146              your target.
147
148
149       --grid-http
150              Try  to  use  GridHTTP  redirection  for  HTTPS URLs. Compatible
151              servers will perform authentication  and  authorization  on  the
152              HTTPS  connection  and  then redirect to HTTP for the GET or PUT
153              file  transfer.  htcp  makes  the   HTTP   request   using   the
154              GRID_AUTH_PASSCODE  single-use  passcode obtained via HTTPS. The
155              --grid-http option will be ignored for directory  operations  or
156              HTTP  URLs.  If  a  redirected transfer isn't possible, a normal
157              HTTPS data transfer will be attempted.
158
159
160       --sitecast
161              Try to use SiteCast to locate  remote  files  which  are  to  be
162              copied  (currently only for the fetching of remote files.) If no
163              location is found via SiteCast, then a direct  request  for  the
164              given  URL  is  tried. (--groups must be used for this option to
165              work.)
166
167
168       --domain <SiteCast domain>
169              Try to use SiteCast to locate  remote  files  which  are  to  be
170              copied  (currently only for the fetching of remote files) if the
171              domain component of the URL matches the SiteCast  domain  given.
172              If  no location is found via SiteCast, then a direct request for
173              the given URL is tried. (--groups must be used for  this  option
174              to work.)
175
176

FILES

178       /tmp/x509up_uID
179              Default GSI Proxy file for Unix UID equal to ID.
180
181
182       /etc/grid-security/certificates
183              Default  location  for trusted Certification Authority root cer‐
184              tificates to use when checking server certificates.
185
186
187       /tmp/.ca-roots-XXXXXX
188              Prior to 7.9.8, the underlying curl library did not support  the
189              CA root certificates directory.  If built with an old version of
190              libcurl, htcp will concatenate the certificates in the CA  roots
191              directory into a unique temporary file and use that.
192
193

ENVIRONMENT

195       X509_CERT_DIR
196              Holds  directory to search for Certification Authority root cer‐
197              tificates when verifying server certificates. (Tried if --capath
198              is not given on the command line.)
199
200
201       X509_USER_PROXY
202              Holds  file  name  of  a  GSI  Proxy to use as user certificate.
203              (Tried if --cert or --key are not given on the command line.)
204
205
206       X509_USER_CERT and X509_USER_KEY
207              Holds file name of X.509 user certificate  and  key.  (Tried  if
208              X509_USER_PROXY is not valid.)
209
210

EXIT CODES

212       0  is  returned on complete success. Curl error codes are returned when
213       reported by the underlying curl library, and  CURLE_HTTP_RETURNED_ERROR
214       (22)  is  returned  when  the HTTP(S) server returns a code outside the
215       range 200-299.  The manpage libcurl-errors(3) lists all the curl  error
216       codes.
217
218

TO DO

220       Recursive  copying.  Server-side  wildcards.  Parallel  streams. Better
221       error recovery.
222
223

AUTHOR

225       Andrew McNab <Andrew.McNab@manchester.ac.uk>
226
227       htcp is part of GridSite: http://www.gridsite.org/
228

SEE ALSO

230       scp(1), curl(1), wget(1), verify(1), libcurl-errors(3)
231
232
233
234htcp                             October 2005                          HTCP(1)
Impressum