1HTCP(1) GridSite Manual HTCP(1)
2
6 htcp, htmv, htrm, htls, htll, htmkdir, htfind, htping - file transfers
7 and queries via HTTP/HTTPS/SiteCast
8
10 htcp, htmv [options] Source-URL[s] Destination-URL
11 htrm, htls, htll, htmkir, htfind [options] Target-URL[s]
13 htping [options]
15
17 htcp is a client to fetch files or directory listings from remote
18 servers using HTTP or HTTPS, or to put or delete files or directories
19 onto remote servers using HTTPS. htcp is similar to scp(1), but uses
20 HTTP/HTTPS rather than ssh as its transfer protocol. htcp can also use
21 the HTCP protocol to query HTTP(S) fileservers via SiteCast.
22 When talking to a fileserver with HTTPS, htcp can run "anonymously",
24 with a standard X.509 user certificate and key, or with a GSI Proxy.
25 This makes htcp very useful in Grid environments where many users have
26 certificates and where jobs and users have access to GSI proxies.
27
30 htcp supports the file:, http: and https: URL schemes as sources and
31 destinations. If no scheme is given, the URL scheme is assumed to be
32 file: and relative to the current directory if not an absolute path.
33 If multiple sources are given during a copy, they will be used in turn
35 and the destination must be a directory (directories are indicated by a
36 trailing /) However, source and destination cannot both refer to remote
37 servers.
38
41 -v/--verbose
42 Turn on debugging information. Used once, this option will
43 enable htcp's messages to stderr. Used twice, will also enable
44 the underlying libcurl messages.
45 --delete
48 Instead of copying files, delete all the URLs given on the com‐
49 mand line. Calling the program as htrm has the same effect.
50 --list Instead of copying files, output lists of files located in the
53 URL-directories given on the command line. Calling the program
54 as htls has the same effect.
55 --long-list
58 Instead of copying files, output long listings of files located
59 in the URL-directories given on the command line. If available,
60 the size in bytes and modification time of each file is given.
61 Calling the program as htll has the same effect.
62 --mkdir
65 Instead of copying files, attempt to create a directory on a
66 remote server with HTTP PUT. The server must support the conven‐
67 tion that PUT to a URL with a trailing slash means create a
68 directory. No file body is sent. Calling the program as htmkdir
69 has the same effect.
70 --move Move/rename files on a single remote server, given the two,
73 absolute URLs of the remote file names. Server must support
74 HTTP/WebDAV MOVE. Calling the program as htmv has the same
75 effect.
76 --ping Query specified multicast groups with the HTCP NOP ("No Opera‐
79 tion") code. SiteCast enabled servers will respond immediately
80 with a NOP reply, and all of the responses will be listed, with
81 the round trip time in milliseconds. Any waiting times speci‐
82 fied in the --groups option will be ignored. Calling the program
83 as htping has the same effect. (--groups must be used for this
84 option to work.)
85 --find Query specified multicast groups with the HTCP TST code. Site‐
88 Cast enabled servers will respond with TST replies if they have
89 the files corresponding to the given SiteCast target URL(s). All
90 of the transfer URLs returned will be listed. Waiting times
91 specified in the --groups option will be used to space out the
92 multicast queries, but the program listens for responses contin‐
93 uously. Calling the program as htfind has the same effect.
94 (--groups must be used for this option to work.)
95 --groups <IP Groups>
98 IP multicast groups to use for SiteCast queries. IP Groups is a
99 comma separated list of groups, in the format:
100 nnn.nnn.nnn.nnn:port[:ttl[:seconds]] The IP number and port must
101 be specified. The IP time-to-live, ttl, controls how many net‐
102 works the multicast packets may pass through - the default, 1,
103 limits packets to the local network. Multiple groups may be
104 specified, separated by commas. If multiple groups are speci‐
105 fied, then seconds is the time to wait before making the next
106 multicast - 1 second is the default.
107 --timeout <seconds>
110 A request timeout used for multicast ping.
111 --anon Do not attempt to use X.509 user certificates or GSI proxies to
114 authenticate to the remote HTTPS server. This means you are
115 "anonymous", but the server's identity may still be verified and
116 the connection is still encrypted.
117 --cert <X.509 cert path> and --key <X.509 key path>
120 Path to the PEM-encoded X.509 or GSI Proxy user certificate and
121 key to use for HTTPS connections, instead of "anonymous mode."
122 If only one of --key or --cert is given, then that will be tried
123 for both. If neither is given, then the following order of
124 precedence is used: the file name held by the variable
125 X509_USER_PROXY; the file /tmp/x509up_uID (with Unix UID equal
126 to ID); the file names held by X509_USER_CERT / X509_USER_KEY;
127 the files ~/.globus/usercert.pem and ~/.globus/userkey.pem
128 (where ~/ is the home directory of the user.)
129 --capath <X.509 CA root certs directory or file>
132 Path to the PEM-encoded CA root certificates to use when verify‐
133 ing remote servers' host certificates in HTTPS connections. Ide‐
134 ally this should be a directory of hash.0 files as described in
135 the OpenSSL verify(1) man page, but a file may be used instead.
136 If --capath is not given, the value of the environment variable
137 X509_CERT_DIR will be tried. If this is not valid, then
138 /etc/grid-security/certificates will be used.
139 --no-verify
142 Do not use CA root certificates to verify remote servers' host
143 certificates. This is useful for testing sites before their
144 certificate is set up properly, but leaves you vulnerable to
145 "man in the middle" attacks by hostile servers masquerading as
146 your target.
147 --grid-http
150 Try to use GridHTTP redirection for HTTPS URLs. Compatible
151 servers will perform authentication and authorization on the
152 HTTPS connection and then redirect to HTTP for the GET or PUT
153 file transfer. htcp makes the HTTP request using the
154 GRID_AUTH_PASSCODE single-use passcode obtained via HTTPS. The
155 --grid-http option will be ignored for directory operations or
156 HTTP URLs. If a redirected transfer isn't possible, a normal
157 HTTPS data transfer will be attempted.
158 --sitecast
161 Try to use SiteCast to locate remote files which are to be
162 copied (currently only for the fetching of remote files.) If no
163 location is found via SiteCast, then a direct request for the
164 given URL is tried. (--groups must be used for this option to
165 work.)
166 --domain <SiteCast domain>
169 Try to use SiteCast to locate remote files which are to be
170 copied (currently only for the fetching of remote files) if the
171 domain component of the URL matches the SiteCast domain given.
172 If no location is found via SiteCast, then a direct request for
173 the given URL is tried. (--groups must be used for this option
174 to work.)
175
178 /tmp/x509up_uID
179 Default GSI Proxy file for Unix UID equal to ID.
180 /etc/grid-security/certificates
183 Default location for trusted Certification Authority root cer‐
184 tificates to use when checking server certificates.
185 /tmp/.ca-roots-XXXXXX
188 Prior to 7.9.8, the underlying curl library did not support the
189 CA root certificates directory. If built with an old version of
190 libcurl, htcp will concatenate the certificates in the CA roots
191 directory into a unique temporary file and use that.
192
195 X509_CERT_DIR
196 Holds directory to search for Certification Authority root cer‐
197 tificates when verifying server certificates. (Tried if --capath
198 is not given on the command line.)
199 X509_USER_PROXY
202 Holds file name of a GSI Proxy to use as user certificate.
203 (Tried if --cert or --key are not given on the command line.)
204 X509_USER_CERT and X509_USER_KEY
207 Holds file name of X.509 user certificate and key. (Tried if
208 X509_USER_PROXY is not valid.)
209
212 0 is returned on complete success. Curl error codes are returned when
213 reported by the underlying curl library, and CURLE_HTTP_RETURNED_ERROR
214 (22) is returned when the HTTP(S) server returns a code outside the
215 range 200-299. The manpage libcurl-errors(3) lists all the curl error
216 codes.
217
220 Recursive copying. Server-side wildcards. Parallel streams. Better
221 error recovery.
222
225 Andrew McNab <Andrew.McNab@manchester.ac.uk>
226 htcp is part of GridSite: http://www.gridsite.org/
228
230 scp(1), curl(1), wget(1), verify(1), libcurl-errors(3)
231htcp October 2005 HTCP(1)