1GLOBUS-URL-COPY(1) Grid Community Toolkit Manual GLOBUS-URL-COPY(1)
2
6 globus-url-copy - globus-url-copy
7
9 globus-url-copy [options] SOURCE-URL DESTINATION-URL
10
12 The globus-url-copy program is a command line tool for multi-protocol
13 data movement. It supports gsiftp:// (GridFTP), ftp://, http://,
14 https://, sshftp:// and file:/// protocol specifiers in the URL.
15
17 -help, -usage
18 Print help.
19 -versions
21 Print the versions of all modules that this program uses
22 -c, -continue-on-error
24 Do not die after any errors. By default, program will exit after
25 most errors.
26 -a, -ascii
28 Convert the file to/from ASCII format to/from local file format
29 -b, -binary
31 Do not apply any conversion to the files. default
32 -f FILENAME
34 Read a list of URL pairs from filename. Each line should contain
35 sourceURL destURL. Enclose URLs with spaces in double qoutes (").
36 Blank lines and lines beginning with # will be ignored.
37 -cd, -create-dest
39 Create destination directory if needed.
40 -r
42 Copy files in subdirectories
43 -fast
45 Recommended when using GridFTP servers. Use MODE E for all data
46 transfers, including reusing data channels between list and
47 transfer operations.
48 -t SECONDS
50 Run the transfer for this number of seconds and then end. Useful
51 for performance testing or forced restart loops.
52 -q, -quiet
54 Suppress all output for successful operation.
55 -v, -verbose
57 Display URLs being transferred
58 -vb, -verbose-perf
60 During the transfer, display the number of bytes transferred and
61 the transfer rate per second. Show URLs being transferred
62 -dbg, -debugftp
64 Debug ftp connections. Prints control channel communication to
65 stderr
66 -rst, -restart
68 Restart failed ftp operations.
69 -rst-retries RETRIES
71 The maximum number of times to retry the operation before giving up
72 on the transfer. Use 0 for infinite. Default is 5.
73 -rst-interval SECONDS
75 The interval in seconds to wait after a failure before retrying the
76 transfer. Use 0 for an exponential backoff. Default is 0.
77 -rst-timeout SECONDS
79 Maximum time after a failure to keep retrying. Use 0 for no
80 timeout. Default is 0.
81 -stall-timeout SECONDS, -st SECONDS
83 How long before cancelling/restarting a transfer with no data
84 movement. Set to 0 to disable. Default is 600 seconds.
85 -df FILENAME, -dumpfile FILENAME
87 Path to a file where untransferred URLs will be saved for later
88 restarting. Resulting file is the same format as the -f input file.
89 If file exists, it will be read and all other URL input will be
90 ignored.
91 -do FILENAME, -dump-only FILENAME
93 Perform no write operations on the destination. Instead, all files
94 that would be transferred are enumerated and dumped to the
95 specified file. Resulting file is the same format as the -f input
96 file. Note: if you intend to use this file as input for a future
97 transfer, the -create-dest option will be required if any
98 destination directories do not already exist.
99 -rp, -relative-paths
101 The path portion of ftp URLs will be interpreted as relative to the
102 user’s starting directory on the server. By default, all paths are
103 root-relative. When this flag is set, the path portion of the ftp
104 URL must start with %2F if it designates a root-relative path.
105 -s SUBJECT, -subject SUBJECT
107 Use this subject to match with both the source and dest servers.
108 -ss SUBJECT, -source-subject SUBJECT
110 Use this subject to match with the source server
111 -ds SUBJECT, -dest-subject SUBJECT
113 Use this subject to match with the destination server.
114 -tcp-bs SIZE, -tcp-buffer-size SIZE
116 Specify the size (in bytes) of the buffer to be used by the
117 underlying ftp data channels.
118 -bs block SIZE, -block-size block SIZE
120 Specify the size (in bytes) of the buffer to be used by the
121 underlying transfer methods.
122 -p PARALLELISM, -parallel PARALLELISM
124 Specify the number of parallel data connections should be used.
125 -notpt, -no-third-party-transfers
127 Turn third-party transfers off (on by default).
128 -nodcau, -no-data-channel-authentication
130 Turn off data channel authentication for ftp transfers.
131 -dcsafe, -data-channel-safe
133 Set data channel protection mode to SAFE
134 -dcpriv, -data-channel-private
136 Set data channel protection mode to PRIVATE
137 Warning
139 Despite having -dcpriv in its command line globus-url-copy will
140 silently fall back to an unencryted data channel when connected to
141 a server that does not support data channel encryption (e.g.
142 dCache)!
143 -off, -partial-offset
145 Offset for partial ftp file transfers, defaults to 0.
146 -len, -partial-length
148 Length for partial ftp file transfers, used only for the source
149 url, defaults the full file.
150 -list URL
152 List the files located at URL.
153 -stripe
155 Enable striped transfers on supported servers.
156 -striped-block-size, -sbs
158 Set layout mode and block size for striped transfers. If not set,
159 server defaults will be used. If set to 0, Partitioned mode will be
160 used. If set to > 0, Blocked mode will be used, with this as the
161 block size.
162 -ipv6
164 Use ipv6 when available (EXPERIMENTAL)
165 -udt
167 Use UDT, a reliable udp based transport protocol, for data
168 transfers
169 -g2, -gridftp2
171 Use GridFTP v2 protocol enhancements when possible.
172 -dp, -delayed-pasv
174 Enable delayed passive.
175 -mn NAME, -module-name NAME
177 Set the back-end storage module to use for both the source and
178 destination in a GridFTP transfer.
179 -mp PARAMETERS, -module-parameters PARAMETERS
181 Set the back-end storage module arguments to use for both the
182 source and destination in a GridFTP transfer.
183 -smn NAME, -src-module-name NAME
185 Set the back-end storage module to use for the source in a GridFTP
186 transfer.
187 -smp PARAMETERS, -src-module-parameters PARAMETERS
189 Set the back-end storage module arguments to use for the source in
190 a GridFTP transfer.
191 -dmn NAME, -dst-module-name NAME
193 Set the back-end storage module to use for the destination in a
194 GridFTP transfer.
195 -dmp PARAMETERS, -dst-module-parameters PARAMETERS
197 Set the back-end storage module arguments to use for the
198 destination in a GridFTP transfer.
199 -aa FILE, -authz-assert FILE
201 Use the assertions in FILE to authorize the access with both source
202 and destination servers.
203 -saa FILE, -src-authz-assert FILE
205 Use the assertions in this file to authorize the access with source
206 server.
207 -daa FILE, -dst-authz-assert FILE
209 Use the assertions in this file to authorize the access with dest
210 server.
211 -cache-aa, -cache-authz-assert
213 Cache the authz assertion for subsequent transfers.
214 -cache-saa, -cache-src-authz-assert
216 Cache the src authz assertion for subsequent transfers.
217 -cache-daa, -cache-dst-authz-assert
219 Cache the dst authz assertion for subsequent transfers.
220 -pipeline, -pp
222 Enable pipelining support for multi-file ftp transfers. Currently
223 third-party transfers benefit from this. EXPERIMENTAL
224 -concurrency, -cc
226 Number of concurrent ftp connections to use for multiple transfers.
227 -nl-bottleneck, -nlb
229 Use NetLogger to estimate speeds of disk and network read/write
230 system calls, and attempt to determine the bottleneck component.
231 -sp COMMANDS, -src-pipe COMMANDS
233 Set the source end of a remote transfer to use piped in input with
234 the given command line. Do not use with -fsstack.
235 -DP COMMANDS, -dst-pipe COMMANDS
237 Set the destination end of a remote transfer to write data to then
238 standard input of the program run via the given command line. Do
239 not use with -fsstack.
240 -pipe COMMANDS
242 Sets both -src-pipe and -dst-pipe to the same thing.
243 -dcstack STACK, -data-channel-stack STACK
245 Set the XIO driver stack for the network on both the source and the
246 destination. Both must be GridFTP servers. The stack should contain
247 all network drivers to use, in the order specified from bottom to
248 top (e.g. -dcstack tcp,gsi). If the gsi driver is not included in
249 the stack and data channel authentication is enabled, it will be
250 inserted above the transport driver in the stack.
251 -fsstack STACK, -file-system-stack STACK
253 Set the XIO driver stack for the disk on both the source and the
254 destination. Both must be GridFTP servers. The stack should contain
255 all file system drivers to use, in the order specified from bottom
256 to top.
257 -src-dcstack STACK, -source-data-channel-stack STACK
259 Set the XIO driver stack for the network on the source GridFTP
260 server. See -dcstack above for description of the STACK string.
261 -src-fsstack STACK, -source-file-system-stack STACK
263 Set the XIO driver stack for the disk on the source GridFTP server.
264 See -fsstack above for description of the STACK string.
265 -dst-dcstack STACK, -dest-data-channel-stack STACK
267 Set the XIO driver stack for the network on the destination GridFTP
268 server. See -dcstack above for description of the STACK string.
269 -dst-fsstack STACK, -dest-file-system-stack STACK
271 Set the XIO driver stack for the disk on the destination GridFTP
272 server. See -fsstack above for description of the STACK string.
273 -cred PATH
275 Set the credentials to use for both ftp connections.
276 -src-cred CRED-FILE, -sc CRED-FILE
278 Set the credentials to use for source ftp connections.
279 -dst-cred CRED-FILE, -dc CRED-FILE
281 Set the credentials to use for destination ftp connections.
282 -af FILENAME, -alias-file FILENAME
284 File with mapping of logical host aliases to lists of physical
285 hosts. When used with multiple concurrent connections, each
286 connection uses the next host in the list. Each line should either
287 be an alias, noted with the @ symbol, or a hostname[:port].
288 Currently, only the aliases @source and @destination are valid, and
289 they are used for every source or destination URL.
290 -sync
292 Only transfer files where the destination does not exist or differs
293 from the source. -sync-level controls how to determine if files
294 differ.
295 -sync-level number
297 Criteria for determining if files differ when performing a sync
298 transfer. The default sync level is 2. The available levels are:
299 · Level 0 will only transfer if the destination does not exist.
301 · Level 1 will transfer if the size of the destination does not
303 match the size of the source.
304 · Level 2 will transfer if the time stamp of the destination is
306 older than the time stamp of the source.
307 · Level 3 will perform a checksum of the source and destination
309 and transfer if the checksums do not match. The default
310 algorithm used for this checksum is MD5, but other algorithms
311 can be specified with the -algo parameter.
312 -checksum-alg CHECKSUM-ALGORITHM
314 Set the algorithm type to use for all checksum operations during
315 the transfer.
316 -verify-checksum
318 Perform a checksum on the source and destination after each file
319 transfer and compare the two. If they do not match, fail the
320 transfer. The default algorithm used for this checksum is MD5, but
321 other algorithms can be specified with the -checksum-alg parameter.
322
324 The following environment variables are recognized:
325 GLOBUS_HOSTNAME
327 Overrides client hostname communicated to the server
328 GLOBUS_DOMAIN_NAME
330 Overrides client domainname communicated to the server
331 GLOBUS_FTP_CLIENT_DATA_IP
333 Overwrites client IP address communicated to the server for the
334 data connection. Used when the receiver is behind a NAT and should
335 be configured to the external address of its gateway.
336 GLOBUS_FTP_CLIENT_SOURCE_PASV
338 Configures sender side (source) instead of receiver side
339 (destination) to open a data port and establishes the data
340 connection from the receiver to the sender. By default the data
341 connection is established from sender to receiver, but if the
342 receiver is behind a NAT this doesn’t work, because the receiver
343 will communicate its own address instead of the external address of
344 its gateway to the sender and its own address can’t be reached
345 directly from the sender.
346
348 Copyright © 1999-2016 University of Chicago
349Grid Community Toolkit 6 09/08/2020 GLOBUS-URL-COPY(1)