1GLOBUS-URL-COPY(1)       Grid Community Toolkit Manual      GLOBUS-URL-COPY(1)
2
3
4

NAME

6       globus-url-copy - globus-url-copy
7

SYNOPSIS

9       globus-url-copy [options] SOURCE-URL DESTINATION-URL
10

DESCRIPTION

12       The globus-url-copy program is a command line tool for multi-protocol
13       data movement. It supports gsiftp:// (GridFTP), ftp://, http://,
14       https://, sshftp:// and file:/// protocol specifiers in the URL.
15

OPTIONS

17       -help, -usage
18           Print help.
19
20       -versions
21           Print the versions of all modules that this program uses
22
23       -c, -continue-on-error
24           Do not die after any errors. By default, program will exit after
25           most errors.
26
27       -a, -ascii
28           Convert the file to/from ASCII format to/from local file format
29
30       -b, -binary
31           Do not apply any conversion to the files.  default
32
33       -f FILENAME
34           Read a list of URL pairs from filename. Each line should contain
35           sourceURL destURL. Enclose URLs with spaces in double qoutes (").
36           Blank lines and lines beginning with # will be ignored.
37
38       -cd, -create-dest
39           Create destination directory if needed.
40
41       -r
42           Copy files in subdirectories
43
44       -fast
45           Recommended when using GridFTP servers. Use MODE E for all data
46           transfers, including reusing data channels between list and
47           transfer operations.
48
49       -t SECONDS
50           Run the transfer for this number of seconds and then end. Useful
51           for performance testing or forced restart loops.
52
53       -q, -quiet
54           Suppress all output for successful operation.
55
56       -v, -verbose
57           Display URLs being transferred
58
59       -vb, -verbose-perf
60           During the transfer, display the number of bytes transferred and
61           the transfer rate per second. Show URLs being transferred
62
63       -dbg, -debugftp
64           Debug ftp connections. Prints control channel communication to
65           stderr
66
67       -rst, -restart
68           Restart failed ftp operations.
69
70       -rst-retries RETRIES
71           The maximum number of times to retry the operation before giving up
72           on the transfer. Use 0 for infinite. Default is 5.
73
74       -rst-interval SECONDS
75           The interval in seconds to wait after a failure before retrying the
76           transfer. Use 0 for an exponential backoff. Default is 0.
77
78       -rst-timeout SECONDS
79           Maximum time after a failure to keep retrying. Use 0 for no
80           timeout. Default is 0.
81
82       -stall-timeout SECONDS, -st SECONDS
83           How long before cancelling/restarting a transfer with no data
84           movement. Set to 0 to disable. Default is 600 seconds.
85
86       -df FILENAME, -dumpfile FILENAME
87           Path to a file where untransferred URLs will be saved for later
88           restarting. Resulting file is the same format as the -f input file.
89           If file exists, it will be read and all other URL input will be
90           ignored.
91
92       -do FILENAME, -dump-only FILENAME
93           Perform no write operations on the destination. Instead, all files
94           that would be transferred are enumerated and dumped to the
95           specified file. Resulting file is the same format as the -f input
96           file. Note: if you intend to use this file as input for a future
97           transfer, the -create-dest option will be required if any
98           destination directories do not already exist.
99
100       -rp, -relative-paths
101           The path portion of ftp URLs will be interpreted as relative to the
102           user’s starting directory on the server. By default, all paths are
103           root-relative. When this flag is set, the path portion of the ftp
104           URL must start with %2F if it designates a root-relative path.
105
106       -s SUBJECT, -subject SUBJECT
107           Use this subject to match with both the source and dest servers.
108
109       -ss SUBJECT, -source-subject SUBJECT
110           Use this subject to match with the source server
111
112       -ds SUBJECT, -dest-subject SUBJECT
113           Use this subject to match with the destination server.
114
115       -tcp-bs SIZE, -tcp-buffer-size SIZE
116           Specify the size (in bytes) of the buffer to be used by the
117           underlying ftp data channels.
118
119       -bs block SIZE, -block-size block SIZE
120           Specify the size (in bytes) of the buffer to be used by the
121           underlying transfer methods.
122
123       -p PARALLELISM, -parallel PARALLELISM
124           Specify the number of parallel data connections should be used.
125
126       -notpt, -no-third-party-transfers
127           Turn third-party transfers off (on by default).
128
129       -nodcau, -no-data-channel-authentication
130           Turn off data channel authentication for ftp transfers.
131
132       -dcsafe, -data-channel-safe
133           Set data channel protection mode to SAFE
134
135       -dcpriv, -data-channel-private
136           Set data channel protection mode to PRIVATE
137
138           Warning
139           Despite having -dcpriv in its command line globus-url-copy will
140           silently fall back to an unencryted data channel when connected to
141           a server that does not support data channel encryption (e.g.
142           dCache)!
143
144       -off, -partial-offset
145           Offset for partial ftp file transfers, defaults to 0.
146
147       -len, -partial-length
148           Length for partial ftp file transfers, used only for the source
149           url, defaults the full file.
150
151       -list URL
152           List the files located at URL.
153
154       -stripe
155           Enable striped transfers on supported servers.
156
157       -striped-block-size, -sbs
158           Set layout mode and block size for striped transfers. If not set,
159           server defaults will be used. If set to 0, Partitioned mode will be
160           used. If set to > 0, Blocked mode will be used, with this as the
161           block size.
162
163       -ipv6
164           Use ipv6 when available (EXPERIMENTAL)
165
166       -udt
167           Use UDT, a reliable udp based transport protocol, for data
168           transfers
169
170       -g2, -gridftp2
171           Use GridFTP v2 protocol enhancements when possible.
172
173       -dp, -delayed-pasv
174           Enable delayed passive.
175
176       -mn NAME, -module-name NAME
177           Set the back-end storage module to use for both the source and
178           destination in a GridFTP transfer.
179
180       -mp PARAMETERS, -module-parameters PARAMETERS
181           Set the back-end storage module arguments to use for both the
182           source and destination in a GridFTP transfer.
183
184       -smn NAME, -src-module-name NAME
185           Set the back-end storage module to use for the source in a GridFTP
186           transfer.
187
188       -smp PARAMETERS, -src-module-parameters PARAMETERS
189           Set the back-end storage module arguments to use for the source in
190           a GridFTP transfer.
191
192       -dmn NAME, -dst-module-name NAME
193           Set the back-end storage module to use for the destination in a
194           GridFTP transfer.
195
196       -dmp PARAMETERS, -dst-module-parameters PARAMETERS
197           Set the back-end storage module arguments to use for the
198           destination in a GridFTP transfer.
199
200       -aa FILE, -authz-assert FILE
201           Use the assertions in FILE to authorize the access with both source
202           and destination servers.
203
204       -saa FILE, -src-authz-assert FILE
205           Use the assertions in this file to authorize the access with source
206           server.
207
208       -daa FILE, -dst-authz-assert FILE
209           Use the assertions in this file to authorize the access with dest
210           server.
211
212       -cache-aa, -cache-authz-assert
213           Cache the authz assertion for subsequent transfers.
214
215       -cache-saa, -cache-src-authz-assert
216           Cache the src authz assertion for subsequent transfers.
217
218       -cache-daa, -cache-dst-authz-assert
219           Cache the dst authz assertion for subsequent transfers.
220
221       -pipeline, -pp
222           Enable pipelining support for multi-file ftp transfers. Currently
223           third-party transfers benefit from this.  EXPERIMENTAL
224
225       -concurrency, -cc
226           Number of concurrent ftp connections to use for multiple transfers.
227
228       -nl-bottleneck, -nlb
229           Use NetLogger to estimate speeds of disk and network read/write
230           system calls, and attempt to determine the bottleneck component.
231
232       -sp COMMANDS, -src-pipe COMMANDS
233           Set the source end of a remote transfer to use piped in input with
234           the given command line. Do not use with -fsstack.
235
236       -DP COMMANDS, -dst-pipe COMMANDS
237           Set the destination end of a remote transfer to write data to then
238           standard input of the program run via the given command line. Do
239           not use with -fsstack.
240
241       -pipe COMMANDS
242           Sets both -src-pipe and -dst-pipe to the same thing.
243
244       -dcstack STACK, -data-channel-stack STACK
245           Set the XIO driver stack for the network on both the source and the
246           destination. Both must be GridFTP servers. The stack should contain
247           all network drivers to use, in the order specified from bottom to
248           top (e.g. -dcstack tcp,gsi). If the gsi driver is not included in
249           the stack and data channel authentication is enabled, it will be
250           inserted above the transport driver in the stack.
251
252       -fsstack STACK, -file-system-stack STACK
253           Set the XIO driver stack for the disk on both the source and the
254           destination. Both must be GridFTP servers. The stack should contain
255           all file system drivers to use, in the order specified from bottom
256           to top.
257
258       -src-dcstack STACK, -source-data-channel-stack STACK
259           Set the XIO driver stack for the network on the source GridFTP
260           server. See -dcstack above for description of the STACK string.
261
262       -src-fsstack STACK, -source-file-system-stack STACK
263           Set the XIO driver stack for the disk on the source GridFTP server.
264           See -fsstack above for description of the STACK string.
265
266       -dst-dcstack STACK, -dest-data-channel-stack STACK
267           Set the XIO driver stack for the network on the destination GridFTP
268           server. See -dcstack above for description of the STACK string.
269
270       -dst-fsstack STACK, -dest-file-system-stack STACK
271           Set the XIO driver stack for the disk on the destination GridFTP
272           server. See -fsstack above for description of the STACK string.
273
274       -cred PATH
275           Set the credentials to use for both ftp connections.
276
277       -src-cred CRED-FILE, -sc CRED-FILE
278           Set the credentials to use for source ftp connections.
279
280       -dst-cred CRED-FILE, -dc CRED-FILE
281           Set the credentials to use for destination ftp connections.
282
283       -af FILENAME, -alias-file FILENAME
284           File with mapping of logical host aliases to lists of physical
285           hosts. When used with multiple concurrent connections, each
286           connection uses the next host in the list. Each line should either
287           be an alias, noted with the @ symbol, or a hostname[:port].
288           Currently, only the aliases @source and @destination are valid, and
289           they are used for every source or destination URL.
290
291       -sync
292           Only transfer files where the destination does not exist or differs
293           from the source. -sync-level controls how to determine if files
294           differ.
295
296       -sync-level number
297           Criteria for determining if files differ when performing a sync
298           transfer. The default sync level is 2. The available levels are:
299
300           ·   Level 0 will only transfer if the destination does not exist.
301
302           ·   Level 1 will transfer if the size of the destination does not
303               match the size of the source.
304
305           ·   Level 2 will transfer if the time stamp of the destination is
306               older than the time stamp of the source.
307
308           ·   Level 3 will perform a checksum of the source and destination
309               and transfer if the checksums do not match. The default
310               algorithm used for this checksum is MD5, but other algorithms
311               can be specified with the -algo parameter.
312
313       -checksum-alg CHECKSUM-ALGORITHM
314           Set the algorithm type to use for all checksum operations during
315           the transfer.
316
317       -verify-checksum
318           Perform a checksum on the source and destination after each file
319           transfer and compare the two. If they do not match, fail the
320           transfer. The default algorithm used for this checksum is MD5, but
321           other algorithms can be specified with the -checksum-alg parameter.
322

ENVIRONMENT

324       The following environment variables are recognized:
325
326       GLOBUS_HOSTNAME
327           Overrides client hostname communicated to the server
328
329       GLOBUS_DOMAIN_NAME
330           Overrides client domainname communicated to the server
331
332       GLOBUS_FTP_CLIENT_DATA_IP
333           Overwrites client IP address communicated to the server for the
334           data connection. Used when the receiver is behind a NAT and should
335           be configured to the external address of its gateway.
336
337       GLOBUS_FTP_CLIENT_SOURCE_PASV
338           Configures sender side (source) instead of receiver side
339           (destination) to open a data port and establishes the data
340           connection from the receiver to the sender. By default the data
341           connection is established from sender to receiver, but if the
342           receiver is behind a NAT this doesn’t work, because the receiver
343           will communicate its own address instead of the external address of
344           its gateway to the sender and its own address can’t be reached
345           directly from the sender.
346

AUTHOR

348       Copyright © 1999-2016 University of Chicago
349
350
351
352Grid Community Toolkit 6          09/08/2020                GLOBUS-URL-COPY(1)
Impressum