1UBERFTP(1C) UBERFTP(1C)
2
6 uberftp - GridFTP-enabled client
7
9 uberftp [options] [host options] [host]
10 uberftp [options] [host options] host “cmd”
12 uberftp [options] srcurl dsturl
14 uberftp [options] -f urlfile
16 uberftp [options] -cmd url
18
21 uberftp is a GridFTP-enabled client that supports both interactive use
22 and FTP commands on the uberftp command line to transfer files between
23 two computers. It is intended for use with computers that have a
24 GridFTP server installed. Uberftp supports GSI authentication, parallel
25 data channels and striping. For more information about GridFTP, see the
26 GridFTP URL in the "SEE ALSO" section below.
27 Only the first usage shown above will create an interactive session. If
29 host is specified, uberftp immediately attempts to establish a connec‐
30 tion to the GridFTP server on host, enters its command interpreter and
31 awaits commands from the user. If host is not specified, uberftp imme‐
32 diately drops into the command interpreter without connecting to any
33 GridFTP server.
34 The second usage option denotes a non interactive session in which
36 “cmd” is a series of one or more commands as described in the COMMANDS
37 section below to run once the control connection is established. These
38 commands are run exactly as if they had been entered from the interac‐
39 tive prompt. This list must be enclosed in quotes. Multiple commands
40 are semicolon or comma delimited. uberftp will execute these commands
41 and then exit. Uberftp will exit upon the first error encountered.
42 The third and forth usage statements use the URL style format for spec‐
44 ifying the source and destination for the files to transfer. The third
45 usage statement places these URLs on the command line. The forth usage
46 allows the user to specify multiple URL pairs in a separate file for
47 Uberftp to transfer one at a time. The supported URL syntaxes are
48 gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file
49 and file://path.
50 The fifth usage statement allows for commands that take pathnames to
52 accept URLs instead. The allowable commands are listed in the -cmds
53 section below.
54
57 -P port
58 Connect to port instead of the default. The default for GSI au‐
59 thentication is 2811. The default for password authentication is
60 21.
61 -u user
63 Connect as user. This is useful for both password and GSI au‐
64 thentication mechanisms.
65 -p pass
67 Use pass when authenticating. If pass equals X, UberFTP will
68 prompt for the password with character echoing turned off.
69
73 -active
74 Use ACTIVE mode for data transfers.
75 -ascii Use ASCII mode for data transfers.
77 -binary
79 Use BINARY mode for data transfers.
80 -blksize n
82 Set the internal buffer size to n.
83 -cksum [on|off]
85 Enable/Disable CRC checks after file transfers.
86 -cos name
88 Set the storage class of service to name. Used with HPSS instal‐
89 lations. Use the class of service name default to allow the re‐
90 mote server to decide which class of service to use.
91 -d Enable debugging. Same as '-debug 3'. Deprecated.
93 -debug n
95 Set the debug level to n.
96 -family name
98 Set the storage family to name. Use the family name default to
99 allow the remote server to decide which family to use.
100 -glob [on|off]
102 Enable/Disable filename expansion.
103 -hash Enable printing of hash marks during transfers.
105 -keepalive n
107 Send control channel keepalive messages every n seconds during
108 data transfers.
109 -mode [E|S]
111 Switch the transfer mode to extended block (E) or streams mode
112 (S).
113 -parallel n
115 Use n parallel data channels during extended block transfers.
116 -passive
118 Use PASSIVE mode for data transfers.
119 -pbsz n
121 Set the data protection buffer size to n n bytes.
122 -prot [C|S|E|P]
124 Set the data protection lelvel to clear (C), safe (S), confiden‐
125 tial (E) or private (P).
126 -retry n
128 Retry commands that fail with transient errors n times.
129 -resume path
131 Retry the recursive transfer starting at path.
132 -tcpbuf n
134 Set the TCP read/write buffers to n bytes.
135 -wait This will cause the client to wait for remote files to stage be‐
137 fore attempting to transfer them.
138 -v Print UberFTP version information and exit.
140 -version
142 Print UberFTP version information and exit.
143 -versions
145 Print version information about all used globus modules and
146 exit.
147
150 -cat url
151 Print to stdout the contents of the remote file.
152 -chgrp [-r] group url
154 Set the group ownership of the remote object(s).
155 -chmod [-r] perms url
157 Set the permissions of the remote object(s).
158 -dir [-r] url
160 List the contents of the remote object.
161 -link url path
163 Create a hardlink named <path> to the remote object.
164 -ls [-r] url
166 List the contents of the remote object.
167 -mkdir url
169 Create the remote directory.
170 -rename url path
172 Rename the remote object to the given path.
173 -rm [-r] url
175 Remove the remote object(s).
176 -rmdir url
178 Remove the remote directory.
179 -size url
181 Return the size of the remote object.
182 -stage [-r] seconds url
184 Attempt to stage the remote object(s) over the time period given
185 in seconds.
186 -symlink url path
188 Create a symlink named <path> to the remote object.
189
192 By default, without any special environment variables, command line op‐
193 tions or commands, uberftp will transfer files in PASSIVE STREAMS mode.
194 PASSIVE means that the client will initiate the data connection which
195 is useful for users behind firewalls. STREAMS mode implies that GRIDFTP
196 features including striping and parallel data connections are not used.
197 In order to take advantage of these features with GridFTP capable
198 servers, you must either change the mode directly using -m command line
199 switch or the mode interactive command, or you can change the mode in‐
200 directly by specifying more than one parallel data connection using the
201 -c command line switch or by using the parallel interactive command.
202
205 By default, uberftp requires a GSI certificate. If you do not already
206 have a certificate, see the following web page to learn how to get one:
207 http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html
209 Once you have a certificate, use the grid-proxy-init command to get a
211 valid proxy.
212
215 ! [command]
216 Run the command using a shell on the local machine. If no com‐
217 mand is given, invoke an interactive shell.
218 ? [command]
220 If command is given, print a (hopefully) helpful blurb about it.
221 Otherwise, list all commands.
222 active Change to ACTIVE mode which causes the server to initiate the
224 data connection. The default is PASSIVE mode unless the variable
225 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
226 a firewall you must use PASSIVE mode.
227 ascii Change the data transfer type to ASCII which causes the server
229 to do some simple transformations to the file being transferred.
230 This is mostly useful for changing EOL (end of line) in text
231 files when moving between platforms. This option is almost
232 never necessary today. The default is BINARY mode also known as
233 IMAGE mode.
234 binary Change the data transfer type to BINARY (aka IMAGE) which causes
236 the server to not perform transformations to the file being
237 transferred. This is the default and is faster than an ASCII
238 transfer.
239 blksize size
241 Change the size of the memory buffer used to read and write data
242 to disks to size bytes. The default block size is 1024*1024
243 (1048576) bytes but it can be changed at compile time. The block
244 size can be increased to improve file transfer performance. This
245 is not related to the extended block mode block size used to de‐
246 termine the ratio of data to header for data transferred on the
247 data channel.
248 bugs Prints information regarding bug reporting and feature requests.
250 bye Close all control and data connections and exit.
252 cat file1 [file2 ... filen]
254 Print the contents of the remote file(s) to stdout.
255 cdup Change the remote working directory up one level.
257 cd [dir]
259 Change the remote working directory to dir. If dir is not given,
260 the client will make every attempt to change to the user's home
261 directory.
262 chgrp [-r] group object [object2 ... objectn]
265 Change group ownership on the remote object(s).
266 -r Recursively chgrp everything in the given directory.
267 chmod [-r] perms object [object2 ... objectn]
270 Change permissions on the remote object(s).
271 -r Recursively chmod everything in the given directory.
272 close Close the control connection to the remote host.
274 cksum [on|off]
276 Enable file cksum comparison after each file transfer. This only
277 works with NCSA's mass storage system.
278 on Enable checksum comparison
279 off Disable checksum comparison
280 cos name
282 Sets the HPSS class of service to name on the FTP service if the
283 service supports it. If name is omitted, the current class of
284 service is printed. Use the class of service name default to
285 allow the remote server to decide which class of service to use.
286 dcau [N|A|S subject]
288 Change the data channel authentication settings. If the service
289 does not support DCAU, these settings are ignored.
290 N Disabled dcau.
291 A Expect the remote identity to be mine. (Default)
292 S subject Expect the remote identity to be subject.
293 debug [0-3]
295 Turn debug statements on/off. If no value is given, this command
296 will toggle between debug(2) and non debug(1) mode. Otherwise
297 the debug level is set to the given level.
298 0 Only errors are printed
299 1 Default. Errors and some helpful messages are printed
300 2 Print useful control channel information
301 3 Print all information
302 family name
304 Sets the tape family to name on the FTP service if the service
305 supports it. If name is omitted, the current family is printed.
306 Use the family name default to allow the remote server to decide
307 which family to use.
308 glob [on|off]
310 Enable or disable filename globbing. If no option is given, this
311 command will toggle the current setting.
312 on Enable filename globbing
313 off Disable filename globbing
314 dir [-r] [target]
316 List the contents of the remote target directory. If target is
317 not given, then the current working directory is used.
318 -r Recursively list target.
319 target Directory or file to list. '.' is used by default.
320 get [-r] source [destination]
322 Retrieve file(s) from the remote service. If source implies mul‐
323 tiple transfers, either through regular expressions or by using
324 the recursive feature, then destination must be a directory. If
325 destination is not specified, source is used.
326 -r Recursively transfer the given directory.
327 hash Print hash marks during data transfers. This does not work dur‐
329 ing third party transfers.
330 help [command]
332 If command is given, print a helpful blurb about command. Oth‐
333 erwise, list all commands.
334 keepalive [seconds]
336 Attempts to keep the control channel from being blocked by fire‐
337 walls during long data channel operations. UberFTP sends a NOOP
338 command to the service at intervals equal to the specified num‐
339 ber of seconds. Setting it to zero will disable keepalive. If
340 seconds are not given, the current timeout is displayed. This
341 feature is disabled by default.
342 seconds number of seconds between NOOPs. Disabled if zero.
343 lcat file1 [file2 ... filen]
345 Print the contents of the local file(s) to stdout.
346 lcd [dir]
348 Change the local working directory to dir. If dir is not given,
349 the client will make every attempt to change to the user's home
350 directory.
351 lcdup Change the local working directory up one level.
353 lchgrp [-r] group object [object2 ... objectn]
355 Change group ownership on the local object(s).
356 -r Recursively chgrp everything in the given directory.
357 lchmod [-r] perms object [object2 ... objectn]
360 Change permissions on the local object(s).
361 -r Recursively chmod everything in the given directory.
362 lclose Close the control connection to the local host.
364 ldir [-r] [target]
366 List the contents of the local target directory. If target is
367 not given, then the current working directory is used.
368 -r Recursively list target.
369 target Directory or file to list. '.' is used by default.
370 link [oldfile] [newfile]
372 Create a hardlink to oldfile named newfile on the remote ser‐
373 vice.
374 llink [oldfile] [newfile]
376 Create a hardlink to oldfile named newfile on the local service.
377 lls [-r] [target]
379 List the contents of the local target directory. If target is
380 not given, then the current working directory is used.
381 -r Recursively list target.
382 target Directory or file to list. '.' is used by default.
383 llscos List the available class of services on the local server (HPSS
385 only).
386 llsfam List the available tape families on the local server (HPSS
388 only).
389 lmkdir dir1 [dir2 ... dirn]
391 Create the local directory(ies).
392 lopen [-P port] [-u user] [-p pass | X] host
394 Opens a control channel to host and that host becomes the 'lo‐
395 cal' machine. After using lopen, all local (l*) commands per‐
396 form their respective operations on host rather than the local
397 machine. This is how third party transfers are accomplished. GSI
398 authentication is used unless the -p option is used.
399 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
400 word).
401 -u user Connect as alternate user.
402 -p pass | X
403 Use password pass when authenticating with host.
404 If pass equals X, read the password from STDIN with
405 character echoing turned off.
406 host Connect to host.
407 lpwd Prints the current local working directory.
409 lrename src dst
411 Rename the local object src to dst.
412 lrm [-r] object1 [object1...objectn]
414 Removes the local file system object(s).
415 -r Recursively remove the given directory.
416 lrmdir dir1 [dir2...dirn]
418 Removes the given directories from the local service.
419 lquote cmd
421 Pass cmd to the local FTP service. This allows the user to use
422 server-specific commands that are not available through the
423 uberftp interface.
424 ls [-r] [target]
426 List the contents of the remote target directory. If [target] is
427 not given, then the current working directory is used.
428 -r Recursively list target.
429 target Directory or file to list. '.' is used by default.
430 lscos List the available class of services on the remote server (HPSS
432 only).
433 lsfam List the available tape families on the remote server (HPSS
435 only).
436 lsize file1 [file2...filen]
438 Prints the size of the given object(s).
439 lstage [-r] seconds object1 [object2...objectn]
441 Attempt to stage all matching files within the given number of
442 seconds on the local service.
443 seconds number of seconds to attempt staging
444 -r Recursively stage all files in the given subdirectory.
445 lsymlink [oldfile] [newfile]
447 Create a symlink to oldfile named newfile on the local service.
448 mput [-r] object1 [object2...objectn]
450 Retrieve file(s) from the remote service. This is similiar to
451 making multiple calls to get without specifying a destination.
452 -r Recursively transfer the given directory.
453 mkdir dir
455 Create the remote directory.
456 mode [E|S]
458 Toggle the data transfer mode between Streams mode and Extended
459 Block mode. The default is Streams mode. If no option is given,
460 it will display the current mode.
461 E Extended block mode
462 S Streams mode
463 mput [-r] object1 [object2...objectn]
465 Store file(s) to the remote service. This is similiar to making
466 multiple calls to put without specifying a destination.
467 -r Recursively transfer the given directory.
468 open [-P port] [-u user] [-p pass | X] host
470 Opens a control channel to host and that host becomes the 're‐
471 mote' machine. GSI authentication is used unless the -p option
472 is used.
473 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
474 word).
475 -u user Connect as user.
476 -p pass | X
477 Use password pass when authenticating with host.
478 If pass equals X, read the password from STDIN with
479 character echoing turned off.
480 host Connect to host.
481 order [type]
483 Changes the order of lists returned from ls and lls to the given
484 scheme. If type is not given, the current order is displayed.
485 type Ordering scheme to use. Value options are:
486 none Do not order listings
487 name Order listings by name
488 size Order listings by size
489 type Order listings by type
490 parallel [number]
492 Set the number of parallel data connections to number. This is
493 only useful for extended block mode transfers. The default num‐
494 ber of data connections is one. If no number is given, the cur‐
495 rent setting for the number of parallel connects is printed.
496 passive
498 Change to PASSIVE mode which causes the client to initiate the
499 data connection. This is the default mode unless the variable
500 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
501 a firewall you must use PASSIVE mode.
502 pbsz [size]
504 Change the length of the protection buffer. The protection buf‐
505 fer is used to encrypt data on the data channel. The length of
506 the protection buffer represents the largest encoded message
507 that is allowed on the data channel. By default, the protection
508 buffer is grown to match the internal buffer used. For efficient
509 transfers, pbsz should be sufficiently larger than blksize so
510 that the wrapped buffer fits within the protection buffer. Oth‐
511 erwise, the blksize buffer is broken into multiple pieces so
512 that each write is less than pbsz when wrapped. If pbsz is not
513 given, the current size is displayed.
514 size length of protection buffer. 0 will set it to its de‐
515 fault.
516 pget offset size srcfile [destfile]
518 Retrieve only the specified portion of the file(s). If srcfile
519 is a regular expression and expands to multiple files, and des‐
520 tination is given, destination must refer to a directory.
521 offset Offset within the file
522 size Amount of data to retrieve
523 srcfile Name of remote file
524 destfile Name of local file. srcfile is used if destfile
525 is not specified
526 pput offset size srcfile [destfile]
528 Store only the specified portion of the file(s). If srcfile is a
529 regular expression and expands to multiple files, and destina‐
530 tion is given, destination must refer to a directory.
531 offset Offset within the file
532 size Amount of data to retrieve
533 srcfile Name of local file
534 destfile Name of remote file. srcfile is used if destfile
535 is not specified
536 prot [C|S|E|P]
538 This command configures the level of security on the data chan‐
539 nel after data channel authentication has completed. Clear means
540 that the data will not be protected. Safe means that the data
541 will be integrity protected meaning that altered data will be
542 detected. Confidential means that the data will be unreadable to
543 third parties. Private mode means the data will be confidential
544 and safe.
545 C Set protection level to clear.
546 S Set protection level to safe.
547 E Set protection level to confidential.
548 P Set protection level to private.
549 put [-r] source [destination]
551 Store file(s) to the remote service. If source implies multiple
552 transfers, either through regular expressions or by using the
553 recursive feature, then destination must be a directory. If des‐
554 tination is not specified, source is used.
555 -r Recursively transfer the given directory.
556 pwd Prints the current working directory.
558 quit Close all control and data connections and exit.
560 quote cmd
562 Pass cmd to the remote FTP service. This allows the user to use
563 server-specific commands that are not available through the
564 uberftp interface.
565 rename src dst
567 Rename the remote object src to dst.
568 retry [cnt]
570 Configures retry on failed commands that have transient errors.
571 cnt represents the number of times a failed command is retried.
572 A value of zero effectively disables retry. Zero is the default.
573 If no value is given the current setting is displayed.
574 cnt Number of times a failed command is retried.
575 resume [-d] path
577 Sets a restart point for recursive transfers. If a long recur‐
578 sive transfer fails, you can set resume to the path that failed
579 and UberFTP will skip all file and directory creations up to the
580 given path.
581 path Path to resume transfer at. If path is not given, print
582 the current
583 resume target.
584 -d Remove the current resume path.
585 rm [-r] object1 [object1...objectn]
587 Removes the remote file system object(s).
588 -r Recursively remove the given directory.
589 rmdir dir1 [dir2...dirn]
591 Removes the given directories from the remote service.
592 runique
594 Toggles the client to store files using unique names during put
595 operations.
596 size file1 [file2...filen]
598 Prints the size of the given object(s).
599 stage [-r] seconds object1 [object2...objectn]
601 Attempt to stage all matching files within the given number of
602 seconds on the remote service.
603 seconds number of seconds to attempt staging
604 -r Recursively stage all files in the given subdirectory.
605 sunique
607 Toggles the client to store files using unique names during get
608 operations.
609 symlink [oldfile] [newfile]
611 Create a symlink to oldfile named newfile on the remote service.
612 tcpbuf [size]
614 Set the data channel TCP buffer size to size bytes. If size is
615 not given, the current TCP buffer size will be printed.
616 versions
618 Prints the versions of all Globus modules being used.
619 wait Toggles whether the client should wait for files to stage before
621 attempting to retrieve them.
622
626 Use the active command to enable active mode FTP when using NCSA's Uni‐
627 Tree mass storage system if possible since it will give much better
628 file transfer performance. When tranferring files over long distances,
629 use a large value (for example, 16777216) for tcpbuf. When there is
630 high network traffic, you may be able to improve performance using the
631 parallel command to increase the number of parallel data connections to
632 2-4.
633
635 In order to perform a third-party transfer, you must log into two FTP
636 servers. Typically, you connect to a single FTP service to "get" files
637 to the local machine and "put" files to the remote service. For third-
638 party transfers, you must connect to a second service thereby replacing
639 the former local machine. In UberFTP terminology, it is referred to as
640 "opening a new local service" since, from the perspective of the user,
641 the new local service will appear as though the user initiated the FTP
642 session from that machine.
643 All remote service commands have "l*" counterparts that allow you to
645 specify that the command is to be performed on the local service,
646 whether that service is the local machine or a new local service. So
647 to open a new local service, use the "l*" version of the open command:
648 UberFTP> lopen mss.ncsa.teragrid.org
650 UberFTP> lclose
651 Once you have connected to both services, files can be transferred as
653 before with the change that files you "get" go to the new local service
654 and files you "put" are sent from the new local service.
655
657 By default, local port selection is managed by the operating system.
658 However, you may wish to specify which ports UberFTP should use for in‐
659 coming and out going connections. This is useful when dealing with
660 firewalls.
661 Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all in‐
663 bound connections to use the specified port range. Likewise, setting
664 UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound
665 connections to use the specified port range.
666 The environment variables GLOBUS_TCP_PORT_RANGE and
668 GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection.
669 These variables behave exactly as their UBERFTP counterparts and are
670 available for backwards compatibility with older versions. The UBERFTP
671 variables take precedence over the GLOBUS variables.
672 The values of the variables specify a port range, a minimum port number
674 and a maximum port number, separated by either a comma or a space. For
675 example, to set the inbound port range, you would set:
676 UBERFTP_TCP_PORT_RANGE=40000,50000
678 Using the space delimiter, this format is also acceptable:
680 UBERFTP_TCP_PORT_RANGE="40000 50000"
682 See your shell documentation for the proper syntax for settings vari‐
684 ables within your environment.
685 Setting the ephemeral port range to an unusable range will cause
687 UberFTP connections to fail. For instance, setting a port range from 10
688 to 100 with a non root process will fail on most operating systems.
689
692 UberFTP will exit with a value of 0 if no errors occurred during the
693 session, otherwise it will exit with a value of 1. In non interactive,
694 commandline mode, it will exit after the first error occurs.
695
697 Set the environment variable to set active mode FTP (improves file
698 transfer performance to the mass storage system). Connect to NCSA's
699 UniTree mass storage system interactively from NCSA's TeraGrid cluster:
700 setenv UBERFTP_ACTIVE_MODE on
702 % uberftp mss.ncsa.teragrid.org
703 ...
704 220 UNIX Archive FTP server ready.
705 230 User consult logged in.
706 UberFTP>
707 Use the command-line interface to copy a file from NCSA's TeraGrid
709 cluster to the UniTree mass storage system. (There is no need to set
710 tcpbuf since it is over a LAN but active mode is turned on to improve
711 file transfer performance to the mass storage system.):
712 uberftp mss.ncsa.teragrid.org \
714 "active; cd work; get file.tar"
715 Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster.
717 (Note that tcpbuf is set to 16777216 since there is a long network la‐
718 tency between NCSA and SDSC):
719 uberftp tg-gridftp.sdsc.teragrid.org \
721 "tcpbuf 16777216; cd scr; put file.tar"
722
724 mssftp(1), msscmd(1), ftp(1),
725 GridFTP:
726 https://gridcf.org/gct-docs/latest/gridftp/
727 TCP Window Size:
728 http://www.vonwelch.com/report/tcp_windows/
729 http://www.psc.edu/tcp-tune
730 Active vs. Passive FTP:
731 http://slacksite.com/other/ftp.html
732 Note: The links above are not under the GridCF's control so they may
734 become obsolete.
735 12 Nov 2020 UBERFTP(1C)