1UBERFTP(1) General Commands Manual UBERFTP(1)
2
6 uberftp - GridFTP-enabled client
7
9 uberftp [options] [host options] [host]
10 uberftp [options] [host options] host “cmd”
12 uberftp [options] srcurl dsturl
14 uberftp [options] -f urlfile
16 uberftp [options] -cmd url
18
21 uberftp is a GridFTP-enabled client that supports both interactive use
22 and FTP commands on the uberftp command line to transfer files between
23 two computers. It is intended for use with computers that have a
24 GridFTP server installed. Uberftp supports GSI authentication, parallel
25 data channels and striping. For more information about GridFTP, see the
26 GridFTP URL in the "SEE ALSO" section below.
27 Only the first usage shown above will create an interactive session. If
29 host is specified, uberftp immediately attempts to establish a connec‐
30 tion to the GridFTP server on host, enters its command interpreter and
31 awaits commands from the user. If host is not specified, uberftp imme‐
32 diately drops into the command interpreter without connecting to any
33 GridFTP server.
34 The second usage option denotes a non interactive session in which
36 “cmd” is a series of one or more commands as described in the COMMANDS
37 section below to run once the control connection is established. These
38 commands are run exactly as if they had been entered from the interac‐
39 tive prompt. This list must be enclosed in quotes. Multiple commands
40 are semicolon or comma delimited. uberftp will execute these commands
41 and then exit. Uberftp will exit upon the first error encountered.
42 The third and forth usage statements use the URL style format for spec‐
44 ifying the source and destination for the files to transfer. The third
45 usage statement places these URLs on the command line. The forth usage
46 allows the user to specify multiple URL pairs in a separate file for
47 Uberftp to transfer one at a time. The supported URL syntaxes are
48 gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file
49 and file://path.
50 The fifth usage statement allows for commands that take pathnames to
52 accept URLs instead. The allowable commands are listed in the -cmds
53 section below.
54
57 -P port
58 Connect to port instead of the default. The default for GSI au‐
59 thentication is 2811. The default for password authentication is
60 21.
61 -u user
63 Connect as user. This is useful for both password and GSI au‐
64 thentication mechanisms.
65 -p pass
67 Use pass when authenticating. If pass equals X, UberFTP will
68 prompt for the password with character echoing turned off.
69
73 -active
74 Use ACTIVE mode for data transfers.
75 -ascii Use ASCII mode for data transfers.
77 -binary
79 Use BINARY mode for data transfers.
80 -blksize n
82 Set the internal buffer size to n.
83 -cksum [on|off]
85 Enable/Disable CRC checks after file transfers.
86 -cos name
88 Set the storage class of service to name. Used with HPSS instal‐
89 lations. Use the class of service name default to allow the re‐
90 mote server to decide which class of service to use.
91 -d Enable debugging. Same as '-debug 3'. Deprecated.
93 -debug n
95 Set the debug level to n.
96 -family name
98 Set the storage family to name. Use the family name default to
99 allow the remote server to decide which family to use.
100 -glob [on|off]
102 Enable/Disable filename expansion.
103 -hash Enable printing of hash marks during transfers.
105 -keepalive n
107 Send control channel keepalive messages every n seconds during
108 data transfers.
109 -mode [E|S]
111 Switch the transfer mode to extended block (E) or streams mode
112 (S).
113 -parallel n
115 Use n parallel data channels during extended block transfers.
116 -passive
118 Use PASSIVE mode for data transfers.
119 -pbsz n
121 Set the data protection buffer size to n n bytes.
122 -prot [C|S|E|P]
124 Set the data protection lelvel to clear (C), safe (S), confiden‐
125 tial (E) or private (P).
126 -retry n
128 Retry commands that fail with transient errors n times.
129 -resume path
131 Retry the recursive transfer starting at path.
132 -tcpbuf n
134 Set the TCP read/write buffers to n bytes.
135 -wait This will cause the client to wait for remote files to stage be‐
137 fore attempting to transfer them.
138 -v Print UberFTP version information and exit.
140 -version
142 Print UberFTP version information and exit.
143 -versions
145 Print version information about all used globus modules and
146 exit.
147
150 -cat url
151 Print to stdout the contents of the remote file. See CAT BEHAV‐
152 IOUR below for details about the behaviour of this functional‐
153 ity.
154 -chgrp [-r] group url
156 Set the group ownership of the remote object(s).
157 -chmod [-r] perms url
159 Set the permissions of the remote object(s).
160 -dir [-r] url
162 List the contents of the remote object.
163 -link url path
165 Create a hardlink named <path> to the remote object.
166 -ls [-r] url
168 List the contents of the remote object.
169 -mkdir url
171 Create the remote directory.
172 -rename url path
174 Rename the remote object to the given path.
175 -rm [-r] url
177 Remove the remote object(s).
178 -rmdir url
180 Remove the remote directory.
181 -size url
183 Return the size of the remote object.
184 -stage [-r] seconds url
186 Attempt to stage the remote object(s) over the time period given
187 in seconds.
188 -symlink url path
190 Create a symlink named <path> to the remote object.
191
194 By default, without any special environment variables, command line op‐
195 tions or commands, uberftp will transfer files in PASSIVE STREAMS mode.
196 PASSIVE means that the client will initiate the data connection which
197 is useful for users behind firewalls. STREAMS mode implies that GRIDFTP
198 features including striping and parallel data connections are not used.
199 In order to take advantage of these features with GridFTP capable
200 servers, you must either change the mode directly using -m command line
201 switch or the mode interactive command, or you can change the mode in‐
202 directly by specifying more than one parallel data connection using the
203 -c command line switch or by using the parallel interactive command.
204
207 By default, uberftp requires a GSI certificate. If you do not already
208 have a certificate, see the following web page to learn how to get one:
209 http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html
211 Once you have a certificate, use the grid-proxy-init command to get a
213 valid proxy.
214
217 ! [command]
218 Run the command using a shell on the local machine. If no com‐
219 mand is given, invoke an interactive shell.
220 ? [command]
222 If command is given, print a (hopefully) helpful blurb about it.
223 Otherwise, list all commands.
224 active Change to ACTIVE mode which causes the server to initiate the
226 data connection. The default is PASSIVE mode unless the variable
227 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
228 a firewall you must use PASSIVE mode.
229 ascii Change the data transfer type to ASCII which causes the server
231 to do some simple transformations to the file being transferred.
232 This is mostly useful for changing EOL (end of line) in text
233 files when moving between platforms. This option is almost
234 never necessary today. The default is BINARY mode also known as
235 IMAGE mode.
236 binary Change the data transfer type to BINARY (aka IMAGE) which causes
238 the server to not perform transformations to the file being
239 transferred. This is the default and is faster than an ASCII
240 transfer.
241 blksize size
243 Change the size of the memory buffer used to read and write data
244 to disks to size bytes. The default block size is 1024*1024
245 (1048576) bytes but it can be changed at compile time. The block
246 size can be increased to improve file transfer performance. This
247 is not related to the extended block mode block size used to de‐
248 termine the ratio of data to header for data transferred on the
249 data channel.
250 bugs Prints information regarding bug reporting and feature requests.
252 bye Close all control and data connections and exit.
254 cat file1 [file2 ... filen]
256 Print the contents of the remote file(s) to stdout. See CAT BE‐
257 HAVIOUR below for details about the behaviour of this function‐
258 ality.
259 cdup Change the remote working directory up one level.
261 cd [dir]
263 Change the remote working directory to dir. If dir is not given,
264 the client will make every attempt to change to the user's home
265 directory.
266 chgrp [-r] group object [object2 ... objectn]
269 Change group ownership on the remote object(s).
270 -r Recursively chgrp everything in the given directory.
271 chmod [-r] perms object [object2 ... objectn]
274 Change permissions on the remote object(s).
275 -r Recursively chmod everything in the given directory.
276 close Close the control connection to the remote host.
278 cksum [on|off]
280 Enable file cksum comparison after each file transfer. This only
281 works with NCSA's mass storage system.
282 on Enable checksum comparison
283 off Disable checksum comparison
284 cos name
286 Sets the HPSS class of service to name on the FTP service if the
287 service supports it. If name is omitted, the current class of
288 service is printed. Use the class of service name default to
289 allow the remote server to decide which class of service to use.
290 dcau [N|A|S subject]
292 Change the data channel authentication settings. If the service
293 does not support DCAU, these settings are ignored.
294 N Disabled dcau.
295 A Expect the remote identity to be mine. (Default)
296 S subject Expect the remote identity to be subject.
297 debug [0-3]
299 Turn debug statements on/off. If no value is given, this command
300 will toggle between debug(2) and non debug(1) mode. Otherwise
301 the debug level is set to the given level.
302 0 Only errors are printed
303 1 Default. Errors and some helpful messages are printed
304 2 Print useful control channel information
305 3 Print all information
306 family name
308 Sets the tape family to name on the FTP service if the service
309 supports it. If name is omitted, the current family is printed.
310 Use the family name default to allow the remote server to decide
311 which family to use.
312 glob [on|off]
314 Enable or disable filename globbing. If no option is given, this
315 command will toggle the current setting.
316 on Enable filename globbing
317 off Disable filename globbing
318 dir [-r] [target]
320 List the contents of the remote target directory. If target is
321 not given, then the current working directory is used.
322 -r Recursively list target.
323 target Directory or file to list. '.' is used by default.
324 get [-r] source [destination]
326 Retrieve file(s) from the remote service. If source implies mul‐
327 tiple transfers, either through regular expressions or by using
328 the recursive feature, then destination must be a directory. If
329 destination is not specified, source is used.
330 -r Recursively transfer the given directory.
331 hash Print hash marks during data transfers. This does not work dur‐
333 ing third party transfers.
334 help [command]
336 If command is given, print a helpful blurb about command. Oth‐
337 erwise, list all commands.
338 keepalive [seconds]
340 Attempts to keep the control channel from being blocked by fire‐
341 walls during long data channel operations. UberFTP sends a NOOP
342 command to the service at intervals equal to the specified num‐
343 ber of seconds. Setting it to zero will disable keepalive. If
344 seconds are not given, the current timeout is displayed. This
345 feature is disabled by default.
346 seconds number of seconds between NOOPs. Disabled if zero.
347 lcat file1 [file2 ... filen]
349 Print the contents of the local file(s) to stdout. See CAT BE‐
350 HAVIOUR below for details about the behaviour of this function‐
351 ality.
352 lcd [dir]
354 Change the local working directory to dir. If dir is not given,
355 the client will make every attempt to change to the user's home
356 directory.
357 lcdup Change the local working directory up one level.
359 lchgrp [-r] group object [object2 ... objectn]
361 Change group ownership on the local object(s).
362 -r Recursively chgrp everything in the given directory.
363 lchmod [-r] perms object [object2 ... objectn]
366 Change permissions on the local object(s).
367 -r Recursively chmod everything in the given directory.
368 lclose Close the control connection to the local host.
370 ldir [-r] [target]
372 List the contents of the local target directory. If target is
373 not given, then the current working directory is used.
374 -r Recursively list target.
375 target Directory or file to list. '.' is used by default.
376 link [oldfile] [newfile]
378 Create a hardlink to oldfile named newfile on the remote ser‐
379 vice.
380 llink [oldfile] [newfile]
382 Create a hardlink to oldfile named newfile on the local service.
383 lls [-r] [target]
385 List the contents of the local target directory. If target is
386 not given, then the current working directory is used.
387 -r Recursively list target.
388 target Directory or file to list. '.' is used by default.
389 llscos List the available class of services on the local server (HPSS
391 only).
392 llsfam List the available tape families on the local server (HPSS
394 only).
395 lmkdir dir1 [dir2 ... dirn]
397 Create the local directory(ies).
398 lopen [-P port] [-u user] [-p pass | X] host
400 Opens a control channel to host and that host becomes the 'lo‐
401 cal' machine. After using lopen, all local (l*) commands per‐
402 form their respective operations on host rather than the local
403 machine. This is how third party transfers are accomplished. GSI
404 authentication is used unless the -p option is used.
405 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
406 word).
407 -u user Connect as alternate user.
408 -p pass | X
409 Use password pass when authenticating with host.
410 If pass equals X, read the password from STDIN with
411 character echoing turned off.
412 host Connect to host.
413 lpwd Prints the current local working directory.
415 lrename src dst
417 Rename the local object src to dst.
418 lrm [-r] object1 [object1...objectn]
420 Removes the local file system object(s).
421 -r Recursively remove the given directory.
422 lrmdir dir1 [dir2...dirn]
424 Removes the given directories from the local service.
425 lquote cmd
427 Pass cmd to the local FTP service. This allows the user to use
428 server-specific commands that are not available through the
429 uberftp interface.
430 ls [-r] [target]
432 List the contents of the remote target directory. If [target] is
433 not given, then the current working directory is used.
434 -r Recursively list target.
435 target Directory or file to list. '.' is used by default.
436 lscos List the available class of services on the remote server (HPSS
438 only).
439 lsfam List the available tape families on the remote server (HPSS
441 only).
442 lsize file1 [file2...filen]
444 Prints the size of the given object(s).
445 lstage [-r] seconds object1 [object2...objectn]
447 Attempt to stage all matching files within the given number of
448 seconds on the local service.
449 seconds number of seconds to attempt staging
450 -r Recursively stage all files in the given subdirectory.
451 lsymlink [oldfile] [newfile]
453 Create a symlink to oldfile named newfile on the local service.
454 mput [-r] object1 [object2...objectn]
456 Retrieve file(s) from the remote service. This is similiar to
457 making multiple calls to get without specifying a destination.
458 -r Recursively transfer the given directory.
459 mkdir dir
461 Create the remote directory.
462 mode [E|S]
464 Toggle the data transfer mode between Streams mode and Extended
465 Block mode. The default is Streams mode. If no option is given,
466 it will display the current mode.
467 E Extended block mode
468 S Streams mode
469 mput [-r] object1 [object2...objectn]
471 Store file(s) to the remote service. This is similiar to making
472 multiple calls to put without specifying a destination.
473 -r Recursively transfer the given directory.
474 open [-P port] [-u user] [-p pass | X] host
476 Opens a control channel to host and that host becomes the 're‐
477 mote' machine. GSI authentication is used unless the -p option
478 is used.
479 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
480 word).
481 -u user Connect as user.
482 -p pass | X
483 Use password pass when authenticating with host.
484 If pass equals X, read the password from STDIN with
485 character echoing turned off.
486 host Connect to host.
487 order [type]
489 Changes the order of lists returned from ls and lls to the given
490 scheme. If type is not given, the current order is displayed.
491 type Ordering scheme to use. Value options are:
492 none Do not order listings
493 name Order listings by name
494 size Order listings by size
495 type Order listings by type
496 parallel [number]
498 Set the number of parallel data connections to number. This is
499 only useful for extended block mode transfers. The default num‐
500 ber of data connections is one. If no number is given, the cur‐
501 rent setting for the number of parallel connects is printed.
502 passive
504 Change to PASSIVE mode which causes the client to initiate the
505 data connection. This is the default mode unless the variable
506 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
507 a firewall you must use PASSIVE mode.
508 pbsz [size]
510 Change the length of the protection buffer. The protection buf‐
511 fer is used to encrypt data on the data channel. The length of
512 the protection buffer represents the largest encoded message
513 that is allowed on the data channel. By default, the protection
514 buffer is grown to match the internal buffer used. For efficient
515 transfers, pbsz should be sufficiently larger than blksize so
516 that the wrapped buffer fits within the protection buffer. Oth‐
517 erwise, the blksize buffer is broken into multiple pieces so
518 that each write is less than pbsz when wrapped. If pbsz is not
519 given, the current size is displayed.
520 size length of protection buffer. 0 will set it to its de‐
521 fault.
522 pget offset size srcfile [destfile]
524 Retrieve only the specified portion of the file(s). If srcfile
525 is a regular expression and expands to multiple files, and des‐
526 tination is given, destination must refer to a directory.
527 offset Offset within the file
528 size Amount of data to retrieve
529 srcfile Name of remote file
530 destfile Name of local file. srcfile is used if destfile
531 is not specified
532 pput offset size srcfile [destfile]
534 Store only the specified portion of the file(s). If srcfile is a
535 regular expression and expands to multiple files, and destina‐
536 tion is given, destination must refer to a directory.
537 offset Offset within the file
538 size Amount of data to retrieve
539 srcfile Name of local file
540 destfile Name of remote file. srcfile is used if destfile
541 is not specified
542 prot [C|S|E|P]
544 This command configures the level of security on the data chan‐
545 nel after data channel authentication has completed. Clear means
546 that the data will not be protected. Safe means that the data
547 will be integrity protected meaning that altered data will be
548 detected. Confidential means that the data will be unreadable to
549 third parties. Private mode means the data will be confidential
550 and safe.
551 C Set protection level to clear.
552 S Set protection level to safe.
553 E Set protection level to confidential.
554 P Set protection level to private.
555 put [-r] source [destination]
557 Store file(s) to the remote service. If source implies multiple
558 transfers, either through regular expressions or by using the
559 recursive feature, then destination must be a directory. If des‐
560 tination is not specified, source is used.
561 -r Recursively transfer the given directory.
562 pwd Prints the current working directory.
564 quit Close all control and data connections and exit.
566 quote cmd
568 Pass cmd to the remote FTP service. This allows the user to use
569 server-specific commands that are not available through the
570 uberftp interface.
571 rename src dst
573 Rename the remote object src to dst.
574 retry [cnt]
576 Configures retry on failed commands that have transient errors.
577 cnt represents the number of times a failed command is retried.
578 A value of zero effectively disables retry. Zero is the default.
579 If no value is given the current setting is displayed.
580 cnt Number of times a failed command is retried.
581 resume [-d] path
583 Sets a restart point for recursive transfers. If a long recur‐
584 sive transfer fails, you can set resume to the path that failed
585 and UberFTP will skip all file and directory creations up to the
586 given path.
587 path Path to resume transfer at. If path is not given, print
588 the current
589 resume target.
590 -d Remove the current resume path.
591 rm [-r] object1 [object1...objectn]
593 Removes the remote file system object(s).
594 -r Recursively remove the given directory.
595 rmdir dir1 [dir2...dirn]
597 Removes the given directories from the remote service.
598 runique
600 Toggles the client to store files using unique names during put
601 operations.
602 size file1 [file2...filen]
604 Prints the size of the given object(s).
605 stage [-r] seconds object1 [object2...objectn]
607 Attempt to stage all matching files within the given number of
608 seconds on the remote service.
609 seconds number of seconds to attempt staging
610 -r Recursively stage all files in the given subdirectory.
611 sunique
613 Toggles the client to store files using unique names during get
614 operations.
615 symlink [oldfile] [newfile]
617 Create a symlink to oldfile named newfile on the remote service.
618 tcpbuf [size]
620 Set the data channel TCP buffer size to size bytes. If size is
621 not given, the current TCP buffer size will be printed.
622 versions
624 Prints the versions of all Globus modules being used.
625 wait Toggles whether the client should wait for files to stage before
627 attempting to retrieve them.
628
632 Use the active command to enable active mode FTP when using NCSA's Uni‐
633 Tree mass storage system if possible since it will give much better
634 file transfer performance. When tranferring files over long distances,
635 use a large value (for example, 16777216) for tcpbuf. When there is
636 high network traffic, you may be able to improve performance using the
637 parallel command to increase the number of parallel data connections to
638 2-4.
639
641 In order to perform a third-party transfer, you must log into two FTP
642 servers. Typically, you connect to a single FTP service to "get" files
643 to the local machine and "put" files to the remote service. For third-
644 party transfers, you must connect to a second service thereby replacing
645 the former local machine. In UberFTP terminology, it is referred to as
646 "opening a new local service" since, from the perspective of the user,
647 the new local service will appear as though the user initiated the FTP
648 session from that machine.
649 All remote service commands have "l*" counterparts that allow you to
651 specify that the command is to be performed on the local service,
652 whether that service is the local machine or a new local service. So
653 to open a new local service, use the "l*" version of the open command:
654 UberFTP> lopen mss.ncsa.teragrid.org
656 UberFTP> lclose
657 Once you have connected to both services, files can be transferred as
659 before with the change that files you "get" go to the new local service
660 and files you "put" are sent from the new local service.
661
663 By default, local port selection is managed by the operating system.
664 However, you may wish to specify which ports UberFTP should use for in‐
665 coming and out going connections. This is useful when dealing with
666 firewalls.
667 Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all in‐
669 bound connections to use the specified port range. Likewise, setting
670 UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound
671 connections to use the specified port range.
672 The environment variables GLOBUS_TCP_PORT_RANGE and
674 GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection.
675 These variables behave exactly as their UBERFTP counterparts and are
676 available for backwards compatibility with older versions. The UBERFTP
677 variables take precedence over the GLOBUS variables.
678 The values of the variables specify a port range, a minimum port number
680 and a maximum port number, separated by either a comma or a space. For
681 example, to set the inbound port range, you would set:
682 UBERFTP_TCP_PORT_RANGE=40000,50000
684 Using the space delimiter, this format is also acceptable:
686 UBERFTP_TCP_PORT_RANGE="40000 50000"
688 See your shell documentation for the proper syntax for settings vari‐
690 ables within your environment.
691 Setting the ephemeral port range to an unusable range will cause
693 UberFTP connections to fail. For instance, setting a port range from 10
694 to 100 with a non root process will fail on most operating systems.
695
698 UberFTP 2.9.1 comes with an additional environment variable:
699 UBERFTP_CAT_CORRECT
701 If this environment variable is present UberFTP omits the odd newline
703 that is otherwise printed out in addition to the file contents by the
704 cat functionality of previous versions of UberFTP for non-empty files.
705 If it is not present, the old behaviour is used for compatibility rea‐
706 sons. The content of this environment variable is not evaluated, just
707 its presence in your environment. Also see:
708 https://github.com/gridcf/UberFTP/issues/22
710
713 UberFTP will exit with a value of 0 if no errors occurred during the
714 session, otherwise it will exit with a value of 1. In non interactive,
715 commandline mode, it will exit after the first error occurs.
716
718 Set the environment variable to set active mode FTP (improves file
719 transfer performance to the mass storage system). Connect to NCSA's
720 UniTree mass storage system interactively from NCSA's TeraGrid cluster:
721 setenv UBERFTP_ACTIVE_MODE on
723 % uberftp mss.ncsa.teragrid.org
724 ...
725 220 UNIX Archive FTP server ready.
726 230 User consult logged in.
727 UberFTP>
728 Use the command-line interface to copy a file from NCSA's TeraGrid
730 cluster to the UniTree mass storage system. (There is no need to set
731 tcpbuf since it is over a LAN but active mode is turned on to improve
732 file transfer performance to the mass storage system.):
733 uberftp mss.ncsa.teragrid.org \
735 "active; cd work; get file.tar"
736 Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster.
738 (Note that tcpbuf is set to 16777216 since there is a long network la‐
739 tency between NCSA and SDSC):
740 uberftp tg-gridftp.sdsc.teragrid.org \
742 "tcpbuf 16777216; cd scr; put file.tar"
743
745 mssftp(1), msscmd(1), ftp(1),
746 GridFTP:
747 https://gridcf.org/gct-docs/latest/gridftp/
748 TCP Window Size:
749 http://www.vonwelch.com/report/tcp_windows/
750 http://www.psc.edu/tcp-tune
751 Active vs. Passive FTP:
752 http://slacksite.com/other/ftp.html
753 Note: The links above are not under the GridCF's control so they may
755 become obsolete.
756 22 Jul 2022 UBERFTP(1)