1UBERFTP(1C) UBERFTP(1C)
2
6 uberftp - GridFTP-enabled client
7
9 uberftp [options] [host options] [host]
10 uberftp [options] [host options] host “cmd”
12 uberftp [options] srcurl dsturl
14 uberftp [options] -f urlfile
16 uberftp [options] -cmd url
18
21 uberftp is a GridFTP-enabled client that supports both interactive use
22 and FTP commands on the uberftp command line to transfer files between
23 two computers. It is intended for use with computers that have a
24 GridFTP server installed. Uberftp supports GSI authentication, parallel
25 data channels and striping. For more information about GridFTP, see the
26 GridFTP URL in the "SEE ALSO" section below.
27 Only the first usage shown above will create an interactive session. If
29 host is specified, uberftp immediately attempts to establish a connec‐
30 tion to the GridFTP server on host, enters its command interpreter and
31 awaits commands from the user. If host is not specified, uberftp imme‐
32 diately drops into the command interpreter without connecting to any
33 GridFTP server.
34 The second usage option denotes a non interactive session in which
36 “cmd” is a series of one or more commands as described in the COMMANDS
37 section below to run once the control connection is established. These
38 commands are run exactly as if they had been entered from the interac‐
39 tive prompt. This list must be enclosed in quotes. Multiple commands
40 are semicolon or comma delimited. uberftp will execute these commands
41 and then exit. Uberftp will exit upon the first error encountered.
42 The third and forth usage statements use the URL style format for spec‐
44 ifying the source and destination for the files to transfer. The third
45 usage statement places these URLs on the command line. The forth usage
46 allows the user to specify multiple URL pairs in a separate file for
47 Uberftp to transfer one at a time. The supported URL syntaxes are
48 gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file
49 and file://path.
50 The fifth usage statement allows for commands that take pathnames to
52 accept URLs instead. The allowable commands are listed in the -cmds
53 section below.
54
57 -P port
58 Connect to port instead of the default. The default for GSI
59 authentication is 2811. The default for password authentication
60 is 21.
61 -u user
63 Connect as user. This is useful for both password and GSI
64 authentication mechanisms.
65 -p pass
67 Use pass when authenticating. If pass equals X, UberFTP will
68 prompt for the password with character echoing turned off.
69
73 -active
74 Use ACTIVE mode for data transfers.
75 -ascii Use ASCII mode for data transfers.
77 -binary
79 Use BINARY mode for data transfers.
80 -blksize n
82 Set the internal buffer size to n.
83 -cksum [on|off]
85 Enable/Disable CRC checks after file transfers.
86 -cos name
88 Set the storage class of service to name. Used with HPSS instal‐
89 lations. Use the class of service name default to allow the
90 remote server to decide which class of service to use.
91 -d Enable debugging. Same as '-debug 3'. Deprecated.
93 -debug n
95 Set the debug level to n.
96 -family name
98 Set the storage family to name. Use the family name default to
99 allow the remote server to decide which family to use.
100 -glob [on|off]
102 Enable/Disable filename expansion.
103 -hash Enable printing of hash marks during transfers.
105 -keepalive n
107 Send control channel keepalive messages every n seconds during
108 data transfers.
109 -mode [E|S]
111 Switch the transfer mode to extended block (E) or streams mode
112 (S).
113 -parallel n
115 Use n parallel data channels during extended block transfers.
116 -passive
118 Use PASSIVE mode for data transfers.
119 -pbsz n
121 Set the data protection buffer size to n n bytes.
122 -prot [C|S|E|P]
124 Set the data protection lelvel to clear (C), safe (S), confiden‐
125 tial (E) or private (P).
126 -retry n
128 Retry commands that fail with transient errors n times.
129 -resume path
131 Retry the recursive transfer starting at path.
132 -tcpbuf n
134 Set the TCP read/write buffers to n bytes.
135 -wait This will cause the client to wait for remote files to stage
137 before attempting to transfer them.
138 -v Print UberFTP version information and exit.
140 -version
142 Print UberFTP version information and exit.
143 -versions
145 Print version information about all used globus modules and
146 exit.
147
150 -cat url
151 Print to stdout the contents of the remote file.
152 -chgrp [-r] group url
154 Set the group ownership of the remote object(s).
155 -chmod [-r] perms url
157 Set the permissions of the remote object(s).
158 -dir [-r] url
160 List the contents of the remote object.
161 -link url path
163 Create a hardlink named <path> to the remote object.
164 -ls [-r] url
166 List the contents of the remote object.
167 -mkdir url
169 Create the remote directory.
170 -rename url path
172 Rename the remote object to the given path.
173 -rm [-r] url
175 Remove the remote object(s).
176 -rmdir url
178 Remove the remote directory.
179 -size url
181 Return the size of the remote object.
182 -stage [-r] seconds url
184 Attempt to stage the remote object(s) over the time period given
185 in seconds.
186 -symlink url path
188 Create a symlink named <path> to the remote object.
189
192 By default, without any special environment variables, command line
193 options or commands, uberftp will transfer files in PASSIVE STREAMS
194 mode. PASSIVE means that the client will initiate the data connection
195 which is useful for users behind firewalls. STREAMS mode implies that
196 GRIDFTP features including striping and parallel data connections are
197 not used. In order to take advantage of these features with GridFTP
198 capable servers, you must either change the mode directly using -m com‐
199 mand line switch or the mode interactive command, or you can change the
200 mode indirectly by specifying more than one parallel data connection
201 using the -c command line switch or by using the parallel interactive
202 command.
203
206 By default, uberftp requires a GSI certificate. If you do not already
207 have a certificate, see the following web page to learn how to get one:
208 http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html
210 Once you have a certificate, use the grid-proxy-init command to get a
212 valid proxy.
213
216 ! [command]
217 Run the command using a shell on the local machine. If no com‐
218 mand is given, invoke an interactive shell.
219 ? [command]
221 If command is given, print a (hopefully) helpful blurb about it.
222 Otherwise, list all commands.
223 active Change to ACTIVE mode which causes the server to initiate the
225 data connection. The default is PASSIVE mode unless the variable
226 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
227 a firewall you must use PASSIVE mode.
228 ascii Change the data transfer type to ASCII which causes the server
230 to do some simple transformations to the file being transferred.
231 This is mostly useful for changing EOL (end of line) in text
232 files when moving between platforms. This option is almost
233 never necessary today. The default is BINARY mode also known as
234 IMAGE mode.
235 binary Change the data transfer type to BINARY (aka IMAGE) which causes
237 the server to not perform transformations to the file being
238 transferred. This is the default and is faster than an ASCII
239 transfer.
240 blksize size
242 Change the size of the memory buffer used to read and write data
243 to disks to size bytes. The default block size is 1024*1024
244 (1048576) bytes but it can be changed at compile time. The block
245 size can be increased to improve file transfer performance. This
246 is not related to the extended block mode block size used to
247 determine the ratio of data to header for data transferred on
248 the data channel.
249 bugs Prints information regarding bug reporting and feature requests.
251 bye Close all control and data connections and exit.
253 cat file1 [file2 ... filen]
255 Print the contents of the remote file(s) to stdout.
256 cdup Change the remote working directory up one level.
258 cd [dir]
260 Change the remote working directory to dir. If dir is not given,
261 the client will make every attempt to change to the user's home
262 directory.
263 chgrp [-r] group object [object2 ... objectn]
266 Change group ownership on the remote object(s).
267 -r Recursively chgrp everything in the given directory.
268 chmod [-r] perms object [object2 ... objectn]
271 Change permissions on the remote object(s).
272 -r Recursively chmod everything in the given directory.
273 close Close the control connection to the remote host.
275 cksum [on|off]
277 Enable file cksum comparison after each file transfer. This only
278 works with NCSA's mass storage system.
279 on Enable checksum comparison
280 off Disable checksum comparison
281 cos name
283 Sets the HPSS class of service to name on the FTP service if the
284 service supports it. If name is omitted, the current class of
285 service is printed. Use the class of service name default to
286 allow the remote server to decide which class of service to use.
287 dcau [N|A|S subject]
289 Change the data channel authentication settings. If the service
290 does not support DCAU, these settings are ignored.
291 N Disabled dcau.
292 A Expect the remote identity to be mine. (Default)
293 S subject Expect the remote identity to be subject.
294 debug [0-3]
296 Turn debug statements on/off. If no value is given, this command
297 will toggle between debug(2) and non debug(1) mode. Otherwise
298 the debug level is set to the given level.
299 0 Only errors are printed
300 1 Default. Errors and some helpful messages are printed
301 2 Print useful control channel information
302 3 Print all information
303 family name
305 Sets the tape family to name on the FTP service if the service
306 supports it. If name is omitted, the current family is printed.
307 Use the family name default to allow the remote server to decide
308 which family to use.
309 glob [on|off]
311 Enable or disable filename globbing. If no option is given, this
312 command will toggle the current setting.
313 on Enable filename globbing
314 off Disable filename globbing
315 dir [-r] [target]
317 List the contents of the remote target directory. If target is
318 not given, then the current working directory is used.
319 -r Recursively list target.
320 target Directory or file to list. '.' is used by default.
321 get [-r] source [destination]
323 Retrieve file(s) from the remote service. If source implies mul‐
324 tiple transfers, either through regular expressions or by using
325 the recursive feature, then destination must be a directory. If
326 destination is not specified, source is used.
327 -r Recursively transfer the given directory.
328 hash Print hash marks during data transfers. This does not work dur‐
330 ing third party transfers.
331 help [command]
333 If command is given, print a helpful blurb about command. Oth‐
334 erwise, list all commands.
335 keepalive [seconds]
337 Attempts to keep the control channel from being blocked by fire‐
338 walls during long data channel operations. UberFTP sends a NOOP
339 command to the service at intervals equal to the specified num‐
340 ber of seconds. Setting it to zero will disable keepalive. If
341 seconds are not given, the current timeout is displayed. This
342 feature is disabled by default.
343 seconds number of seconds between NOOPs. Disabled if zero.
344 lcat file1 [file2 ... filen]
346 Print the contents of the local file(s) to stdout.
347 lcd [dir]
349 Change the local working directory to dir. If dir is not given,
350 the client will make every attempt to change to the user's home
351 directory.
352 lcdup Change the local working directory up one level.
354 lchgrp [-r] group object [object2 ... objectn]
356 Change group ownership on the local object(s).
357 -r Recursively chgrp everything in the given directory.
358 lchmod [-r] perms object [object2 ... objectn]
361 Change permissions on the local object(s).
362 -r Recursively chmod everything in the given directory.
363 lclose Close the control connection to the local host.
365 ldir [-r] [target]
367 List the contents of the local target directory. If target is
368 not given, then the current working directory is used.
369 -r Recursively list target.
370 target Directory or file to list. '.' is used by default.
371 link [oldfile] [newfile]
373 Create a hardlink to oldfile named newfile on the remote ser‐
374 vice.
375 llink [oldfile] [newfile]
377 Create a hardlink to oldfile named newfile on the local service.
378 lls [-r] [target]
380 List the contents of the local target directory. If target is
381 not given, then the current working directory is used.
382 -r Recursively list target.
383 target Directory or file to list. '.' is used by default.
384 llscos List the available class of services on the local server (HPSS
386 only).
387 llsfam List the available tape families on the local server (HPSS
389 only).
390 lmkdir dir1 [dir2 ... dirn]
392 Create the local directory(ies).
393 lopen [-P port] [-u user] [-p pass | X] host
395 Opens a control channel to host and that host becomes the
396 'local' machine. After using lopen, all local (l*) commands
397 perform their respective operations on host rather than the
398 local machine. This is how third party transfers are accom‐
399 plished. GSI authentication is used unless the -p option is
400 used.
401 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
402 word).
403 -u user Connect as alternate user.
404 -p pass | X
405 Use password pass when authenticating with host.
406 If pass equals X, read the password from STDIN with
407 character echoing turned off.
408 host Connect to host.
409 lpwd Prints the current local working directory.
411 lrename src dst
413 Rename the local object src to dst.
414 lrm [-r] object1 [object1...objectn]
416 Removes the local file system object(s).
417 -r Recursively remove the given directory.
418 lrmdir dir1 [dir2...dirn]
420 Removes the given directories from the local service.
421 lquote cmd
423 Pass cmd to the local FTP service. This allows the user to use
424 server-specific commands that are not available through the
425 uberftp interface.
426 ls [-r] [target]
428 List the contents of the remote target directory. If [target] is
429 not given, then the current working directory is used.
430 -r Recursively list target.
431 target Directory or file to list. '.' is used by default.
432 lscos List the available class of services on the remote server (HPSS
434 only).
435 lsfam List the available tape families on the remote server (HPSS
437 only).
438 lsize file1 [file2...filen]
440 Prints the size of the given object(s).
441 lstage [-r] seconds object1 [object2...objectn]
443 Attempt to stage all matching files within the given number of
444 seconds on the local service.
445 seconds number of seconds to attempt staging
446 -r Recursively stage all files in the given subdirectory.
447 lsymlink [oldfile] [newfile]
449 Create a symlink to oldfile named newfile on the local service.
450 mput [-r] object1 [object2...objectn]
452 Retrieve file(s) from the remote service. This is similiar to
453 making multiple calls to get without specifying a destination.
454 -r Recursively transfer the given directory.
455 mkdir dir
457 Create the remote directory.
458 mode [E|S]
460 Toggle the data transfer mode between Streams mode and Extended
461 Block mode. The default is Streams mode. If no option is given,
462 it will display the current mode.
463 E Extended block mode
464 S Streams mode
465 mput [-r] object1 [object2...objectn]
467 Store file(s) to the remote service. This is similiar to making
468 multiple calls to put without specifying a destination.
469 -r Recursively transfer the given directory.
470 open [-P port] [-u user] [-p pass | X] host
472 Opens a control channel to host and that host becomes the
473 'remote' machine. GSI authentication is used unless the -p
474 option is used.
475 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
476 word).
477 -u user Connect as user.
478 -p pass | X
479 Use password pass when authenticating with host.
480 If pass equals X, read the password from STDIN with
481 character echoing turned off.
482 host Connect to host.
483 order [type]
485 Changes the order of lists returned from ls and lls to the given
486 scheme. If type is not given, the current order is displayed.
487 type Ordering scheme to use. Value options are:
488 none Do not order listings
489 name Order listings by name
490 size Order listings by size
491 type Order listings by type
492 parallel [number]
494 Set the number of parallel data connections to number. This is
495 only useful for extended block mode transfers. The default num‐
496 ber of data connections is one. If no number is given, the cur‐
497 rent setting for the number of parallel connects is printed.
498 passive
500 Change to PASSIVE mode which causes the client to initiate the
501 data connection. This is the default mode unless the variable
502 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
503 a firewall you must use PASSIVE mode.
504 pbsz [size]
506 Change the length of the protection buffer. The protection buf‐
507 fer is used to encrypt data on the data channel. The length of
508 the protection buffer represents the largest encoded message
509 that is allowed on the data channel. By default, the protection
510 buffer is grown to match the internal buffer used. For efficient
511 transfers, pbsz should be sufficiently larger than blksize so
512 that the wrapped buffer fits within the protection buffer. Oth‐
513 erwise, the blksize buffer is broken into multiple pieces so
514 that each write is less than pbsz when wrapped. If pbsz is not
515 given, the current size is displayed.
516 size length of protection buffer. 0 will set it to its
517 default.
518 pget offset size srcfile [destfile]
520 Retrieve only the specified portion of the file(s). If srcfile
521 is a regular expression and expands to multiple files, and des‐
522 tination is given, destination must refer to a directory.
523 offset Offset within the file
524 size Amount of data to retrieve
525 srcfile Name of remote file
526 destfile Name of local file. srcfile is used if destfile
527 is not specified
528 pput offset size srcfile [destfile]
530 Store only the specified portion of the file(s). If srcfile is a
531 regular expression and expands to multiple files, and destina‐
532 tion is given, destination must refer to a directory.
533 offset Offset within the file
534 size Amount of data to retrieve
535 srcfile Name of local file
536 destfile Name of remote file. srcfile is used if destfile
537 is not specified
538 prot [C|S|E|P]
540 This command configures the level of security on the data chan‐
541 nel after data channel authentication has completed. Clear means
542 that the data will not be protected. Safe means that the data
543 will be integrity protected meaning that altered data will be
544 detected. Confidential means that the data will be unreadable to
545 third parties. Private mode means the data will be confidential
546 and safe.
547 C Set protection level to clear.
548 S Set protection level to safe.
549 E Set protection level to confidential.
550 P Set protection level to private.
551 put [-r] source [destination]
553 Store file(s) to the remote service. If source implies multiple
554 transfers, either through regular expressions or by using the
555 recursive feature, then destination must be a directory. If des‐
556 tination is not specified, source is used.
557 -r Recursively transfer the given directory.
558 pwd Prints the current working directory.
560 quit Close all control and data connections and exit.
562 quote cmd
564 Pass cmd to the remote FTP service. This allows the user to use
565 server-specific commands that are not available through the
566 uberftp interface.
567 rename src dst
569 Rename the remote object src to dst.
570 retry [cnt]
572 Configures retry on failed commands that have transient errors.
573 cnt represents the number of times a failed command is retried.
574 A value of zero effectively disables retry. Zero is the default.
575 If no value is given the current setting is displayed.
576 cnt Number of times a failed command is retried.
577 resume [-d] path
579 Sets a restart point for recursive transfers. If a long recur‐
580 sive transfer fails, you can set resume to the path that failed
581 and UberFTP will skip all file and directory creations up to the
582 given path.
583 path Path to resume transfer at. If path is not given, print
584 the current
585 resume target.
586 -d Remove the current resume path.
587 rm [-r] object1 [object1...objectn]
589 Removes the remote file system object(s).
590 -r Recursively remove the given directory.
591 rmdir dir1 [dir2...dirn]
593 Removes the given directories from the remote service.
594 runique
596 Toggles the client to store files using unique names during put
597 operations.
598 size file1 [file2...filen]
600 Prints the size of the given object(s).
601 stage [-r] seconds object1 [object2...objectn]
603 Attempt to stage all matching files within the given number of
604 seconds on the remote service.
605 seconds number of seconds to attempt staging
606 -r Recursively stage all files in the given subdirectory.
607 sunique
609 Toggles the client to store files using unique names during get
610 operations.
611 symlink [oldfile] [newfile]
613 Create a symlink to oldfile named newfile on the remote service.
614 tcpbuf [size]
616 Set the data channel TCP buffer size to size bytes. If size is
617 not given, the current TCP buffer size will be printed.
618 versions
620 Prints the versions of all Globus modules being used.
621 wait Toggles whether the client should wait for files to stage before
623 attempting to retrieve them.
624
628 Use the active command to enable active mode FTP when using NCSA's Uni‐
629 Tree mass storage system if possible since it will give much better
630 file transfer performance. When tranferring files over long distances,
631 use a large value (for example, 16777216) for tcpbuf. When there is
632 high network traffic, you may be able to improve performance using the
633 parallel command to increase the number of parallel data connections to
634 2-4.
635
637 In order to perform a third-party transfer, you must log into two FTP
638 servers. Typically, you connect to a single FTP service to "get" files
639 to the local machine and "put" files to the remote service. For third-
640 party transfers, you must connect to a second service thereby replacing
641 the former local machine. In UberFTP terminology, it is referred to as
642 "opening a new local service" since, from the perspective of the user,
643 the new local service will appear as though the user initiated the FTP
644 session from that machine.
645 All remote service commands have "l*" counterparts that allow you to
647 specify that the command is to be performed on the local service,
648 whether that service is the local machine or a new local service. So
649 to open a new local service, use the "l*" version of the open command:
650 UberFTP> lopen mss.ncsa.teragrid.org
652 UberFTP> lclose
653 Once you have connected to both services, files can be transferred as
655 before with the change that files you "get" go to the new local service
656 and files you "put" are sent from the new local service.
657
659 By default, local port selection is managed by the operating system.
660 However, you may wish to specify which ports UberFTP should use for
661 incoming and out going connections. This is useful when dealing with
662 firewalls.
663 Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all
665 inbound connections to use the specified port range. Likewise, setting
666 UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound
667 connections to use the specified port range.
668 The environment variables GLOBUS_TCP_PORT_RANGE and
670 GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection.
671 These variables behave exactly as their UBERFTP counterparts and are
672 available for backwards compatibility with older versions. The UBERFTP
673 variables take precedence over the GLOBUS variables.
674 The values of the variables specify a port range, a minimum port number
676 and a maximum port number, separated by either a comma or a space. For
677 example, to set the inbound port range, you would set:
678 UBERFTP_TCP_PORT_RANGE=40000,50000
680 Using the space delimiter, this format is also acceptable:
682 UBERFTP_TCP_PORT_RANGE="40000 50000"
684 See your shell documentation for the proper syntax for settings vari‐
686 ables within your environment.
687 Setting the ephemeral port range to an unusable range will cause
689 UberFTP connections to fail. For instance, setting a port range from 10
690 to 100 with a non root process will fail on most operating systems.
691
694 UberFTP will exit with a value of 0 if no errors occurred during the
695 session, otherwise it will exit with a value of 1. In non interactive,
696 commandline mode, it will exit after the first error occurs.
697
699 Set the environment variable to set active mode FTP (improves file
700 transfer performance to the mass storage system). Connect to NCSA's
701 UniTree mass storage system interactively from NCSA's TeraGrid cluster:
702 setenv UBERFTP_ACTIVE_MODE on
704 % uberftp mss.ncsa.teragrid.org
705 ...
706 220 UNIX Archive FTP server ready.
707 230 User consult logged in.
708 UberFTP>
709 Use the command-line interface to copy a file from NCSA's TeraGrid
711 cluster to the UniTree mass storage system. (There is no need to set
712 tcpbuf since it is over a LAN but active mode is turned on to improve
713 file transfer performance to the mass storage system.):
714 uberftp mss.ncsa.teragrid.org \
716 "active; cd work; get file.tar"
717 Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster.
719 (Note that tcpbuf is set to 16777216 since there is a long network
720 latency between NCSA and SDSC):
721 uberftp tg-gridftp.sdsc.teragrid.org \
723 "tcpbuf 16777216; cd scr; put file.tar"
724
726 mssftp(1), msscmd(1), ftp(1),
727 GridFTP:
728 http://www.globus.org/datagrid/gridftp.html
729 TCP Window Size:
730 http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html
731 Active vs. Passive FTP:
732 http://slacksite.com/other/ftp.html
733 Note: The links above are not under NCSA's control so they may become
735 obsolete.
736 16 May 2008 UBERFTP(1C)