1UBERFTP(1C) UBERFTP(1C)
2
6 uberftp - GridFTP-enabled client
7
9 uberftp [options] [host options] [host]
10 uberftp [options] [host options] host “cmd”
12 uberftp [options] srcurl dsturl
14 uberftp [options] -f urlfile
16 uberftp [options] -cmd url
18
21 uberftp is a GridFTP-enabled client that supports both interactive use
22 and FTP commands on the uberftp command line to transfer files between
23 two computers. It is intended for use with computers that have a
24 GridFTP server installed. Uberftp supports GSI authentication, parallel
25 data channels and striping. For more information about GridFTP, see the
26 GridFTP URL in the "SEE ALSO" section below.
27 Only the first usage shown above will create an interactive session. If
29 host is specified, uberftp immediately attempts to establish a connec‐
30 tion to the GridFTP server on host, enters its command interpreter and
31 awaits commands from the user. If host is not specified, uberftp imme‐
32 diately drops into the command interpreter without connecting to any
33 GridFTP server.
34 The second usage option denotes a non interactive session in which
36 “cmd” is a series of one or more commands as described in the COMMANDS
37 section below to run once the control connection is established. These
38 commands are run exactly as if they had been entered from the interac‐
39 tive prompt. This list must be enclosed in quotes. Multiple commands
40 are semicolon or comma delimited. uberftp will execute these commands
41 and then exit. Uberftp will exit upon the first error encountered.
42 The third and forth usage statements use the URL style format for spec‐
44 ifying the source and destination for the files to transfer. The third
45 usage statement places these URLs on the command line. The forth usage
46 allows the user to specify multiple URL pairs in a separate file for
47 Uberftp to transfer one at a time. The supported URL syntaxes are
48 gsiftp://[user@]host[:port]/file, ftp://[user[:pass]@]host[:port]/file
49 and file://path.
50 The fifth usage statement allows for commands that take pathnames to
52 accept URLs instead. The allowable commands are listed in the -cmds
53 section below.
54
57 -P port
58 Connect to port instead of the default. The default for GSI
59 authentication is 2811. The default for password authentication
60 is 21.
61 -u user
63 Connect as user. This is useful for both password and GSI
64 authentication mechanisms.
65 -p pass
67 Use pass when authenticating. If pass equals X, UberFTP will
68 prompt for the password with character echoing turned off.
69
73 -active
74 Use ACTIVE mode for data transfers.
75 -ascii Use ASCII mode for data transfers.
77 -binary
79 Use BINARY mode for data transfers.
80 -blksize n
82 Set the internal buffer size to n.
83 -cksum [on|off]
85 Enable/Disable CRC checks after file transfers.
86 -debug n
88 Set the debug level to n.
89 -family name
91 Set the remote storage family to name.
92 -glob [on|off]
94 Enable/Disable filename expansion.
95 -hash Enable printing of hash marks during transfers.
97 -keepalive n
99 Send control channel keepalive messages every n seconds during
100 data transfers.
101 -mode [E|S]
103 Switch the transfer mode to extended block (E) or streams mode
104 (S).
105 -parallel n
107 Use n parallel data channels during extended block transfers.
108 -passive
110 Use PASSIVE mode for data transfers.
111 -pbsz n
113 Set the data protection buffer size to n n bytes.
114 -prot [C|S|E|P]
116 Set the data protection lelvel to clear (C), safe (S), confiden‐
117 tial (E) or private (P).
118 -retry n
120 Retry commands that fail with transient errors n times.
121 -resume path
123 Retry the recursive transfer starting at path.
124 -tcpbuf n
126 Set the TCP read/write buffers to n bytes.
127 -wait This will cause the client to wait for remote files to stage
129 before attempting to transfer them.
130 -v Print UberFTP version information and exit.
132 -version
134 Print UberFTP version information and exit.
135 -versions
137 Print version information about all used globus modules and
138 exit.
139
142 -cat url
143 Print to stdout the contents of the remote file.
144 -chgrp [-r] group url
146 Set the group ownership of the remote object(s).
147 -chmod [-r] perms url
149 Set the permissions of the remote object(s).
150 -dir [-r] url
152 List the contents of the remote object.
153 -ls [-r] url
155 List the contents of the remote object.
156 -mkdir url
158 Create the remote directory.
159 -rename url path
161 Rename the remote object to the given path.
162 -rm [-r] url
164 Remove the remote object(s).
165 -rmdir url
167 Remove the remote directory.
168 -size url
170 Return the size of the remote object.
171 -stage [-r] seconds url
173 Attempt to stage the remote object(s) over the time period given
174 in seconds.
175
178 By default, without any special environment variables, command line
179 options or commands, uberftp will transfer files in PASSIVE STREAMS
180 mode. PASSIVE means that the client will initiate the data connection
181 which is useful for users behind firewalls. STREAMS mode implies that
182 GRIDFTP features including striping and parallel data connections are
183 not used. In order to take advantage of these features with GridFTP
184 capable servers, you must either change the mode directly using -m com‐
185 mand line switch or the mode interactive command, or you can change the
186 mode indirectly by specifying more than one parallel data connection
187 using the -c command line switch or by using the parallel interactive
188 command.
189
192 By default, uberftp requires a GSI certificate. If you do not already
193 have a certificate, see the following web page to learn how to get one:
194 http://www.ncsa.uiuc.edu/UserInfo/Grid/Security/GetUserCert.html
196 Once you have a certificate, use the grid-proxy-init command to get a
198 valid proxy.
199
202 ! [command]
203 Run the command using a shell on the local machine. If no com‐
204 mand is given, invoke an interactive shell.
205 ? [command]
207 If command is given, print a (hopefully) helpful blurb about it.
208 Otherwise, list all commands.
209 active Change to ACTIVE mode which causes the server to initiate the
211 data connection. The default is PASSIVE mode unless the variable
212 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
213 a firewall you must use PASSIVE mode.
214 ascii Change the data transfer type to ASCII which causes the server
216 to do some simple transformations to the file being transferred.
217 This is mostly useful for changing EOL (end of line) in text
218 files when moving between platforms. This option is almost
219 never necessary today. The default is BINARY mode also known as
220 IMAGE mode.
221 binary Change the data transfer type to BINARY (aka IMAGE) which causes
223 the server to not perform transformations to the file being
224 transferred. This is the default and is faster than an ASCII
225 transfer.
226 blksize size
228 Change the size of the memory buffer used to read and write data
229 to disks to size bytes. The default block size is 1024*1024
230 (1048576) bytes but it can be changed at compile time. The block
231 size can be increased to improve file transfer performance. This
232 is not related to the extended block mode block size used to
233 determine the ratio of data to header for data transferred on
234 the data channel.
235 bugs Prints information regarding bug reporting and feature requests.
237 bye Close all control and data connections and exit.
239 cat file1 [file2 ... filen]
241 Print the contents of the remote file(s) to stdout.
242 cdup Change the remote working directory up one level.
244 cd [dir]
246 Change the remote working directory to dir. If dir is not given,
247 the client will make every attempt to change to the user's home
248 directory.
249 chgrp [-r] group object [object2 ... objectn]
252 Change group ownership on the remote object(s).
253 -r Recursively chgrp everything in the given directory.
254 chmod [-r] perms object [object2 ... objectn]
257 Change permissions on the remote object(s).
258 -r Recursively chmod everything in the given directory.
259 close Close the control connection to the remote host.
261 cksum [on|off]
263 Enable file cksum comparison after each file transfer. This only
264 works with NCSA's mass storage system.
265 on Enable checksum comparison
266 off Disable checksum comparison
267 dcau [N|A|S subject]
269 Change the data channel authentication settings. If the service
270 does not support DCAU, these settings are ignored.
271 N Disabled dcau.
272 A Expect the remote identity to be mine. (Default)
273 S subject Expect the remote identity to be subject.
274 debug [0-3]
276 Turn debug statements on/off. If no value is given, this command
277 will toggle between debug(2) and non debug(1) mode. Otherwise
278 the debug level is set to the given level.
279 0 Only errors are printed
280 1 Default. Errors and some helpful messages are printed
281 2 Print useful control channel information
282 3 Print all information
283 family name
285 Sets the tape family to name on the FTP service if the service
286 supports it. If name is omitted, the current family is printed.
287 glob [on|off]
289 Enable or disable filename globbing. If no option is given, this
290 command will toggle the current setting.
291 on Enable filename globbing
292 off Disable filename globbing
293 dir [-r] [target]
295 List the contents of the remote target directory. If target is
296 not given, then the current working directory is used.
297 -r Recursively list target.
298 target Directory or file to list. '.' is used by default.
299 get [-r] source [destination]
301 Retreive file(s) from the remote service. If source implies mul‐
302 tiple transfers, either through regular expressions or by using
303 the recursive feature, then destination must be a directory. If
304 destination is not specified, source is used.
305 -r Recursively transfer the given directory.
306 hash Print hash marks during data transfers. This does not work dur‐
308 ing third party transfers.
309 help [command]
311 If command is given, print a helpful blurb about command. Oth‐
312 erwise, list all commands.
313 keepalive [seconds]
315 Attempts to keep the control channel from being blocked by fire‐
316 walls during long data channel operations. UberFTP sends a NOOP
317 command to the service at intervals equal to the specified num‐
318 ber of seconds. Setting it to zero will disable keepalive. If
319 seconds are not given, the current timeout is displayed. This
320 feature is disabled by default.
321 seconds number of seconds between NOOPs. Disabled if zero.
322 lcat file1 [file2 ... filen]
324 Print the contents of the local file(s) to stdout.
325 lcd [dir]
327 Change the local working directory to dir. If dir is not given,
328 the client will make every attempt to change to the user's home
329 directory.
330 lcdup Change the local working directory up one level.
332 lchgrp [-r] group object [object2 ... objectn]
334 Change group ownership on the local object(s).
335 -r Recursively chgrp everything in the given directory.
336 lchmod [-r] perms object [object2 ... objectn]
339 Change permissions on the local object(s).
340 -r Recursively chmod everything in the given directory.
341 lclose Close the control connection to the local host.
343 ldir [-r] [target]
345 List the contents of the local target directory. If target is
346 not given, then the current working directory is used.
347 -r Recursively list target.
348 target Directory or file to list. '.' is used by default.
349 lls [-r] [target]
351 List the contents of the local target directory. If target is
352 not given, then the current working directory is used.
353 -r Recursively list target.
354 target Directory or file to list. '.' is used by default.
355 lmkdir dir1 [dir2 ... dirn]
357 Create the local directory(ies).
358 lopen [-P port] [-u user] [-p pass | X] host
360 Opens a control channel to host and that host becomes the
361 'local' machine. After using lopen, all local (l*) commands
362 perform their respective operations on host rather than the
363 local machine. This is how third party transfers are accom‐
364 plished. GSI authentication is used unless the -p option is
365 used.
366 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
367 word).
368 -u user Connect as alternate user.
369 -p pass | X
370 Use password pass when authenticating with host.
371 If pass equals X, read the password from STDIN with
372 character echoing turned off.
373 host Connect to host.
374 lpwd Prints the current local working directory.
376 lrename src dst
378 Rename the local object src to dst.
379 lrm [-r] object1 [object1...objectn]
381 Removes the local file system object(s).
382 -r Recursively remove the given directory.
383 lrmdir dir1 [dir2...dirn]
385 Removes the given directories from the local service.
386 lquote cmd
388 Pass cmd to the local FTP service. This allows the user to use
389 server-specific commands that are not available through the
390 uberftp interface.
391 ls [-r] [target]
393 List the contents of the remote target directory. If [target] is
394 not given, then the current working directory is used.
395 -r Recursively list target.
396 target Directory or file to list. '.' is used by default.
397 lsize file1 [file2...filen]
399 Prints the size of the given object(s).
400 lstage [-r] seconds object1 [object2...objectn]
402 Attempt to stage all matching files within the given number of
403 seconds on the local service.
404 seconds number of seconds to attempt staging
405 -r Recursively stage all files in the given subdirectory.
406 mput [-r] object1 [object2...objectn]
408 Retrieve file(s) from the remote service. This is similiar to
409 making multiple calls to get without specifying a destination.
410 -r Recursively transfer the given directory.
411 mkdir dir
413 Create the remote directory.
414 mode [E|S]
416 Toggle the data transfer mode between Streams mode and Extended
417 Block mode. The default is Streams mode. If no option is given,
418 it will display the current mode.
419 E Extended block mode
420 S Streams mode
421 mput [-r] object1 [object2...objectn]
423 Store file(s) to the remote service. This is similiar to making
424 multiple calls to put without specifying a destination.
425 -r Recursively transfer the given directory.
426 open [-P port] [-u user] [-p pass | X] host
428 Opens a control channel to host and that host becomes the
429 'remote' machine. GSI authentication is used unless the -p
430 option is used.
431 -P port Connect to port (Default 2811 for GSI, 21 for pass‐
432 word).
433 -u user Connect as user.
434 -p pass | X
435 Use password pass when authenticating with host.
436 If pass equals X, read the password from STDIN with
437 character echoing turned off.
438 host Connect to host.
439 order [type]
441 Changes the order of lists returned from ls and lls to the given
442 scheme. If type is not given, the current order is displayed.
443 type Ordering scheme to use. Value options are:
444 none Do not order listings
445 name Order listings by name
446 size Order listings by size
447 type Order listings by type
448 parallel [number]
450 Set the number of parallel data connections to number. This is
451 only useful for extended block mode transfers. The default num‐
452 ber of data connections is one. If no number is given, the cur‐
453 rent setting for the number of parallel connects is printed.
454 passive
456 Change to PASSIVE mode which causes the client to initiate the
457 data connection. This is the default mode unless the variable
458 UBERFTP_ACTIVE_MODE is set in the environment. If you are behind
459 a firewall you must use PASSIVE mode.
460 pbsz [size]
462 Change the length of the protection buffer. The protection buf‐
463 fer is used to encrypt data on the data channel. The length of
464 the protection buffer represents the largest encoded message
465 that is allowed on the data channel. By default, the protection
466 buffer is grown to match the internal buffer used. For efficient
467 transfers, pbsz should be sufficiently larger than blksize so
468 that the wrapped buffer fits within the protection buffer. Oth‐
469 erwise, the blksize buffer is broken into multiple pieces so
470 that each write is less than pbsz when wrapped. If pbsz is not
471 given, the current size is displayed.
472 size length of protection buffer. 0 will set it to its
473 default.
474 pget offset size srcfile [destfile]
476 Retrieve only the specified portion of the file(s). If srcfile
477 is a regular expression and expands to multiple files, and des‐
478 tination is given, destination must refer to a directory.
479 offset Offset within the file
480 size Amount of data to retrieve
481 srcfile Name of remote file
482 destfile Name of local file. srcfile is used if destfile
483 is not specified
484 pput offset size srcfile [destfile]
486 Store only the specified portion of the file(s). If srcfile is a
487 regular expression and expands to multiple files, and destina‐
488 tion is given, destination must refer to a directory.
489 offset Offset within the file
490 size Amount of data to retrieve
491 srcfile Name of local file
492 destfile Name of remote file. srcfile is used if destfile
493 is not specified
494 prot [C|S|E|P]
496 This command configures the level of security on the data chan‐
497 nel after data channel authentication has completed. Clear means
498 that the data will not be protected. Safe means that the data
499 will be integrity protected meaning that altered data will be
500 detected. Confidential means that the data will be unreadable to
501 third parties. Private mode means the data will be confidential
502 and safe.
503 C Set protection level to clear.
504 S Set protection level to safe.
505 E Set protection level to confidential.
506 P Set protection level to private.
507 get [-r] source [destination]
509 Store file(s) to the remote service. If source implies multiple
510 transfers, either through regular expressions or by using the
511 recursive feature, then destination must be a directory. If des‐
512 tination is not specified, source is used.
513 -r Recursively transfer the given directory.
514 pwd Prints the current working directory.
516 quit Close all control and data connections and exit.
518 quote cmd
520 Pass cmd to the remote FTP service. This allows the user to use
521 server-specific commands that are not available through the
522 uberftp interface.
523 rename src dst
525 Rename the remote object src to dst.
526 retry [cnt]
528 Configures retry on failed commands that have transient errors.
529 cnt represents the number of times a failed command is retried.
530 A value of zero effectively disables retry. Zero is the default.
531 If no value is given the current setting is displayed.
532 cnt Number of times a failed command is retried.
533 resume [-d] path
535 Sets a restart point for recursive transfers. If a long recur‐
536 sive transfer fails, you can set resume to the path that failed
537 and UberFTP will skip all file and directory creations up to the
538 given path.
539 path Path to resume transfer at. If path is not given, print
540 the current
541 resume target.
542 -d Remove the current resume path.
543 rm [-r] object1 [object1...objectn]
545 Removes the remote file system object(s).
546 -r Recursively remove the given directory.
547 rmdir dir1 [dir2...dirn]
549 Removes the given directories from the remote service.
550 runique
552 Toggles the client to store files using unique names during put
553 operations.
554 size file1 [file2...filen]
556 Prints the size of the given object(s).
557 stage [-r] seconds object1 [object2...objectn]
559 Attempt to stage all matching files within the given number of
560 seconds on the remote service.
561 seconds number of seconds to attempt staging
562 -r Recursively stage all files in the given subdirectory.
563 sunique
565 Toggles the client to store files using unique names during get
566 operations.
567 tcpbuf [size]
569 Set the data channel TCP buffer size to size bytes. If size is
570 not given, the current TCP buffer size will be printed.
571 versions
573 Prints the versions of all Globus modules being used.
574 wait Toggles whether the client should wait for files to stage before
576 attempting to retrieve them.
577
581 Use the active command to enable active mode FTP when using NCSA's Uni‐
582 Tree mass storage system if possible since it will give much better
583 file transfer performance. When tranferring files over long distances,
584 use a large value (for example, 16777216) for tcpbuf. When there is
585 high network traffic, you may be able to improve performance using the
586 parallel command to increase the number of parallel data connections to
587 2-4.
588
590 In order to perform a third-party transfer, you must log into two FTP
591 servers. Typically, you connect to a single FTP service to "get" files
592 to the local machine and "put" files to the remote service. For third-
593 party transfers, you must connect to a second service thereby replacing
594 the former local machine. In UberFTP terminology, it is referred to as
595 "opening a new local service" since, from the perspective of the user,
596 the new local service will appear as though the user initiated the FTP
597 session from that machine.
598 All remote service commands have "l*" counterparts that allow you to
600 specify that the command is to be performed on the local service,
601 whether that service is the local machine or a new local service. So
602 to open a new local service, use the "l*" version of the open command:
603 UberFTP> lopen mss.ncsa.teragrid.org
605 UberFTP> lclose
606 Once you have connected to both services, files can be transferred as
608 before with the change that files you "get" go to the new local service
609 and files you "put" are sent from the new local service.
610
612 By default, local port selection is managed by the operating system.
613 However, you may wish to specify which ports UberFTP should use for
614 incoming and out going connections. This is useful when dealing with
615 firewalls.
616 Setting UBERFTP_TCP_PORT_RANGE in your environment will cause all
618 inbound connections to use the specified port range. Likewise, setting
619 UBERFTP_TCP_SOURCE_RANGE in your environment will cause all outbound
620 connections to use the specified port range.
621 The environment variables GLOBUS_TCP_PORT_RANGE and
623 GLOBUS_TCP_SOURCE_RANGE will also control the ephemeral port selection.
624 These variables behave exactly as their UBERFTP counterparts and are
625 available for backwards compatibility with older versions. The UBERFTP
626 variables take precedence over the GLOBUS variables.
627 The values of the variables specify a port range, a minimum port number
629 and a maximum port number, separated by either a comma or a space. For
630 example, to set the inbound port range, you would set:
631 UBERFTP_TCP_PORT_RANGE=40000,50000
633 Using the space delimiter, this format is also acceptable:
635 UBERFTP_TCP_PORT_RANGE="40000 50000"
637 See your shell documentation for the proper syntax for settings vari‐
639 ables within your environment.
640 Setting the ephemeral port range to an unusable range will cause
642 UberFTP connections to fail. For instance, setting a port range from 10
643 to 100 with a non root process will fail on most operating systems.
644
647 UberFTP will exit with a value of 0 if no errors occurred during the
648 session, otherwise it will exit with a value of 1. In non interactive,
649 commandline mode, it will exit after the first error occurs.
650
652 Set the environment variable to set active mode FTP (improves file
653 transfer performance to the mass storage system). Connect to NCSA's
654 UniTree mass storage system interactively from NCSA's TeraGrid cluster:
655 setenv UBERFTP_ACTIVE_MODE on
657 % uberftp mss.ncsa.teragrid.org
658 ...
659 220 UNIX Archive FTP server ready.
660 230 User consult logged in.
661 UberFTP>
662 Use the command-line interface to copy a file from NCSA's TeraGrid
664 cluster to the UniTree mass storage system. (There is no need to set
665 tcpbuf since it is over a LAN but active mode is turned on to improve
666 file transfer performance to the mass storage system.):
667 uberftp mss.ncsa.teragrid.org \
669 "active; cd work; get file.tar"
670 Copy a file from SDSC's TeraGrid cluster to NCSA's TeraGrid cluster.
672 (Note that tcpbuf is set to 16777216 since there is a long network
673 latency between NCSA and SDSC):
674 uberftp tg-gridftp.sdsc.teragrid.org \
676 "tcpbuf 16777216; cd scr; put file.tar"
677
679 mssftp(1), msscmd(1), ftp(1),
680 GridFTP:
681 http://www.globus.org/datagrid/gridftp.html
682 TCP Window Size:
683 http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html
684 Active vs. Passive FTP:
685 http://slacksite.com/other/ftp.html
686 Note: The links above are not under NCSA's control so they may become
688 obsolete.
689 16 May 2008 UBERFTP(1C)