1ncftpspooler(1) General Commands Manual ncftpspooler(1)
2
6 ncftpspooler - Global batch FTP job processor daemon
7
9 ncftpspooler -d [options]
10 ncftpspooler -l [options]
12
14 Command line flags:
15 -d Begin background processing of FTP jobs in the designated FTP
16 job queue directory.
17 -q XX Use this option to specify a directory to use as the FTP job
19 queue instead of the default directory, /var/spool/ncftp.
20 -o XX Use this option to specify a filename to use as the log file.
22 By default, (and rather inappropriately) the program simply
23 uses a file called log in the job queue directory. If you
24 don't want a log, use this option to specify /dev/null.
25 -l Lists the contents of the job queue directory.
27 -s XX When the job queue is empty, the program sleeps 120 seconds and
29 then checks again to see if a new job has been submitted. Use
30 this option to change the number of seconds used for this
31 delay.
32
34 The ncftpspooler program evolved from the ncftpbatch program. The
35 ncftpbatch program was originally designed as a ``personal FTP
36 spooler'' which would process a single background job a particular user
37 and exit when it finished; the ncftpspooler program is a ``global FTP
38 spooler'' which stays running and processes background jobs as they are
39 submitted.
40 The job queue directory is monitored for specially-named and formatted
42 text files. Each file serves as a single FTP job. The name of the job
43 file contains the type of FTP job (get or put), a timestamp indicating
44 the earliest the job should be processed, and optionally some addi‐
45 tional information to make it easier to create unique job files (i.e. a
46 sequence number). The contents of the job files have information such
47 as the remote server machine to FTP to, username, password, remote
48 pathname, etc.
49 Your job queue directory must be readable and writable by the user that
51 you plan to run ncftpspooler as, so that jobs can be removed or renamed
52 within the queue.
53 More importantly, the user that is running the program will need ade‐
55 quate privileges to access the local files that are involved in the
56 FTPing. I.e., if your spooler is going to be processing jobs which
57 upload files to remote servers, then the user will need read permission
58 on the local files that will be uploaded (and directory access permis‐
59 sion the parent directories). Likewise, if your spooler is going to be
60 processing jobs which download files, then the user would need to be
61 able to write to the local directories.
62 Once you have created your spool directory with appropriate permissions
64 and ownerships, you can run ncftpspooler -d to launch the spooler dae‐
65 mon. You can run additional spoolers if you want to process more than
66 FTP job from the same job queue directory simultaneously. You can then
67 monitor the log file (i.e., using tail -f ) to track the progress of
68 the spooler. Most of the time it won't be doing anything, unless job
69 files have appeared in the job queue directory.
70
72 When the ncftpspooler program monitors the job queue directory, it
73 ignores any files that do not follow the naming convention for job
74 files. The job files must be prefixed in the format of X-YYYYM‐
75 MDD-hhmmss where X denotes a job type, YYYY is the four-digit year, MM
76 is the two-digit month number, DD is the two-digit day of the month, hh
77 is the two-digit hour of the day (00-23), mm is the two-digit minute,
78 and ss is the two-digit second. The date and time represent the earli‐
79 est time you want the job to be run.
80 The job type can be g for a get (download from remote host), or p for
82 aput (upload to remote host).
83 As an example, if you wanted to schedule an upload to occur at 11:45 PM
85 on December 7, 2001, a job file could be named
86 p-20011207-234500
88 In practice, the job files include additional information such as a
90 sequence number or process ID. This makes it easier to create unique
91 job file names. Here is the same example, with a process ID and a
92 sequence number:
93 p-20011207-234500-1234-2
95 When submitting job files to the queue directory, be sure to use a dash
97 character after the hhmmss field if you choose to append any additional
98 data to the job file name.
99
101 Job files are ordinary text files, so that they can be created by hand.
102 Each line of the file is a key-pair in the format variable=value, or is
103 a comment line beginning with an octothorpe character (#), or is a
104 blank line. Here is an example job file:
105 # This is a NcFTP spool file entry.
107 job-name=g-20011016-100656-008299-1
108 op=get
109 hostname=ftp.freebsd.org
110 xtype=I
111 passive=1
112 remote-dir=pub/FreeBSD
113 local-dir=/tmp
114 remote-file=README.TXT
115 local-file=readme.txt
116 Job files are flexible since they follow an easy-to-use format and do
118 not have many requirements, but there are a few mandatory parameters
119 that must appear for the spooler to be able to process the job.
120 op The operation (job type) to perform. Valid values are get and
122 put.
123 hostname
125 The remote host to FTP to. This may be an IP address or a DNS
126 name (i.e. ftp.example.com).
127 For a regular get job, these parameters are required:
129 remote-file
131 The pathname of the file to download from the remote server.
132 local-file
134 The pathname to use on the local server for the downloaded
135 file.
136 For a regular put job, these parameters are required:
138 local-file
140 The pathname of the file to upload to the remote server.
141 remote-file
143 The pathname to use on the remote server for the uploaded file.
144 For a recursive get job, these parameters are required:
146 remote-file
148 The pathname of the file or directory to download from the
149 remote server.
150 local-dir
152 The directory pathname to use on the local server to contain
153 the downloaded items.
154 For a recursive put job, these parameters are required:
156 local-file
158 The pathname of the file or directory to upload to the remote
159 server.
160 remote-dir
162 The directory pathname to use on the remote server to contain
163 the uploaded items.
164 The rest of the parameters are optional. The spooler will attempt to
166 use reasonable defaults for these parameters if necessary.
167 user The username to use to login to the remote server. Defaults to
169 ``anonymous'' for guest access.
170 pass The password to use in conjunction with the username to login
172 to the remote server.
173 acct The account to use in conjunction with the username to login to
175 the remote server. The need to specify this parameter is
176 extremely rare.
177 port The port number to use in conjunction with the remote hostname
179 to connect to the remote server. Defaults to the standard FTP
180 port number, 21.
181 host-ip The IP address to use in conjunction with the remote hostname
183 to connect to the remote server. This parameter can be used in
184 place of the hostname parameter, but one or the other must be
185 used. This parameter is commonly included along with the host‐
186 name parameter as supplemental information.
187 xtype The transfer type to use. Defaults to binary transfer type
189 (TYPE I). Valid values are I for binary, A for ASCII text.
190 passive Whether to use FTP passive data connections (PASV) or FTP
192 active data connections (PORT). Valid values are 0 for active,
193 1 for passive, or 2 to try passive, then fallback to active.
194 The default is 2.
195 recursive
197 This can be used to transfer entire directory trees. By
198 default, only a single file is transferred. Valid values are
199 yes or no.
200 delete This can be used to delete the source file on the source
202 machine after successfully transferring the file to the desti‐
203 nation machine. By default, source files are not deleted.
204 Valid values are yes or no.
205 job-name
207 This isn't used by the program, but can be used by an entity
208 which is automatically generating job files. As an example,
209 when using the -bbb flag with ncftpput, it creates a job file
210 on stdout with a job-name parameter so you can easily copy the
211 file to the job queue directory with the suggested job name as
212 the job file name.
213 pre-ftp-command
215 post-ftp-command
217 These parameters correspond to the -W, and -Y options of
218 ncftpget and ncftpput. It is important to note that these
219 refer to RFC959 File Transfer Protocol commands and not shell
220 commands, nor commands used from within /usr/bin/ftp or ncftp.
221 pre-shell-command
223 post-shell-command
225 These parameters provide hooks so you can run a custom program
226 when an item is processed by the spooler. Valid values are
227 pathnames to scripts or executable programs. Note that the
228 value must not contain any command-line arguments -- if you
229 want to do that, create a shell script and have it run your
230 program with the command-line arguments it requires.
231 Generally speaking, post-shell-command is much more useful than
233 pre-shell-command since if you need to use these options you're more
234 likely to want to do something after the FTP transfer has completed
235 rather than before. For example, you might want to run a shell script
236 which pages an administrator to notify her that her 37 gigabyte file
237 download has completed.
238 When your custom program is run, it receives on standard input the con‐
240 tents of the job file (i.e. several lines of variable=value key-pairs),
241 as well as additional data the spooler may provide, such as a result
242 key-pair with a textual description of the job's completion status.
243 post-shell-command update a log file named /var/log/ncftp_spooler.
245 #!/usr/bin/perl -w
247 my ($line);
249 my (%params) = ();
250 while (defined($line = <STDIN>)) {
252 $params{$1} = $2
253 if ($line =~ /^([^=\#\s]+)=(.*)/);
254 }
255 if ((defined($params{"result"})) &&
257 ($params{"result"} =~ /^Succeeded/))
258 {
259 open(LOG, ">> /var/log/ncftp_spooler.log")
260 or exit(1);
261 print LOG "DOWNLOAD" if ($params{"op"} eq "get");
262 print LOG "UPLOAD" if ($params{"op"} eq "put");
263 print LOG " ", $params{"local-file"}, "\n";
264 close(LOG);
265 }
266
268 The log file should be examined to determine if any ncftpspooler pro‐
269 cesses are actively working on jobs. The log contains copious amounts
270 of useful information, including the entire FTP control connection con‐
271 versation between the FTP client and server.
272
274 The recursive option may not be reliable since ncftpspooler depends on
275 functionality which may or may not be present in the remote server
276 software. Additionally, even if the functionality is available, ncftp‐
277 spooler may need to use heuristics which cannot be considered 100%
278 accurate. Therefore it is best to create individual jobs for each file
279 in the directory tree, rather than a single recursive directory job.
280 For resumption of downloads to work, the remote server must support the
282 FTP SIZE and MDTM primitives. Most modern FTP server software can do
283 this, but there are still a number of bare-bones ftpd implementations
284 which do not. In these cases, ncftpspooler will re-download the file
285 in entirety each time until the download succeeds.
286 The program needs to be improved to detect jobs that have no chance of
288 ever completing successfully. There are still a number of cases where
289 jobs can get spooled but get retried over and over again until a vigi‐
290 lant sysadmin manually removes the jobs.
291 The spool files may contain usernames and passwords stored in cleart‐
293 ext. These files should not be readable by any user except the user
294 running the program!
295
297 Mike Gleason, NcFTP Software (http://www.ncftp.com).
298
300 ncftpbatch(1), ncftp(1), ncftpput(1), ncftpget(1), uucp(1).
301ncftpspooler NcFTP Software ncftpspooler(1)