1sitecopy(1) User Manuals sitecopy(1)
2
6 sitecopy - maintain remote copies of web sites
7
9 sitecopy [options] [operation mode] sitename ...
10
12 sitecopy is for copying locally stored web sites to remote web servers.
13 A single command will upload files to the server which have changed
14 locally, and delete files from the server which have been removed
15 locally, to keep the remote site synchronized with the local site. The
16 aim is to remove the hassle of uploading and deleting individual files
17 using an FTP client. sitecopy will also optionally try to spot files
18 you move locally, and move them remotely.
19 FTP, WebDAV and other HTTP-based authoring servers (for instance,
21 AOLserver and Netscape Enterprise) are supported.
22
25 This section covers how to start maintaining a web site using sitecopy.
26 After introducing the basics, two situations are covered: first, where
27 you have already upload the site to the remote server; second, where
28 you haven't. Lastly, normal site maintenance activities are explained.
29 Introducing the Basics
32 If you have not already done so, you need to create an rcfile, which
33 will store information about the sites you wish to administer. You also
34 need to create a storage directory, which sitecopy uses to record the
35 state of the files on each of the remote sites. The rcfile and storage
36 directory must both be accessible only by you - sitecopy will not run
37 otherwise. To create the storage directory with the correct permis‐
38 sions, use the command
39 mkdir -m 700 .sitecopy
40 from your home directory. To create the rcfile, use the commands
41 touch .sitecopyrc
42 chmod 600 .sitecopyrc
43 from your home directory. Once this is done, edit the rcfile to enter
44 your site details as shown in the CONFIGURATION section.
45 Existing Remote Site
48 If you have already uploaded the site to the remote server, ensure your
49 local files are synchronized with the remote files. Then, run
50 sitecopy --catchup sitename
51 where sitename is the name of the site you used after the site keyword
52 in the rcfile.
53 If you do not have a local copy of the remote site, then you can use
55 fetch mode to discover what is on the remote site, and synchronize mode
56 to download it. Fetch mode works well for WebDAV servers, and might
57 work if you're lucky for FTP servers. Run
58 sitecopy --fetch sitename
59 to fetch the site - if this succeeds, then run
60 sitecopy --synch sitename
61 to download a local copy. Do NOT do this if you already have a local
62 copy of your site.
63 New Remote Site
66 Ensure that the root directory of the site has been created on the
67 server by the server administrator. Run
68 sitecopy --init sitename
69 where sitename is the name of the site you used after the site keyword
70 in the rcfile.
71 Site Maintenance
74 After setting up the site as given in one of the two above sections,
75 you can now start editing your local files as normal. When you have
76 finished a set of changes, and you want to update the remote copy of
77 the site, run:
78 sitecopy --update sitename
79 and all the changed files will be uploaded to the server. Any files you
80 delete locally will be deleted remotely too, unless the nodelete option
81 is specified in the rcfile. If you move any files between directories,
82 the remote files will be deleted from the server then uploaded again
83 unless you specify the checkmoved option in the rcfile.
84 At any time, if you wish to see what changes you have made to the local
86 site since the last update, you can run
87 sitecopy sitename
88 which will display the list of differences.
89 Synchronization Problems
92 In some circumstances, the actual files which make up the remote site
93 will be different from what sitecopy thinks is on the remote site. This
94 can happen, for instance, if the connection to the server is broken
95 during an update. When this situation arises, Fetch Mode should be used
96 to fetch the list of files making up the site from the remote server.
97
100 In normal operation, specify a single operation mode, followed by any
101 options you choose, then one or more site names. For instance,
102 sitecopy --update --quiet mainsite anothersite
103 will quietly update the sites named 'mainsite' and 'anothersite'.
104
107 -l, --list
108 List Mode - produces a listing of all the differences between
109 the local files and the remote copy for the specified sites.
110 -ll, --flatlist
112 Flat list Mode - like list mode, except the output produced is
113 suitable for parsing by an external script or program. An AWK
114 script, changes.awk. is provided which produces an HTML page
115 from this mode.
116 -u, --update
118 Update Mode - updates the remote copy of the specified sites.
119 -f, --fetch
121 Fetch Mode - fetches the list of files from the remote server.
122 Note that this mode has only limited support in FTP - the server
123 must accept the MDTM command, and use a Unix-style 'ls' for LIST
124 implementation.
125 -s, --synchronize
127 Synchronize Mode - updates the local site from the remote copy.
128 WARNING: This mode overwrites local files. Use with care.
129 -i, --initialize
131 Initialization Mode - initializes the sites specified - making
132 sitecopy think there are NO files on the remote server.
133 -c, --catchup
135 Catchup Mode - makes sitecopy think the local site is exactly
136 the same as the remote copy.
137 -v, --view
139 View Mode - displays all the site definitions from the rcfile.
140 -h, --help
142 Display help information.
143 -V, --version
145 Display version information.
146
148 -y, --prompting
149 Applicable in Update Mode only, will prompt the user for confir‐
150 mation for each update (i.e., creating a directory, uploading a
151 file etc.).
152 -r RCFILE, --rcfile=RCFILE
154 Specify an alternate run control file location.
155 -p PATH, --storepath=PATH
157 Specify an alternate location to use for the remote site storage
158 directory.
159 -q, --quiet
161 Quiet output - display the filename only for each update per‐
162 formed.
163 -qq, --silent
165 Very quiet output - display nothing for each update performed.
166 -o, --show-progress
168 Applicable in Update Mode only, displays the progress (percent‐
169 age complete) of data transfer.
170 -k, --keep-going
172 Keep going past errors in Update Mode or Synch Mode
173 -a, --allsites
176 Perform the given operation on all sites - applicable for all
177 modes except View Mode, for which it has no effect.
178 -d MASK, --debug=KEY[,KEY...]
180 Turns on debugging. A list of comma-separated keywords should
181 be given. Each keyword may be one of:
182 socket Socket handling
183 files File handling
184 rcfile rcfile parser
185 http HTTP driver
186 httpbody Display response bodies in HTTP
187 ftp FTP driver
188 xml XML parsing information
189 xmlparse Low-level XML parsing information
190 httpauth HTTP authentication information
191 cleartext Display passwords in plain text
192 Passwords will be obscured in the debug output unless the clear‐
194 text keyword is used. An example use of debugging is to debug
195 FTP fetch mode:
196 sitecopy --debug=ftp,socket --fetch sitename
198
201 The stored state of a site is the snapshot of the state of the site
202 saved into the storage directory (~/.sitecopy/). The storage file is
203 used to record this state between invocations. In update mode, sitecopy
204 builds up a files list for each site by scanning the local directory,
205 reading in the stored state, and comparing the two - determining which
206 files have changed, which have moved, and so on.
207
210 Configuration is performed via the run control file (rcfile). This
211 file contains a set of site definitions. A unique name is assigned to
212 every site definition, which is used on the command line to refer to
213 the site.
214 Each site definition contains the details of the server the site is
216 stored on, how the site may be accessed at that server, where the site
217 is held locally and remotely, and any other options for the site.
218 Site Definition
221 A site definition is made up of a series of lines:
222 site sitename
224 server server-name
225 remote remote-root-directory
226 local local-root-directory
227 [ port port-number ]
228 [ username username ]
229 [ password password ]
230 [ proxy-server proxy-name
231 proxy-port port-number ]
232 [ url siteURL ]
233 [ protocol { ftp | webdav } ]
234 [ ftp nopasv ]
235 [ ftp showquit ]
236 [ ftp { usecwd | nousecwd } ]
237 [ http expect ]
238 [ http secure ]
239 [ safe ]
240 [ state { checksum | timesize } ]
241 [ permissions { ignore | exec | all | dir } ]
242 [ symlinks { ignore | follow | maintain } ]
243 [ nodelete ]
244 [ nooverwrite ]
245 [ checkmoved [renames] ]
246 [ tempupload ]
247 [ exclude pattern ]...
248 [ ignore pattern ]...
249 [ ascii pattern ]...
250 Anything after a hash (#) in a line is ignored as a comment. Values
252 may be quoted and characters may be backslash-escaped. For example, to
253 use the exclude pattern *#, use the following line:
254 exclude "*#"
255 Remote Server Options
258 The server key is used to specify the remote server the site is stored
259 on. This may be either a DNS name or IP address. A connection is made
260 to the default port for the protocol used, or that given by the port
261 key. sitecopy supports the WebDAV or FTP protocols - the protocol key
262 specifies which to use, taking the value of either webdav or ftp
263 respectively. By default, FTP will be used.
264 The proxy-server and proxy-port keys may be used to specify a proxy
266 server to use. Proxy servers are currently only supported for WebDAV.
267 If the FTP server does not support passive (PASV) mode, then the key
269 ftp nopasv should be used. To display the message returned by the
270 server on closing the connection, use the ftp showquit option. If the
271 server only supports uploading files in the current working directory,
272 use the key ftp usecwd (possible symptom: "overwrite permission
273 denied"). Note that the remote-directory (keyword remote) must be an
274 absolute path (starting with '/'), or usecwd will be ignored.
275 If the WebDAV server correctly supports the 100-continue expectation,
277 e.g. Apache 1.3.9 and later, the key http expect should be used. Doing
278 so can save some bandwidth and time in an update.
279 If the WebDAV server supports access via SSL, the key http secure can
281 be used. Doing so will cause the transfers between sitecopy and the
282 host to be performed using an secure, encrypted link. The first time
283 SSL is used to access the server, the user will be prompted to verify
284 the SSL certificate, if it's not signed by a CA trusted in the system's
285 CA root bundle.
286 To authenticate the user with the server, the username and password
288 keys are used. If it exists, the ~/.netrc will be searched for a pass‐
289 word if one is not specified. See ftp(1) for the syntax of this file.
290 Basic and digest authentication are supported for WebDAV. Note that
292 basic authentication must not be used unless the connection is known to
293 be secure.
294 The full URL that is used to access the site can optionally be speci‐
296 fied in the url key. This is used only in flat list mode, so the site
297 URL can be inserted in 'Recent Changes' pages. The URL must not have a
298 trailing slash; a valid example is
299 url http://www.site.com/mysite
300 If the tempupload option is given, new or changed files are upload with
302 a ".in." prefix, then moved to the true filename when the upload is
303 complete.
304 File State
307 File state is stored in the storage files (~/.sitecopy/*), and is used
308 to discover when a file has been changed. Two methods are supported,
309 and can be selected using the state option, with either parameter:
310 timesize (the default), and checksum.
311 timesize uses the last-modification date and the size of files to
313 detect when they have changed. checksum uses an MD5 checksum to detect
314 any changes to the file contents.
315 Note that MD5 checksumming involves reading in the entire file, and is
317 slower than simply using the last-modification date and size. It may be
318 useful for instance if a versioning system is in use which updates the
319 last-modification date on a 'checkout', but this doesn't actually
320 change the file contents.
321 Safe Mode
324 Safe Mode is enabled by using the safe key. When enabled, each time a
325 file is uploaded to the server, the modification time of the file as on
326 the server is recorded. Subsequently, whenever this file has been
327 changed locally and is to be uploaded again, the current modification
328 time of the file on the server is retrieved, and compared with the
329 stored value. If these differ, then the remote copy of the file has
330 been altered by a foreign party. A warning message is issued, and your
331 local copy of the file will not be uploaded over it, to prevent losing
332 any changes.
333 Safe Mode can be used with FTP or WebDAV servers, but if Apache/mod_dav
335 is used, mod_dav 0.9.11 or later is required.
336 Note Safe mode cannot be used in conjunction with the nooverwrite
338 option (see below).
339 File Storage Locations
342 The remote key specifies the root directory of the remote copy of the
343 site. It may be in the form of an absolute pathname, e.g.
344 remote /www/mysite/
345 For FTP, the directory may also be specified relative to the login
346 directory, in which case it must be prefixed by "~/", for example:
347 remote ~/public_html/
348 The local key specifies the directory in which the site is stored
350 locally. This may be given relative to your home directory (as given
351 by the environment variable $HOME), again using the "~/" prefix.
352 local ~/html/foosite/
353 local /home/fred/html/foosite/
354 are equivalent, if $HOME is set to "/home/fred".
355 For both the local and remote keywords, a trailing slash may be used,
357 but is not required.
358 File Permissions Handling
361 File permissions handling is dictated by the permissions key, which may
362 be given one of three values:
363 ignore to ignore file permissions completely (the default),
365 exec to mirror the permissions of executable files only,
367 all to mirror the permissions of all files.
369 This can be used, for instance, to ensure the permissions of CGI files
371 are set. The option is currently ignored for WebDAV servers. For FTP
372 servers, a chmod is performed remotely to set the permissions.
373 To handle directory permissions, the key:
375 permissions dir
376 may be used in addition to a permissions key of either exec, local or
377 all. Note that permissions all does not imply permissions dir.
378 Symbolic Link Handling
381 Symlinks found in the local site can be either ignored, followed, or
382 maintained. In 'follow' mode, the files references by the symlinks will
383 be uploaded in their place. In 'maintain' mode, the link will be cre‐
384 ated remotely as well, see below for more information. The mode used
385 for each site is specified with the symlinks rcfile key, which may take
386 the value of ignore, follow or maintain to select the mode as appropri‐
387 ate.
388 The default mode is follow, i.e. symbolic links found in the local site
390 are followed.
391 Symbolic link Maintain Mode
394 This mode is currently only supported by the WebDAV driver, and will
395 work only with servers which implement WebDAV Advanced Collections,
396 which is a work-in-progress. The target of the link on the server is
397 literally copied from the target of the symlink. Hint: you can use
398 URL's if you like:
399 ln -s "http://www.somewhere.org/" somewherehome
400 In this way, a "302 Redirect" can be easily set up from the client,
402 without having to alter the server configuration.
403 Deleting and Moving Remote Files
406 The nodelete option may be used to prevent remote files from ever being
407 deleted. This may be useful if you keep large amounts of data on the
408 remote server which you do not need to store locally as well.
409 If your server does not allow you to upload changed files over existing
411 files, then you can use the nooverwrite option. When this is used,
412 before uploading a changed file, the remote file will be deleted.
413 If the checkmoved option is used, sitecopy will look for any files
415 which have been moved locally. If any are found, when the remote site
416 is updated, the files will be moved remotely.
417 If the checkmoved renames option is used, sitecopy will look for any
419 files which have been moved or renamed locally. This option may only be
420 used in conjunction with the state checksum option.
421 WARNING
423 If you are not using MD5 checksumming (i.e. the state checksum option)
425 to determine file state, do NOT use the checkmoved option if you tend
426 to hold files in different directories with identical sizes, modifica‐
427 tion times and names and ever move them about. This seems unlikely, but
428 don't say you haven't been warned.
429 Excluding Files
432 Files may be excluded from the files list by use of the exclude key,
433 which accepts shell-style globbing patterns. For example, use
434 exclude *.bak
435 exclude *~
436 exclude "#*#"
437 to exclude all files which have a .bak extension, end in a tilde (~)
438 character, or which begin and end with a a hash. Don't forget to quote
439 or escape the value if it includes a hash!
440 To exclude certain files within an particular directory, simply prefix
442 the pattern with the directory name - including a leading slash. For
443 instance:
444 exclude /docs/*.m4
445 exclude /files/*.gz
446 which will exclude all files with the .m4 extension in the 'docs' sub‐
447 directory of the site, and all files with the .gz extension in the
448 files subdirectory.
449 An entire directory can also be excluded - simply use the directory
451 name with no trailing slash. For example
452 exclude /foo/bar
453 exclude /where/else
454 to exclude the 'foo/bar' and 'where/else' subdirectories of the site.
455 Exclude patterns are consulted when scanning the local directory, and
457 when scanning the remote site during a --fetch. Any file which matches
458 any exclude pattern is not added to the files list. This means that a
459 file which has already been uploaded by sitecopy, and subsequently
460 matches an exclude pattern will be deleted from the server.
461 Ignoring Local Changes to Files
464 The ignore option is used to instruct sitecopy to ignore any local
465 changes made to a file. If a change is made to the contents of an
466 ignored file, this file will not be uploaded by update mode. Ignored
467 files will be created, moved and deleted as normal.
468 The ignore option is used in the same way as the exclude option.
470 Note that synchronize mode will overwrite changes made to ignored
472 files.
473 FTP Transfer Mode
476 To specify the FTP transfer mode for files, use the ascii key. Any
477 files which are transferred using ASCII mode have CRLF/LF translation
478 performed appropriately. For example, use
479 ascii *.pl
480 to upload all files with the .pl extension as ASCII text. This key has
481 no effect with WebDAV (currently).
482
485 Return values are specified for different operation modes. If multiple
486 sites are specified on the command line, the return value is in respect
487 to the last site given.
488 Update Mode
491 -1 ... update never even started - configuration problem
492 0 ... update was entirely successful.
493 1 ... update went wrong somewhere
494 2 ... could not connect or login to server
495 List Mode (default mode of operation)
498 -1 ... could not form list - configuration problem
499 0 ... the remote site does not need updating
500 1 ... the remote site needs updating
501
504 FTP Server, Simple Usage
505 Fred's site is uploaded to the FTP server 'my.server.com' and held in
506 the directory 'public_html', which is in the login directory. The site
507 is stored locally in the directory /home/fred/html.
508 site mysite
510 server my.server.com
511 url http://www.server.com/fred
512 username fred
513 password juniper
514 local /home/fred/html/
515 remote ~/public_html/
516 FTP Server, Complex Usage
519 Here, Freda's site is uploaded to the FTP server ´ftp.elsewhere.com´,
520 where it is held in the directory /www/freda/. The local site is stored
521 in /home/freda/sites/elsewhere/
522 site anothersite
524 server ftp.elsewhere.com
525 username freda
526 password blahblahblah
527 local /home/freda/sites/elsewhere/
528 remote /www/freda/
529 # Freda wants files with a .bak extension or a
530 # trailing ~ to be ignored:
531 exclude *.bak
532 exclude *~
533 WebDAV Server, Simple Usage
536 This example shows use of a WebDAV server.
537 site supersite
539 server dav.wow.com
540 protocol webdav
541 username pow
542 password zap
543 local /home/joe/www/super/
544 remote /
545
548 ~/.sitecopyrc Default run control file location.
549 ~/.sitecopy/ Remote site information storage directory
550 ~/.netrc Remote server accounts information
551
554 Known problems: Fetch + synch modes are NOT reliable for FTP. If you
555 need reliable operation of fetch or synch modes, you shouldn't be using
556 sitecopy. Try rsync instead.
557 Please send bug reports and feature requests to <sitecopy@lyra.org>
560 rather than to the author, since the mailing list is archived and can
561 be a useful resource for others.
562
565 rsync(1), ftp(1), mirror(1)
566
569 [Listed for reference only, no claim of compliance to any of the below
570 standards is made.]
571 RFC 959 - File Transfer Protocol (FTP)
573 RFC 1521 - Multipurpose Internet Mail Extensions Part One
574 RFC 1945 - Hypertext Transfer Protocol -- HTTP/1.0
575 RFC 2396 - Uniform Resource Identifiers: Generic Syntax
576 RFC 2518 - HTTP Extensions for Distributed Authoring -- WEBDAV
577 RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1
578 RFC 2617 - HTTP Authentication
579 REC-XML - Extensible Markup Language (XML) 1.0
580 REC-XML-NAMES - Namespaces in XML
581
584 draft-ietf-ftpext-mlst-05.txt - Extensions to FTP
585 draft-ietf-webdav-collections-protocol-03.txt - WebDAV Advanced Collec‐
586 tions Protocol
587
589 Joe Orton and others.
590 e-mail: sitecopy@lyra.org
591 www: http://www.lyra.org/sitecopy/
592sitecopy June 2001 sitecopy(1)