1FPSYNC(1) BSD General Commands Manual FPSYNC(1)
2
4 fpsync — Synchronize directories in parallel using fpart + rsync or cpio.
5
7 fpsync [-h] [-v] [-m tool] [-n jobs] [-f files] [-s size] [-E] [-w wrks]
8 [-d shdir] [-t tmpdir] [-r jobname] [-o toolopts] [-O fpartopts]
9 [-S] src_dir/ dst_url/
10
12 The fpsync tool synchronizes directories in parallel using fpart(1) and
13 rsync(1) or cpio(1). It computes subsets of src_dir/ and spawns jobs to
14 synchronize them to dst_url/.
15 Synchronization jobs can be executed either locally or remotely (using
17 SSH workers, see option -w) and are executed on-the-fly while filesystem
18 crawling goes on. This makes fpsync a good tool for migrating large
19 filesystems.
20
22 -h Print help
23 -v Verbose mode. Can be be specified several times to increase ver‐
25 bosity level.
26 -m tool
28 External copy tool used to synchronize files. Currently supported
29 tools are: rsync and cpio. Default: rsync. When using cpio and
30 more than one worker, directory timestamps may not be replicated.
31 A second pass will fix them. See the README file for more
32 details.
33 -n jobs
35 Start jobs concurrent sync jobs (either locally or remotely, see
36 below). Default: 2
37 -f files
39 Transfer at most files files or directories per sync job.
40 Default: 2000
41 -s size
43 Transfer at most size bytes per sync job.
44 Default: 4294967296 (4 GB)
45 -E Work on a per-directory basis (rsync tool only). In that mode,
47 fpsync works with lists of directories instead of files. That
48 mode may generate coarse-grained lists but enables rsync(1) 's
49 --delete option by default ( WARNING!!! ), making it a good can‐
50 didate for a final (cleaning) pass after several incremental
51 passes using standard (file) mode.
52 -w wrks
54 Use remote SSH wrks to synchronize files. Synchronization jobs
55 are executed locally when this option is not set. wrks is a
56 space-separated list of login@machine connection strings and can
57 be specified several times. You must be allowed to connect to
58 those machines using a SSH key to avoid user interaction.
59 -d shdir
61 Set fpsync shared directory to shdir. This option is mandatory
62 when using SSH workers and set by default to tmpdir when running
63 locally. The specified directory must be an absolute path ; it
64 will be used to handle communications with SSH hosts (sharing
65 partitions and log files) and, as a consequence, must be made
66 available to all participating hosts (e.g. through a r/w NFS
67 mount), including the master one running fpsync.
68 -t tmpdir
70 Set fpsync temporary directory to tmpdir. This directory remains
71 local and does not need to be shared amongst SSH workers when
72 using the -w option. Default: /tmp/fpsync
73 -r jobname
75 Resume job jobname and restart synchronizing remaining partitions
76 from a previous run. jobname can be obtained using verbose mode
77 (see option -v). Note that filesystem crawling is skipped when
78 resuming a previous run. As a consequence, options -f, -s, -o,
79 -O, -S, src_dir/, and dst_url/ are ignored.
80 -o toolopts
82 Override default rsync(1) or cpio(1) options with toolopts. Use
83 this option with care as certain options are incompatible with a
84 parallel usage (e.g. rsync's --delete). Default for rsync:
85 -lptgoD -v --numeric-ids. Empty for cpio.
86 -O fpartopts
88 Override default fpart(1) options with fpartopts.
89 Default: -x .zfs -x .snapshot* -x .ckpt
90 -S Sudo mode. Use sudo(8) for filesystem crawling and synchroniza‐
92 tions.
93 src_dir/
95 Source directory. It must be absolute and available on all par‐
96 ticipating hosts (including the master one, running fpsync).
97 dst_url/
99 Destination directory or URL (rsync tool only). If a remote URL
100 is provided, it must be supported by rsync(1). All participating
101 workers must be able to reach that target.
102
104 Each fpsync run generates a unique jobname, which is displayed in verbose
105 mode (see option -v) and within log files. You can use that jobname to
106 resume a previous run (see option -r). fpsync will then restart synchro‐
107 nizing data from the parts that were being synchonized at the time it
108 stopped.
109 This unique feature gives the administrator the ability to stop fpsync
111 and restart it later, without having to restart the whole filesystem
112 crawling and synchronization process. Note that resuming is only possible
113 when filesystem crawling step has finished.
114 During synchronization, you can press CTRL-C to interrupt the process.
116 The first CTRL-C prevents new synchronizations from being submitted and
117 the process will wait for current synchronizations to be finished before
118 exiting. If you press CTRL-C again, current synchronizations will be
119 killed and fpsync will exit immediately. When using option -E to enable
120 directory mode and rsync's --delete option, keep in mind that killing
121 rsync processes may lead to a situation where certain files have been
122 updated and others not deleted yet (because the deletion process is post‐
123 poned using rsync's --delete-after option).
124 On certain systems, CTRL-T can be pressed to get the status of current
126 and remaining parts to be synchronized. This can also be achieved by
127 sending a SIGINFO to the fpsync process.
128 Whether you use verbose mode or not, everything is logged within
130 shdir/log/.
131
133 Here are some examples:
134 fpsync -n 4 /usr/src/ /var/src/
136 Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
138 fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
140 /mnt/dst/
141 Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe‐
143 cuted remotely on 2 SSH workers (machine1 and machine2). The
144 shared directory is set to /mnt/fpsync and mounted on the machine
145 running fpsync, as well as on machine1 and machine2. The source
146 directory (/mnt/src/) is also available on those 3 machines,
147 while the destination directory (/mnt/dst/) is mounted on SSH
148 workers only (machine1 and machine2).
149
151 Parallelizing rsync(1) or cpio(1) makes several options not usable, such
152 as rsync's --delete. If your source directory is live while fpsync is
153 running, you will have to delete extra files from destination directory.
154 This is traditionally done by using a final -offline- rsync(1) pass that
155 will use this option, but you can also use fpsync and option -E to per‐
156 form the same task using several workers.
157 fpsync enqueues synchronization jobs on disk, within the tmpdir/queue
159 directory. Be careful to host this queue on a filesystem that can handle
160 fine-grained mtime timestamps (i.e. with a sub-second precision) if you
161 want the queue to be processed in order when fpart(1) generates several
162 jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
163 using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
164 Contrary to rsync(1), fpsync enforces the final '/' on the source direc‐
166 tory. It means that directory contents are synchronized, not the source
167 directory itself (i.e. you will not get a subdirectory of the name of the
168 source directory in the target directory after synchronization).
169
171 cpio(1), fpart(1), rsync(1), sudo(8)
172
174 Fpsync has been written by Ganaël LAPLANCHE and is available under the
175 BSD license on
176 http://contribs.martymac.org
177
179 No bug known (yet).
180BSD January 27, 2015 BSD