1FPSYNC(1) BSD General Commands Manual FPSYNC(1)
2
4 fpsync — Synchronize directories in parallel using fpart and an external
5 tool
6
8 fpsync [-p] [-n jobs] [-w wrks] [-m tool] [-T path] [-f files] [-s size]
9 [-E] [-o toolopts] [-O fpartopts] [-S] [-t tmpdir] [-d shdir]
10 [-M mailaddr] [-v] src_dir/ dst_url/
11 fpsync -l
12 fpsync -r runid [-R] [OPTIONS...]
13 fpsync -a runid
14 fpsync -D runid
15
17 The fpsync tool synchronizes directories in parallel using fpart(1) and
18 rsync(1), cpio(1) or tar(1). It computes subsets of src_dir/ and spawns
19 jobs to synchronize them to dst_url/.
20 Synchronization jobs can be executed either locally or remotely (using
22 SSH workers, see option -w) and are executed on-the-fly while filesystem
23 crawling goes on. This makes fpsync a good tool for migrating large
24 filesystems.
25
27 -t tmpdir
28 Set fpsync temporary directory to tmpdir. This directory remains
29 local and does not need to be shared amongst SSH workers when us‐
30 ing the -w option. Default: /tmp/fpsync
31 -d shdir
33 Set fpsync shared directory to shdir. This option is mandatory
34 when using SSH workers and set by default to tmpdir when running
35 locally. The specified directory must be an absolute path ; it
36 will be used to handle communications with SSH hosts (sharing
37 partitions and log files) and, as a consequence, must be made
38 available to all participating hosts (e.g. through a r/w NFS
39 mount), including the master one running fpsync.
40 -M mailaddr
42 Send an e-mail to mailaddr after a run. Multiple -space-sepa‐
43 rated- addresses can be specified. That option requires the
44 mail(1) client to be installed and configured on the master host
45 running fpsync.
46 -v Verbose mode. Can be be specified several times to increase ver‐
48 bosity level.
49 -h Print help
51
53 -m tool
54 External copy tool used to synchronize files. Currently sup‐
55 ported tools are: rsync, cpio, tar, and tarify. Default: rsync.
56 When using cpio or tar and more than one worker, directory time‐
57 stamps may not be replicated. A second pass will fix them. Spe‐
58 cial tool tarify generates tarballs into destination directory.
59 -T path
61 Specify absolute path of copy tool (guessed by default). If you
62 force a specific path, the copy tool must be present at that path
63 on each worker. That path cannot be changed when resuming a run.
64 -f files
66 Transfer at most files files or directories per sync job. 0
67 means unlimited but you must at least specify one file or size
68 limit. Default: 2000
69 -s size
71 Transfer at most size bytes per sync job. 0 means unlimited but
72 you must at least specify one file or size limit. You can use a
73 human-friendly unit suffix here (k, m, g, t, p).
74 Default: 4294967296 (4 GB)
75 -E Work on a per-directory basis (rsync tool only). In that mode,
77 fpsync works with lists of directories instead of files. That
78 mode may generate coarse-grained lists but enables rsync(1) 's
79 --delete option by default ( WARNING!!! ), making it a good can‐
80 didate for a final (cleaning) pass after several incremental
81 passes using standard (file) mode. When option -E is specified
82 twice, it enables 'aggressive' mode which isolates erroneous di‐
83 rectories and enables recursive synchronization for them. This
84 advanced mode can be useful to try to overcome transcient errors
85 such as Linux SMB client deferring opendir() calls to support
86 compound SMB requests.
87 -o toolopts
89 Override default rsync(1), cpio(1) or tar(1) options with
90 toolopts. Use this option with care as certain options are in‐
91 compatible with a parallel usage (e.g. rsync's --delete). De‐
92 fault for rsync: “-lptgoD -v --numeric-ids”. Empty for cpio, tar
93 and tarify.
94 -O fpartopts
96 Override default fpart(1) options with fpartopts. Options and
97 values must be separated by a pipe character.
98 Default: “-x|.zfs|-x|.snapshot*|-x|.ckpt”.
99 -S Sudo mode. Use sudo(8) for filesystem crawling and synchroniza‐
101 tions.
102 src_dir/
104 Source directory. It must be absolute and available on all par‐
105 ticipating hosts (including the master one, running fpsync).
106 dst_url/
108 Destination directory or URL (rsync tool only). If a remote URL
109 is provided, it must be supported by rsync(1). All participating
110 workers must be able to reach that target.
111
113 -n jobs
114 Start jobs concurrent sync jobs (either locally or remotely, see
115 below) per run. Default: 2
116 -w wrks
118 Use remote SSH wrks to synchronize files. Synchronization jobs
119 are executed locally when this option is not set. wrks is a
120 space-separated list of login@machine connection strings and can
121 be specified several times. You must be allowed to connect to
122 those machines using a SSH key to avoid user interaction.
123
125 -p Prepare mode: prepare, test synchronization environment, start
126 fpart(1) and create partitions but do not actually start trans‐
127 fers. That mode can be used to create a run that can then be re‐
128 sumed using option -r.
129 -l List previous runs and their status.
131 -r runid
133 Resume run runid and restart synchronizing remaining partitions
134 from a previous run. runid is displayed when using verbose mode
135 (see option -v) or prepare mode (option -p) and can be retrieved
136 afterwards using option -l. Note that filesystem crawling is
137 skipped when resuming a previous run. As a consequence, options
138 -m, -f, -s, -E, -o, -O, -S, src_dir/, and dst_url/ are ignored.
139 -R Replay mode: when using option -r, force re-synchronizing run's
141 all partitions instead of remaining ones only. That mode can be
142 useful to skip filesystem crawling when you have to replay a fi‐
143 nal pass several times and you know directory structure has not
144 changed in the meantime (you may miss files if you use replay
145 mode with a standard, file-based, run).
146 -a runid
148 Archive run runid (including partition files, logs, queue and
149 work directories) to tmpdir. That option requires the tar(1)
150 client to be installed on the master host running fpsync.
151 -D runid
153 Delete run runid (including partition files, logs, queue and work
154 directories).
155
157 Each fpsync run generates a unique runid, which is displayed in verbose
158 mode (see option -v) and within log files. You can use that runid to re‐
159 sume a previous run (see option -r). fpsync will then restart synchro‐
160 nizing data from the parts that were being synchronized at the time it
161 stopped.
162 This unique feature gives the administrator the ability to stop fpsync
164 and restart it later, without having to restart the whole filesystem
165 crawling and synchronization process. Note that resuming is only possi‐
166 ble when filesystem crawling step has finished.
167 During synchronization, you can press CTRL-C to interrupt the process.
169 The first CTRL-C prevents new synchronizations from being submitted and
170 the process will wait for current synchronizations to be finished before
171 exiting. If you press CTRL-C again, current synchronizations will be
172 killed and fpsync will exit immediately. When using option -E to enable
173 directory mode and rsync's --delete option, keep in mind that killing
174 rsync processes may lead to a situation where certain files have been up‐
175 dated and others not deleted yet (because the deletion process is post‐
176 poned using rsync's --delete-after option).
177 On certain systems, CTRL-T can be pressed to get the status of current
179 and remaining parts to be synchronized. This can also be achieved by
180 sending a SIGINFO to the fpsync process.
181 Whether you use verbose mode or not, everything is logged within
183 shdir/log/.
184
186 Here are some examples:
187 fpsync -n 4 /usr/src/ /var/src/
189 Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
191 fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
193 /mnt/dst/
194 Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe‐
196 cuted remotely on 2 SSH workers (machine1 and machine2). The
197 shared directory is set to /mnt/fpsync and mounted on the machine
198 running fpsync, as well as on machine1 and machine2. The source
199 directory (/mnt/src/) is also available on those 3 machines,
200 while the destination directory (/mnt/dst/) is mounted on SSH
201 workers only (machine1 and machine2).
202
204 Parallelizing rsync(1) can make several options not usable, such as
205 --delete. If your source directory is live while fpsync is running, you
206 will have to delete extra files from destination directory. This is tra‐
207 ditionally done by using a final -offline- rsync(1) pass that will use
208 this option, but you can also use fpsync and option -E to perform the
209 same task using several workers.
210 fpsync enqueues synchronization jobs on disk, within the tmpdir/queue di‐
212 rectory. Be careful to host this queue on a filesystem that can handle
213 fine-grained mtime timestamps (i.e. with a sub-second precision) if you
214 want the queue to be processed in order when fpart(1) generates several
215 jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
216 using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
217 Contrary to rsync(1), fpsync enforces the final '/' on the source direc‐
219 tory. It means that directory contents are synchronized, not the source
220 directory itself (i.e. you will not get a subdirectory of the name of the
221 source directory in the target directory after synchronization).
222 Before starting filesystem crawling, fpsync changes its current working
224 directory to src_dir/ and generates partitions containing relative paths
225 (all starting with './'). This is important to keep in mind when modify‐
226 ing toolopts or fpartopts dealing with file or directory paths.
227
229 cpio(1), fpart(1), mail(1), rsync(1), tar(1), sudo(8)
230
232 Fpsync has been written by Ganaël LAPLANCHE and is available under the
233 BSD license on http://contribs.martymac.org
234
236 No bug known (yet).
237BSD January 27, 2015 BSD