1FPSYNC(1) BSD General Commands Manual FPSYNC(1)
2
4 fpsync — Synchronize directories in parallel using fpart and an external
5 tool
6
8 fpsync [-p] [-n jobs] [-w wrks] [-m tool] [-f files] [-s size] [-E]
9 [-o toolopts] [-O fpartopts] [-S] [-t tmpdir] [-d shdir]
10 [-M mailaddr] [-v] src_dir/ dst_url/
11 fpsync -l
12 fpsync -r runid [-R] [OPTIONS...]
13 fpsync -a runid
14 fpsync -D runid
15
17 The fpsync tool synchronizes directories in parallel using fpart(1) and
18 rsync(1), cpio(1) or tar(1). It computes subsets of src_dir/ and spawns
19 jobs to synchronize them to dst_url/.
20 Synchronization jobs can be executed either locally or remotely (using
22 SSH workers, see option -w) and are executed on-the-fly while filesystem
23 crawling goes on. This makes fpsync a good tool for migrating large
24 filesystems.
25
27 -t tmpdir
28 Set fpsync temporary directory to tmpdir. This directory remains
29 local and does not need to be shared amongst SSH workers when us‐
30 ing the -w option. Default: /tmp/fpsync
31 -d shdir
33 Set fpsync shared directory to shdir. This option is mandatory
34 when using SSH workers and set by default to tmpdir when running
35 locally. The specified directory must be an absolute path ; it
36 will be used to handle communications with SSH hosts (sharing
37 partitions and log files) and, as a consequence, must be made
38 available to all participating hosts (e.g. through a r/w NFS
39 mount), including the master one running fpsync.
40 -M mailaddr
42 Send an e-mail to mailaddr after a run. Multiple -space-sepa‐
43 rated- addresses can be specified. That option requires the
44 mail(1) client to be installed and configured on the master host
45 running fpsync.
46 -v Verbose mode. Can be be specified several times to increase ver‐
48 bosity level.
49 -h Print help
51
53 -m tool
54 External copy tool used to synchronize files. Currently sup‐
55 ported tools are: rsync, cpio, tar, and tarify. Default: rsync.
56 When using cpio or tar and more than one worker, directory time‐
57 stamps may not be replicated. A second pass will fix them. Spe‐
58 cial tool tarify generates tarballs into destination directory.
59 -f files
61 Transfer at most files files or directories per sync job. 0
62 means unlimited but you must at least specify one file or size
63 limit. Default: 2000
64 -s size
66 Transfer at most size bytes per sync job. 0 means unlimited but
67 you must at least specify one file or size limit. You can use a
68 human-friendly unit suffix here (k, m, g, t, p).
69 Default: 4294967296 (4 GB)
70 -E Work on a per-directory basis (rsync tool only). In that mode,
72 fpsync works with lists of directories instead of files. That
73 mode may generate coarse-grained lists but enables rsync(1) 's
74 --delete option by default ( WARNING!!! ), making it a good can‐
75 didate for a final (cleaning) pass after several incremental
76 passes using standard (file) mode.
77 -o toolopts
79 Override default rsync(1), cpio(1) or tar(1) options with
80 toolopts. Use this option with care as certain options are in‐
81 compatible with a parallel usage (e.g. rsync's --delete). De‐
82 fault for rsync: “-lptgoD -v --numeric-ids”. Empty for cpio, tar
83 and tarify.
84 -O fpartopts
86 Override default fpart(1) options with fpartopts. Options and
87 values must be separated by a pipe character.
88 Default: “-x|.zfs|-x|.snapshot*|-x|.ckpt”.
89 -S Sudo mode. Use sudo(8) for filesystem crawling and synchroniza‐
91 tions.
92 src_dir/
94 Source directory. It must be absolute and available on all par‐
95 ticipating hosts (including the master one, running fpsync).
96 dst_url/
98 Destination directory or URL (rsync tool only). If a remote URL
99 is provided, it must be supported by rsync(1). All participating
100 workers must be able to reach that target.
101
103 -n jobs
104 Start jobs concurrent sync jobs (either locally or remotely, see
105 below) per run. Default: 2
106 -w wrks
108 Use remote SSH wrks to synchronize files. Synchronization jobs
109 are executed locally when this option is not set. wrks is a
110 space-separated list of login@machine connection strings and can
111 be specified several times. You must be allowed to connect to
112 those machines using a SSH key to avoid user interaction.
113
115 -p Prepare mode: prepare, test synchronization environment, start
116 fpart(1) and create partitions but do not actually start trans‐
117 fers. That mode can be used to create a run that can then be re‐
118 sumed using option -r.
119 -l List previous runs and their status.
121 -r runid
123 Resume run runid and restart synchronizing remaining partitions
124 from a previous run. runid is displayed when using verbose mode
125 (see option -v) or prepare mode (option -p) and can be retrieved
126 afterwards using option -l. Note that filesystem crawling is
127 skipped when resuming a previous run. As a consequence, options
128 -m, -f, -s, -E, -o, -O, -S, src_dir/, and dst_url/ are ignored.
129 -R Replay mode: when using option -r, force re-synchronizing run's
131 all partitions instead of remaining ones only. That mode can be
132 useful to skip filesystem crawling when you have to replay a fi‐
133 nal pass several times and you know directory structure has not
134 changed in the meantime (you may miss files if you use replay
135 mode with a standard, file-based, run).
136 -a runid
138 Archive run runid (including partition files, logs, queue and
139 work directories) to tmpdir. That option requires the tar(1)
140 client to be installed on the master host running fpsync.
141 -D runid
143 Delete run runid (including partition files, logs, queue and work
144 directories).
145
147 Each fpsync run generates a unique runid, which is displayed in verbose
148 mode (see option -v) and within log files. You can use that runid to re‐
149 sume a previous run (see option -r). fpsync will then restart synchro‐
150 nizing data from the parts that were being synchronized at the time it
151 stopped.
152 This unique feature gives the administrator the ability to stop fpsync
154 and restart it later, without having to restart the whole filesystem
155 crawling and synchronization process. Note that resuming is only possi‐
156 ble when filesystem crawling step has finished.
157 During synchronization, you can press CTRL-C to interrupt the process.
159 The first CTRL-C prevents new synchronizations from being submitted and
160 the process will wait for current synchronizations to be finished before
161 exiting. If you press CTRL-C again, current synchronizations will be
162 killed and fpsync will exit immediately. When using option -E to enable
163 directory mode and rsync's --delete option, keep in mind that killing
164 rsync processes may lead to a situation where certain files have been up‐
165 dated and others not deleted yet (because the deletion process is post‐
166 poned using rsync's --delete-after option).
167 On certain systems, CTRL-T can be pressed to get the status of current
169 and remaining parts to be synchronized. This can also be achieved by
170 sending a SIGINFO to the fpsync process.
171 Whether you use verbose mode or not, everything is logged within
173 shdir/log/.
174
176 Here are some examples:
177 fpsync -n 4 /usr/src/ /var/src/
179 Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
181 fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
183 /mnt/dst/
184 Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe‐
186 cuted remotely on 2 SSH workers (machine1 and machine2). The
187 shared directory is set to /mnt/fpsync and mounted on the machine
188 running fpsync, as well as on machine1 and machine2. The source
189 directory (/mnt/src/) is also available on those 3 machines,
190 while the destination directory (/mnt/dst/) is mounted on SSH
191 workers only (machine1 and machine2).
192
194 Parallelizing rsync(1) can make several options not usable, such as
195 --delete. If your source directory is live while fpsync is running, you
196 will have to delete extra files from destination directory. This is tra‐
197 ditionally done by using a final -offline- rsync(1) pass that will use
198 this option, but you can also use fpsync and option -E to perform the
199 same task using several workers.
200 fpsync enqueues synchronization jobs on disk, within the tmpdir/queue di‐
202 rectory. Be careful to host this queue on a filesystem that can handle
203 fine-grained mtime timestamps (i.e. with a sub-second precision) if you
204 want the queue to be processed in order when fpart(1) generates several
205 jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
206 using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
207 Contrary to rsync(1), fpsync enforces the final '/' on the source direc‐
209 tory. It means that directory contents are synchronized, not the source
210 directory itself (i.e. you will not get a subdirectory of the name of the
211 source directory in the target directory after synchronization).
212 Before starting filesystem crawling, fpsync changes its current working
214 directory to src_dir/ and generates partitions containing relative paths
215 (all starting with './'). This is important to keep in mind when modify‐
216 ing toolopts or fpartopts dealing with file or directory paths.
217
219 cpio(1), fpart(1), mail(1), rsync(1), tar(1), sudo(8)
220
222 Fpsync has been written by Ganaël LAPLANCHE and is available under the
223 BSD license on http://contribs.martymac.org
224
226 No bug known (yet).
227BSD January 27, 2015 BSD