1FPSYNC(1) BSD General Commands Manual FPSYNC(1)
2
4 fpsync — Synchronize directories in parallel using fpart + rsync or cpio
5
7 fpsync [-h] [-v] [-m tool] [-n jobs] [-f files] [-s size] [-E] [-w wrks]
8 [-d shdir] [-t tmpdir] [-r jobname] [-o toolopts] [-O fpartopts]
9 [-S] src_dir/ dst_url/
10
12 The fpsync tool synchronizes directories in parallel using fpart(1) and
13 rsync(1) or cpio(1). It computes subsets of src_dir/ and spawns jobs to
14 synchronize them to dst_url/.
15 Synchronization jobs can be executed either locally or remotely (using
17 SSH workers, see option -w) and are executed on-the-fly while filesystem
18 crawling goes on. This makes fpsync a good tool for migrating large
19 filesystems.
20
22 -h Print help
23 -v Verbose mode. Can be be specified several times to increase ver‐
25 bosity level.
26 -m tool
28 External copy tool used to synchronize files. Currently sup‐
29 ported tools are: rsync and cpio. Default: rsync. When using
30 cpio and more than one worker, directory timestamps may not be
31 replicated. A second pass will fix them. See the README file
32 for more details.
33 -n jobs
35 Start jobs concurrent sync jobs (either locally or remotely, see
36 below). Default: 2
37 -f files
39 Transfer at most files files or directories per sync job. 0
40 means unlimited but you must at least specify one file or size
41 limit. Default: 2000
42 -s size
44 Transfer at most size bytes per sync job. 0 means unlimited but
45 you must at least specify one file or size limit. You can use a
46 human-friendly unit suffix here (k, m, g, t, p).
47 Default: 4294967296 (4 GB)
48 -E Work on a per-directory basis (rsync tool only). In that mode,
50 fpsync works with lists of directories instead of files. That
51 mode may generate coarse-grained lists but enables rsync(1) 's
52 --delete option by default ( WARNING!!! ), making it a good can‐
53 didate for a final (cleaning) pass after several incremental
54 passes using standard (file) mode.
55 -w wrks
57 Use remote SSH wrks to synchronize files. Synchronization jobs
58 are executed locally when this option is not set. wrks is a
59 space-separated list of login@machine connection strings and can
60 be specified several times. You must be allowed to connect to
61 those machines using a SSH key to avoid user interaction.
62 -d shdir
64 Set fpsync shared directory to shdir. This option is mandatory
65 when using SSH workers and set by default to tmpdir when running
66 locally. The specified directory must be an absolute path ; it
67 will be used to handle communications with SSH hosts (sharing
68 partitions and log files) and, as a consequence, must be made
69 available to all participating hosts (e.g. through a r/w NFS
70 mount), including the master one running fpsync.
71 -t tmpdir
73 Set fpsync temporary directory to tmpdir. This directory remains
74 local and does not need to be shared amongst SSH workers when
75 using the -w option. Default: /tmp/fpsync
76 -r jobname
78 Resume job jobname and restart synchronizing remaining partitions
79 from a previous run. jobname can be obtained using verbose mode
80 (see option -v). Note that filesystem crawling is skipped when
81 resuming a previous run. As a consequence, options -f, -s, -o,
82 -O, -S, src_dir/, and dst_url/ are ignored.
83 -o toolopts
85 Override default rsync(1) or cpio(1) options with toolopts. Use
86 this option with care as certain options are incompatible with a
87 parallel usage (e.g. rsync's --delete). Default for rsync:
88 -lptgoD -v --numeric-ids. Empty for cpio.
89 -O fpartopts
91 Override default fpart(1) options with fpartopts.
92 Default: -x .zfs -x .snapshot* -x .ckpt
93 -S Sudo mode. Use sudo(8) for filesystem crawling and synchroniza‐
95 tions.
96 src_dir/
98 Source directory. It must be absolute and available on all par‐
99 ticipating hosts (including the master one, running fpsync).
100 dst_url/
102 Destination directory or URL (rsync tool only). If a remote URL
103 is provided, it must be supported by rsync(1). All participating
104 workers must be able to reach that target.
105
107 Each fpsync run generates a unique jobname, which is displayed in verbose
108 mode (see option -v) and within log files. You can use that jobname to
109 resume a previous run (see option -r). fpsync will then restart synchro‐
110 nizing data from the parts that were being synchronized at the time it
111 stopped.
112 This unique feature gives the administrator the ability to stop fpsync
114 and restart it later, without having to restart the whole filesystem
115 crawling and synchronization process. Note that resuming is only possi‐
116 ble when filesystem crawling step has finished.
117 During synchronization, you can press CTRL-C to interrupt the process.
119 The first CTRL-C prevents new synchronizations from being submitted and
120 the process will wait for current synchronizations to be finished before
121 exiting. If you press CTRL-C again, current synchronizations will be
122 killed and fpsync will exit immediately. When using option -E to enable
123 directory mode and rsync's --delete option, keep in mind that killing
124 rsync processes may lead to a situation where certain files have been
125 updated and others not deleted yet (because the deletion process is post‐
126 poned using rsync's --delete-after option).
127 On certain systems, CTRL-T can be pressed to get the status of current
129 and remaining parts to be synchronized. This can also be achieved by
130 sending a SIGINFO to the fpsync process.
131 Whether you use verbose mode or not, everything is logged within
133 shdir/log/.
134
136 Here are some examples:
137 fpsync -n 4 /usr/src/ /var/src/
139 Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
141 fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
143 /mnt/dst/
144 Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe‐
146 cuted remotely on 2 SSH workers (machine1 and machine2). The
147 shared directory is set to /mnt/fpsync and mounted on the machine
148 running fpsync, as well as on machine1 and machine2. The source
149 directory (/mnt/src/) is also available on those 3 machines,
150 while the destination directory (/mnt/dst/) is mounted on SSH
151 workers only (machine1 and machine2).
152
154 Parallelizing rsync(1) or cpio(1) makes several options not usable, such
155 as rsync's --delete. If your source directory is live while fpsync is
156 running, you will have to delete extra files from destination directory.
157 This is traditionally done by using a final -offline- rsync(1) pass that
158 will use this option, but you can also use fpsync and option -E to per‐
159 form the same task using several workers.
160 fpsync enqueues synchronization jobs on disk, within the tmpdir/queue
162 directory. Be careful to host this queue on a filesystem that can handle
163 fine-grained mtime timestamps (i.e. with a sub-second precision) if you
164 want the queue to be processed in order when fpart(1) generates several
165 jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
166 using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
167 Contrary to rsync(1), fpsync enforces the final '/' on the source direc‐
169 tory. It means that directory contents are synchronized, not the source
170 directory itself (i.e. you will not get a subdirectory of the name of the
171 source directory in the target directory after synchronization).
172
174 cpio(1), fpart(1), rsync(1), sudo(8)
175
177 Fpsync has been written by Ganaël LAPLANCHE and is available under the
178 BSD license on
179 http://contribs.martymac.org
180
182 No bug known (yet).
183BSD January 27, 2015 BSD