1nbdcopy(1)                          LIBNBD                          nbdcopy(1)
2
3
4

NAME

6       nbdcopy - copy to and from an NBD server
7

SYNOPSIS

9        nbdcopy [--allocated] [-C N|--connections=N]
10                [--destination-is-zero|--target-is-zero] [--flush]
11                [--no-extents] [-p|--progress|--progress=FD]
12                [--request-size=N] [-R N|--requests=N] [-S N|--sparse=N]
13                [--synchronous] [-T N|--threads=N] [-v|--verbose]
14                SOURCE DESTINATION
15
16        SOURCE, DESTINATION := - | FILE | DEVICE | NBD-URI | [ CMD ARGS ... ]
17        DESTINATION += null:
18
19        nbdcopy --help
20
21        nbdcopy -V|--version
22

EXAMPLES

24   nbdcopy nbd://example.com local.img
25       This copies everything from the NBD server at "example.com" to a local
26       file called local.img.
27
28   nbdcopy nbd://example.com - | file -
29       This streams the first part of the disk on the NBD server at
30       "example.com" into the file(1) command.  Note here that "-" means to
31       stream to stdout (and therefore into the pipe to the file command).
32
33       nbdinfo(1) is another way to detect the content on an NBD server.
34
35   nbdcopy -p /dev/sdX "nbd+unix:///?socket=/tmp/unixsock"
36       Copy the full local hard disk "/dev/sdX" to the NBD server listening on
37       the Unix domain socket /tmp/unixsock.  Because of the -p option this
38       will print a progress bar.
39
40   nbdcopy nbd://server1 nbd://server2
41       Copy a full disk from one NBD server to another.
42
43   nbdcopy -- [ qemu-nbd -r -f qcow2 https://example.com/disk.qcow2 ] -
44       Run qemu-nbd(8) as a subprocess to open URL
45       "https://example.com/disk.qcow2", which is then streamed to stdout
46       ("-"), converted to blocks of raw format data.  Note "--" to prevent
47       qemu-nbd flags from being interpreted as nbdcopy flags.
48
49   cat disk1 disk2 | nbdcopy -- - [ qemu-nbd -f qcow2 output.qcow2 ]
50       Concatenate two raw-format disk images into the qcow2 file
51       output.qcow2.  The output file has to be precreated.
52
53   nbdcopy nbd://server null:
54       Read the contents of the NBD server as fast as possible and throw it
55       away (write it to a virtual null device).  This is useful for
56       benchmarking NBD servers and/or nbdcopy.
57

DESCRIPTION

59       nbdcopy copies to and from an NBD server.  It can upload a local file
60       to an NBD server, or download the contents of an NBD server to a local
61       file, device or stdin/stdout.  It can also copy between NBD servers.
62
63       The local file can be a file, a block device (eg. "/dev/cdrom"), or "-"
64       which means stream in from stdin or stream out to stdout.
65
66       The NBD server can be specified using an NBD URI (like
67       "nbd://localhost").  The NBD server can be local or remote, and
68       encryption can be used if libnbd was built with encryption support.
69       Alternately you can use square brackets around a qemu-nbd(8) or
70       nbdkit(1) command to run the NBD server as a subprocess of nbdcopy.
71
72       The destination may be the special word "null:" to throw away the
73       output.
74
75       For more complex copying operations including converting between disk
76       formats use "qemu-img convert", see qemu-img(1).
77

OPTIONS

79       --help
80           Display brief command line help and exit.
81
82       --allocated
83           Normally nbdcopy tries to create sparse output (with holes) if the
84           destination supports that.  It does this in two ways: either using
85           extent information from the source to copy holes (see
86           --no-extents), or by detecting runs of zeroes (see -S).  If you use
87           --allocated then nbdcopy creates a fully allocated, non-sparse
88           output on the destination.
89
90       -C N
91       --connections=N
92           Set the maximum number of NBD connections ("multi-conn").  By
93           default nbdcopy will try to use multi-conn with up to 4 connections
94           if the NBD server supports it.  If copying between NBD servers then
95           nbdcopy cannot use multi-conn if either of the servers does not
96           support it.
97
98       --destination-is-zero
99       --target-is-zero
100           Assume the destination is already zeroed.  This allows nbdcopy to
101           skip copying blocks of zeroes from the source to the destination.
102           This is not safe unless the destination device is already zeroed.
103           (--target-is-zero is provided for compatibility with qemu-img(1).)
104
105       --flush
106           Flush writes to ensure that everything is written to persistent
107           storage before nbdcopy exits.
108
109       --no-extents
110           Normally nbdcopy uses extent metadata to skip over parts of the
111           source disk which contain holes.  If you use this flag, nbdcopy
112           ignores extent information and reads everything, which is usually
113           slower.  You might use this flag in two situations: the source NBD
114           server has incorrect metadata information; or the source has very
115           slow extent querying so it's faster to simply read all of the data.
116
117       -p
118       --progress
119           Display a progress bar.
120
121       --progress=FD
122           Write a progress bar to the file descriptor "FD" (a number) in a
123           format which is easily parsable by other programs.  nbdcopy will
124           periodically write the string "N/100\n" (where N is an integer
125           between 0 and 100) to the file descriptor.
126
127           To get nbdcopy to write the progress bar to a file you can use the
128           following shell commands:
129
130            exec 3>/tmp/progress
131            nbdcopy --progress=3 ...
132            exec 3>&-
133
134       --request-size=N
135           Set the maximum request size in bytes. The maximum value is 32 MiB,
136           specified by the NBD protocol.
137
138       -R N
139       --requests=N
140           Set the maximum number of requests in flight per NBD connection.
141
142       -S N
143       --sparse=N
144           Detect all zero blocks of size N (bytes) and make them sparse on
145           the output.  You can also turn off sparse detection using -S 0.
146           The default is 4096 bytes.
147
148       --synchronous
149           Force synchronous copying using the libnbd(3) synchronous ("high
150           level") API.  This is slow but may be necessary for some broken NBD
151           servers which cannot handle multiple requests in flight.  This mode
152           is also used when streaming to and from stdio, pipes and sockets.
153
154       -T N
155       --threads=N
156           Use up to N threads for copying.  By default this is set to the
157           number of processor cores available.
158
159           Note --threads=0 means autodetect and --threads=1 means use a
160           single thread.
161
162       -v
163       --verbose
164           Verbose output.  This enables debug in libnbd (see
165           nbd_set_debug(3)) as well as printing other useful information.
166
167       -V
168       --version
169           Display the package name and version and exit.
170

MULTI-CONN, THREADS, REQUESTS IN FLIGHT

172       The three options --connections, --threads and --requests are related
173       and control the amount of parallelism available.  The defaults should
174       ensure a reasonable amount of parallelism if possible and you don’t
175       need to adjust them, but this section tries to describe what is going
176       on.
177
178       Firstly if either side of the copy is streaming to or from stdio, a
179       pipe, or a socket, or if you use the --synchronous option, then nbdcopy
180       works in synchronous mode with no parallelism, and nothing else in this
181       section applies.
182
183       The --connections=N option controls NBD multi-conn (see "Multi-conn" in
184       libnbd(3)), opening up to N connections to the NBD server (or to both
185       NBD servers if copying between NBD servers).  This defaults to 4.  The
186       NBD servers must support and advertise multi-conn.  For nbdkit(1)
187       availability of multi-conn can depend on the plugin.  You can use
188       nbdinfo(1) to find out if a particular NBD server is advertising multi-
189       conn.  If the NBD server doesn’t advertise multi-conn then only one
190       connection will be opened regardless of the --connections flag.
191
192       When copying between two NBD servers, the number of connections is
193       limited to the minimum multi-conn supported on both sides.  For the
194       purposes of this calculation, you can consider local files and block
195       devices as supporting infinite multi-conn.
196
197       When you run an NBD server as a subprocess (using the "[ ... ]" syntax)
198       multi-conn cannot be used.
199
200       The --threads=N option allows nbdcopy to start up to N threads
201       (defaulting to the number of cores).  However nbdcopy cannot use more
202       threads than the number of NBD connections.
203
204       The --requests=N option controls the maximum number of requests in
205       flight on each NBD connection.  This enables the NBD server to process
206       requests in parallel even when multi-conn isn’t available or when using
207       a single thread.  The default is chosen to allow a reasonable amount of
208       parallelism without using too much memory.
209
210       Because of this parallelism, nbdcopy does not read or write blocks in
211       order.  If for some reason you require that blocks are copied in strict
212       order then you must use --synchronous.
213

RUNNING NBD SERVER AS A SUBPROCESS

215       Instead of connecting to an already running server using an NBD URI,
216       you can run an NBD server as a subprocess using:
217
218        nbdcopy -- [ CMD ARGS ... ] ...
219
220       This requires the server to support systemd socket activation, which
221       both qemu-nbd(8) and nbdkit(1) support (see also
222       nbd_connect_systemd_socket_activation(3)).
223
224       "[" and "]" must be separate command line parameters.  You will usually
225       need to use "--" to stop nbdcopy from misinterpreting NBD server flags
226       as nbdcopy flags.  Both the source and destination may be subprocesses.
227       nbdcopy cleans up the subprocess on exit.
228
229       Some examples follow.
230
231   nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] - | hexdump -C
232       In this example, qemu-nbd(8) is run as a subprocess.  The subprocess
233       opens disk.qcow2 and exposes it as NBD to nbdcopy.  nbdcopy streams
234       this to stdout ("-") into the pipe which is read by hexdump(1).
235
236   nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] [ nbdkit memory 1G ]
237       Two subprocesses are created, qemu-nbd(8) as the source and nbdkit(1)
238       as the destination.  The qcow2 file is converted to raw and stored
239       temporarily in the RAM disk (nbdkit-memory-plugin(1)).
240
241       When nbdcopy exits both servers are killed and the RAM disk goes away,
242       so this command has no overall effect, but is useful for testing.
243

COMPARING NBD SERVER CONTENT

245       You can use nbdcopy, cmp(1) and bash(1) process substitution to compare
246       the content of two NBD servers for equality:
247
248        cmp <( nbdcopy nbd://server1 - ) <( nbdcopy nbd://server2 - )
249
250       Note this tests that the content is logically equal.  It does not
251       compare the NBD metadata such as sparseness (see nbdinfo(1) --map
252       option).  Thus for example a run of allocated zeroes in one server will
253       match a hole in the other.
254

SEE ALSO

256       libnbd(3), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdkit(1), qemu-img(1).
257

AUTHORS

259       Richard W.M. Jones
260
262       Copyright (C) 2020-2021 Red Hat Inc.
263

LICENSE

265       This library is free software; you can redistribute it and/or modify it
266       under the terms of the GNU Lesser General Public License as published
267       by the Free Software Foundation; either version 2 of the License, or
268       (at your option) any later version.
269
270       This library is distributed in the hope that it will be useful, but
271       WITHOUT ANY WARRANTY; without even the implied warranty of
272       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
273       Lesser General Public License for more details.
274
275       You should have received a copy of the GNU Lesser General Public
276       License along with this library; if not, write to the Free Software
277       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
278       02110-1301 USA
279
280
281
282libnbd-1.7.12                     2021-05-29                        nbdcopy(1)
Impressum