1nbdcopy(1)                          LIBNBD                          nbdcopy(1)
2
3
4

NAME

6       nbdcopy - copy to and from an NBD server
7

SYNOPSIS

9        nbdcopy [--allocated] [-C N|--connections=N]
10                [--destination-is-zero|--target-is-zero] [--flush]
11                [--no-extents] [-p|--progress|--progress=FD]
12                [--queue-size=N] [--request-size=N] [-R N|--requests=N]
13                [-S N|--sparse=N] [--synchronous] [-T N|--threads=N]
14                [-v|--verbose]
15                SOURCE DESTINATION
16
17        SOURCE, DESTINATION := - | FILE | DEVICE | NBD-URI | [ CMD ARGS ... ]
18        DESTINATION += null:
19
20        nbdcopy --help
21
22        nbdcopy -V|--version
23

EXAMPLES

25   nbdcopy nbd://example.com local.img
26       This copies everything from the NBD server at "example.com" to a local
27       file called local.img.
28
29   nbdcopy nbd://example.com - | file -
30       This streams the first part of the disk on the NBD server at
31       "example.com" into the file(1) command.  Note here that "-" means to
32       stream to stdout (and therefore into the pipe to the file command).
33
34       nbdinfo(1) is another way to detect the content on an NBD server.
35
36   nbdcopy -p /dev/sdX "nbd+unix:///?socket=/tmp/unixsock"
37       Copy the full local hard disk "/dev/sdX" to the NBD server listening on
38       the Unix domain socket /tmp/unixsock.  Because of the -p option this
39       will print a progress bar.
40
41   nbdcopy nbd://server1 nbd://server2
42       Copy a full disk from one NBD server to another.
43
44   nbdcopy -- [ qemu-nbd -r -f qcow2 https://example.com/disk.qcow2 ] -
45       Run qemu-nbd(8) as a subprocess to open URL
46       "https://example.com/disk.qcow2", which is then streamed to stdout
47       ("-"), converted to blocks of raw format data.  Note "--" to prevent
48       qemu-nbd flags from being interpreted as nbdcopy flags.
49
50   cat disk1 disk2 | nbdcopy -- - [ qemu-nbd -f qcow2 output.qcow2 ]
51       Concatenate two raw-format disk images into the qcow2 file
52       output.qcow2.  The output file has to be precreated.
53
54   nbdcopy nbd://server null:
55       Read the contents of the NBD server as fast as possible and throw it
56       away (write it to a virtual null device).  This is useful for
57       benchmarking NBD servers and/or nbdcopy.
58

DESCRIPTION

60       nbdcopy copies to and from an NBD server.  It can upload a local file
61       to an NBD server, or download the contents of an NBD server to a local
62       file, device or stdin/stdout.  It can also copy between NBD servers.
63
64       The local file can be a file, a block device (eg. "/dev/cdrom"), or "-"
65       which means stream in from stdin or stream out to stdout.
66
67       The NBD server can be specified using an NBD URI (like
68       "nbd://localhost").  The NBD server can be local or remote, and
69       encryption can be used if libnbd was built with encryption support.
70       Alternately you can use square brackets around a qemu-nbd(8) or
71       nbdkit(1) command to run the NBD server as a subprocess of nbdcopy.
72
73       The destination may be the special word "null:" to throw away the
74       output.
75
76       For more complex copying operations including converting between disk
77       formats use "qemu-img convert", see qemu-img(1).
78

OPTIONS

80       --help
81           Display brief command line help and exit.
82
83       --allocated
84           Normally nbdcopy tries to create sparse output (with holes) if the
85           destination supports that.  It does this in two ways: either using
86           extent information from the source to copy holes (see
87           --no-extents), or by detecting runs of zeroes (see -S).  If you use
88           --allocated then nbdcopy creates a fully allocated, non-sparse
89           output on the destination.
90
91       -C N
92       --connections=N
93           Set the maximum number of NBD connections ("multi-conn").  By
94           default nbdcopy will try to use multi-conn with up to 4 connections
95           if the NBD server supports it.  If copying between NBD servers then
96           nbdcopy cannot use multi-conn if either of the servers does not
97           support it.
98
99       --destination-is-zero
100       --target-is-zero
101           Assume the destination is already zeroed.  This allows nbdcopy to
102           skip copying blocks of zeroes from the source to the destination.
103           This is not safe unless the destination device is already zeroed.
104           (--target-is-zero is provided for compatibility with qemu-img(1).)
105
106       --flush
107           Flush writes to ensure that everything is written to persistent
108           storage before nbdcopy exits.
109
110       --no-extents
111           Normally nbdcopy uses extent metadata to skip over parts of the
112           source disk which contain holes.  If you use this flag, nbdcopy
113           ignores extent information and reads everything, which is usually
114           slower.  You might use this flag in two situations: the source NBD
115           server has incorrect metadata information; or the source has very
116           slow extent querying so it's faster to simply read all of the data.
117
118       -p
119       --progress
120           Display a progress bar.
121
122       --progress=FD
123           Write a progress bar to the file descriptor "FD" (a number) in a
124           format which is easily parsable by other programs.  nbdcopy will
125           periodically write the string "N/100\n" (where N is an integer
126           between 0 and 100) to the file descriptor.
127
128           To get nbdcopy to write the progress bar to a file you can use the
129           following shell commands:
130
131            exec 3>/tmp/progress
132            nbdcopy --progress=3 ...
133            exec 3>&-
134
135       --queue-size=N
136           Set the maximum number of bytes to queue for in flight requests.
137           The default value is 16 MiB, allowing up to 64 256k requests per
138           NBD connection. If you use larger --request-size you may want to
139           increase this value.
140
141       --request-size=N
142           Set the maximum request size in bytes. The maximum value is 32 MiB,
143           specified by the NBD protocol.
144
145       -R N
146       --requests=N
147           Set the maximum number of requests in flight per NBD connection.
148
149       -S N
150       --sparse=N
151           Detect all zero blocks of size N (bytes) and make them sparse on
152           the output.  You can also turn off sparse detection using -S 0.
153           The default is 4096 bytes, or the destination preferred block size,
154           whichever is larger.
155
156       --synchronous
157           Force synchronous copying using the libnbd(3) synchronous ("high
158           level") API.  This is slow but may be necessary for some broken NBD
159           servers which cannot handle multiple requests in flight.  This mode
160           is also used when streaming to and from stdio, pipes and sockets.
161
162       -T N
163       --threads=N
164           Use up to N threads for copying.  By default this is set to the
165           number of processor cores available.
166
167           Note --threads=0 means autodetect and --threads=1 means use a
168           single thread.
169
170       -v
171       --verbose
172           Verbose output.  This enables debug in libnbd (see
173           nbd_set_debug(3)) as well as printing other useful information.
174
175       -V
176       --version
177           Display the package name and version and exit.
178

MULTI-CONN, THREADS, REQUESTS IN FLIGHT

180       The three options --connections, --threads and --requests are related
181       and control the amount of parallelism available.  The defaults should
182       ensure a reasonable amount of parallelism if possible and you don’t
183       need to adjust them, but this section tries to describe what is going
184       on.
185
186       Firstly if either side of the copy is streaming to or from stdio, a
187       pipe, or a socket, or if you use the --synchronous option, then nbdcopy
188       works in synchronous mode with no parallelism, and nothing else in this
189       section applies.
190
191       The --connections=N option controls NBD multi-conn (see "Multi-conn" in
192       libnbd(3)), opening up to N connections to the NBD server (or to both
193       NBD servers if copying between NBD servers).  This defaults to 4.  The
194       NBD servers must support and advertise multi-conn.  For nbdkit(1)
195       availability of multi-conn can depend on the plugin.  You can use
196       nbdinfo(1) to find out if a particular NBD server is advertising multi-
197       conn.  If the NBD server doesn’t advertise multi-conn then only one
198       connection will be opened regardless of the --connections flag.
199
200       When copying between two NBD servers, the number of connections is
201       limited to the minimum multi-conn supported on both sides.  For the
202       purposes of this calculation, you can consider local files and block
203       devices as supporting infinite multi-conn.
204
205       When you run an NBD server as a subprocess (using the "[ ... ]" syntax)
206       multi-conn cannot be used.
207
208       The --threads=N option allows nbdcopy to start up to N threads
209       (defaulting to the number of cores).  However nbdcopy cannot use more
210       threads than the number of NBD connections.
211
212       The --requests=N option controls the maximum number of requests in
213       flight on each NBD connection.  This enables the NBD server to process
214       requests in parallel even when multi-conn isn’t available or when using
215       a single thread.  The default is chosen to allow a reasonable amount of
216       parallelism without using too much memory.
217
218       Because of this parallelism, nbdcopy does not read or write blocks in
219       order.  If for some reason you require that blocks are copied in strict
220       order then you must use --synchronous.
221

RUNNING NBD SERVER AS A SUBPROCESS

223       Instead of connecting to an already running server using an NBD URI,
224       you can run an NBD server as a subprocess using:
225
226        nbdcopy -- [ CMD ARGS ... ] ...
227
228       This requires the server to support systemd socket activation, which
229       both qemu-nbd(8) and nbdkit(1) support (see also
230       nbd_connect_systemd_socket_activation(3)).
231
232       "[" and "]" must be separate command line parameters.  You will usually
233       need to use "--" to stop nbdcopy from misinterpreting NBD server flags
234       as nbdcopy flags.  Both the source and destination may be subprocesses.
235       nbdcopy cleans up the subprocess on exit.
236
237       Some examples follow.
238
239   nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] - | hexdump -C
240       In this example, qemu-nbd(8) is run as a subprocess.  The subprocess
241       opens disk.qcow2 and exposes it as NBD to nbdcopy.  nbdcopy streams
242       this to stdout ("-") into the pipe which is read by hexdump(1).  (See
243       also nbddump(1))
244
245   nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] [ nbdkit memory 1G ]
246       Two subprocesses are created, qemu-nbd(8) as the source and nbdkit(1)
247       as the destination.  The qcow2 file is converted to raw and stored
248       temporarily in the RAM disk (nbdkit-memory-plugin(1)).
249
250       When nbdcopy exits both servers are killed and the RAM disk goes away,
251       so this command has no overall effect, but is useful for testing.
252

SEE ALSO

254       libnbd(3), nbddump(1), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdublk(1),
255       nbdkit(1), qemu-img(1).
256

AUTHORS

258       Richard W.M. Jones
259
261       Copyright Red Hat
262

LICENSE

264       This library is free software; you can redistribute it and/or modify it
265       under the terms of the GNU Lesser General Public License as published
266       by the Free Software Foundation; either version 2 of the License, or
267       (at your option) any later version.
268
269       This library is distributed in the hope that it will be useful, but
270       WITHOUT ANY WARRANTY; without even the implied warranty of
271       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
272       Lesser General Public License for more details.
273
274       You should have received a copy of the GNU Lesser General Public
275       License along with this library; if not, write to the Free Software
276       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
277       02110-1301 USA
278
279
280
281libnbd-1.18.1                     2023-10-31                        nbdcopy(1)
Impressum