1nbdcopy(1) LIBNBD nbdcopy(1)
2
6 nbdcopy - copy to and from an NBD server
7
9 nbdcopy [--allocated] [-C N|--connections=N]
10 [--destination-is-zero|--target-is-zero] [--flush]
11 [--no-extents] [-p|--progress|--progress=FD]
12 [--queue-size=N] [--request-size=N] [-R N|--requests=N]
13 [-S N|--sparse=N] [--synchronous] [-T N|--threads=N]
14 [-v|--verbose]
15 SOURCE DESTINATION
16 SOURCE, DESTINATION := - | FILE | DEVICE | NBD-URI | [ CMD ARGS ... ]
18 DESTINATION += null:
19 nbdcopy --help
21 nbdcopy -V|--version
23
25 nbdcopy nbd://example.com local.img
26 This copies everything from the NBD server at "example.com" to a local
27 file called local.img.
28 nbdcopy nbd://example.com - | file -
30 This streams the first part of the disk on the NBD server at
31 "example.com" into the file(1) command. Note here that "-" means to
32 stream to stdout (and therefore into the pipe to the file command).
33 nbdinfo(1) is another way to detect the content on an NBD server.
35 nbdcopy -p /dev/sdX "nbd+unix:///?socket=/tmp/unixsock"
37 Copy the full local hard disk "/dev/sdX" to the NBD server listening on
38 the Unix domain socket /tmp/unixsock. Because of the -p option this
39 will print a progress bar.
40 nbdcopy nbd://server1 nbd://server2
42 Copy a full disk from one NBD server to another.
43 nbdcopy -- [ qemu-nbd -r -f qcow2 https://example.com/disk.qcow2 ] -
45 Run qemu-nbd(8) as a subprocess to open URL
46 "https://example.com/disk.qcow2", which is then streamed to stdout
47 ("-"), converted to blocks of raw format data. Note "--" to prevent
48 qemu-nbd flags from being interpreted as nbdcopy flags.
49 cat disk1 disk2 | nbdcopy -- - [ qemu-nbd -f qcow2 output.qcow2 ]
51 Concatenate two raw-format disk images into the qcow2 file
52 output.qcow2. The output file has to be precreated.
53 nbdcopy nbd://server null:
55 Read the contents of the NBD server as fast as possible and throw it
56 away (write it to a virtual null device). This is useful for
57 benchmarking NBD servers and/or nbdcopy.
58
60 nbdcopy copies to and from an NBD server. It can upload a local file
61 to an NBD server, or download the contents of an NBD server to a local
62 file, device or stdin/stdout. It can also copy between NBD servers.
63 The local file can be a file, a block device (eg. "/dev/cdrom"), or "-"
65 which means stream in from stdin or stream out to stdout.
66 The NBD server can be specified using an NBD URI (like
68 "nbd://localhost"). The NBD server can be local or remote, and
69 encryption can be used if libnbd was built with encryption support.
70 Alternately you can use square brackets around a qemu-nbd(8) or
71 nbdkit(1) command to run the NBD server as a subprocess of nbdcopy.
72 The destination may be the special word "null:" to throw away the
74 output.
75 For more complex copying operations including converting between disk
77 formats use "qemu-img convert", see qemu-img(1).
78
80 --help
81 Display brief command line help and exit.
82 --allocated
84 Normally nbdcopy tries to create sparse output (with holes) if the
85 destination supports that. It does this in two ways: either using
86 extent information from the source to copy holes (see
87 --no-extents), or by detecting runs of zeroes (see -S). If you use
88 --allocated then nbdcopy creates a fully allocated, non-sparse
89 output on the destination.
90 -C N
92 --connections=N
93 Set the maximum number of NBD connections ("multi-conn"). By
94 default nbdcopy will try to use multi-conn with up to 4 connections
95 if the NBD server supports it. If copying between NBD servers then
96 nbdcopy cannot use multi-conn if either of the servers does not
97 support it.
98 --destination-is-zero
100 --target-is-zero
101 Assume the destination is already zeroed. This allows nbdcopy to
102 skip copying blocks of zeroes from the source to the destination.
103 This is not safe unless the destination device is already zeroed.
104 (--target-is-zero is provided for compatibility with qemu-img(1).)
105 --flush
107 Flush writes to ensure that everything is written to persistent
108 storage before nbdcopy exits.
109 --no-extents
111 Normally nbdcopy uses extent metadata to skip over parts of the
112 source disk which contain holes. If you use this flag, nbdcopy
113 ignores extent information and reads everything, which is usually
114 slower. You might use this flag in two situations: the source NBD
115 server has incorrect metadata information; or the source has very
116 slow extent querying so it's faster to simply read all of the data.
117 -p
119 --progress
120 Display a progress bar.
121 --progress=FD
123 Write a progress bar to the file descriptor "FD" (a number) in a
124 format which is easily parsable by other programs. nbdcopy will
125 periodically write the string "N/100\n" (where N is an integer
126 between 0 and 100) to the file descriptor.
127 To get nbdcopy to write the progress bar to a file you can use the
129 following shell commands:
130 exec 3>/tmp/progress
132 nbdcopy --progress=3 ...
133 exec 3>&-
134 --queue-size=N
136 Set the maximum number of bytes to queue for in flight requests.
137 The default value is 16 MiB, allowing up to 64 256k requests per
138 NBD connection. If you use larger --request-size you may want to
139 increase this value.
140 --request-size=N
142 Set the maximum request size in bytes. The maximum value is 32 MiB,
143 specified by the NBD protocol.
144 -R N
146 --requests=N
147 Set the maximum number of requests in flight per NBD connection.
148 -S N
150 --sparse=N
151 Detect all zero blocks of size N (bytes) and make them sparse on
152 the output. You can also turn off sparse detection using -S 0.
153 The default is 4096 bytes.
154 --synchronous
156 Force synchronous copying using the libnbd(3) synchronous ("high
157 level") API. This is slow but may be necessary for some broken NBD
158 servers which cannot handle multiple requests in flight. This mode
159 is also used when streaming to and from stdio, pipes and sockets.
160 -T N
162 --threads=N
163 Use up to N threads for copying. By default this is set to the
164 number of processor cores available.
165 Note --threads=0 means autodetect and --threads=1 means use a
167 single thread.
168 -v
170 --verbose
171 Verbose output. This enables debug in libnbd (see
172 nbd_set_debug(3)) as well as printing other useful information.
173 -V
175 --version
176 Display the package name and version and exit.
177
179 The three options --connections, --threads and --requests are related
180 and control the amount of parallelism available. The defaults should
181 ensure a reasonable amount of parallelism if possible and you don’t
182 need to adjust them, but this section tries to describe what is going
183 on.
184 Firstly if either side of the copy is streaming to or from stdio, a
186 pipe, or a socket, or if you use the --synchronous option, then nbdcopy
187 works in synchronous mode with no parallelism, and nothing else in this
188 section applies.
189 The --connections=N option controls NBD multi-conn (see "Multi-conn" in
191 libnbd(3)), opening up to N connections to the NBD server (or to both
192 NBD servers if copying between NBD servers). This defaults to 4. The
193 NBD servers must support and advertise multi-conn. For nbdkit(1)
194 availability of multi-conn can depend on the plugin. You can use
195 nbdinfo(1) to find out if a particular NBD server is advertising multi-
196 conn. If the NBD server doesn’t advertise multi-conn then only one
197 connection will be opened regardless of the --connections flag.
198 When copying between two NBD servers, the number of connections is
200 limited to the minimum multi-conn supported on both sides. For the
201 purposes of this calculation, you can consider local files and block
202 devices as supporting infinite multi-conn.
203 When you run an NBD server as a subprocess (using the "[ ... ]" syntax)
205 multi-conn cannot be used.
206 The --threads=N option allows nbdcopy to start up to N threads
208 (defaulting to the number of cores). However nbdcopy cannot use more
209 threads than the number of NBD connections.
210 The --requests=N option controls the maximum number of requests in
212 flight on each NBD connection. This enables the NBD server to process
213 requests in parallel even when multi-conn isn’t available or when using
214 a single thread. The default is chosen to allow a reasonable amount of
215 parallelism without using too much memory.
216 Because of this parallelism, nbdcopy does not read or write blocks in
218 order. If for some reason you require that blocks are copied in strict
219 order then you must use --synchronous.
220
222 Instead of connecting to an already running server using an NBD URI,
223 you can run an NBD server as a subprocess using:
224 nbdcopy -- [ CMD ARGS ... ] ...
226 This requires the server to support systemd socket activation, which
228 both qemu-nbd(8) and nbdkit(1) support (see also
229 nbd_connect_systemd_socket_activation(3)).
230 "[" and "]" must be separate command line parameters. You will usually
232 need to use "--" to stop nbdcopy from misinterpreting NBD server flags
233 as nbdcopy flags. Both the source and destination may be subprocesses.
234 nbdcopy cleans up the subprocess on exit.
235 Some examples follow.
237 nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] - | hexdump -C
239 In this example, qemu-nbd(8) is run as a subprocess. The subprocess
240 opens disk.qcow2 and exposes it as NBD to nbdcopy. nbdcopy streams
241 this to stdout ("-") into the pipe which is read by hexdump(1).
242 nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] [ nbdkit memory 1G ]
244 Two subprocesses are created, qemu-nbd(8) as the source and nbdkit(1)
245 as the destination. The qcow2 file is converted to raw and stored
246 temporarily in the RAM disk (nbdkit-memory-plugin(1)).
247 When nbdcopy exits both servers are killed and the RAM disk goes away,
249 so this command has no overall effect, but is useful for testing.
250
252 You can use nbdcopy, cmp(1) and bash(1) process substitution to compare
253 the content of two NBD servers for equality:
254 cmp <( nbdcopy nbd://server1 - ) <( nbdcopy nbd://server2 - )
256 Note this tests that the content is logically equal. It does not
258 compare the NBD metadata such as sparseness (see nbdinfo(1) --map
259 option). Thus for example a run of allocated zeroes in one server will
260 match a hole in the other.
261
263 libnbd(3), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdkit(1), qemu-img(1).
264
266 Richard W.M. Jones
267
269 Copyright (C) 2020-2021 Red Hat Inc.
270
272 This library is free software; you can redistribute it and/or modify it
273 under the terms of the GNU Lesser General Public License as published
274 by the Free Software Foundation; either version 2 of the License, or
275 (at your option) any later version.
276 This library is distributed in the hope that it will be useful, but
278 WITHOUT ANY WARRANTY; without even the implied warranty of
279 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
280 Lesser General Public License for more details.
281 You should have received a copy of the GNU Lesser General Public
283 License along with this library; if not, write to the Free Software
284 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
285 02110-1301 USA
286libnbd-1.12.5 2022-07-10 nbdcopy(1)