1nbdcopy(1) LIBNBD nbdcopy(1)
2
6 nbdcopy - copy to and from an NBD server
7
9 nbdcopy [--allocated] [-C N|--connections=N]
10 [--destination-is-zero|--target-is-zero] [--flush]
11 [--no-extents] [-p|--progress|--progress=FD]
12 [-R N|--requests=N] [-S N|--sparse=N]
13 [--synchronous] [-T N|--threads=N]
14 SOURCE DESTINATION
15 SOURCE, DESTINATION := - | FILE | DEVICE | NBD-URI | [ CMD ARGS ... ]
17 nbdcopy --help
19 nbdcopy -V|--version
21
23 nbdcopy nbd://example.com local.img
24 This copies everything from the NBD server at "example.com" to a local
25 file called local.img.
26 nbdcopy nbd://example.com - | file -
28 This streams the first part of the disk on the NBD server at
29 "example.com" into the file(1) command. Note here that "-" means to
30 stream to stdout (and therefore into the pipe to the file command).
31 nbdinfo(1) is another way to detect the content on an NBD server.
33 nbdcopy -p /dev/sdX "nbd+unix:///?socket=/tmp/unixsock"
35 Copy the full local hard disk "/dev/sdX" to the NBD server listening on
36 the Unix domain socket /tmp/unixsock. Because of the -p option this
37 will print a progress bar.
38 nbdcopy nbd://server1 nbd://server2
40 Copy a full disk from one NBD server to another.
41 nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow ] disk.raw
43 Run qemu-nbd(8) as a subprocess to open disk.qcow2, which is then
44 copied to disk.raw. Note "--" to prevent qemu-nbd flags from being
45 interpreted as nbdcopy flags.
46 cat disk1 disk2 | nbdcopy -- - [ qemu-nbd -f qcow2 output.qcow2 ]
48 Concatenate two raw-format disk images into the qcow2 file
49 output.qcow2. The output file has to be precreated.
50
52 nbdcopy copies to and from an NBD server. It can upload a local file
53 to an NBD server, or download the contents of an NBD server to a local
54 file, device or stdin/stdout. It can also copy between NBD servers.
55 The local file can be a file, a block device (eg. "/dev/cdrom"), or "-"
57 which means stream in from stdin or stream out to stdout.
58 The NBD server can be specified using an NBD URI (like
60 "nbd://localhost"). The NBD server can be local or remote, and
61 encryption can be used if libnbd was built with encryption support.
62 Alternately you can use square brackets around a qemu-nbd(8) or
63 nbdkit(1) command to run the NBD server as a subprocess of nbdcopy.
64 For more complex copying operations including converting between disk
66 formats use "qemu-img convert", see qemu-img(1).
67
69 --help
70 Display brief command line help and exit.
71 --allocated
73 Normally nbdcopy tries to create sparse output (with holes) if the
74 destination supports that. It does this in two ways: either using
75 extent information from the source to copy holes (see
76 --no-extents), or by detecting runs of zeroes (see -S). If you use
77 --allocated then nbdcopy creates a fully allocated, non-sparse
78 output on the destination.
79 -C N
81 --connections=N
82 Set the maximum number of NBD connections ("multi-conn"). By
83 default nbdcopy will try to use multi-conn with up to 4 connections
84 if the NBD server supports it. If copying between NBD servers then
85 nbdcopy cannot use multi-conn if either of the servers does not
86 support it.
87 --destination-is-zero
89 --target-is-zero
90 Assume the destination is already zeroed. This allows nbdcopy to
91 skip copying blocks of zeroes from the source to the destination.
92 This is not safe unless the destination device is already zeroed.
93 (--target-is-zero is provided for compatibility with qemu-img(1).)
94 --flush
96 Flush writes to ensure that everything is written to persistent
97 storage before nbdcopy exits.
98 --no-extents
100 Normally nbdcopy uses extent metadata to skip over parts of the
101 source disk which contain holes. If you use this flag, nbdcopy
102 ignores extent information and reads everything, which is usually
103 slower. You might use this flag in two situations: the source NBD
104 server has incorrect metadata information; or the source has very
105 slow extent querying so it's faster to simply read all of the data.
106 -p
108 --progress
109 Display a progress bar.
110 --progress=FD
112 Write a progress bar to the file descriptor "FD" (a number) in a
113 format which is easily parsable by other programs. nbdcopy will
114 periodically write the string "N/100\n" (where N is an integer
115 between 0 and 100) to the file descriptor.
116 To get nbdcopy to write the progress bar to a file you can use the
118 following shell commands:
119 exec 3>/tmp/progress
121 nbdcopy --progress=3 ...
122 exec 3>&-
123 -R N
125 --requests=N
126 Set the maximum number of requests in flight per NBD connection.
127 -S N
129 --sparse=N
130 Detect all zero blocks of size N (bytes) and make them sparse on
131 the output. You can also turn off sparse detection using -S 0.
132 The default is 4096 bytes.
133 --synchronous
135 Force synchronous copying using the libnbd(3) synchronous ("high
136 level") API. This is slow but may be necessary for some broken NBD
137 servers which cannot handle multiple requests in flight. This mode
138 is also used when streaming to and from stdio, pipes and sockets.
139 -T N
141 --threads=N
142 Use up to N threads for copying. By default this is set to the
143 number of processor cores available.
144 Note --threads=0 means autodetect and --threads=1 means use a
146 single thread.
147 -V
149 --version
150 Display the package name and version and exit.
151
153 The three options --connections, --threads and --requests are related
154 and control the amount of parallelism available. The defaults should
155 ensure a reasonable amount of parallelism if possible and you don’t
156 need to adjust them, but this section tries to describe what is going
157 on.
158 Firstly if either side of the copy is streaming to or from stdio, a
160 pipe, or a socket, or if you use the --synchronous option, then nbdcopy
161 works in synchronous mode with no parallelism, and nothing else in this
162 section applies.
163 The --connections=N option controls NBD multi-conn (see "Multi-conn" in
165 libnbd(3)), opening up to N connections to the NBD server (or to both
166 NBD servers if copying between NBD servers). This defaults to 4. The
167 NBD servers must support and advertise multi-conn. For nbdkit(1)
168 availability of multi-conn can depend on the plugin. You can use
169 nbdinfo(1) to find out if a particular NBD server is advertising multi-
170 conn. If the NBD server doesn’t advertise multi-conn then only one
171 connection will be opened regardless of the --connections flag.
172 When copying between two NBD servers, the number of connections is
174 limited to the minimum multi-conn supported on both sides. For the
175 purposes of this calculation, you can consider local files and block
176 devices as supporting infinite multi-conn.
177 When you run an NBD server as a subprocess (using the "[ ... ]" syntax)
179 multi-conn cannot be used.
180 The --threads=N option allows nbdcopy to start up to N threads
182 (defaulting to the number of cores). However nbdcopy cannot use more
183 threads than the number of NBD connections.
184 The --requests=N option controls the maximum number of requests in
186 flight on each NBD connection. This enables the NBD server to process
187 requests in parallel even when multi-conn isn’t available or when using
188 a single thread. The default is chosen to allow a reasonable amount of
189 parallelism without using too much memory.
190 Because of this parallelism, nbdcopy does not read or write blocks in
192 order. If for some reason you require that blocks are copied in strict
193 order then you must use --synchronous.
194
196 Instead of connecting to an already running server using an NBD URI,
197 you can run an NBD server as a subprocess using:
198 nbdcopy -- [ CMD ARGS ... ] ...
200 This requires the server to support systemd socket activation, which
202 both qemu-nbd(8) and nbdkit(1) support (see also
203 nbd_connect_systemd_socket_activation(3)).
204 "[" and "]" must be separate command line parameters. You will usually
206 need to use "--" to stop nbdcopy from misinterpreting NBD server flags
207 as nbdcopy flags. Both the source and destination may be subprocesses.
208 nbdcopy cleans up the subprocess on exit.
209 Some examples follow.
211 nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] - | hexdump -C
213 In this example, qemu-nbd(8) is run as a subprocess. The subprocess
214 opens disk.qcow2 and exposes it as NBD to nbdcopy. nbdcopy streams
215 this to stdout ("-") into the pipe which is read by hexdump(1).
216 nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow2 ] [ nbdkit memory 1G ]
218 Two subprocesses are created, qemu-nbd(8) as the source and nbdkit(1)
219 as the destination. The qcow2 file is converted to raw and stored
220 temporarily in the RAM disk (nbdkit-memory-plugin(1)).
221 When nbdcopy exits both servers are killed and the RAM disk goes away,
223 so this command has no overall effect, but is useful for testing.
224
226 You can use nbdcopy, cmp(1) and bash(1) process substitution to compare
227 the content of two NBD servers for equality:
228 cmp <( nbdcopy nbd://server1 - ) <( nbdcopy nbd://server2 - )
230 Note this tests that the content is logically equal. It does not
232 compare the NBD metadata such as sparseness (see nbdinfo(1) --map
233 option). Thus for example a run of allocated zeroes in one server will
234 match a hole in the other.
235
237 libnbd(3), nbdfuse(1), nbdinfo(1), nbdsh(1), nbdkit(1), qemu-img(1).
238
240 Richard W.M. Jones
241
243 Copyright (C) 2020 Red Hat Inc.
244
246 This library is free software; you can redistribute it and/or modify it
247 under the terms of the GNU Lesser General Public License as published
248 by the Free Software Foundation; either version 2 of the License, or
249 (at your option) any later version.
250 This library is distributed in the hope that it will be useful, but
252 WITHOUT ANY WARRANTY; without even the implied warranty of
253 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
254 Lesser General Public License for more details.
255 You should have received a copy of the GNU Lesser General Public
257 License along with this library; if not, write to the Free Software
258 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
259 02110-1301 USA
260libnbd-1.6.2 2021-03-02 nbdcopy(1)