1libnbd(3) LIBNBD libnbd(3)
2
6 libnbd - network block device (NBD) client library in userspace
7
9 #include <libnbd.h>
10 struct nbd_handle *nbd;
12 char buf[512];
13 if ((nbd = nbd_create ()) == NULL ||
15 nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
16 nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
17 fprintf (stderr, "%s\n", nbd_get_error ());
18 exit (EXIT_FAILURE);
19 }
20 nbd_close (nbd);
21 cc prog.c -o prog -lnbd
23 or:
24 cc prog.c -o prog `pkg-config libnbd --cflags --libs`
25
27 Network Block Device (NBD) is a network protocol for accessing block
28 devices over the network. Block devices are hard disks and things that
29 behave like hard disks such as disk images and virtual machines.
30 Libnbd is a client library for the NBD protocol which can access most
32 of the features of NBD while being simple to use and powerful.
33 This manual page gives an overview of libnbd, using C as an example,
35 but the library is available from other programming languages.
36 libnbd-api(3)
38 Covers the C API in detail.
39
41 To use the API at all you must first open a handle by calling
42 "nbd_create" (or its equivalent in other languages):
43 struct nbd_handle *nbd;
45 nbd = nbd_create ();
47 This creates and returns a handle, which is associated with one
49 connection to an NBD server, initially not connected.
50 Each handle is a complex state machine which can be in states such as
52 created, connected to a remote server, handshaking, idle and ready to
53 issue commands, or busy sending or receiving commands.
54 There are two levels of API available. A simple high level synchronous
56 API lets you give the handle high level instructions like “connect to
57 the server”, “read a block”, “write a block”, etc. Each of these
58 functions will run to completion before returning. A more complicated
59 low level non-blocking asynchronous API is also available where you can
60 integrate with poll(2) or another main loop.
61 You can freely mix the two APIs on the same handle. You can also call
63 APIs on a single handle from multiple threads.
64
66 This is the simplest way to use the API, with the possible drawback
67 that each libnbd function blocks until it is finished.
68 Create a handle and connect to the server:
70 struct nbd_handle *nbd;
72 nbd = nbd_create ();
74 if (!nbd) {
75 fprintf (stderr, "%s\n", nbd_get_error ());
76 exit (EXIT_FAILURE);
77 }
78 if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
79 fprintf (stderr, "%s\n", nbd_get_error ());
80 exit (EXIT_FAILURE);
81 }
82 Read the first sector (512 bytes) from the NBD export:
84 char buf[512];
86 if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
88 fprintf (stderr, "%s\n", nbd_get_error ());
89 exit (EXIT_FAILURE);
90 }
91 Close the handle:
93 nbd_close (nbd);
95 You can call the high level API from multiple threads, but each libnbd
97 API call takes a lock on the handle and so commands will not run in
98 parallel.
99
101 The low level API is useful if you want to use libnbd in non-blocking
102 code; or if you want to issue commands in parallel from multiple
103 threads; or if you need more control especially over having multiple
104 commands in-flight on a single connection.
105 To use the low level API you will need to integrate with poll(2) or
107 another “main loop” such as the GLib main event loop. A simple
108 implementation using poll(2) is available called "nbd_poll", and it is
109 also useful to examine how this is implemented (lib/poll.c in the
110 libnbd source code) because that will tell you how to integrate libnbd
111 with other main loops.
112 As with the high level API, it all starts by creating a handle:
114 struct nbd_handle *nbd;
116 nbd = nbd_create ();
118 if (nbd == NULL) {
119 fprintf (stderr, "%s\n", nbd_get_error ());
120 exit (EXIT_FAILURE);
121 }
122 To connect to the server asynchronously, we start the connection using
124 "nbd_aio_connect" and then enter our main loop to check for events
125 until the connection becomes ready:
126 int fd;
128 struct sockaddr_un addr;
129 socklen_t len;
130 /* some code to set up addr,
132 then ... */
133 if (nbd_aio_connect (nbd, &addr, len) == -1) {
134 fprintf (stderr, "%s\n", nbd_get_error ());
135 exit (EXIT_FAILURE);
136 }
137 while (!nbd_aio_is_ready (nbd)) {
138 if (nbd_poll (nbd, -1) == -1) {
139 fprintf (stderr, "%s\n", nbd_get_error ());
140 exit (EXIT_FAILURE);
141 }
142 }
143 To read data asynchronously, start an asynchronous read command, which
145 returns a 64 bit command ID, and enter the main loop until the command
146 has completed:
147 int64_t id;
149 char buf[512];
150 id = nbd_aio_pread (nbd, buf, sizeof buf, offset, 0)
152 if (id == -1) {
153 fprintf (stderr, "%s\n", nbd_get_error ());
154 exit (EXIT_FAILURE);
155 }
156 while (!nbd_aio_command_completed (nbd, id)) {
157 if (nbd_poll (nbd, -1) == -1) {
158 fprintf (stderr, "%s\n", nbd_get_error ());
159 exit (EXIT_FAILURE);
160 }
161 }
162 For almost all high level synchronous calls (eg. "nbd_pread") there is
164 a low level asynchronous equivalent (eg. "nbd_aio_pread").
165
167 When any API call returns an error ("-1" or "NULL" depending on the
168 API), an error message and sometimes an errno value are available. You
169 can retrieve the error message and/or errno of the most recently failed
170 call using "nbd_get_error" and "nbd_get_errno". For example:
171 if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
173 fprintf (stderr,
174 "failed to connect to remote server: %s (errno = %d)\n",
175 nbd_get_error (), nbd_get_errno ());
176 }
177 These functions use thread-local storage to return the most recent
179 error in the current thread. This is why you don't need to pass the
180 handle to these calls. They even work if "nbd_create" returns "NULL"
181 when there is no handle at all.
182 For this reason you cannot call them from a different thread. You
184 should call them immediately after the failed API call, from the same
185 thread. Furthermore the error string returned by "nbd_get_error" is
186 only valid until the next libnbd API call in the current thread, so if
187 you need to keep the string you must copy it (eg. using strdup(3)).
188 Even when a call returns an error, "nbd_get_errno" might return 0.
190 This means no additional errno information is available for this error.
191
193 Libnbd can print lots of debugging messages, useful if you have a
194 problem with the library. Either enable debugging after creating the
195 handle:
196 nbd = nbd_create ();
198 nbd_set_debug (nbd, true);
199 or set the "LIBNBD_DEBUG=1" environment variable which will enable
201 debugging by default on all new handles.
202 Debugging messages are sent to stderr by default, but you can redirect
204 them to a logging system using "nbd_set_debug_callback".
205
207 There are several ways to connect to NBD servers, and you can even run
208 a server from libnbd. Normally you would connect to a server which is
209 already running, over a local Unix domain socket or a remote TCP
210 connection. The high level API calls are:
211 nbd_connect_unix (nbd, "socket");
213 nbd_connect_tcp (nbd, "localhost", "nbd");
214 For "nbd_connect_tcp" the third parameter is the port name or number,
216 which can either be a name from /etc/services or the port number as a
217 string (eg. "10809").
218 Connecting to a subprocess
220 Some NBD servers — notably nbdkit(1) with the "-s" parameter — can also
221 accept a single NBD connection on stdin/stdout. You can run these
222 servers as a subprocess of your main program. This example creates a
223 1G writable RAM disk for testing which is discarded as soon as the
224 libnbd handle is closed:
225 char *argv[] = { "nbdkit", "-s", "memory", "1G", NULL };
227 nbd_connect_command (nbd, argv);
228 Connecting to a URI
230 libnbd supports the NBD URI specification.
231 This specification is currently evolving, and discussion about it can
233 be found on the NBD mailing list. A final link to the specification
234 will be added to this documentation when it is available.
235 To connect to a URI via the high level API, use:
237 nbd_connect_uri (nbd, "nbd://example.com/");
239 This feature is implemented by calling other libnbd APIs to set up the
241 export name, TLS parameters, and finally connect over a Unix domain
242 socket or TCP.
243 URI support is an optional feature of the library, requiring libxml2 at
245 compile time. The "nbd_connect_uri" and "nbd_aio_connect_uri" calls
246 will raise an error (with errno = "ENOTSUP") if it was not built with
247 this feature, and you can also test for it explicitly using
248 "nbd_supports_uri".
249
251 It is possible for NBD servers to serve different content on different
252 “exports”. For this you must pass the right export name to the server.
253 Call this API before connecting:
254 nbd_set_export_name (nbd, "/export");
256 Note that some servers (like nbdkit(1)) ignore this, and others (like
258 qemu-nbd(8)) require it to be set correctly but cannot serve different
259 content.
260 After connecting the server will send back a set of flags describing
262 the export, such as whether it is writable and if it can support flush
263 to permanent storage. These flags can be accessed from libnbd using
264 APIs such as:
265 int is_read_only = nbd_read_only (nbd);
267 int can_flush = nbd_can_flush (nbd);
268 (and several more, see libnbd-api(3)).
270 To get the size of the export in bytes, use:
272 int64_t size = nbd_get_size (nbd);
274
276 You can read and write data from the NBD server using "pread" and
277 "pwrite" or their asynchronous equivalents.
278 Some servers also support:
280 trim/discard
282 If "nbd_can_trim" return true, "nbd_trim" can be used to “punch
283 holes” in the backing storage of the disk on the server. Normally
284 (although not in every case) the holes read back as zeroes but take
285 up no space.
286 zeroing
288 If "nbd_can_zero" returns true, "nbd_zero" can be used to
289 efficiently zero parts of the disk without having to send large
290 amounts of zero bytes over the network (as would be necessary if
291 using "nbd_pwrite").
292 This is slightly different from trimming because the backing
294 storage is still allocated. For some storage types this can make
295 future writes more efficient and/or less likely to fail because of
296 out of space errors.
297 flushing
299 Some servers can commit data to permanent storage and tell you that
300 this has happened reliably. There are two export flags associated
301 with this: "nbd_can_flush" and "nbd_can_fua".
302 The "nbd_flush" call (available if "nbd_can_flush" returns true)
304 flushes all pending writes to disk and does not complete until that
305 operation has finished. It is similar to using sync(2) on POSIX
306 systems.
307 A more efficient way to achieve this is to set the flag
309 "LIBNBD_CMD_FLAG_FUA" on write-like calls (like write, trim and
310 zero). This flag means the call will not complete until committed
311 to permanent storage, but it does not involve flushing the entire
312 disk.
313 prefetching
315 Some servers can prefetch data, making subsequent reads faster.
316 The "nbd_cache" call (available if "nbd_can_cache" returns true) is
317 used to prefetch.
318
320 Issuing multiple in-flight requests
321 NBD servers which properly implement the specification can handle
322 multiple requests in flight over the same connection at the same time.
323 Libnbd supports this when using the low level API.
324 To use it you simply issue more requests as needed (eg. using calls
326 like "nbd_aio_pread", "nbd_aio_pwrite") without waiting for previous
327 commands to complete. You need to be careful that requests in flight
328 do not overlap with other write-like commands in flight — a parallel
329 read may see indeterminate data, and a parallel write may even cause
330 disk corruption where the resulting disk contents do not match either
331 of the two writes.
332 Each request is identified by a unique 64 bit handle (assigned by
334 libnbd), allowing libnbd and callers to match replies to requests.
335 Replies may arrive out of order.
336 Although in theory you can have an indefinite number of requests in
338 flight at the same time, in practice it's a good idea to limit them to
339 some number. Libnbd will queue commands in the handle even if it
340 cannot write them to the server, so this limit is largely to prevent a
341 backlog of commands from consuming too much memory. It is suggested to
342 start with a limit of 64 requests in flight (per NBD connection), and
343 measure how adjusting the limit up and down affects performance for
344 your local configuration.
345 There is a full example using multiple in-flight requests available at
347 https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and-writes.c
348 Multi-conn
350 Some NBD servers advertise “multi-conn” which means that it is safe to
351 make multiple connections to the server and load-balance commands
352 across all of the connections.
353 To do this you should open a single connection first and test it for
355 this feature. Without error handling it would look like this:
356 struct nbd_handle nbd[4];
358 size_t i;
359 bool supports_multi_conn;
360 nbd[0] = nbd_create ();
362 nbd[0]_connect_tcp (nbd[0], "server", "10809");
363 supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;
364 If multi-conn is supported then you can open further connections:
366 for (i = 1; i <= 3; ++i) {
368 nbd[i] = nbd_create ();
369 nbd[i]_connect_tcp (nbd[i], "server", "10809");
370 }
371 If you are issuing multiple in-flight requests (see above) and limiting
373 the number, then the limit should be applied to each individual NBD
374 connection.
375
377 The NBD protocol and libnbd supports TLS (sometimes incorrectly called
378 “SSL”) for encryption of the data stream and authentication of clients
379 and servers. Libnbd defaults to TLS disabled for maximum
380 interoperability. To enable it on a handle you must call "nbd_set_tls"
381 before connecting:
382 nbd_set_tls (nbd, 1); // to allow TLS, but fall back to unencrypted
384 nbd_set_tls (nbd, 2); // to require TLS, and fail otherwise
385 It may also be necessary to verify that the server’s identity is
387 correct. For some servers it may be necessary to verify to the server
388 that the client is permitted to connect. This can be done using either
389 X.509 certificates, or TLS Pre-Shared Keys (PSK). Certificates are
390 more secure. PSK is far more convenient, but you must have an existing
391 secure channel to distribute the keys.
392 Setting up X.509 using system certificate authorities (CAs)
394 This is the default if you don’t call any other "nbd_set_tls_*"
395 functions. In this case the server must have a public (eg. HTTPS)
396 certificate which can be verified against the CAs registered on your
397 system (eg. under /etc/pki).
398 To disable server name verification — which opens you up to a potential
400 Man-In-The-Middle (MITM) attack — use:
401 nbd_set_tls_verify_peer (nbd, false);
403 Setting up an X.509 certificate authority (CA)
405 You can set up your own CA and register clients and servers with it,
406 issuing client and server certificates which will reliably authenticate
407 your clients and servers to each other.
408 Doing this is described in detail in the nbdkit-tls(1) manual. The
410 only differences for libnbd are:
411 · Non-root certificates must be placed in "$HOME/.pki/libnbd/" or
413 "$HOME/.config/pki/libnbd/"
414 · Libnbd reads client-cert.pem and client-key.pem (instead of
416 server-cert.pem and server-key.pem).
417 Once you have set up the directory containing the certificates, call:
419 nbd_set_tls_certificates (nbd, "/path/to/directory");
421 Setting up Pre-Shared Keys (PSK)
423 TLS Pre-Shared Keys are a much more convenient method of setting up
424 TLS, and more appropriate for NBD, but you should have an existing
425 secure method available to distribute the keys. They are therefore
426 ideal if you want to set up an NBD service as an adjunct to an existing
427 secure REST API.
428 Use psktool(1) to create a file of "username:key" pairs:
430 psktool -u username -p keys.psk
432 and pass this path to libnbd:
434 nbd_set_tls_psk_file (nbd, "keys.psk");
436 If necessary you may need to set the client username (otherwise libnbd
438 will use your login name):
439 nbd_set_tls_username (nbd, "username");
441
443 libnbd-api(3), certtool(1), nbdkit(1), nbdkit-tls(1), nbd-server(1),
444 psktool(1), qemu(1), qemu-nbd(8).
445
447 Eric Blake
448 Richard W.M. Jones
450
452 Copyright (C) 2019 Red Hat Inc.
453
455 This library is free software; you can redistribute it and/or modify it
456 under the terms of the GNU Lesser General Public License as published
457 by the Free Software Foundation; either version 2 of the License, or
458 (at your option) any later version.
459 This library is distributed in the hope that it will be useful, but
461 WITHOUT ANY WARRANTY; without even the implied warranty of
462 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
463 Lesser General Public License for more details.
464 You should have received a copy of the GNU Lesser General Public
466 License along with this library; if not, write to the Free Software
467 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
468 02110-1301 USA
469libnbd-0.1.4 2019-06-09 libnbd(3)