1libnbd(3) LIBNBD libnbd(3)
2
6 libnbd - network block device (NBD) client library in userspace
7
9 #include <libnbd.h>
10 struct nbd_handle *nbd;
12 char buf[512];
13 if ((nbd = nbd_create ()) == NULL ||
15 nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
16 nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
17 fprintf (stderr, "%s\n", nbd_get_error ());
18 nbd_close (nbd);
19 exit (EXIT_FAILURE);
20 }
21 nbd_close (nbd);
22 cc prog.c -o prog -lnbd
24 or:
25 cc prog.c -o prog `pkg-config libnbd --cflags --libs`
26
28 Network Block Device (NBD) is a network protocol for accessing block
29 devices over the network. Block devices are hard disks and things that
30 behave like hard disks such as disk images and virtual machines.
31 Libnbd is a client library for the NBD protocol which can access most
33 of the features of NBD while being simple to use and powerful.
34 This manual page gives an overview of libnbd, using C as an example,
36 but the library is available from other programming languages.
37 nbd_create(3), nbd_pread(3), etc.
39 Each manual page covers one function from the C API in detail.
40 There is a full list in section "C API" below.
41 libnbd-ocaml(3)
43 Using the API from OCaml.
44 libnbd-golang(3)
46 Using the API from Go.
47 nbdsh(1)
49 Using the NBD shell (nbdsh) for command line and Python scripting.
50
52 To use the API at all you must first open a handle by calling
53 nbd_create(3) (or its equivalent in other languages):
54 struct nbd_handle *nbd;
56 nbd = nbd_create ();
58 This creates and returns a handle, which is associated with one
60 connection to an NBD server, initially not connected.
61 Each handle is a complex state machine which can be in states such as
63 created, connected to a remote server, handshaking, idle and ready to
64 issue commands, or busy sending or receiving commands.
65 Handles have a name used in debugging messages. The name is normally
67 generated ("nbd1", "nbd2" etc) but you can set a friendly name with
68 nbd_set_handle_name(3). Also there is a private field in the handle
69 for use by the application, see nbd_set_private_data(3).
70 When you have finished with the handle you must call nbd_close(3) which
72 closes the underlying socket (if necessary) and frees up all associated
73 resources.
74
76 There are two levels of API available. A simple high level synchronous
77 API lets you give the handle high level instructions like “connect to
78 the server”, “read a block”, “write a block”, etc. Each of these
79 functions will run to completion, blocking the current thread before
80 returning. A more complicated low level non-blocking asynchronous API
81 is also available where you can integrate with poll(2) or another main
82 loop.
83 You can freely mix the two APIs on the same handle. You can also call
85 APIs on a single handle from multiple threads. Single API calls on the
86 handle are atomic — they either take a lock on the handle while they
87 run or are careful to access handle fields atomically.
88 Libnbd does not create its own threads.
90
92 This is the simplest way to use the API, with the possible drawback
93 that each libnbd function blocks until it is finished.
94 Create a handle and connect to the server:
96 struct nbd_handle *nbd;
98 nbd = nbd_create ();
100 if (!nbd) {
101 fprintf (stderr, "%s\n", nbd_get_error ());
102 nbd_close (nbd);
103 exit (EXIT_FAILURE);
104 }
105 if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
106 fprintf (stderr, "%s\n", nbd_get_error ());
107 nbd_close (nbd);
108 exit (EXIT_FAILURE);
109 }
110 Read the first sector (512 bytes) from the NBD export:
112 char buf[512];
114 if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
116 fprintf (stderr, "%s\n", nbd_get_error ());
117 nbd_close (nbd);
118 exit (EXIT_FAILURE);
119 }
120 Close the handle:
122 nbd_close (nbd);
124 You can call the high level API from multiple threads, but each libnbd
126 API call takes a lock on the handle and so commands will not run in
127 parallel.
128
130 The low level API is useful if you want to use libnbd in non-blocking
131 code; or if you want to issue commands in parallel from multiple
132 threads; or if you need more control especially over having multiple
133 commands in-flight on a single connection.
134 To use the low level API you will need to integrate with poll(2) or
136 another “main loop” such as the GLib main event loop.
137 Issuing asynchronous commands
139 Use the "nbd_aio_*" variants to issue commands asynchronously (without
140 waiting for the command to complete before returning). For example the
141 asynchronous variant of nbd_pread(3) is:
142 int64_t cookie;
144 cookie = nbd_aio_pread (nbd, buf, sizeof buf,
146 NBD_NULL_COMPLETION, 0);
147 if (cookie == -1) {
148 fprintf (stderr, "%s\n", nbd_get_error ());
149 nbd_close (nbd);
150 exit (EXIT_FAILURE);
151 }
152 There are several things to note here:
154 • This only starts the command. The command is (usually) still in
156 flight when the call returns success, where you must rely on
157 subsequent API calls for learning the final command outcome and
158 trigger any remaining callbacks. However, you must also be able to
159 handle the case where system load allows the state machine to
160 advance far enough to invoke callbacks before the asynchronous API
161 returns.
162 • A buffer ("buf") has been assigned to collect the result of the
164 read, but it is not guaranteed to be filled with data until the
165 command has completed (see examples below). The buffer must not be
166 freed until the command has finished running.
167 • You can issue multiple commands on the same handle at the same
169 time.
170 • A cookie is returned which identifies this command in subsequent
172 calls. The cookie is unique (per libnbd handle) and ≥ 1.
173 • You may register a function which is called when the command
175 completes, see "Completion callbacks" below. In this case we have
176 specified a null completion callback. If a completion callback is
177 specified, it will only be called if the asynchronous command was
178 successfully submitted (if the asynchronous API itself returns an
179 error, there is nothing further to be completed).
180 Socket and direction
182 Each libnbd handle has an associated socket (once it has started
183 connecting). You can read the file descriptor of the socket using:
184 int fd = nbd_aio_get_fd (nbd);
186 The socket is non-blocking. Between calls into libnbd it is in the
188 "would block" condition. You can find out if libnbd is expecting to
189 read or write from the socket next by calling:
190 int dir = nbd_aio_get_direction (nbd);
192 which returns one of "LIBNBD_AIO_DIRECTION_READ",
194 "LIBNBD_AIO_DIRECTION_WRITE" or "LIBNBD_AIO_DIRECTION_BOTH" (=
195 "READ|WRITE"). And so to set up the next call to poll(2) or other main
196 loop you must translate this to "POLLIN", "POLLOUT" or "POLLIN|POLLOUT"
197 (or whatever mechanism your main loop uses).
198 Notifying libnbd when an event happens
200 When you detect (eg. using poll(2)) that a read or write event has
201 happened on the socket, you must then tell libnbd about it. You have
202 to check the direction again (since it may have been changed by another
203 thread), and notify libnbd:
204 int r = 0;
206 dir = nbd_aio_get_direction (nbd);
208 if ((dir & LIBNBD_AIO_DIRECTION_READ) &&
210 a_read_event_occurred ())
211 r = nbd_aio_notify_read (nbd);
212 else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) &&
213 a_write_event_occurred ())
214 r = nbd_aio_notify_write (nbd);
215 if (r == -1) {
217 fprintf (stderr, "%s\n", nbd_get_error ());
218 // ...
219 }
220 The notify calls move the state machine along, reading and writing from
222 the socket possibly multiple times, until the socket would block again,
223 at which point they return control to the caller.
224 Simple implementation with nbd_poll(3)
226 In fact if you want to use poll(2) on a single handle, a simple
227 implementation has already been written called nbd_poll(3). It is also
228 useful to examine how this is implemented (lib/poll.c in the libnbd
229 source code) because that will tell you how to integrate libnbd with
230 more complex main loops.
231 Some examples of using nbd_poll(3) follow.
233 As with the high level API, it all starts by creating a handle:
235 struct nbd_handle *nbd;
237 nbd = nbd_create ();
239 if (nbd == NULL) {
240 fprintf (stderr, "%s\n", nbd_get_error ());
241 nbd_close (nbd);
242 exit (EXIT_FAILURE);
243 }
244 To connect to the server asynchronously, we start the connection using
246 nbd_aio_connect(3) and then enter our main loop to check for events
247 until the connection becomes ready:
248 int fd;
250 struct sockaddr_un addr;
251 socklen_t len;
252 /* some code to set up addr,
254 then ... */
255 if (nbd_aio_connect (nbd, &addr, len) == -1) {
256 fprintf (stderr, "%s\n", nbd_get_error ());
257 nbd_close (nbd);
258 exit (EXIT_FAILURE);
259 }
260 while (! nbd_aio_is_ready (nbd)) {
261 if (nbd_poll (nbd, -1) == -1) {
262 fprintf (stderr, "%s\n", nbd_get_error ());
263 nbd_close (nbd);
264 exit (EXIT_FAILURE);
265 }
266 }
267 To read data asynchronously, start an asynchronous read command, which
269 returns a 64 bit command cookie, and enter the main loop until the
270 command has completed:
271 int64_t cookie;
273 char buf[512];
274 cookie = nbd_aio_pread (nbd, buf, sizeof buf, offset,
276 NBD_NULL_COMPLETION, 0);
277 if (cookie == -1) {
278 fprintf (stderr, "%s\n", nbd_get_error ());
279 nbd_close (nbd);
280 exit (EXIT_FAILURE);
281 }
282 while (! nbd_aio_command_completed (nbd, cookie)) {
283 if (nbd_poll (nbd, -1) == -1) {
284 fprintf (stderr, "%s\n", nbd_get_error ());
285 nbd_close (nbd);
286 exit (EXIT_FAILURE);
287 }
288 }
289 For almost all high level synchronous calls (eg. nbd_pread(3)) there is
291 a low level asynchronous equivalent (eg. nbd_aio_pread(3)) for starting
292 a command.
293 glib2 integration
295 See
296 https://gitlab.com/nbdkit/libnbd/blob/master/examples/glib-main-loop.c
297 libev integration
299 See https://gitlab.com/nbdkit/libnbd/blob/master/examples/copy-libev.c
300
302 When any API call returns an error (-1 or "NULL" depending on the API),
303 an error message and sometimes an errno value are available. You can
304 retrieve the error message and/or errno of the most recently failed
305 call using nbd_get_error(3) and nbd_get_errno(3). For example:
306 if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
308 fprintf (stderr,
309 "failed to connect to remote server: %s (errno = %d)\n",
310 nbd_get_error (), nbd_get_errno ());
311 }
312 These functions use thread-local storage to return the most recent
314 error in the current thread. This is why you don't need to pass the
315 handle to these calls. They even work if nbd_create(3) returns "NULL"
316 when there is no handle at all.
317 For this reason you cannot call them from a different thread. You
319 should call them immediately after the failed API call, from the same
320 thread. Furthermore the error string returned by nbd_get_error(3) is
321 only valid until the next libnbd API call in the current thread, so if
322 you need to keep the string you must copy it (eg. using strdup(3)).
323 Errno
325 For some errors, a system call error number (see errno(3)) is
326 available. You can find the error number by calling nbd_get_errno(3).
327 It works the same way as nbd_get_error(3) with respect to threads.
328 Even when a call returns an error, nbd_get_errno(3) might return 0.
330 This does not mean there was no error. It means no additional errno
331 information is available for this error.
332 The error number is often the raw error returned by a system call that
334 failed.
335 It can also be used to indicate special conditions. The most common
337 cases are:
338 "EINVAL"
340 Invalid parameters or state for the current libnbd call. (This can
341 also indicate that requests are not aligned to "Block size
342 constraints").
343 "ENOTSUP"
345 The libnbd call is not available in this build of libnbd (eg. when
346 using a TLS API if the library was compiled without TLS support).
347 "ENOMEM"
349 The library ran out of memory while performing some operation.
350 "ERANGE"
352 A request is too large, for example if you try to read too many
353 bytes in a single nbd_pread(3) call.
354 "EFAULT"
356 A pointer parameter was "NULL" when it should be non-NULL. See the
357 section below.
358 Non-NULL parameters
360 Almost all libnbd functions when called from C take one or more pointer
361 parameters that must not be "NULL". For example, the handle parameter,
362 strings and buffers should usually not be "NULL".
363 If a "NULL" is passed as one of these parameters, libnbd attempts to
365 return an error with nbd_get_errno(3) returning "EFAULT".
366 However it may cause other compiler-related warnings and even undefined
368 behaviour, so you should try to avoid this programming mistake.
369
371 Libnbd can print lots of debugging messages, useful if you have a
372 problem with the library. Either enable debugging after creating the
373 handle:
374 nbd = nbd_create ();
376 nbd_set_debug (nbd, true);
377 or set the "LIBNBD_DEBUG=1" environment variable which will enable
379 debugging by default on all new handles.
380 Debugging messages are sent to stderr by default, but you can redirect
382 them to a logging system using nbd_set_debug_callback(3).
383
385 There are several ways to connect to NBD servers, and you can even run
386 a server from libnbd. Normally you would connect to a server which is
387 already running, over a local Unix domain socket or a remote TCP
388 connection. The high level API calls are:
389 nbd_connect_unix (nbd, "socket");
391 nbd_connect_tcp (nbd, "localhost", "nbd");
392 For nbd_connect_tcp(3) the third parameter is the port name or number,
394 which can either be a name from /etc/services or the port number as a
395 string (eg. "10809").
396 Connecting to an NBD URI
398 libnbd supports the NBD URI specification. The format of URIs is
399 documented in nbd_connect_uri(3).
400 You can connect to a URI as in these examples (using the high level
402 API):
403 nbd_connect_uri (nbd, "nbd://example.com/");
405 nbd_connect_uri (nbd, "nbds+unix:///export?socket=/tmp/nbd.sock");
407 This feature is implemented by calling other libnbd APIs to set up the
409 export name, TLS parameters, and finally connect over a Unix domain
410 socket or TCP.
411 URI support is an optional feature of the library, requiring libxml2 at
413 compile time. The nbd_connect_uri(3) and nbd_aio_connect_uri(3) calls
414 will raise an error (with nbd_get_errno(3) returning "ENOTSUP") if it
415 was not built with this feature, and you can also test for it
416 explicitly using nbd_supports_uri(3).
417 Connecting to a subprocess
419 Some NBD servers — notably nbdkit(1) with the -s parameter, and
420 nbd-server(1) with the port parameter set to 0 — can also accept a
421 single NBD connection on stdin/stdout. You can run these servers as a
422 subprocess of your main program using nbd_connect_command(3). This
423 example creates a 1G writable RAM disk:
424 char *argv[] = { "nbdkit", "-s", "--exit-with-parent",
426 "memory", "1G", NULL };
427 nbd_connect_command (nbd, argv);
428 When the handle is closed the nbdkit subprocess is killed, which in
430 this case means the RAM disk is discarded, so this is useful for
431 testing.
432 Connecting to a subprocess using systemd socket activation
434 Some NBD servers — notably nbdkit(1) and qemu-nbd(1) — support systemd
435 socket activation allowing libnbd to pass a socket to the subprocess.
436 This works very similarly to nbd_connect_command(3) described above,
437 but you must use nbd_connect_systemd_socket_activation(3) instead.
438 Connecting to any socket
440 If none of the other nbd_connect* methods are suitable you can create a
441 connected socket yourself and pass it to nbd_connect_socket(3).
442 One use for this is in fuzzing where we use socketpair(2) to create the
444 socket, then fork, then have the test harness in the child process
445 connected to libnbd over the socket pair (see:
446 https://gitlab.com/nbdkit/libnbd/-/blob/master/fuzzing/libnbd-fuzz-wrapper.c).
447 Another use is to connect libnbd to an address family that it does not
449 support natively, such as XDP or IB.
450
452 By default, when beginning a connection, libnbd will handle all
453 negotiation with the server, using only the configuration (eg.
454 nbd_set_export_name(3) or nbd_add_meta_context(3)) that was requested
455 before the connection attempt; this phase continues until
456 nbd_aio_is_connecting(3) no longer returns true, at which point, either
457 data commands are ready to use or else the connection has failed with
458 an error.
459 But there are scenarios in which it is useful to also control the
461 handshaking commands sent during negotiation, such as asking the server
462 for a list of available exports prior to selecting which one to use.
463 This is done by calling nbd_set_opt_mode(3) before connecting; then
464 after requesting a connection, the state machine will pause at
465 nbd_aio_is_negotiating(3) at any point that the user can decide which
466 handshake command to send next. Note that the negotiation state is
467 only reachable from newstyle servers; older servers cannot negotiate
468 and will progress all the way to the ready state.
469 When the negotiating state is reached, you can initiate option commands
471 such as nbd_opt_list(3) or their asynchronous equivalents, as well as
472 alter configuration such as export name that previously had to be set
473 before connection. Since the NBD protocol does not allow parallel
474 negotiating commands, no cookie is involved, and you can track
475 completion of each command when the state is no longer
476 nbd_aio_is_connecting(3). If nbd_opt_go(3) fails but the connection is
477 still live, you will be back in negotiation state, where you can
478 request a different export name and try again. Exiting the negotiation
479 state is only possible with a successful nbd_opt_go(3) which moves to
480 the data phase, or nbd_opt_abort(3) which performs a clean shutdown of
481 the connection by skipping the data phase.
482
484 It is possible for NBD servers to serve different content on different
485 “exports”. For this you must pass the right export name to the server.
486 Call this API before connecting:
487 nbd_set_export_name (nbd, "export");
489 Note that there are some servers (like nbdkit(1) ≤ 1.14) which ignore
491 this, and other servers (like qemu-nbd(8)) which require it to be set
492 correctly but cannot serve different content.
493 These APIs are also available after a successful nbd_opt_info(3) during
495 the negotiation phase, if you used nbd_set_opt_mode(3) prior to
496 connecting.
497 Flag calls
499 After connecting the server will send back a set of flags describing
500 the export, such as whether it is writable and if it can support flush
501 to permanent storage. These flags can be accessed from libnbd using
502 APIs such as:
503 int is_read_only = nbd_is_read_only (nbd);
505 int can_flush = nbd_can_flush (nbd);
506 Flag calls are: nbd_can_cache(3), nbd_can_df(3), nbd_can_fast_zero(3),
508 nbd_can_flush(3), nbd_can_fua(3), nbd_can_meta_context(3),
509 nbd_can_multi_conn(3), nbd_can_trim(3), nbd_can_zero(3),
510 nbd_is_read_only(3), nbd_is_rotational(3).
511 Size of the export
513 To get the size of the export in bytes, use nbd_get_size(3):
514 int64_t size = nbd_get_size (nbd);
516 Block size constraints
518 Some NBD servers cannot handle requests at any byte boundary. They
519 might, for example, require all requests to be aligned to 512 byte
520 sectors.
521 Also some servers advertise a preferred block size. This is not a
523 requirement, but is the minimum block size that can be accessed
524 efficiently (usually without triggering expensive read-modify-write
525 cycles inside the server).
526 These are referred to as block size constraints and can be queried by
528 calling nbd_get_block_size(3). Pay attention in particular to the
529 "LIBNBD_SIZE_MINIMUM" constraint as some servers will fail requests
530 which are smaller or not aligned to this block size with "EINVAL"
531 ("Invalid argument") errors.
532
534 You can read and write data from the NBD server using nbd_pread(3) and
535 nbd_pwrite(3) or their asynchronous equivalents.
536 All data commands support a "flags" argument (mandatory in C, but
538 optional in languages where it can default to 0). For convenience, the
539 constant "LIBNBD_CMD_FLAG_MASK" is defined with the set of flags
540 currently recognized by libnbd, where future NBD protocol extensions
541 may result in additional flags being supported; but in general,
542 specific data commands only accept a subset of known flags.
543 Libnbd defaults to performing some client-side sanity checking in each
545 of its data commands; for example, attempts to write to a server that
546 has advertised a read-only connection are rejected. It is possible to
547 override aspects of this checking by using nbd_set_strict_mode(3).
548 Some servers also support:
550 trim/discard
552 If nbd_can_trim(3) returns true, nbd_trim(3) can be used to “punch
553 holes” in the backing storage of the disk on the server. Normally
554 (although not in every case) the holes read back as zeroes but take
555 up no space.
556 zeroing
558 If nbd_can_zero(3) returns true, nbd_zero(3) can be used to
559 efficiently zero parts of the disk without having to send large
560 amounts of zero bytes over the network (as would be necessary if
561 using nbd_pwrite(3)).
562 This is slightly different from trimming because the backing
564 storage is still allocated. For some storage types this can make
565 future writes more efficient and/or less likely to fail because of
566 out of space errors.
567 flushing
569 Some servers can commit data to permanent storage and tell you that
570 this has happened reliably. There are two export flags associated
571 with this: nbd_can_flush(3) and nbd_can_fua(3).
572 The nbd_flush(3) call (available if nbd_can_flush(3) returns true)
574 flushes all pending writes to disk and does not complete until that
575 operation has finished. It is similar to using sync(2) on POSIX
576 systems.
577 A more efficient way to achieve this is to set the flag
579 "LIBNBD_CMD_FLAG_FUA" on write-like calls (like write, trim and
580 zero). This flag means the call will not complete until committed
581 to permanent storage, but it does not involve flushing the entire
582 disk.
583 prefetching
585 Some servers can prefetch data, making subsequent reads faster.
586 The nbd_cache(3) call (available if nbd_can_cache(3) returns true)
587 is used to prefetch.
588 block status
590 Some servers are able to provide information about the various
591 extents within the image, via the notion of one or more meta
592 contexts. The most common meta context is "base:allocation"
593 (available in libnbd.h as "LIBNBD_CONTEXT_BASE_ALLOCATION"), which
594 can be used to learn which portions of a file are allocated or read
595 as zero. Other contexts may be available; for example, qemu-nbd(8)
596 can expose a meta context "qemu:dirty-bitmap:NAME" for tracking
597 which portions of a file are tracked by a qcow2 dirty bitmap.
598 In order to utilize block status, the client must call
600 nbd_add_meta_context(3) prior to connecting, for each meta context
601 in which it is interested, then check nbd_can_meta_context(3) after
602 connection to see which contexts the server actually supports. If
603 a context is supported, the client can then use nbd_block_status(3)
604 with a callback function that will receive an array of 32-bit
605 integer pairs describing consecutive extents within a context. In
606 each pair, the first integer is the length of the extent, the
607 second is a bitmask description of that extent (for the
608 "base:allocation" context, the bitmask may include
609 "LIBNBD_STATE_HOLE" for unallocated portions of the file, and/or
610 "LIBNBD_STATE_ZERO" for portions of the file known to read as
611 zero).
612 There is a full example of requesting meta context and using block
614 status available at
615 https://gitlab.com/nbdkit/libnbd/blob/master/interop/dirty-bitmap.c
616
618 Issuing multiple in-flight requests
619 NBD servers which properly implement the specification can handle
620 multiple data requests in flight over the same connection at the same
621 time. Libnbd supports this when using the low level API.
622 To use it you simply issue more requests as needed (eg. using calls
624 like nbd_aio_pread(3), nbd_aio_pwrite(3)) without waiting for previous
625 commands to complete. You need to be careful that requests in flight
626 do not overlap with disk offsets of other write-like commands in flight
627 — an overlapping read may see indeterminate data, and an overlapping
628 write may even cause disk corruption where the resulting disk contents
629 do not match either of the two writes.
630 Each request is identified by a unique 64 bit cookie (assigned by
632 libnbd), allowing libnbd and callers to match replies to requests.
633 Replies may arrive out of order. A request that is rejected client-
634 side for failing a sanity check (such as attempting to write to a read-
635 only server, see nbd_set_strict_mode(3)) will fail rather than
636 returning a cookie, although closure cleanup is still performed.
637 Although in theory you can have an indefinite number of requests in
639 flight at the same time, in practice it's a good idea to limit them to
640 some number. Libnbd will queue commands in the handle even if it
641 cannot write them to the server, so this limit is largely to prevent a
642 backlog of commands from consuming too much memory. It is suggested to
643 start with a limit of 64 requests in flight (per NBD connection), and
644 measure how adjusting the limit up and down affects performance for
645 your local configuration.
646 There is a full example using multiple in-flight requests available at
648 https://gitlab.com/nbdkit/libnbd/blob/master/examples/threaded-reads-and-writes.c
649 Multi-conn
651 Some NBD servers advertise “multi-conn” which means that it is safe to
652 make multiple connections to the server and load-balance commands
653 across all of the connections.
654 To do this you should open a single connection first and test for this
656 feature using nbd_can_multi_conn(3). Without error handling it would
657 look like this:
658 struct nbd_handle *nbd[4];
660 size_t i;
661 bool supports_multi_conn;
662 nbd[0] = nbd_create ();
664 nbd_connect_tcp (nbd[0], "server", "10809");
665 supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;
666 If multi-conn is supported then you can open further connections:
668 if (supports_multi_conn) {
670 for (i = 1; i <= 3; ++i) {
671 nbd[i] = nbd_create ();
672 nbd_connect_tcp (nbd[i], "server", "10809");
673 }
674 }
675 If you are issuing multiple in-flight requests (see above) and limiting
677 the number, then the limit should be applied to each individual NBD
678 connection.
679
681 The NBD protocol and libnbd supports TLS (sometimes incorrectly called
682 “SSL”) for encryption of the data stream and authentication of clients
683 and servers. Libnbd defaults to TLS disabled for maximum
684 interoperability. To enable it on a handle you must call
685 nbd_set_tls(3) before connecting.
686 To allow TLS, but fall back to unencrypted:
688 nbd_set_tls (nbd, LIBNBD_TLS_ALLOW);
690 Use nbd_get_tls_negotiated(3) to find out if TLS negotiation was
692 successful. Avoid "LIBNBD_TLS_ALLOW" if man-in-the-middle attacks are
693 a concern.
694 The most secure mode is to require TLS and fail to connect if the
696 server does not support it:
697 nbd_set_tls (nbd, LIBNBD_TLS_REQUIRE);
699 It may also be necessary to verify that the server’s identity is
701 correct. For some servers it may be necessary to verify to the server
702 that the client is permitted to connect. This can be done using either
703 X.509 certificates, or TLS Pre-Shared Keys (PSK). Certificates are
704 more secure. PSK is far more convenient, but you must have an existing
705 secure channel to distribute the keys.
706 Setting up X.509 using system certificate authorities (CAs)
708 This is the default if you don’t call any other "nbd_set_tls_*"
709 functions. In this case the server must have a public (eg. HTTPS)
710 certificate which can be verified against the CAs registered on your
711 system (eg. under /etc/pki).
712 To disable server name verification — which opens you up to a potential
714 Man-In-The-Middle (MITM) attack — use:
715 nbd_set_tls_verify_peer (nbd, false);
717 Setting up an X.509 certificate authority (CA)
719 You can set up your own CA and register clients and servers with it,
720 issuing client and server certificates which will reliably authenticate
721 your clients and servers to each other.
722 Doing this is described in detail in the nbdkit-tls(1) manual. The
724 only differences for libnbd are:
725 • Non-root certificates must be placed in "$HOME/.pki/libnbd/" or
727 "$HOME/.config/pki/libnbd/"
728 • Libnbd reads client-cert.pem and client-key.pem (instead of
730 server-cert.pem and server-key.pem).
731 Once you have set up the directory containing the certificates, call:
733 nbd_set_tls_certificates (nbd, "/path/to/directory");
735 Setting up Pre-Shared Keys (PSK)
737 TLS Pre-Shared Keys are a much more convenient method of setting up
738 TLS, and more appropriate for NBD, but you should have an existing
739 secure method available to distribute the keys. They are therefore
740 ideal if you want to set up an NBD service as an adjunct to an existing
741 secure REST API.
742 Use psktool(1) to create a file of "username:key" pairs:
744 psktool -u username -p keys.psk
746 and pass this path to libnbd:
748 nbd_set_tls_psk_file (nbd, "keys.psk");
750 If necessary you may need to set the client username (otherwise libnbd
752 will use your login name):
753 nbd_set_tls_username (nbd, "username");
755
757 Some libnbd calls take callbacks (eg. nbd_set_debug_callback(3),
758 nbd_aio_pread(3)). Libnbd can call these functions while processing.
759 In the C API these libnbd calls take a structure which contains the
761 function pointer and an optional opaque "void *user_data" pointer:
762 nbd_aio_pread (nbd, buf, sizeof buf, offset,
764 (nbd_completion_callback) { .callback = my_fn,
765 .user_data = my_data },
766 0);
767 For optional callbacks, if you don't want the callback, either set
769 ".callback" to "NULL" or use the equivalent macros (such as
770 "NBD_NULL_COMPLETION") defined in "libnbd.h":
771 nbd_aio_pread (nbd, buf, sizeof buf, offset,
773 NBD_NULL_COMPLETION, 0);
774 From other languages the structure and opaque pointer are not needed
776 because you can use closures to achieve the same effect.
777 Callback lifetimes
779 You can associate an optional free function with callbacks. Libnbd
780 will call this function when the callback will not be called again by
781 libnbd, including in the case where the API fails.
782 This can be used to free associated "user_data". For example:
784 void *my_data = malloc (...);
786 nbd_aio_pread_structured (nbd, buf, sizeof buf, offset,
788 (nbd_chunk_callback) { .callback = my_fn,
789 .user_data = my_data,
790 .free = free },
791 NBD_NULL_COMPLETION,
792 0);
793 will call free(3) once on "my_data" after the point where it is known
795 that the "chunk.callback = my_fn" function can no longer be called,
796 regardless of how many times "my_fn" was actually called. If both a
797 mid-command and completion callback are supplied, the functions will be
798 reached in this order: mid-function callbacks, completion callback,
799 mid-function free, and finally completion free.
800 The free function is only accessible in the C API as it is not needed
802 in garbage collected programming languages.
803 Callbacks with ".callback=NULL" and ".free!=NULL"
805 It is possible to register a callback like this:
806 ...
808 (nbd_completion_callback) { .callback = NULL,
809 .user_data = my_data,
810 .free = free },
811 ...
812 The meaning of this is that the callback is never called, but the free
814 function is still called after the last time the callback would have
815 been called. This is useful for applying generic freeing actions when
816 asynchronous commands are retired.
817 Callbacks and locking
819 The callbacks are invoked at a point where the libnbd lock is held,
820 typically during a call to "nbd_aio_notify_read",
821 "nbd_aio_notify_write", "nbd_aio_poll", or other call that can advance
822 libnbd's state machine. Depending on system load, it is even possible
823 for a callback to be reached before completion of the "nbd_aio_*" call
824 that specified the callback. As such, it is unsafe for the callback to
825 call any "nbd_*" APIs on the same nbd object, as it would cause
826 deadlock.
827 Completion callbacks
829 All of the asychronous commands have an optional completion callback
830 function that is used if the call to the asynchronous API reports
831 success. The completion callback is invoked when the submitted command
832 is eventually marked complete, after any mid-command callbacks have
833 finished, and before any free functions. The completion callback is
834 not reached if the asynchronous API itself fails, while free callbacks
835 are reached regardless of the result of the initial asynchronous API.
836 When the completion callback returns 1, the command is automatically
838 retired (there is no need to call nbd_aio_command_completed(3)); for
839 any other return value, the command still needs to be manually retired
840 (otherwise, the command will tie up resources until nbd_close(3) is
841 eventually reached).
842 Callbacks with "int *error" parameter
844 Some of the high-level commands (nbd_pread_structured(3),
845 nbd_block_status(3)) involve the use of a callback function invoked by
846 the state machine at appropriate points in the server's reply before
847 the overall command is complete. These callback functions, along with
848 all of the completion callbacks, include a parameter "error" which is a
849 pointer containing the value of any error detected so far. If a
850 callback function fails and wants to change the resulting error of the
851 overall command visible later in the API sequence, it should assign
852 back into "error" and return -1 in the C API. Assignments into "error"
853 are ignored for any other return value; similarly, assigning 0 into
854 "error" does not have an effect. In other language bindings, reporting
855 callback errors is generally done by raising an exception rather than
856 by return value.
857 Note that a mid-command callback might never be reached, such as if
859 libnbd detects that the command was invalid to send (see
860 nbd_set_strict_mode(3)) or if the server reports a failure that
861 concludes the command. It is safe for a mid-command callback to ignore
862 non-zero "error": all the other parameters to the mid-command callback
863 will still be valid (corresponding to the current portion of the
864 server's reply), and the overall command will still fail (at the
865 completion callback or nbd_aio_command_completed(3) for an asynchronous
866 command, or as the result of the overall synchronous command).
867 Returing -1 from a mid-command callback does not prevent that callback
868 from being reached again, if the server sends more mid-command replies
869 that warrant another use of that callback. A mid-command callback may
870 be reached more times than expected if the server is non-compliant.
871 On the other hand, if a completion callback is supplied (only possible
873 with asynchronous commands), it will not be called if the initial API
874 call fails (such as attempting an asynchronous command in the wrong
875 state - there is nothing to be completed since the command was not
876 queued), but will otherwise be called exactly once, and the completion
877 callback must not ignore the value pointed to by "error". In
878 particular, the content of a buffer passed to nbd_aio_pread(3) or
879 nbd_aio_pread_structured(3) is undefined if *error is non-zero on entry
880 to the completion callback. It is recommended that if you choose to
881 use automatic command retirement (where the completion callback returns
882 1 to avoid needing to call nbd_aio_command_completed(3) later), your
883 completion function should return 1 on all control paths, even when
884 handling errors (note that with automatic retirement, assigning into
885 "error" is pointless as there is no later API to see that value).
886
888 Libnbd tracks several statistics counters, useful for tracking how much
889 traffic was sent over the connection. The counters track the number of
890 plaintext bytes sent and received by the NBD protocol (not necessarily
891 the number of bytes sent over the socket, particularly when TLS is
892 enabled), as well as the number of protocol chunks (a group of bytes
893 delineated by a magic number, and not the same as the number of TCP
894 packets).
895 printf ("bytes: sent=%" PRIu64 " received=%" PRIu64,
897 nbd_stats_bytes_sent (nbd), nbd_stats_bytes_received (nbd));
898 printf ("chunks: sent=%" PRIu64 " received=%" PRIu64,
899 nbd_stats_chunks_sent (nbd), nbd_stats_chunks_received (nbd));
900
902 Libnbd does not install signal handlers. It attempts to disable
903 "SIGPIPE" when writing to the NBD socket using the "MSG_NOSIGNAL" flag
904 of send(2), or the "SO_NOSIGPIPE" socket option, on platforms that
905 support those.
906 On some old Linux or newer non-Linux platforms the main program may
908 wish to register a signal handler to ignore SIGPIPE:
909 signal (SIGPIPE, SIG_IGN);
911
913 On most systems, C programs that use libnbd can be compiled like this:
914 cc prog.c -o prog -lnbd
916 To detect if the libnbd library and header file is installed, the
918 preferred method is to use pkg-config(1) or pkgconf(1):
919 pkg-config libnbd --exists || fail libnbd is required
921 In case the library or header file are not installed in the usual
923 system locations, you can compile your program like this, using pkg-
924 config to detect the proper location of libnbd:
925 cc prog.c -o prog `pkg-config libnbd --cflags --libs`
927 To compile an external project against a built copy of the libnbd
929 source tree which hasn't been installed, see the ./run script.
930 Autoconf projects
932 External projects which use autoconf and need to check if libnbd is
933 installed should use the "PKG_CHECK_MODULES" macro in configure.ac like
934 this:
935 PKG_CHECK_MODULES([LIBNBD], [libnbd])
937 This will define "@LIBNBD_CFLAGS@" and "@LIBNBD_LIBS@" which you will
939 need to add to your Makefile.am.
940 CMake projects
942 For CMake projects use:
943 find_package(PkgConfig REQUIRED)
945 pkg_check_modules(LIBNBD REQUIRED libnbd)
946 target_link_libraries(prog ${LIBNBD_LIBRARIES})
947 target_include_directories(prog PUBLIC ${LIBNBD_INCLUDE_DIRS})
948 target_compile_options(prog PUBLIC ${LIBNBD_CFLAGS_OTHER})
949 Meson projects
951 For meson projects use:
952 nbd_dep = dependency('libnbd')
954 executable('prog', 'prog.c', dependencies : [nbd_dep])
955
957 "HOME"
958 Used in some situations to find TLS certificates. See
959 nbd_set_tls_certificates(3).
960 "LIBNBD_DEBUG"
962 If this is set to the exact string 1 when the handle is created
963 then debugging is enabled. See "DEBUGGING MESSAGES" above.
964 "LOGNAME"
966 The default TLS username. See nbd_set_tls_username(3).
967
969 C API
970 nbd_add_meta_context(3), nbd_aio_block_status(3), nbd_aio_cache(3),
971 nbd_aio_command_completed(3), nbd_aio_connect(3),
972 nbd_aio_connect_command(3), nbd_aio_connect_socket(3),
973 nbd_aio_connect_systemd_socket_activation(3), nbd_aio_connect_tcp(3),
974 nbd_aio_connect_unix(3), nbd_aio_connect_uri(3),
975 nbd_aio_connect_vsock(3), nbd_aio_disconnect(3), nbd_aio_flush(3),
976 nbd_aio_get_direction(3), nbd_aio_get_fd(3), nbd_aio_in_flight(3),
977 nbd_aio_is_closed(3), nbd_aio_is_connecting(3), nbd_aio_is_created(3),
978 nbd_aio_is_dead(3), nbd_aio_is_negotiating(3),
979 nbd_aio_is_processing(3), nbd_aio_is_ready(3), nbd_aio_notify_read(3),
980 nbd_aio_notify_write(3), nbd_aio_opt_abort(3), nbd_aio_opt_go(3),
981 nbd_aio_opt_info(3), nbd_aio_opt_list(3),
982 nbd_aio_opt_list_meta_context(3),
983 nbd_aio_opt_list_meta_context_queries(3),
984 nbd_aio_opt_set_meta_context(3),
985 nbd_aio_opt_set_meta_context_queries(3), nbd_aio_opt_starttls(3),
986 nbd_aio_opt_structured_reply(3), nbd_aio_peek_command_completed(3),
987 nbd_aio_pread(3), nbd_aio_pread_structured(3), nbd_aio_pwrite(3),
988 nbd_aio_trim(3), nbd_aio_zero(3), nbd_block_status(3), nbd_cache(3),
989 nbd_can_cache(3), nbd_can_df(3), nbd_can_fast_zero(3),
990 nbd_can_flush(3), nbd_can_fua(3), nbd_can_meta_context(3),
991 nbd_can_multi_conn(3), nbd_can_trim(3), nbd_can_zero(3),
992 nbd_clear_debug_callback(3), nbd_clear_meta_contexts(3), nbd_close(3),
993 nbd_connect_command(3), nbd_connect_socket(3),
994 nbd_connect_systemd_socket_activation(3), nbd_connect_tcp(3),
995 nbd_connect_unix(3), nbd_connect_uri(3), nbd_connect_vsock(3),
996 nbd_connection_state(3), nbd_create(3), nbd_flush(3),
997 nbd_get_block_size(3), nbd_get_canonical_export_name(3),
998 nbd_get_debug(3), nbd_get_errno(3), nbd_get_error(3),
999 nbd_get_export_description(3), nbd_get_export_name(3),
1000 nbd_get_full_info(3), nbd_get_handle_name(3),
1001 nbd_get_handshake_flags(3), nbd_get_meta_context(3),
1002 nbd_get_nr_meta_contexts(3), nbd_get_opt_mode(3),
1003 nbd_get_package_name(3), nbd_get_pread_initialize(3),
1004 nbd_get_private_data(3), nbd_get_protocol(3),
1005 nbd_get_request_block_size(3), nbd_get_request_meta_context(3),
1006 nbd_get_request_structured_replies(3), nbd_get_size(3),
1007 nbd_get_socket_activation_name(3), nbd_get_strict_mode(3),
1008 nbd_get_structured_replies_negotiated(3), nbd_get_tls(3),
1009 nbd_get_tls_negotiated(3), nbd_get_tls_username(3),
1010 nbd_get_tls_verify_peer(3), nbd_get_uri(3), nbd_get_version(3),
1011 nbd_is_read_only(3), nbd_is_rotational(3), nbd_kill_subprocess(3),
1012 nbd_opt_abort(3), nbd_opt_go(3), nbd_opt_info(3), nbd_opt_list(3),
1013 nbd_opt_list_meta_context(3), nbd_opt_list_meta_context_queries(3),
1014 nbd_opt_set_meta_context(3), nbd_opt_set_meta_context_queries(3),
1015 nbd_opt_starttls(3), nbd_opt_structured_reply(3), nbd_poll(3),
1016 nbd_poll2(3), nbd_pread(3), nbd_pread_structured(3), nbd_pwrite(3),
1017 nbd_set_debug(3), nbd_set_debug_callback(3), nbd_set_export_name(3),
1018 nbd_set_full_info(3), nbd_set_handle_name(3),
1019 nbd_set_handshake_flags(3), nbd_set_opt_mode(3),
1020 nbd_set_pread_initialize(3), nbd_set_private_data(3),
1021 nbd_set_request_block_size(3), nbd_set_request_meta_context(3),
1022 nbd_set_request_structured_replies(3),
1023 nbd_set_socket_activation_name(3), nbd_set_strict_mode(3),
1024 nbd_set_tls(3), nbd_set_tls_certificates(3), nbd_set_tls_psk_file(3),
1025 nbd_set_tls_username(3), nbd_set_tls_verify_peer(3),
1026 nbd_set_uri_allow_local_file(3), nbd_set_uri_allow_tls(3),
1027 nbd_set_uri_allow_transports(3), nbd_shutdown(3),
1028 nbd_stats_bytes_received(3), nbd_stats_bytes_sent(3),
1029 nbd_stats_chunks_received(3), nbd_stats_chunks_sent(3),
1030 nbd_supports_tls(3), nbd_supports_uri(3), nbd_supports_vsock(3),
1031 nbd_trim(3), nbd_zero(3).
1032 Servers
1034 nbdkit(1), nbd-server(1), qemu-nbd(8).
1035 Encryption tools
1037 certtool(1), nbdkit-tls(1), psktool(1).
1038 Standards
1040 https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md,
1041 https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md.
1042 Release notes
1044 libnbd-release-notes-1.16(1), libnbd-release-notes-1.14(1),
1045 libnbd-release-notes-1.12(1), libnbd-release-notes-1.10(1),
1046 libnbd-release-notes-1.8(1), libnbd-release-notes-1.6(1),
1047 libnbd-release-notes-1.4(1), libnbd-release-notes-1.2(1).
1048 Other
1050 libnbd-security(3), nbdcopy(1), nbddump(1), nbdfuse(1), nbdinfo(1),
1051 nbdsh(1), nbdublk(1), qemu(1).
1052
1054 Eric Blake
1055 Richard W.M. Jones
1057
1059 Copyright Red Hat
1060
1062 This library is free software; you can redistribute it and/or modify it
1063 under the terms of the GNU Lesser General Public License as published
1064 by the Free Software Foundation; either version 2 of the License, or
1065 (at your option) any later version.
1066 This library is distributed in the hope that it will be useful, but
1068 WITHOUT ANY WARRANTY; without even the implied warranty of
1069 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
1070 Lesser General Public License for more details.
1071 You should have received a copy of the GNU Lesser General Public
1073 License along with this library; if not, write to the Free Software
1074 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
1075 02110-1301 USA
1076libnbd-1.16.5 2023-09-26 libnbd(3)