1libnbd(3) LIBNBD libnbd(3)
2
6 libnbd - network block device (NBD) client library in userspace
7
9 #include <libnbd.h>
10 struct nbd_handle *nbd;
12 char buf[512];
13 if ((nbd = nbd_create ()) == NULL ||
15 nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
16 nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
17 fprintf (stderr, "%s\n", nbd_get_error ());
18 nbd_close (nbd);
19 exit (EXIT_FAILURE);
20 }
21 nbd_close (nbd);
22 cc prog.c -o prog -lnbd
24 or:
25 cc prog.c -o prog `pkg-config libnbd --cflags --libs`
26
28 Network Block Device (NBD) is a network protocol for accessing block
29 devices over the network. Block devices are hard disks and things that
30 behave like hard disks such as disk images and virtual machines.
31 Libnbd is a client library for the NBD protocol which can access most
33 of the features of NBD while being simple to use and powerful.
34 This manual page gives an overview of libnbd, using C as an example,
36 but the library is available from other programming languages.
37 nbd_create(3), nbd_pread(3), etc.
39 Each manual page covers one function from the C API in detail.
40 There is a full list in section "C API" below.
41 libnbd-ocaml(3)
43 Using the API from OCaml.
44 libnbd-golang(3)
46 Using the API from Go.
47 libnbd-rust(3)
49 Using the API from Rust.
50 nbdsh(1)
52 Using the NBD shell (nbdsh) for command line and Python scripting.
53
55 To use the API at all you must first open a handle by calling
56 nbd_create(3) (or its equivalent in other languages):
57 struct nbd_handle *nbd;
59 nbd = nbd_create ();
61 This creates and returns a handle, which is associated with one
63 connection to an NBD server, initially not connected.
64 Each handle is a complex state machine which can be in states such as
66 created, connected to a remote server, handshaking, idle and ready to
67 issue commands, or busy sending or receiving commands.
68 Handles have a name used in debugging messages. The name is normally
70 generated ("nbd1", "nbd2" etc) but you can set a friendly name with
71 nbd_set_handle_name(3). Also there is a private field in the handle
72 for use by the application, see nbd_set_private_data(3).
73 When you have finished with the handle you must call nbd_close(3) which
75 closes the underlying socket (if necessary) and frees up all associated
76 resources.
77
79 There are two levels of API available. A simple high level synchronous
80 API lets you give the handle high level instructions like “connect to
81 the server”, “read a block”, “write a block”, etc. Each of these
82 functions will run to completion, blocking the current thread before
83 returning. A more complicated low level non-blocking asynchronous API
84 is also available where you can integrate with poll(2) or another main
85 loop.
86 You can freely mix the two APIs on the same handle. You can also call
88 APIs on a single handle from multiple threads. Single API calls on the
89 handle are atomic — they either take a lock on the handle while they
90 run or are careful to access handle fields atomically.
91 Libnbd does not create its own threads.
93
95 This is the simplest way to use the API, with the possible drawback
96 that each libnbd function blocks until it is finished.
97 Create a handle and connect to the server:
99 struct nbd_handle *nbd;
101 nbd = nbd_create ();
103 if (!nbd) {
104 fprintf (stderr, "%s\n", nbd_get_error ());
105 nbd_close (nbd);
106 exit (EXIT_FAILURE);
107 }
108 if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
109 fprintf (stderr, "%s\n", nbd_get_error ());
110 nbd_close (nbd);
111 exit (EXIT_FAILURE);
112 }
113 Read the first sector (512 bytes) from the NBD export:
115 char buf[512];
117 if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
119 fprintf (stderr, "%s\n", nbd_get_error ());
120 nbd_close (nbd);
121 exit (EXIT_FAILURE);
122 }
123 Close the handle:
125 nbd_close (nbd);
127 You can call the high level API from multiple threads, but each libnbd
129 API call takes a lock on the handle and so commands will not run in
130 parallel.
131
133 The low level API is useful if you want to use libnbd in non-blocking
134 code; or if you want to issue commands in parallel from multiple
135 threads; or if you need more control especially over having multiple
136 commands in-flight on a single connection.
137 To use the low level API you will need to integrate with poll(2) or
139 another “main loop” such as the GLib main event loop.
140 Issuing asynchronous commands
142 Use the "nbd_aio_*" variants to issue commands asynchronously (without
143 waiting for the command to complete before returning). For example the
144 asynchronous variant of nbd_pread(3) is:
145 int64_t cookie;
147 cookie = nbd_aio_pread (nbd, buf, sizeof buf,
149 NBD_NULL_COMPLETION, 0);
150 if (cookie == -1) {
151 fprintf (stderr, "%s\n", nbd_get_error ());
152 nbd_close (nbd);
153 exit (EXIT_FAILURE);
154 }
155 There are several things to note here:
157 • This only starts the command. The command is (usually) still in
159 flight when the call returns success, where you must rely on
160 subsequent API calls for learning the final command outcome and
161 trigger any remaining callbacks. However, you must also be able to
162 handle the case where system load allows the state machine to
163 advance far enough to invoke callbacks before the asynchronous API
164 returns.
165 • A buffer ("buf") has been assigned to collect the result of the
167 read, but it is not guaranteed to be filled with data until the
168 command has completed (see examples below). The buffer must not be
169 freed until the command has finished running.
170 • You can issue multiple commands on the same handle at the same
172 time.
173 • A cookie is returned which identifies this command in subsequent
175 calls. The cookie is unique (per libnbd handle) and ≥ 1.
176 • You may register a function which is called when the command
178 completes, see "Completion callbacks" below. In this case we have
179 specified a null completion callback. If a completion callback is
180 specified, it will only be called if the asynchronous command was
181 successfully submitted (if the asynchronous API itself returns an
182 error, there is nothing further to be completed).
183 Socket and direction
185 Each libnbd handle has an associated socket (once it has started
186 connecting). You can read the file descriptor of the socket using:
187 int fd = nbd_aio_get_fd (nbd);
189 The socket is non-blocking. Between calls into libnbd it is in the
191 "would block" condition. You can find out if libnbd is expecting to
192 read or write from the socket next by calling:
193 int dir = nbd_aio_get_direction (nbd);
195 which returns one of "LIBNBD_AIO_DIRECTION_READ",
197 "LIBNBD_AIO_DIRECTION_WRITE" or "LIBNBD_AIO_DIRECTION_BOTH" (=
198 "READ|WRITE"). And so to set up the next call to poll(2) or other main
199 loop you must translate this to "POLLIN", "POLLOUT" or "POLLIN|POLLOUT"
200 (or whatever mechanism your main loop uses).
201 Notifying libnbd when an event happens
203 When you detect (eg. using poll(2)) that a read or write event has
204 happened on the socket, you must then tell libnbd about it. You have
205 to check the direction again (since it may have been changed by another
206 thread), and notify libnbd:
207 int r = 0;
209 dir = nbd_aio_get_direction (nbd);
211 if ((dir & LIBNBD_AIO_DIRECTION_READ) &&
213 a_read_event_occurred ())
214 r = nbd_aio_notify_read (nbd);
215 else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) &&
216 a_write_event_occurred ())
217 r = nbd_aio_notify_write (nbd);
218 if (r == -1) {
220 fprintf (stderr, "%s\n", nbd_get_error ());
221 // ...
222 }
223 The notify calls move the state machine along, reading and writing from
225 the socket possibly multiple times, until the socket would block again,
226 at which point they return control to the caller.
227 Simple implementation with nbd_poll(3)
229 In fact if you want to use poll(2) on a single handle, a simple
230 implementation has already been written called nbd_poll(3). It is also
231 useful to examine how this is implemented (lib/poll.c in the libnbd
232 source code) because that will tell you how to integrate libnbd with
233 more complex main loops.
234 Some examples of using nbd_poll(3) follow.
236 As with the high level API, it all starts by creating a handle:
238 struct nbd_handle *nbd;
240 nbd = nbd_create ();
242 if (nbd == NULL) {
243 fprintf (stderr, "%s\n", nbd_get_error ());
244 nbd_close (nbd);
245 exit (EXIT_FAILURE);
246 }
247 To connect to the server asynchronously, we start the connection using
249 nbd_aio_connect(3) and then enter our main loop to check for events
250 until the connection becomes ready:
251 int fd;
253 struct sockaddr_un addr;
254 socklen_t len;
255 /* some code to set up addr,
257 then ... */
258 if (nbd_aio_connect (nbd, &addr, len) == -1) {
259 fprintf (stderr, "%s\n", nbd_get_error ());
260 nbd_close (nbd);
261 exit (EXIT_FAILURE);
262 }
263 while (! nbd_aio_is_ready (nbd)) {
264 if (nbd_poll (nbd, -1) == -1) {
265 fprintf (stderr, "%s\n", nbd_get_error ());
266 nbd_close (nbd);
267 exit (EXIT_FAILURE);
268 }
269 }
270 To read data asynchronously, start an asynchronous read command, which
272 returns a 64 bit command cookie, and enter the main loop until the
273 command has completed:
274 int64_t cookie;
276 char buf[512];
277 cookie = nbd_aio_pread (nbd, buf, sizeof buf, offset,
279 NBD_NULL_COMPLETION, 0);
280 if (cookie == -1) {
281 fprintf (stderr, "%s\n", nbd_get_error ());
282 nbd_close (nbd);
283 exit (EXIT_FAILURE);
284 }
285 while (! nbd_aio_command_completed (nbd, cookie)) {
286 if (nbd_poll (nbd, -1) == -1) {
287 fprintf (stderr, "%s\n", nbd_get_error ());
288 nbd_close (nbd);
289 exit (EXIT_FAILURE);
290 }
291 }
292 For almost all high level synchronous calls (eg. nbd_pread(3)) there is
294 a low level asynchronous equivalent (eg. nbd_aio_pread(3)) for starting
295 a command.
296 glib2 integration
298 See
299 https://gitlab.com/nbdkit/libnbd/blob/master/examples/glib-main-loop.c
300 libev integration
302 See https://gitlab.com/nbdkit/libnbd/blob/master/examples/copy-libev.c
303
305 When any API call returns an error (-1 or "NULL" depending on the API),
306 an error message and sometimes an errno value are available. You can
307 retrieve the error message and/or errno of the most recently failed
308 call using nbd_get_error(3) and nbd_get_errno(3). For example:
309 if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
311 fprintf (stderr,
312 "failed to connect to remote server: %s (errno = %d)\n",
313 nbd_get_error (), nbd_get_errno ());
314 }
315 These functions use thread-local storage to return the most recent
317 error in the current thread. This is why you don't need to pass the
318 handle to these calls. They even work if nbd_create(3) returns "NULL"
319 when there is no handle at all.
320 For this reason you cannot call them from a different thread. You
322 should call them immediately after the failed API call, from the same
323 thread. Furthermore the error string returned by nbd_get_error(3) is
324 only valid until the next libnbd API call in the current thread, so if
325 you need to keep the string you must copy it (eg. using strdup(3)).
326 Errno
328 For some errors, a system call error number (see errno(3)) is
329 available. You can find the error number by calling nbd_get_errno(3).
330 It works the same way as nbd_get_error(3) with respect to threads.
331 Even when a call returns an error, nbd_get_errno(3) might return 0.
333 This does not mean there was no error. It means no additional errno
334 information is available for this error.
335 The error number is often the raw error returned by a system call that
337 failed.
338 It can also be used to indicate special conditions. The most common
340 cases are:
341 "EINVAL"
343 Invalid parameters or state for the current libnbd call. (This can
344 also indicate that requests are not aligned to "Block size
345 constraints").
346 "ENOTSUP"
348 The libnbd call is not available in this build of libnbd (eg. when
349 using a TLS API if the library was compiled without TLS support).
350 "ENOMEM"
352 The library ran out of memory while performing some operation.
353 "ERANGE"
355 A request is too large, for example if you try to read too many
356 bytes in a single nbd_pread(3) call.
357 "EFAULT"
359 A pointer parameter was "NULL" when it should be non-NULL. See the
360 section below.
361 Non-NULL parameters
363 Almost all libnbd functions when called from C take one or more pointer
364 parameters that must not be "NULL". For example, the handle parameter,
365 strings and buffers should usually not be "NULL".
366 If a "NULL" is passed as one of these parameters, libnbd attempts to
368 return an error with nbd_get_errno(3) returning "EFAULT".
369 However it may cause other compiler-related warnings and even undefined
371 behaviour, so you should try to avoid this programming mistake.
372
374 Libnbd can print lots of debugging messages, useful if you have a
375 problem with the library. Either enable debugging after creating the
376 handle:
377 nbd = nbd_create ();
379 nbd_set_debug (nbd, true);
380 or set the "LIBNBD_DEBUG=1" environment variable which will enable
382 debugging by default on all new handles.
383 Debugging messages are sent to stderr by default, but you can redirect
385 them to a logging system using nbd_set_debug_callback(3).
386
388 There are several ways to connect to NBD servers, and you can even run
389 a server from libnbd. Normally you would connect to a server which is
390 already running, over a local Unix domain socket or a remote TCP
391 connection. The high level API calls are:
392 nbd_connect_unix (nbd, "socket");
394 nbd_connect_tcp (nbd, "localhost", "nbd");
395 For nbd_connect_tcp(3) the third parameter is the port name or number,
397 which can either be a name from /etc/services or the port number as a
398 string (eg. "10809").
399 Connecting to an NBD URI
401 libnbd supports the NBD URI specification. The format of URIs is
402 documented in nbd_connect_uri(3).
403 You can connect to a URI as in these examples (using the high level
405 API):
406 nbd_connect_uri (nbd, "nbd://example.com/");
408 nbd_connect_uri (nbd, "nbds+unix:///export?socket=/tmp/nbd.sock");
410 This feature is implemented by calling other libnbd APIs to set up the
412 export name, TLS parameters, and finally connect over a Unix domain
413 socket or TCP.
414 URI support is an optional feature of the library, requiring libxml2 at
416 compile time. The nbd_connect_uri(3) and nbd_aio_connect_uri(3) calls
417 will raise an error (with nbd_get_errno(3) returning "ENOTSUP") if it
418 was not built with this feature, and you can also test for it
419 explicitly using nbd_supports_uri(3).
420 Connecting to a subprocess
422 Some NBD servers — notably nbdkit(1) with the -s parameter, and
423 nbd-server(1) with the port parameter set to 0 — can also accept a
424 single NBD connection on stdin/stdout. You can run these servers as a
425 subprocess of your main program using nbd_connect_command(3). This
426 example creates a 1G writable RAM disk:
427 char *argv[] = { "nbdkit", "-s", "--exit-with-parent",
429 "memory", "1G", NULL };
430 nbd_connect_command (nbd, argv);
431 When the handle is closed the nbdkit subprocess is killed, which in
433 this case means the RAM disk is discarded, so this is useful for
434 testing.
435 Connecting to a subprocess using systemd socket activation
437 Some NBD servers — notably nbdkit(1) and qemu-nbd(1) — support systemd
438 socket activation allowing libnbd to pass a socket to the subprocess.
439 This works very similarly to nbd_connect_command(3) described above,
440 but you must use nbd_connect_systemd_socket_activation(3) instead.
441 Connecting to any socket
443 If none of the other nbd_connect* methods are suitable you can create a
444 connected socket yourself and pass it to nbd_connect_socket(3).
445 One use for this is in fuzzing where we use socketpair(2) to create the
447 socket, then fork, then have the test harness in the child process
448 connected to libnbd over the socket pair (see:
449 https://gitlab.com/nbdkit/libnbd/-/blob/master/fuzzing/libnbd-fuzz-wrapper.c).
450 Another use is to connect libnbd to an address family that it does not
452 support natively, such as XDP or IB.
453
455 By default, when beginning a connection, libnbd will handle all
456 negotiation with the server, using only the configuration (eg.
457 nbd_set_export_name(3) or nbd_add_meta_context(3)) that was requested
458 before the connection attempt; this phase continues until
459 nbd_aio_is_connecting(3) no longer returns true, at which point, either
460 data commands are ready to use or else the connection has failed with
461 an error.
462 But there are scenarios in which it is useful to also control the
464 handshaking commands sent during negotiation, such as asking the server
465 for a list of available exports prior to selecting which one to use.
466 This is done by calling nbd_set_opt_mode(3) before connecting; then
467 after requesting a connection, the state machine will pause at
468 nbd_aio_is_negotiating(3) at any point that the user can decide which
469 handshake command to send next. Note that the negotiation state is
470 only reachable from newstyle servers; older servers cannot negotiate
471 and will progress all the way to the ready state.
472 When the negotiating state is reached, you can initiate option commands
474 such as nbd_opt_list(3) or their asynchronous equivalents, as well as
475 alter configuration such as export name that previously had to be set
476 before connection. Since the NBD protocol does not allow parallel
477 negotiating commands, no cookie is involved, and you can track
478 completion of each command when the state is no longer
479 nbd_aio_is_connecting(3). If nbd_opt_go(3) fails but the connection is
480 still live, you will be back in negotiation state, where you can
481 request a different export name and try again. Exiting the negotiation
482 state is only possible with a successful nbd_opt_go(3) which moves to
483 the data phase, or nbd_opt_abort(3) which performs a clean shutdown of
484 the connection by skipping the data phase.
485
487 It is possible for NBD servers to serve different content on different
488 “exports”. For this you must pass the right export name to the server.
489 Call this API before connecting:
490 nbd_set_export_name (nbd, "export");
492 Note that there are some servers (like nbdkit(1) ≤ 1.14) which ignore
494 this, and other servers (like qemu-nbd(8)) which require it to be set
495 correctly but cannot serve different content.
496 These APIs are also available after a successful nbd_opt_info(3) during
498 the negotiation phase, if you used nbd_set_opt_mode(3) prior to
499 connecting.
500 Flag calls
502 After connecting the server will send back a set of flags describing
503 the export, such as whether it is writable and if it can support flush
504 to permanent storage. These flags can be accessed from libnbd using
505 APIs such as:
506 int is_read_only = nbd_is_read_only (nbd);
508 int can_flush = nbd_can_flush (nbd);
509 Flag calls are: nbd_can_block_status_payload(3), nbd_can_cache(3),
511 nbd_can_df(3), nbd_can_fast_zero(3), nbd_can_flush(3), nbd_can_fua(3),
512 nbd_can_meta_context(3), nbd_can_multi_conn(3), nbd_can_trim(3),
513 nbd_can_zero(3), nbd_is_read_only(3), nbd_is_rotational(3).
514 Size of the export
516 To get the size of the export in bytes, use nbd_get_size(3):
517 int64_t size = nbd_get_size (nbd);
519 Block size constraints
521 Some NBD servers cannot handle requests at any byte boundary. They
522 might, for example, require all requests to be aligned to 512 byte
523 sectors.
524 Also some servers advertise a preferred block size. This is not a
526 requirement, but is the minimum block size that can be accessed
527 efficiently (usually without triggering expensive read-modify-write
528 cycles inside the server).
529 These are referred to as block size constraints and can be queried by
531 calling nbd_get_block_size(3). Pay attention in particular to the
532 "LIBNBD_SIZE_MINIMUM" constraint as some servers will fail requests
533 which are smaller or not aligned to this block size with "EINVAL"
534 ("Invalid argument") errors.
535
537 You can read and write data from the NBD server using nbd_pread(3) and
538 nbd_pwrite(3) or their asynchronous equivalents.
539 All data commands support a "flags" argument (mandatory in C, but
541 optional in languages where it can default to 0). For convenience, the
542 constant "LIBNBD_CMD_FLAG_MASK" is defined with the set of flags
543 currently recognized by libnbd, where future NBD protocol extensions
544 may result in additional flags being supported; but in general,
545 specific data commands only accept a subset of known flags.
546 Libnbd defaults to performing some client-side sanity checking in each
548 of its data commands; for example, attempts to write to a server that
549 has advertised a read-only connection are rejected. It is possible to
550 override aspects of this checking by using nbd_set_strict_mode(3).
551 Some servers also support:
553 trim/discard
555 If nbd_can_trim(3) returns true, nbd_trim(3) can be used to “punch
556 holes” in the backing storage of the disk on the server. Normally
557 (although not in every case) the holes read back as zeroes but take
558 up no space.
559 zeroing
561 If nbd_can_zero(3) returns true, nbd_zero(3) can be used to
562 efficiently zero parts of the disk without having to send large
563 amounts of zero bytes over the network (as would be necessary if
564 using nbd_pwrite(3)).
565 This is slightly different from trimming because the backing
567 storage is still allocated. For some storage types this can make
568 future writes more efficient and/or less likely to fail because of
569 out of space errors.
570 flushing
572 Some servers can commit data to permanent storage and tell you that
573 this has happened reliably. There are two export flags associated
574 with this: nbd_can_flush(3) and nbd_can_fua(3).
575 The nbd_flush(3) call (available if nbd_can_flush(3) returns true)
577 flushes all pending writes to disk and does not complete until that
578 operation has finished. It is similar to using sync(2) on POSIX
579 systems.
580 A more efficient way to achieve this is to set the flag
582 "LIBNBD_CMD_FLAG_FUA" on write-like calls (like write, trim and
583 zero). This flag means the call will not complete until committed
584 to permanent storage, but it does not involve flushing the entire
585 disk.
586 prefetching
588 Some servers can prefetch data, making subsequent reads faster.
589 The nbd_cache(3) call (available if nbd_can_cache(3) returns true)
590 is used to prefetch.
591 block status
593 Some servers are able to provide information about the various
594 extents within the image, via the notion of one or more meta
595 contexts. The most common meta context is "base:allocation"
596 (available in libnbd.h as "LIBNBD_CONTEXT_BASE_ALLOCATION"), which
597 can be used to learn which portions of a file are allocated or read
598 as zero. Other contexts may be available; for example, qemu-nbd(8)
599 can expose a meta context "qemu:dirty-bitmap:NAME" for tracking
600 which portions of a file are tracked by a qcow2 dirty bitmap.
601 In order to utilize block status, the client must call
603 nbd_add_meta_context(3) prior to connecting, for each meta context
604 in which it is interested, then check nbd_can_meta_context(3) after
605 connection to see which contexts the server actually supports. If
606 a context is supported, the client can then use
607 nbd_block_status_64(3) with a callback function that will receive
608 an array of structs describing consecutive extents within a
609 context. Each struct gives the length of the extent, then a
610 bitmask description of that extent (for the "base:allocation"
611 context, the bitmask may include "LIBNBD_STATE_HOLE" for
612 unallocated portions of the file, and/or "LIBNBD_STATE_ZERO" for
613 portions of the file known to read as zero).
614 There is a full example of requesting meta context and using block
616 status available at
617 https://gitlab.com/nbdkit/libnbd/blob/master/interop/dirty-bitmap.c
618
620 Issuing multiple in-flight requests
621 NBD servers which properly implement the specification can handle
622 multiple data requests in flight over the same connection at the same
623 time. Libnbd supports this when using the low level API.
624 To use it you simply issue more requests as needed (eg. using calls
626 like nbd_aio_pread(3), nbd_aio_pwrite(3)) without waiting for previous
627 commands to complete. You need to be careful that requests in flight
628 do not overlap with disk offsets of other write-like commands in flight
629 — an overlapping read may see indeterminate data, and an overlapping
630 write may even cause disk corruption where the resulting disk contents
631 do not match either of the two writes.
632 Each request is identified by a unique 64 bit cookie (assigned by
634 libnbd), allowing libnbd and callers to match replies to requests.
635 Replies may arrive out of order. A request that is rejected client-
636 side for failing a sanity check (such as attempting to write to a read-
637 only server, see nbd_set_strict_mode(3)) will fail rather than
638 returning a cookie, although closure cleanup is still performed.
639 Although in theory you can have an indefinite number of requests in
641 flight at the same time, in practice it's a good idea to limit them to
642 some number. Libnbd will queue commands in the handle even if it
643 cannot write them to the server, so this limit is largely to prevent a
644 backlog of commands from consuming too much memory. It is suggested to
645 start with a limit of 64 requests in flight (per NBD connection), and
646 measure how adjusting the limit up and down affects performance for
647 your local configuration.
648 There is a full example using multiple in-flight requests available at
650 https://gitlab.com/nbdkit/libnbd/blob/master/examples/threaded-reads-and-writes.c
651 Multi-conn
653 Some NBD servers advertise “multi-conn” which means that it is safe to
654 make multiple connections to the server and load-balance commands
655 across all of the connections.
656 To do this you should open a single connection first and test for this
658 feature using nbd_can_multi_conn(3). Without error handling it would
659 look like this:
660 struct nbd_handle *nbd[4];
662 size_t i;
663 bool supports_multi_conn;
664 nbd[0] = nbd_create ();
666 nbd_connect_tcp (nbd[0], "server", "10809");
667 supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;
668 If multi-conn is supported then you can open further connections:
670 if (supports_multi_conn) {
672 for (i = 1; i <= 3; ++i) {
673 nbd[i] = nbd_create ();
674 nbd_connect_tcp (nbd[i], "server", "10809");
675 }
676 }
677 If you are issuing multiple in-flight requests (see above) and limiting
679 the number, then the limit should be applied to each individual NBD
680 connection.
681
683 The NBD protocol and libnbd supports TLS (sometimes incorrectly called
684 “SSL”) for encryption of the data stream and authentication of clients
685 and servers. Libnbd defaults to TLS disabled for maximum
686 interoperability. To enable it on a handle you must call
687 nbd_set_tls(3) before connecting.
688 To allow TLS, but fall back to unencrypted:
690 nbd_set_tls (nbd, LIBNBD_TLS_ALLOW);
692 Use nbd_get_tls_negotiated(3) to find out if TLS negotiation was
694 successful. Avoid "LIBNBD_TLS_ALLOW" if man-in-the-middle attacks are
695 a concern.
696 The most secure mode is to require TLS and fail to connect if the
698 server does not support it:
699 nbd_set_tls (nbd, LIBNBD_TLS_REQUIRE);
701 It may also be necessary to verify that the server’s identity is
703 correct. For some servers it may be necessary to verify to the server
704 that the client is permitted to connect. This can be done using either
705 X.509 certificates, or TLS Pre-Shared Keys (PSK). Certificates are
706 more secure. PSK is far more convenient, but you must have an existing
707 secure channel to distribute the keys.
708 Setting up X.509 using system certificate authorities (CAs)
710 This is the default if you don’t call any other "nbd_set_tls_*"
711 functions. In this case the server must have a public (eg. HTTPS)
712 certificate which can be verified against the CAs registered on your
713 system (eg. under /etc/pki).
714 To disable server name verification — which opens you up to a potential
716 Man-In-The-Middle (MITM) attack — use:
717 nbd_set_tls_verify_peer (nbd, false);
719 Setting up an X.509 certificate authority (CA)
721 You can set up your own CA and register clients and servers with it,
722 issuing client and server certificates which will reliably authenticate
723 your clients and servers to each other.
724 Doing this is described in detail in the nbdkit-tls(1) manual. The
726 only differences for libnbd are:
727 • Non-root certificates must be placed in "$HOME/.pki/libnbd/" or
729 "$HOME/.config/pki/libnbd/"
730 • Libnbd reads client-cert.pem and client-key.pem (instead of
732 server-cert.pem and server-key.pem).
733 Once you have set up the directory containing the certificates, call:
735 nbd_set_tls_certificates (nbd, "/path/to/directory");
737 Setting up Pre-Shared Keys (PSK)
739 TLS Pre-Shared Keys are a much more convenient method of setting up
740 TLS, and more appropriate for NBD, but you should have an existing
741 secure method available to distribute the keys. They are therefore
742 ideal if you want to set up an NBD service as an adjunct to an existing
743 secure REST API.
744 Use psktool(1) to create a file of "username:key" pairs:
746 psktool -u username -p keys.psk
748 and pass this path to libnbd:
750 nbd_set_tls_psk_file (nbd, "keys.psk");
752 If necessary you may need to set the client username (otherwise libnbd
754 will use your login name):
755 nbd_set_tls_username (nbd, "username");
757
759 Some libnbd calls take callbacks (eg. nbd_set_debug_callback(3),
760 nbd_aio_pread(3)). Libnbd can call these functions while processing.
761 In the C API these libnbd calls take a structure which contains the
763 function pointer and an optional opaque "void *user_data" pointer:
764 nbd_aio_pread (nbd, buf, sizeof buf, offset,
766 (nbd_completion_callback) { .callback = my_fn,
767 .user_data = my_data },
768 0);
769 For optional callbacks, if you don't want the callback, either set
771 ".callback" to "NULL" or use the equivalent macros (such as
772 "NBD_NULL_COMPLETION") defined in "libnbd.h":
773 nbd_aio_pread (nbd, buf, sizeof buf, offset,
775 NBD_NULL_COMPLETION, 0);
776 From other languages the structure and opaque pointer are not needed
778 because you can use closures to achieve the same effect.
779 Callback lifetimes
781 You can associate an optional free function with callbacks. Libnbd
782 will call this function when the callback will not be called again by
783 libnbd, including in the case where the API fails.
784 This can be used to free associated "user_data". For example:
786 void *my_data = malloc (...);
788 nbd_aio_pread_structured (nbd, buf, sizeof buf, offset,
790 (nbd_chunk_callback) { .callback = my_fn,
791 .user_data = my_data,
792 .free = free },
793 NBD_NULL_COMPLETION,
794 0);
795 will call free(3) once on "my_data" after the point where it is known
797 that the "chunk.callback = my_fn" function can no longer be called,
798 regardless of how many times "my_fn" was actually called. If both a
799 mid-command and completion callback are supplied, the functions will be
800 reached in this order: mid-function callbacks, completion callback,
801 mid-function free, and finally completion free.
802 The free function is only accessible in the C API as it is not needed
804 in garbage collected programming languages.
805 Callbacks with ".callback=NULL" and ".free!=NULL"
807 It is possible to register a callback like this:
808 ...
810 (nbd_completion_callback) { .callback = NULL,
811 .user_data = my_data,
812 .free = free },
813 ...
814 The meaning of this is that the callback is never called, but the free
816 function is still called after the last time the callback would have
817 been called. This is useful for applying generic freeing actions when
818 asynchronous commands are retired.
819 Callbacks and locking
821 The callbacks are invoked at a point where the libnbd lock is held,
822 typically during a call to "nbd_aio_notify_read",
823 "nbd_aio_notify_write", "nbd_aio_poll", or other call that can advance
824 libnbd's state machine. Depending on system load, it is even possible
825 for a callback to be reached before completion of the "nbd_aio_*" call
826 that specified the callback. As such, it is unsafe for the callback to
827 call any "nbd_*" APIs on the same nbd object, as it would cause
828 deadlock.
829 Completion callbacks
831 All of the asychronous commands have an optional completion callback
832 function that is used if the call to the asynchronous API reports
833 success. The completion callback is invoked when the submitted command
834 is eventually marked complete, after any mid-command callbacks have
835 finished, and before any free functions. The completion callback is
836 not reached if the asynchronous API itself fails, while free callbacks
837 are reached regardless of the result of the initial asynchronous API.
838 When the completion callback returns 1, the command is automatically
840 retired (there is no need to call nbd_aio_command_completed(3)); for
841 any other return value, the command still needs to be manually retired
842 (otherwise, the command will tie up resources until nbd_close(3) is
843 eventually reached).
844 Callbacks with "int *error" parameter
846 Some of the high-level commands (nbd_pread_structured(3),
847 nbd_block_status_64(3)) involve the use of a callback function invoked
848 by the state machine at appropriate points in the server's reply before
849 the overall command is complete. These callback functions, along with
850 all of the completion callbacks, include a parameter "error" which is a
851 pointer containing the value of any error detected so far. If a
852 callback function fails and wants to change the resulting error of the
853 overall command visible later in the API sequence, it should assign
854 back into "error" and return -1 in the C API. Assignments into "error"
855 are ignored for any other return value; similarly, assigning 0 into
856 "error" does not have an effect. In other language bindings, reporting
857 callback errors is generally done by raising an exception rather than
858 by return value.
859 Note that a mid-command callback might never be reached, such as if
861 libnbd detects that the command was invalid to send (see
862 nbd_set_strict_mode(3)) or if the server reports a failure that
863 concludes the command. It is safe for a mid-command callback to ignore
864 non-zero "error": all the other parameters to the mid-command callback
865 will still be valid (corresponding to the current portion of the
866 server's reply), and the overall command will still fail (at the
867 completion callback or nbd_aio_command_completed(3) for an asynchronous
868 command, or as the result of the overall synchronous command).
869 Returing -1 from a mid-command callback does not prevent that callback
870 from being reached again, if the server sends more mid-command replies
871 that warrant another use of that callback. A mid-command callback may
872 be reached more times than expected if the server is non-compliant.
873 On the other hand, if a completion callback is supplied (only possible
875 with asynchronous commands), it will not be called if the initial API
876 call fails (such as attempting an asynchronous command in the wrong
877 state - there is nothing to be completed since the command was not
878 queued), but will otherwise be called exactly once, and the completion
879 callback must not ignore the value pointed to by "error". In
880 particular, the content of a buffer passed to nbd_aio_pread(3) or
881 nbd_aio_pread_structured(3) is undefined if *error is non-zero on entry
882 to the completion callback. It is recommended that if you choose to
883 use automatic command retirement (where the completion callback returns
884 1 to avoid needing to call nbd_aio_command_completed(3) later), your
885 completion function should return 1 on all control paths, even when
886 handling errors (note that with automatic retirement, assigning into
887 "error" is pointless as there is no later API to see that value).
888
890 Libnbd tracks several statistics counters, useful for tracking how much
891 traffic was sent over the connection. The counters track the number of
892 plaintext bytes sent and received by the NBD protocol (not necessarily
893 the number of bytes sent over the socket, particularly when TLS is
894 enabled), as well as the number of protocol chunks (a group of bytes
895 delineated by a magic number, and not the same as the number of TCP
896 packets).
897 printf ("bytes: sent=%" PRIu64 " received=%" PRIu64,
899 nbd_stats_bytes_sent (nbd), nbd_stats_bytes_received (nbd));
900 printf ("chunks: sent=%" PRIu64 " received=%" PRIu64,
901 nbd_stats_chunks_sent (nbd), nbd_stats_chunks_received (nbd));
902
904 Libnbd does not install signal handlers. It attempts to disable
905 "SIGPIPE" when writing to the NBD socket using the "MSG_NOSIGNAL" flag
906 of send(2), or the "SO_NOSIGPIPE" socket option, on platforms that
907 support those.
908 On some old Linux or newer non-Linux platforms the main program may
910 wish to register a signal handler to ignore SIGPIPE:
911 signal (SIGPIPE, SIG_IGN);
913
915 On most systems, C programs that use libnbd can be compiled like this:
916 cc prog.c -o prog -lnbd
918 To detect if the libnbd library and header file is installed, the
920 preferred method is to use pkg-config(1) or pkgconf(1):
921 pkg-config libnbd --exists || fail libnbd is required
923 In case the library or header file are not installed in the usual
925 system locations, you can compile your program like this, using pkg-
926 config to detect the proper location of libnbd:
927 cc prog.c -o prog `pkg-config libnbd --cflags --libs`
929 To compile an external project against a built copy of the libnbd
931 source tree which hasn't been installed, see the ./run script.
932 Autoconf projects
934 External projects which use autoconf and need to check if libnbd is
935 installed should use the "PKG_CHECK_MODULES" macro in configure.ac like
936 this:
937 PKG_CHECK_MODULES([LIBNBD], [libnbd])
939 This will define "@LIBNBD_CFLAGS@" and "@LIBNBD_LIBS@" which you will
941 need to add to your Makefile.am.
942 CMake projects
944 For CMake projects use:
945 find_package(PkgConfig REQUIRED)
947 pkg_check_modules(LIBNBD REQUIRED libnbd)
948 target_link_libraries(prog ${LIBNBD_LIBRARIES})
949 target_include_directories(prog PUBLIC ${LIBNBD_INCLUDE_DIRS})
950 target_compile_options(prog PUBLIC ${LIBNBD_CFLAGS_OTHER})
951 Meson projects
953 For meson projects use:
954 nbd_dep = dependency('libnbd')
956 executable('prog', 'prog.c', dependencies : [nbd_dep])
957
959 "HOME"
960 Used in some situations to find TLS certificates. See
961 nbd_set_tls_certificates(3).
962 "LIBNBD_DEBUG"
964 If this is set to the exact string 1 when the handle is created
965 then debugging is enabled. See "DEBUGGING MESSAGES" above.
966 "LOGNAME"
968 The default TLS username. See nbd_set_tls_username(3).
969
971 C API
972 nbd_add_meta_context(3), nbd_aio_block_status(3),
973 nbd_aio_block_status_64(3), nbd_aio_block_status_filter(3),
974 nbd_aio_cache(3), nbd_aio_command_completed(3), nbd_aio_connect(3),
975 nbd_aio_connect_command(3), nbd_aio_connect_socket(3),
976 nbd_aio_connect_systemd_socket_activation(3), nbd_aio_connect_tcp(3),
977 nbd_aio_connect_unix(3), nbd_aio_connect_uri(3),
978 nbd_aio_connect_vsock(3), nbd_aio_disconnect(3), nbd_aio_flush(3),
979 nbd_aio_get_direction(3), nbd_aio_get_fd(3), nbd_aio_in_flight(3),
980 nbd_aio_is_closed(3), nbd_aio_is_connecting(3), nbd_aio_is_created(3),
981 nbd_aio_is_dead(3), nbd_aio_is_negotiating(3),
982 nbd_aio_is_processing(3), nbd_aio_is_ready(3), nbd_aio_notify_read(3),
983 nbd_aio_notify_write(3), nbd_aio_opt_abort(3),
984 nbd_aio_opt_extended_headers(3), nbd_aio_opt_go(3),
985 nbd_aio_opt_info(3), nbd_aio_opt_list(3),
986 nbd_aio_opt_list_meta_context(3),
987 nbd_aio_opt_list_meta_context_queries(3),
988 nbd_aio_opt_set_meta_context(3),
989 nbd_aio_opt_set_meta_context_queries(3), nbd_aio_opt_starttls(3),
990 nbd_aio_opt_structured_reply(3), nbd_aio_peek_command_completed(3),
991 nbd_aio_pread(3), nbd_aio_pread_structured(3), nbd_aio_pwrite(3),
992 nbd_aio_trim(3), nbd_aio_zero(3), nbd_block_status(3),
993 nbd_block_status_64(3), nbd_block_status_filter(3), nbd_cache(3),
994 nbd_can_block_status_payload(3), nbd_can_cache(3), nbd_can_df(3),
995 nbd_can_fast_zero(3), nbd_can_flush(3), nbd_can_fua(3),
996 nbd_can_meta_context(3), nbd_can_multi_conn(3), nbd_can_trim(3),
997 nbd_can_zero(3), nbd_clear_debug_callback(3),
998 nbd_clear_meta_contexts(3), nbd_close(3), nbd_connect_command(3),
999 nbd_connect_socket(3), nbd_connect_systemd_socket_activation(3),
1000 nbd_connect_tcp(3), nbd_connect_unix(3), nbd_connect_uri(3),
1001 nbd_connect_vsock(3), nbd_connection_state(3), nbd_create(3),
1002 nbd_flush(3), nbd_get_block_size(3), nbd_get_canonical_export_name(3),
1003 nbd_get_debug(3), nbd_get_errno(3), nbd_get_error(3),
1004 nbd_get_export_description(3), nbd_get_export_name(3),
1005 nbd_get_extended_headers_negotiated(3), nbd_get_full_info(3),
1006 nbd_get_handle_name(3), nbd_get_handshake_flags(3),
1007 nbd_get_meta_context(3), nbd_get_nr_meta_contexts(3),
1008 nbd_get_opt_mode(3), nbd_get_package_name(3),
1009 nbd_get_pread_initialize(3), nbd_get_private_data(3),
1010 nbd_get_protocol(3), nbd_get_request_block_size(3),
1011 nbd_get_request_extended_headers(3), nbd_get_request_meta_context(3),
1012 nbd_get_request_structured_replies(3), nbd_get_size(3),
1013 nbd_get_socket_activation_name(3), nbd_get_strict_mode(3),
1014 nbd_get_structured_replies_negotiated(3), nbd_get_tls(3),
1015 nbd_get_tls_negotiated(3), nbd_get_tls_username(3),
1016 nbd_get_tls_verify_peer(3), nbd_get_uri(3), nbd_get_version(3),
1017 nbd_is_read_only(3), nbd_is_rotational(3), nbd_kill_subprocess(3),
1018 nbd_opt_abort(3), nbd_opt_extended_headers(3), nbd_opt_go(3),
1019 nbd_opt_info(3), nbd_opt_list(3), nbd_opt_list_meta_context(3),
1020 nbd_opt_list_meta_context_queries(3), nbd_opt_set_meta_context(3),
1021 nbd_opt_set_meta_context_queries(3), nbd_opt_starttls(3),
1022 nbd_opt_structured_reply(3), nbd_poll(3), nbd_poll2(3), nbd_pread(3),
1023 nbd_pread_structured(3), nbd_pwrite(3), nbd_set_debug(3),
1024 nbd_set_debug_callback(3), nbd_set_export_name(3),
1025 nbd_set_full_info(3), nbd_set_handle_name(3),
1026 nbd_set_handshake_flags(3), nbd_set_opt_mode(3),
1027 nbd_set_pread_initialize(3), nbd_set_private_data(3),
1028 nbd_set_request_block_size(3), nbd_set_request_extended_headers(3),
1029 nbd_set_request_meta_context(3), nbd_set_request_structured_replies(3),
1030 nbd_set_socket_activation_name(3), nbd_set_strict_mode(3),
1031 nbd_set_tls(3), nbd_set_tls_certificates(3), nbd_set_tls_psk_file(3),
1032 nbd_set_tls_username(3), nbd_set_tls_verify_peer(3),
1033 nbd_set_uri_allow_local_file(3), nbd_set_uri_allow_tls(3),
1034 nbd_set_uri_allow_transports(3), nbd_shutdown(3),
1035 nbd_stats_bytes_received(3), nbd_stats_bytes_sent(3),
1036 nbd_stats_chunks_received(3), nbd_stats_chunks_sent(3),
1037 nbd_supports_tls(3), nbd_supports_uri(3), nbd_supports_vsock(3),
1038 nbd_trim(3), nbd_zero(3).
1039 Servers
1041 nbdkit(1), nbd-server(1), qemu-nbd(8).
1042 Encryption tools
1044 certtool(1), nbdkit-tls(1), psktool(1).
1045 Standards
1047 https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md,
1048 https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md.
1049 Release notes
1051 libnbd-release-notes-1.18(1), libnbd-release-notes-1.16(1),
1052 libnbd-release-notes-1.14(1), libnbd-release-notes-1.12(1),
1053 libnbd-release-notes-1.10(1), libnbd-release-notes-1.8(1),
1054 libnbd-release-notes-1.6(1), libnbd-release-notes-1.4(1),
1055 libnbd-release-notes-1.2(1).
1056 Other
1058 libnbd-security(3), nbdcopy(1), nbddump(1), nbdfuse(1), nbdinfo(1),
1059 nbdsh(1), nbdublk(1), qemu(1).
1060
1062 Eric Blake
1063 Richard W.M. Jones
1065
1067 Copyright Red Hat
1068
1070 This library is free software; you can redistribute it and/or modify it
1071 under the terms of the GNU Lesser General Public License as published
1072 by the Free Software Foundation; either version 2 of the License, or
1073 (at your option) any later version.
1074 This library is distributed in the hope that it will be useful, but
1076 WITHOUT ANY WARRANTY; without even the implied warranty of
1077 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
1078 Lesser General Public License for more details.
1079 You should have received a copy of the GNU Lesser General Public
1081 License along with this library; if not, write to the Free Software
1082 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
1083 02110-1301 USA
1084libnbd-1.18.1 2023-10-31 libnbd(3)