1nbdkit-filter(3)                    NBDKIT                    nbdkit-filter(3)
2
3
4

NAME

6       nbdkit-filter - how to write nbdkit filters
7

SYNOPSIS

9        #include <nbdkit-filter.h>
10
11        static int
12        myfilter_config (nbdkit_next_config *next, void *nxdata,
13                         const char *key, const char *value)
14        {
15          if (strcmp (key, "myparameter") == 0) {
16            // ...
17            return 0;
18          }
19          else {
20            // pass through to next filter or plugin
21            return next (nxdata, key, value);
22          }
23        }
24
25        static struct nbdkit_filter filter = {
26          .name              = "filter",
27          .config            = myfilter_config,
28          /* etc */
29        };
30
31        NBDKIT_REGISTER_FILTER(filter)
32
33       When this has been compiled to a shared library, do:
34
35        nbdkit [--args ...] --filter=./myfilter.so plugin [key=value ...]
36
37       When debugging, use the -fv options:
38
39        nbdkit -fv --filter=./myfilter.so plugin [key=value ...]
40

DESCRIPTION

42       One or more nbdkit filters can be placed in front of an nbdkit plugin
43       to modify the behaviour of the plugin.  This manual page describes how
44       to create an nbdkit filter.
45
46       Filters can be used for example to limit requests to an offset/limit,
47       add copy-on-write support, or inject delays or errors (for testing).
48
49       Different filters can be stacked:
50
51            NBD     ┌─────────┐    ┌─────────┐          ┌────────┐
52         client ───▶│ filter1 │───▶│ filter2 │── ─ ─ ──▶│ plugin │
53        request     └─────────┘    └─────────┘          └────────┘
54
55       Each filter intercepts plugin functions (see nbdkit-plugin(3)) and can
56       call the next filter or plugin in the chain, modifying parameters,
57       calling before the filter function, in the middle or after.  Filters
58       may even short-cut the chain.  As an example, to process its own
59       parameters the filter can intercept the ".config" method:
60
61        static int
62        myfilter_config (nbdkit_next_config *next, void *nxdata,
63                         const char *key, const char *value)
64        {
65          if (strcmp (key, "myparameter") == 0) {
66            // ...
67            // here you would handle this key, value
68            // ...
69            return 0;
70          }
71          else {
72            // pass through to next filter or plugin
73            return next (nxdata, key, value);
74          }
75        }
76
77        static struct nbdkit_filter filter = {
78          // ...
79          .config            = myfilter_config,
80          // ...
81        };
82
83       The call to "next (nxdata, ...)" calls the ".config" method of the next
84       filter or plugin in the chain.  In the example above any instances of
85       "myparameter=..." on the command line would not be seen by the plugin.
86
87       To see example filters:
88       https://gitlab.com/nbdkit/nbdkit/tree/master/filters
89
90       Filters must be written in C.
91
92       Unlike plugins, where we provide a stable ABI guarantee that permits
93       operation across version differences, filters can only be run with the
94       same version of nbdkit that they were compiled with.  The reason for
95       this is two-fold: the filter API includes access to struct
96       nbdkit_next_ops that is likely to change if new callbacks are added
97       (old nbdkit cannot safely run new filters that access new methods); and
98       if we added new methods then an old filter would not see them and so
99       they would be passed unmodified through the filter, and in some cases
100       that leads to data corruption (new nbdkit cannot safely run old filters
101       unaware of new methods).  Therefore, unlike plugins, you should not
102       expect to distribute filters separately from nbdkit.
103

"#include <nbdkit-filter.h>"

105       All filters should start by including this header file.
106

"struct nbdkit_filter"

108       All filters must define and register one "struct nbdkit_filter", which
109       contains the name of the filter and pointers to plugin methods that the
110       filter wants to intercept.
111
112        static struct nbdkit_filter filter = {
113          .name              = "filter",
114          .longname          = "My Filter",
115          .description       = "This is my great filter for nbdkit",
116          .config            = myfilter_config,
117          /* etc */
118        };
119
120        NBDKIT_REGISTER_FILTER(filter)
121
122       The ".name" field is the name of the filter.  This is the only field
123       which is required.
124

NEXT PLUGIN

126       nbdkit-filter.h defines some function types ("nbdkit_next_config",
127       "nbdkit_next_config_complete", "nbdkit_next_preconnect",
128       "nbdkit_next_list_exports", "nbdkit_next_default_export",
129       "nbdkit_next_open") and a structure called "struct nbdkit_next_ops".
130       These abstract the next plugin or filter in the chain.  There is also
131       an opaque pointer "backend", "context" or "nxdata" which must be passed
132       along when calling these functions.  The value of "backend" is stable
133       between ".after_fork", ".preconnect", ".list_exports", and
134       ".default_export", and can also be obtained by using
135       "nbdkit_context_get_backend" on the "context" parameter to ".open".
136
137       Meanwhile, if the filter does not use "nbdkit_context_set_next", the
138       value of "next" passed to ".prepare" has a stable lifetime that lasts
139       to the corresponding ".finalize", with all intermediate functions (such
140       as ".pread") receiving the same value for convenience.  Functions where
141       "nxdata" is not reused are ".config", ".config_complete", and
142       ".get_ready", which are all called during initialization outside any
143       connections.  The value of "backend" passed to ".after_fork" also
144       occurs without connections, but is shared with ".preconnect",
145       ".list_exports", and ".default_export", and can also be obtained from
146       the "context" passed to ".open", and has a lifetime that lasts to
147       ".cleanup" for use by "nbdkit_next_context_open".  In turn, the value
148       of "context" passed to ".open" has a lifetime that lasts until the
149       matching ".close" for use by "nbdkit_context_get_backend" and
150       "nbdkit_context_set_next".
151
152   Next config, open and close
153       The filter’s ".config", ".config_complete", ".get_ready",
154       ".after_fork", ".preconnect", ".list_exports", ".default_export" and
155       ".open" methods may only call the next ".config", ".config_complete",
156       ".get_ready", ".after_fork", ".preconnect", ".list_exports",
157       ".default_export" and ".open" method in the chain (optionally for
158       ".config" and ".open").
159
160       The filter’s ".close" method is called when an old connection closed,
161       and this has no "next" parameter because it cannot be short-circuited.
162
163   "nbdkit_next"
164       The filter generally needs to call into the underlying plugin, which is
165       done via a pointer to "struct nbdkit_next_ops", also available as the
166       typedef "nbdkit_next".  The most common behavior is to create a next
167       context per connection by calling the "next_open" parameter during
168       ".open", at which point the next context will be automatically provided
169       to the filter’s other methods like ".prepare", ".get_size", ".pread"
170       etc.  The "nbdkit_next" struct contains a comparable set of accessors
171       to plugin methods that can be called during a connection.  When using
172       automatic registration, the "next" parameter is stable between
173       ".prepare" and ".finalize", and nbdkit automatically prepares,
174       finalizes, and closes the next context at the right point in the filter
175       connection lifecycle.
176
177       Alternatively, the filter can manage plugin contexts manually, whether
178       to multiplex multiple client connections through a single context into
179       the plugin, or to open multiple plugin contexts to perform retries or
180       otherwise service a single client connection more efficiently.  In this
181       mode of operation, the filter uses "nbdkit_next_context_open" to open a
182       plugin context using the "backend" parameter passed to ".after_fork",
183       ".preconnect", ".list_exports", ".default_export", or obtained from
184       using "nbdkit_context_get_backend" on the "context" parameter to
185       ".open".  The resulting next context has a lifecycle under manual
186       control, where the filter must use "next->prepare (next)" before using
187       any other function pointers within the next context, and must reclaim
188       the memory using "next->finalize (next)" and
189       "nbdkit_next_context_close" when done.  A filter using manual lifecycle
190       management may use "nbdkit_context_set_next" to associate the next
191       context into the current connection, which lets nbdkit then pass that
192       context as the "next" parameter to future connection-related functions
193       like ".pread" and take over lifecycle responsibility.
194
195       "nbdkit_context_get_backend"
196
197       "nbdkit_next_context_open"
198
199       "nbdkit_next_context_close"
200
201       "nbdkit_context_set_next"
202
203        nbdkit_backend *nbdkit_context_get_backend (nbdkit_context *context);
204
205       Obtains the backend pointer from the "context" parameter to ".open",
206       matching the backend pointer available to ".after_fork", ".preconnect",
207       ".list_exports", and ".default_export".  This backend pointer has a
208       stable lifetime from the time of ".after_fork" until ".cleanup".
209
210        nbdkit_next *nbdkit_next_context_open (nbdkit_backend *backend,
211                                               int readonly, const char *exportname,
212                                               int shared);
213
214       This function attempts to open a new context into the plugin in
215       relation to the filter's current "backend".  The "readonly" and
216       "exportname" parameters behave the same as documented in ".open".  The
217       resulting context will be under the filter's manual lifecycle control
218       unless the filter associates it into the connection with
219       "nbdkit_context_set_next".  The filter should be careful to not violate
220       any threading model restrictions of the plugin if it opens more than
221       one context.
222
223       If "shared" is false, this function must be called while servicing an
224       existing client connection, and the new context will share the same
225       connection details (export name, tls status, and shorter interned
226       string lifetimes) as the current connection, and thus should not be
227       used after the client connection ends.  Conversely, if "shared" is
228       true, this function may be called outside of a current client
229       connection (such as during ".after_fork"), and the resulting context
230       may be freely shared among multiple client connections.  In shared
231       mode, it will not be possible for the plugin to differentiate content
232       based on the client export name, the result of the plugin calling
233       nbdkit_is_tls() will depend solely whether --tls=require was on the
234       command line, the lifetime of interned strings (via
235       "nbdkit_strdup_intern" and friends) lasts for the life of the filter,
236       and the filter must take care to not expose potentially-secure
237       information from the backend to an insecure client.
238
239        void nbdkit_next_context_close (nbdkit_next *next);
240
241       This function closes a context into the plugin.  If the context has
242       previously been prepared, it should first be finalized before using
243       this function.  This function does not need to be called for a plugin
244       context that has been associated with the filter connection via
245       "nbdkit_context_set_next" prior to the ".close" callback.
246
247        nbdkit_next *nbdkit_context_set_next (nbdkit_context *context,
248                                              nbdkit_next *next);
249
250       This function associates a plugin context with the filter's current
251       connection context, given by the "context" parameter to ".open".  Once
252       associated, this plugin context will be given as the "next" parameter
253       to all other connection-specific callbacks.  If associated during
254       ".open", nbdkit will take care of preparing the context prior to
255       ".prepare"; if still associated before ".finalize", nbdkit will take
256       care of finalizing the context, and also for closing it.  A filter may
257       also pass "NULL" for "next", to remove any association; if no plugin
258       context is associated with the connection, then filter callbacks such
259       as ".pread" will receive "NULL" for their "next" parameter.
260
261       This function returns the previous context that had been associated
262       with the connection prior to switching the association to "next"; this
263       result will be "NULL" if there was no previous association.  The filter
264       assumes manual responsibility for any remaining lifecycle functions
265       that must be called on the returned context.
266
267   Using "nbdkit_next"
268       Regardless of whether the plugin context is managed automatically or
269       manually, it is possible for a filter to issue (for example) extra
270       "next->pread" calls in response to a single ".pwrite" call.
271
272       The "next" parameter serves two purposes: it serves as the struct to
273       access the pointers to all the plugin connection functions, and it
274       serves as the opaque data that must be passed as the first parameter to
275       those functions.  For example, calling the plugin's can_flush
276       functionality would be done via
277
278        next->can_flush (next)
279
280       Note that the semantics of the functions in "struct nbdkit_next_ops"
281       are slightly different from what a plugin implements: for example, when
282       a plugin's ".pread" returns -1 on error, the error value to advertise
283       to the client is implicit (via the plugin calling "nbdkit_set_error" or
284       setting "errno"), whereas "next->pread" exposes this via an explicit
285       parameter, allowing a filter to learn or modify this error if desired.
286
287       Use of "next->prepare" and "next->finalize" is only needed when
288       manually managing the plugin context lifetime.
289
290   Other considerations
291       You can modify parameters when you call the "next" function.  However
292       be careful when modifying strings because for some methods (eg.
293       ".config") the plugin may save the string pointer that you pass along.
294       So you may have to ensure that the string is not freed for the lifetime
295       of the server; you may find "nbdkit_strdup_intern" helpful for avoiding
296       a memory leak while still obeying lifecycle constraints.
297
298       Note that if your filter registers a callback but in that callback it
299       doesn't call the "next" function then the corresponding method in the
300       plugin will never be called.  In particular, your ".open" method, if
301       you have one, must call the "next" method if you want the underlying
302       plugin to be available to all further "nbdkit_next" use.
303

CALLBACKS

305       "struct nbdkit_filter" has some static fields describing the filter and
306       optional callback functions which can be used to intercept plugin
307       methods.
308
309   ".name"
310        const char *name;
311
312       This field (a string) is required, and must contain only ASCII
313       alphanumeric characters or non-leading dashes, and be unique amongst
314       all filters.
315
316   ".longname"
317        const char *longname;
318
319       An optional free text name of the filter.  This field is used in error
320       messages.
321
322   ".description"
323        const char *description;
324
325       An optional multi-line description of the filter.
326
327   ".load"
328        void load (void);
329
330       This is called once just after the filter is loaded into memory.  You
331       can use this to perform any global initialization needed by the filter.
332
333   ".unload"
334        void unload (void);
335
336       This may be called once just before the filter is unloaded from memory.
337       Note that it's not guaranteed that ".unload" will always be called (eg.
338       the server might be killed or segfault), so you should try to make the
339       filter as robust as possible by not requiring cleanup.  See also
340       "SHUTDOWN" in nbdkit-plugin(3).
341
342   ".config"
343        int (*config) (nbdkit_next_config *next, void *nxdata,
344                       const char *key, const char *value);
345
346       This intercepts the plugin ".config" method and can be used by the
347       filter to parse its own command line parameters.  You should try to
348       make sure that command line parameter keys that the filter uses do not
349       conflict with ones that could be used by a plugin.
350
351       If there is an error, ".config" should call "nbdkit_error" with an
352       error message and return "-1".
353
354   ".config_complete"
355        int (*config_complete) (nbdkit_next_config_complete *next, void *nxdata);
356
357       This intercepts the plugin ".config_complete" method and can be used to
358       ensure that all parameters needed by the filter were supplied on the
359       command line.
360
361       If there is an error, ".config_complete" should call "nbdkit_error"
362       with an error message and return "-1".
363
364   ".config_help"
365        const char *config_help;
366
367       This optional multi-line help message should summarize any "key=value"
368       parameters that it takes.  It does not need to repeat what already
369       appears in ".description".
370
371       If the filter doesn't take any config parameters you should probably
372       omit this.
373
374   ".thread_model"
375        int (*thread_model) (void);
376
377       Filters may tighten (but not relax) the thread model of the plugin, by
378       defining this callback.  Note that while plugins use a compile-time
379       definition of "THREAD_MODEL", filters do not need to declare a model at
380       compile time; instead, this callback is called after ".config_complete"
381       and before any connections are created.  See "THREADS" in
382       nbdkit-plugin(3) for a discussion of thread models.
383
384       The final thread model used by nbdkit is the smallest (ie. most
385       serialized) out of all the filters and the plugin, and applies for all
386       connections.  Requests for a model larger than permitted by the plugin
387       are silently ignored. It is acceptable for decisions made during
388       ".config" and ".config_complete" to determine which model to request.
389
390       This callback is optional; if it is not present, the filter must be
391       written to handle fully parallel requests, including when multiple
392       requests are issued in parallel on the same connection, similar to a
393       plugin requesting "NBDKIT_THREAD_MODEL_PARALLEL".  This ensures the
394       filter doesn't slow down other filters or plugins.
395
396       If there is an error, ".thread_model" should call "nbdkit_error" with
397       an error message and return "-1".
398
399   ".get_ready"
400        int (*get_ready) (int thread_model);
401
402       This optional callback is reached if the plugin ".get_ready" method
403       succeeded (if the plugin failed, nbdkit has already exited), and can be
404       used by the filter to get ready to serve requests.
405
406       The "thread_model" parameter informs the filter about the final thread
407       model chosen by nbdkit after considering the results of ".thread_model"
408       of all filters in the chain after ".config_complete".
409
410       If there is an error, ".get_ready" should call "nbdkit_error" with an
411       error message and return "-1".
412
413   ".after_fork"
414        int (*after_fork) (nbdkit_backend *backend);
415
416       This optional callback is reached after the plugin ".after_fork" method
417       has succeeded (if the plugin failed, nbdkit has already exited), and
418       can be used by the filter to start background threads.  The "backend"
419       parameter is valid until ".cleanup", for creating manual contexts into
420       the backend with "nbdkit_next_context_open".
421
422       If there is an error, ".after_fork" should call "nbdkit_error" with an
423       error message and return "-1".
424
425   ".cleanup"
426        int (cleanup) (nbdkit_backend *backend);
427
428       This optional callback is reached once after all client connections
429       have been closed, but before the underlying plugin ".cleanup" or any
430       ".unload" callbacks.  It can be used by the filter to gracefully close
431       any background threads created during ".after_fork", as well as close
432       any manual contexts into "backend" previously opened with
433       "nbdkit_next_context_open".
434
435       Note that it's not guaranteed that ".cleanup" will always be called
436       (eg. the server might be killed or segfault), so you should try to make
437       the filter as robust as possible by not requiring cleanup.  See also
438       "SHUTDOWN" in nbdkit-plugin(3).
439
440   ".preconnect"
441        int (*preconnect) (nbdkit_next_preconnect *next, nbdkit_backend *nxdata,
442                           int readonly);
443
444       This intercepts the plugin ".preconnect" method and can be used to
445       filter access to the server.
446
447       If there is an error, ".preconnect" should call "nbdkit_error" with an
448       error message and return "-1".
449
450   ".list_exports"
451        int (*list_exports) (nbdkit_next_list_exports *next, nbdkit_backend *nxdata,
452                             int readonly, int is_tls,
453                             struct nbdkit_exports *exports);
454
455       This intercepts the plugin ".list_exports" method and can be used to
456       filter which exports are advertised.
457
458       The "readonly" parameter matches what is passed to <.preconnect> and
459       ".open", and may be changed by the filter when calling into the plugin.
460       The "is_tls" parameter informs the filter whether TLS negotiation has
461       been completed by the client, but is not passed on to "next" because it
462       cannot be altered.
463
464       It is possible for filters to transform the exports list received back
465       from the layer below.  Without error checking it would look like this:
466
467        myfilter_list_exports (...)
468        {
469          size_t i;
470          struct nbdkit_exports *exports2;
471          struct nbdkit_export e;
472          char *name, *desc;
473
474          exports2 = nbdkit_exports_new ();
475          next_list_exports (nxdata, readonly, exports);
476          for (i = 0; i < nbdkit_exports_count (exports2); ++i) {
477            e = nbdkit_get_export (exports2, i);
478            name = adjust (e.name);
479            desc = adjust (e.desc);
480            nbdkit_add_export (exports, name, desc);
481            free (name);
482            free (desc);
483          }
484          nbdkit_exports_free (exports2);
485        }
486
487       If there is an error, ".list_exports" should call "nbdkit_error" with
488       an error message and return "-1".
489
490       Allocating and freeing nbdkit_exports list
491
492       Two functions are provided to filters only for allocating and freeing
493       the list:
494
495        struct nbdkit_exports *nbdkit_exports_new (void);
496
497       Allocates and returns a new, empty exports list.
498
499       On error this function can return "NULL".  In this case it calls
500       "nbdkit_error" as required.  "errno" will be set to a suitable value.
501
502        void nbdkit_exports_free (struct nbdkit_exports *);
503
504       Frees an existing exports list.
505
506       Iterating over nbdkit_exports list
507
508       Two functions are provided to filters only to iterate over the exports
509       in order:
510
511        size_t nbdkit_exports_count (const struct nbdkit_exports *);
512
513       Returns the number of exports in the list.
514
515        struct nbdkit_export {
516          char *name;
517          char *description;
518        };
519        const struct nbdkit_export nbdkit_get_export (const struct nbdkit_exports *,
520                                                      size_t i);
521
522       Returns a copy of the "i"'th export.
523
524   ".default_export"
525        const char *default_export (nbdkit_next_default_export *next,
526                                    nbdkit_backend *nxdata,
527                                    int readonly, int is_tls)
528
529       This intercepts the plugin ".default_export" method and can be used to
530       alter the canonical export name used in place of the default "".
531
532       The "readonly" parameter matches what is passed to <.preconnect> and
533       ".open", and may be changed by the filter when calling into the plugin.
534       The "is_tls" parameter informs the filter whether TLS negotiation has
535       been completed by the client, but is not passed on to "next" because it
536       cannot be altered.
537
538   ".open"
539        void * (*open) (nbdkit_next_open *next, nbdkit_context *context,
540                        int readonly, const char *exportname, int is_tls);
541
542       This is called when a new client connection is opened and can be used
543       to allocate any per-connection data structures needed by the filter.
544       The handle (which is not the same as the plugin handle) is passed back
545       to other filter callbacks and could be freed in the ".close" callback.
546
547       Note that the handle is completely opaque to nbdkit, but it must not be
548       NULL.  If you don't need to use a handle, return
549       "NBDKIT_HANDLE_NOT_NEEDED" which is a static non-NULL pointer.
550
551       If there is an error, ".open" should call "nbdkit_error" with an error
552       message and return "NULL".
553
554       This callback is optional, but if provided, it should call "next",
555       passing "readonly" and "exportname" possibly modified according to how
556       the filter plans to use the plugin ("is_tls" is not passed, because a
557       filter cannot modify it).  Typically, the filter passes the same values
558       as it received, or passes readonly=true to provide a writable layer on
559       top of a read-only backend.  However, it is also acceptable to attempt
560       write access to the plugin even if this filter is readonly, such as
561       when a file system mounted read-only still requires write access to the
562       underlying device in case a journal needs to be replayed for
563       consistency as part of the mounting process.
564
565       The "exportname" string is only guaranteed to be available during the
566       call (different than the lifetime for the return of
567       "nbdkit_export_name" used by plugins).  If the filter needs to use it
568       (other than immediately passing it down to the next layer) it must take
569       a copy, although "nbdkit_strdup_intern" is useful for this task.  The
570       "exportname" and "is_tls" parameters are provided so that filters do
571       not need to use the plugin-only interfaces of "nbdkit_export_name" and
572       "nbdkit_is_tls".
573
574       The filter should generally call "next" as its first step, to allocate
575       from the plugin outwards, so that ".close" running from the outer
576       filter to the plugin will be in reverse.  Skipping a call to "next" is
577       acceptable if the filter will not access "nbdkit_next" during any of
578       the remaining callbacks reached on the same connection.  The "next"
579       function is provided for convenience; the same functionality can be
580       obtained manually (other than error checking) by using the following:
581
582        nbdkit_context_set_next (context, nbdkit_next_context_open
583           (nbdkit_context_get_backend (context), readonly, exportname, false));
584
585       The value of "context" in this call has a lifetime that lasts until the
586       counterpart ".close", and it is this value that may be passed to
587       "nbdkit_context_get_backend" to obtain the "backend" parameter used to
588       open a plugin context with "nbdkit_next_context_open", as well as the
589       "context" parameter used to associate a plugin context into the current
590       connection with "nbdkit_context_set_next".
591
592   ".close"
593        void (*close) (void *handle);
594
595       This is called when the client closes the connection.  It should clean
596       up any per-connection resources used by the filter.  It is called
597       beginning with the outermost filter and ending with the plugin (the
598       opposite order of ".open" if all filters call "next" first), although
599       this order technically does not matter since the callback cannot report
600       failures or access the underlying plugin.
601
602   ".prepare"
603   ".finalize"
604         int (*prepare) (nbdkit_next *next, void *handle, int readonly);
605         int (*finalize) (nbdkit_next *next, void *handle);
606
607       These two methods can be used to perform any necessary operations just
608       after opening the connection (".prepare") or just before closing the
609       connection (".finalize").
610
611       For example if you need to scan the underlying disk to check for a
612       partition table, you could do it in your ".prepare" method (calling the
613       plugin's ".get_size" and ".pread" methods via "next").  Or if you need
614       to cleanly update superblock data in the image on close you can do it
615       in your ".finalize" method (calling the plugin's ".pwrite" method).
616       Doing these things in the filter's ".open" or ".close" method is not
617       possible without using manual context lifecycle management.
618
619       For ".prepare", the value of "readonly" is the same as was passed to
620       ".open", declaring how this filter will be used.
621
622       Note that nbdkit performs sanity checking on requests made to the
623       underlying plugin; for example, "next->pread" cannot be called on a
624       given connection unless "next->get_size" has first been called at least
625       once in the same connection (to ensure the read requests are in
626       bounds), and "next->pwrite" further requires an earlier successful call
627       to "next->can_write".  In many filters, these prerequisites will be
628       automatically called during the client negotiation phase, but there are
629       cases where a filter overrides query functions or makes I/O calls into
630       the plugin before handshaking is complete, where the filter needs to
631       make those prerequisite calls manually during ".prepare".
632
633       While there are "next->prepare" and "next->finalize" functions, these
634       are different from other filter methods, in that any plugin context
635       associated with the current connection (via the "next" parameter to
636       ".open", or via "nbdkit_context_set_next", is prepared and finalized
637       automatically by nbdkit, so they are only used during manual lifecycle
638       management.  Prepare methods are called starting with the filter
639       closest to the plugin and proceeding outwards (matching the order of
640       ".open" if all filters call "next" before doing anything locally), and
641       only when an outer filter did not skip the "next" call during ".open".
642       Finalize methods are called in the reverse order of prepare methods,
643       with the outermost filter first (and matching the order of ".close"),
644       and only if the prepare method succeeded.
645
646       If there is an error, both callbacks should call "nbdkit_error" with an
647       error message and return "-1".  An error in ".prepare" is reported to
648       the client, but leaves the connection open (a client may try again with
649       a different export name, for example); while an error in ".finalize"
650       forces the client to disconnect.
651
652   ".get_size"
653        int64_t (*get_size) (nbdkit_next *next, void *handle);
654
655       This intercepts the plugin ".get_size" method and can be used to read
656       or modify the apparent size of the block device that the NBD client
657       will see.
658
659       The returned size must be ≥ 0.  If there is an error, ".get_size"
660       should call "nbdkit_error" with an error message and return "-1".  This
661       function is only called once per connection and cached by nbdkit.
662       Similarly, repeated calls to "next->get_size" will return a cached
663       value.
664
665   ".export_description"
666        const char *export_description (nbdkit_next *next, void *handle);
667
668       This intercepts the plugin ".export_description" method and can be used
669       to read or modify the export description that the NBD client will see.
670
671   ".block_size"
672        int block_size (nbdkit_next *next, void *handle, uint32_t *minimum,
673                        uint32_t *preferred, uint32_t *maximum);
674
675       This intercepts the plugin ".block_size" method and can be used to read
676       or modify the block size constraints that the NBD client will see.
677
678   ".can_write"
679   ".can_flush"
680   ".is_rotational"
681   ".can_trim"
682   ".can_zero"
683   ".can_fast_zero"
684   ".can_extents"
685   ".can_fua"
686   ".can_multi_conn"
687   ".can_cache"
688        int (*can_write) (nbdkit_next *next, void *handle);
689        int (*can_flush) (nbdkit_next *next, void *handle);
690        int (*is_rotational) (nbdkit_next *next, void *handle);
691        int (*can_trim) (nbdkit_next *next, void *handle);
692        int (*can_zero) (nbdkit_next *next, void *handle);
693        int (*can_fast_zero) (nbdkit_next *next, void *handle);
694        int (*can_extents) (nbdkit_next *next, void *handle);
695        int (*can_fua) (nbdkit_next *next, void *handle);
696        int (*can_multi_conn) (nbdkit_next *next, void *handle);
697        int (*can_cache) (nbdkit_next *next, void *handle);
698
699       These intercept the corresponding plugin methods, and control feature
700       bits advertised to the client.
701
702       Of note, the semantics of ".can_zero" callback in the filter are
703       slightly different from the plugin, and must be one of three success
704       values visible only to filters:
705
706       "NBDKIT_ZERO_NONE"
707           Completely suppress advertisement of write zero support (this can
708           only be done from filters, not plugins).
709
710       "NBDKIT_ZERO_EMULATE"
711           Inform nbdkit that write zeroes should immediately fall back to
712           ".pwrite" emulation without trying ".zero" (this value is returned
713           by "next->can_zero" if the plugin returned false in its
714           ".can_zero").
715
716       "NBDKIT_ZERO_NATIVE"
717           Inform nbdkit that write zeroes should attempt to use ".zero",
718           although it may still fall back to ".pwrite" emulation for
719           "ENOTSUP" or "EOPNOTSUPP" failures (this value is returned by
720           "next->can_zero" if the plugin returned true in its ".can_zero").
721
722       Remember that most of the feature check functions return merely a
723       boolean success value, while ".can_zero", ".can_fua" and ".can_cache"
724       have three success values.
725
726       The difference between ".can_fua" values may affect choices made in the
727       filter: when splitting a write request that requested FUA from the
728       client, if "next->can_fua" returns "NBDKIT_FUA_NATIVE", then the filter
729       should pass the FUA flag on to each sub-request; while if it is known
730       that FUA is emulated by a flush because of a return of
731       "NBDKIT_FUA_EMULATE", it is more efficient to only flush once after all
732       sub-requests have completed (often by passing "NBDKIT_FLAG_FUA" on to
733       only the final sub-request, or by dropping the flag and ending with a
734       direct call to "next->flush").
735
736       If there is an error, the callback should call "nbdkit_error" with an
737       error message and return "-1".  These functions are called at most once
738       per connection and cached by nbdkit. Similarly, repeated calls to any
739       of the "nbdkit_next" counterparts will return a cached value; by
740       calling into the plugin during ".prepare", you can ensure that later
741       use of the cached values during data commands like <.pwrite> will not
742       fail.
743
744   ".pread"
745        int (*pread) (nbdkit_next *next,
746                      void *handle, void *buf, uint32_t count, uint64_t offset,
747                      uint32_t flags, int *err);
748
749       This intercepts the plugin ".pread" method and can be used to read or
750       modify data read by the plugin.
751
752       The parameter "flags" exists in case of future NBD protocol extensions;
753       at this time, it will be 0 on input, and the filter should not pass any
754       flags to "next->pread".
755
756       If there is an error (including a short read which couldn't be
757       recovered from), ".pread" should call "nbdkit_error" with an error
758       message and return -1 with "err" set to the positive errno value to
759       return to the client.
760
761   ".pwrite"
762        int (*pwrite) (nbdkit_next *next,
763                       void *handle,
764                       const void *buf, uint32_t count, uint64_t offset,
765                       uint32_t flags, int *err);
766
767       This intercepts the plugin ".pwrite" method and can be used to modify
768       data written by the plugin.
769
770       This function will not be called if ".can_write" returned false; in
771       turn, the filter should not call "next->pwrite" if "next->can_write"
772       did not return true.
773
774       The parameter "flags" may include "NBDKIT_FLAG_FUA" on input based on
775       the result of ".can_fua".  In turn, the filter should only pass
776       "NBDKIT_FLAG_FUA" on to "next->pwrite" if "next->can_fua" returned a
777       positive value.
778
779       If there is an error (including a short write which couldn't be
780       recovered from), ".pwrite" should call "nbdkit_error" with an error
781       message and return -1 with "err" set to the positive errno value to
782       return to the client.
783
784   ".flush"
785        int (*flush) (nbdkit_next *next,
786                      void *handle, uint32_t flags, int *err);
787
788       This intercepts the plugin ".flush" method and can be used to modify
789       flush requests.
790
791       This function will not be called if ".can_flush" returned false; in
792       turn, the filter should not call "next->flush" if "next->can_flush" did
793       not return true.
794
795       The parameter "flags" exists in case of future NBD protocol extensions;
796       at this time, it will be 0 on input, and the filter should not pass any
797       flags to "next->flush".
798
799       If there is an error, ".flush" should call "nbdkit_error" with an error
800       message and return -1 with "err" set to the positive errno value to
801       return to the client.
802
803   ".trim"
804        int (*trim) (nbdkit_next *next,
805                     void *handle, uint32_t count, uint64_t offset,
806                     uint32_t flags, int *err);
807
808       This intercepts the plugin ".trim" method and can be used to modify
809       trim requests.
810
811       This function will not be called if ".can_trim" returned false; in
812       turn, the filter should not call "next->trim" if "next->can_trim" did
813       not return true.
814
815       The parameter "flags" may include "NBDKIT_FLAG_FUA" on input based on
816       the result of ".can_fua".  In turn, the filter should only pass
817       "NBDKIT_FLAG_FUA" on to "next->trim" if "next->can_fua" returned a
818       positive value.
819
820       If there is an error, ".trim" should call "nbdkit_error" with an error
821       message and return -1 with "err" set to the positive errno value to
822       return to the client.
823
824   ".zero"
825        int (*zero) (nbdkit_next *next,
826                     void *handle, uint32_t count, uint64_t offset, uint32_t flags,
827                     int *err);
828
829       This intercepts the plugin ".zero" method and can be used to modify
830       zero requests.
831
832       This function will not be called if ".can_zero" returned
833       "NBDKIT_ZERO_NONE" or "NBDKIT_ZERO_EMULATE"; in turn, the filter should
834       not call "next->zero" if "next->can_zero" returned "NBDKIT_ZERO_NONE".
835
836       On input, the parameter "flags" may include "NBDKIT_FLAG_MAY_TRIM"
837       unconditionally, "NBDKIT_FLAG_FUA" based on the result of ".can_fua",
838       and "NBDKIT_FLAG_FAST_ZERO" based on the result of ".can_fast_zero".
839       In turn, the filter may pass "NBDKIT_FLAG_MAY_TRIM" unconditionally,
840       but should only pass "NBDKIT_FLAG_FUA" or "NBDKIT_FLAG_FAST_ZERO" on to
841       "next->zero" if the corresponding "next->can_fua" or
842       "next->can_fast_zero" returned a positive value.
843
844       Note that unlike the plugin ".zero" which is permitted to fail with
845       "ENOTSUP" or "EOPNOTSUPP" to force a fallback to ".pwrite", the
846       function "next->zero" will not fail with "err" set to "ENOTSUP" or
847       "EOPNOTSUPP" unless "NBDKIT_FLAG_FAST_ZERO" was used, because otherwise
848       the fallback has already taken place.
849
850       If there is an error, ".zero" should call "nbdkit_error" with an error
851       message and return -1 with "err" set to the positive errno value to
852       return to the client.  The filter should not fail with "ENOTSUP" or
853       "EOPNOTSUPP" unless "flags" includes "NBDKIT_FLAG_FAST_ZERO" (while
854       plugins have automatic fallback to ".pwrite", filters do not).
855
856   ".extents"
857        int (*extents) (nbdkit_next *next,
858                        void *handle, uint32_t count, uint64_t offset, uint32_t flags,
859                        struct nbdkit_extents *extents,
860                        int *err);
861
862       This intercepts the plugin ".extents" method and can be used to modify
863       extent requests.
864
865       This function will not be called if ".can_extents" returned false; in
866       turn, the filter should not call "next->extents" if "next->can_extents"
867       did not return true.
868
869       It is possible for filters to transform the extents list received back
870       from the layer below.  Without error checking it would look like this:
871
872        myfilter_extents (..., uint32_t count, uint64_t offset, ...)
873        {
874          size_t i;
875          struct nbdkit_extents *extents2;
876          struct nbdkit_extent e;
877          int64_t size;
878
879          size = next->get_size (next);
880          extents2 = nbdkit_extents_new (offset + shift, size);
881          next->extents (next, count, offset + shift, flags, extents2, err);
882          for (i = 0; i < nbdkit_extents_count (extents2); ++i) {
883            e = nbdkit_get_extent (extents2, i);
884            e.offset -= shift;
885            nbdkit_add_extent (extents, e.offset, e.length, e.type);
886          }
887          nbdkit_extents_free (extents2);
888        }
889
890       If there is an error, ".extents" should call "nbdkit_error" with an
891       error message and return -1 with "err" set to the positive errno value
892       to return to the client.
893
894       Allocating and freeing nbdkit_extents list
895
896       Two functions are provided to filters only for allocating and freeing
897       the map:
898
899        struct nbdkit_extents *nbdkit_extents_new (uint64_t start, uint64_t end);
900
901       Allocates and returns a new, empty extents list.  The "start" parameter
902       is the start of the range described in the list, and the "end"
903       parameter is the offset of the byte beyond the end.  Normally you would
904       pass in "offset" as the start and the size of the plugin as the end,
905       but for filters which adjust offsets, they should pass in the adjusted
906       offset.
907
908       On error this function can return "NULL".  In this case it calls
909       "nbdkit_error" and/or "nbdkit_set_error" as required.  "errno" will be
910       set to a suitable value.
911
912        void nbdkit_extents_free (struct nbdkit_extents *);
913
914       Frees an existing extents list.
915
916       Iterating over nbdkit_extents list
917
918       Two functions are provided to filters only to iterate over the extents
919       in order:
920
921        size_t nbdkit_extents_count (const struct nbdkit_extents *);
922
923       Returns the number of extents in the list.
924
925        struct nbdkit_extent {
926          uint64_t offset;
927          uint64_t length;
928          uint32_t type;
929        };
930        struct nbdkit_extent nbdkit_get_extent (const struct nbdkit_extents *,
931                                                size_t i);
932
933       Returns a copy of the "i"'th extent.
934
935       Reading the full extents from the plugin
936
937       A convenience function is provided to filters only which makes one or
938       more requests to the underlying plugin until we have a full set of
939       extents covering the region "[offset..offset+count-1]".
940
941        struct nbdkit_extents *nbdkit_extents_full (
942                                    nbdkit_next *next,
943                                    uint32_t count, uint64_t offset,
944                                    uint32_t flags, int *err);
945
946       Note this allocates a new "struct nbdkit_extents" which the caller must
947       free.  "flags" is passed through to the underlying plugin, but
948       "NBDKIT_FLAG_REQ_ONE" is removed from the set of flags so that the
949       plugin returns as much information as possible (this is usually what
950       you want).
951
952       On error this function can return "NULL".  In this case it calls
953       "nbdkit_error" and/or "nbdkit_set_error" as required.  *err will be set
954       to a suitable value.
955
956       Enforcing alignment of an nbdkit_extents list
957
958       A convenience function is provided to filters only which makes it
959       easier to ensure that the client only encounters aligned extents.
960
961        int nbdkit_extents_aligned (nbdkit_next *next,
962                                    uint32_t count, uint64_t offset,
963                                    uint32_t flags, uint32_t align,
964                                    struct nbdkit_extents *extents, int *err);
965
966       Calls "next->extents" as needed until at least "align" bytes are
967       obtained, where "align" is a power of 2.  Anywhere the underlying
968       plugin returns differing extents within "align" bytes, this function
969       treats that portion of the disk as a single extent with zero and sparse
970       status bits determined by the intersection of all underlying extents.
971       It is an error to call this function with "count" or "offset" that is
972       not already aligned.
973
974   ".cache"
975        int (*cache) (nbdkit_next *next,
976                      void *handle, uint32_t count, uint64_t offset,
977                      uint32_t flags, int *err);
978
979       This intercepts the plugin ".cache" method and can be used to modify
980       cache requests.
981
982       This function will not be called if ".can_cache" returned
983       "NBDKIT_CACHE_NONE" or "NBDKIT_CACHE_EMULATE"; in turn, the filter
984       should not call "next->cache" if "next->can_cache" returned
985       "NBDKIT_CACHE_NONE".
986
987       The parameter "flags" exists in case of future NBD protocol extensions;
988       at this time, it will be 0 on input, and the filter should not pass any
989       flags to "next->cache".
990
991       If there is an error, ".cache" should call "nbdkit_error" with an error
992       message and return -1 with "err" set to the positive errno value to
993       return to the client.
994

ERROR HANDLING

996       If there is an error in the filter itself, the filter should call
997       "nbdkit_error" to report an error message.  If the callback is involved
998       in serving data, the explicit "err" parameter determines the error code
999       that will be sent to the client; other callbacks should return the
1000       appropriate error indication, eg. "NULL" or "-1".
1001
1002       "nbdkit_error" has the following prototype and works like printf(3):
1003
1004        void nbdkit_error (const char *fs, ...);
1005        void nbdkit_verror (const char *fs, va_list args);
1006
1007       For convenience, "nbdkit_error" preserves the value of "errno", and
1008       also supports the glibc extension of a single %m in a format string
1009       expanding to "strerror(errno)", even on platforms that don't support
1010       that natively.
1011

DEBUGGING

1013       Run the server with -f and -v options so it doesn't fork and you can
1014       see debugging information:
1015
1016        nbdkit -fv --filter=./myfilter.so plugin [key=value [key=value [...]]]
1017
1018       To print debugging information from within the filter, call
1019       "nbdkit_debug", which has the following prototype and works like
1020       printf(3):
1021
1022        void nbdkit_debug (const char *fs, ...);
1023        void nbdkit_vdebug (const char *fs, va_list args);
1024
1025       For convenience, "nbdkit_debug" preserves the value of "errno", and
1026       also supports the glibc extension of a single %m in a format string
1027       expanding to "strerror(errno)", even on platforms that don't support
1028       that natively.  Note that "nbdkit_debug" only prints things when the
1029       server is in verbose mode (-v option).
1030
1031   Debug Flags
1032       Debug Flags in filters work exactly the same way as plugins.  See
1033       "Debug Flags" in nbdkit-plugin(3).
1034

INSTALLING THE FILTER

1036       The filter is a "*.so" file and possibly a manual page.  You can of
1037       course install the filter "*.so" file wherever you want, and users will
1038       be able to use it by running:
1039
1040        nbdkit --filter=/path/to/filter.so plugin [args]
1041
1042       However if the shared library has a name of the form
1043       "nbdkit-name-filter.so" and if the library is installed in the
1044       $filterdir directory, then users can be run it by only typing:
1045
1046        nbdkit --filter=name plugin [args]
1047
1048       The location of the $filterdir directory is set when nbdkit is compiled
1049       and can be found by doing:
1050
1051        nbdkit --dump-config
1052
1053       If using the pkg-config/pkgconf system then you can also find the
1054       filter directory at compile time by doing:
1055
1056        pkg-config nbdkit --variable=filterdir
1057

PKG-CONFIG/PKGCONF

1059       nbdkit provides a pkg-config/pkgconf file called "nbdkit.pc" which
1060       should be installed on the correct path when the nbdkit development
1061       environment is installed.  You can use this in autoconf configure.ac
1062       scripts to test for the development environment:
1063
1064        PKG_CHECK_MODULES([NBDKIT], [nbdkit >= 1.2.3])
1065
1066       The above will fail unless nbdkit ≥ 1.2.3 and the header file is
1067       installed, and will set "NBDKIT_CFLAGS" and "NBDKIT_LIBS" appropriately
1068       for compiling filters.
1069
1070       You can also run pkg-config/pkgconf directly, for example:
1071
1072        if ! pkg-config nbdkit --exists; then
1073          echo "you must install the nbdkit development environment"
1074          exit 1
1075        fi
1076
1077       You can also substitute the filterdir variable by doing:
1078
1079        PKG_CHECK_VAR([NBDKIT_FILTERDIR], [nbdkit], [filterdir])
1080
1081       which defines "$(NBDKIT_FILTERDIR)" in automake-generated Makefiles.
1082

WRITING FILTERS IN C++

1084       Instead of using C, it is possible to write filters in C++.  However
1085       for inclusion in upstream nbdkit we would generally prefer filters
1086       written in C.
1087
1088       Filters in C++ work almost exactly like those in C, but the way you
1089       define the "nbdkit_filter" struct is slightly different:
1090
1091        namespace {
1092          nbdkit_filter create_filter() {
1093            nbdkit_filter filter = nbdkit_filter ();
1094            filter.name   = "myfilter";
1095            filter.config = myfilter_config;
1096            return filter;
1097          }
1098        }
1099        static struct nbdkit_filter filter = create_filter ();
1100        NBDKIT_REGISTER_FILTER(filter)
1101

SEE ALSO

1103       nbdkit(1), nbdkit-plugin(3).
1104
1105       Standard filters provided by nbdkit:
1106
1107       nbdkit-blocksize-filter(1), nbdkit-blocksize-policy-filter(1),
1108       nbdkit-cache-filter(1), nbdkit-cacheextents-filter(1),
1109       nbdkit-checkwrite-filter(1), nbdkit-cow-filter(1),
1110       nbdkit-ddrescue-filter(1), nbdkit-delay-filter(1),
1111       nbdkit-error-filter(1), nbdkit-exitlast-filter(1),
1112       nbdkit-exitwhen-filter(1), nbdkit-exportname-filter(1),
1113       nbdkit-ext2-filter(1), nbdkit-extentlist-filter(1),
1114       nbdkit-fua-filter(1), nbdkit-gzip-filter(1), nbdkit-ip-filter(1),
1115       nbdkit-limit-filter(1), nbdkit-log-filter(1), nbdkit-luks-filter(1),
1116       nbdkit-multi-conn-filter(1), nbdkit-nocache-filter(1),
1117       nbdkit-noextents-filter(1), nbdkit-nofilter-filter(1),
1118       nbdkit-noparallel-filter(1), nbdkit-nozero-filter(1),
1119       nbdkit-offset-filter(1), nbdkit-partition-filter(1),
1120       nbdkit-pause-filter(1), nbdkit-protect-filter(1),
1121       nbdkit-rate-filter(1), nbdkit-readahead-filter(1),
1122       nbdkit-retry-filter(1), nbdkit-retry-request-filter(1),
1123       nbdkit-scan-filter(1), nbdkit-stats-filter(1), nbdkit-swab-filter(1),
1124       nbdkit-tar-filter(1), nbdkit-tls-fallback-filter(1),
1125       nbdkit-truncate-filter(1), nbdkit-xz-filter(1) .
1126

AUTHORS

1128       Eric Blake
1129
1130       Richard W.M. Jones
1131
1133       Copyright (C) 2013-2020 Red Hat Inc.
1134

LICENSE

1136       Redistribution and use in source and binary forms, with or without
1137       modification, are permitted provided that the following conditions are
1138       met:
1139
1140       •   Redistributions of source code must retain the above copyright
1141           notice, this list of conditions and the following disclaimer.
1142
1143       •   Redistributions in binary form must reproduce the above copyright
1144           notice, this list of conditions and the following disclaimer in the
1145           documentation and/or other materials provided with the
1146           distribution.
1147
1148       •   Neither the name of Red Hat nor the names of its contributors may
1149           be used to endorse or promote products derived from this software
1150           without specific prior written permission.
1151
1152       THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
1153       EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
1154       IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
1155       PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
1156       LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
1157       CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
1158       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
1159       BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
1160       WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
1161       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
1162       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
1163
1164
1165
1166nbdkit-1.32.5                     2023-01-03                  nbdkit-filter(3)
Impressum