1nbdkit-curl-plugin(1) NBDKIT nbdkit-curl-plugin(1)
2
6 nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7
9 nbdkit -r curl [url=]http://example.com/disk.img
10
12 "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13 served over HTTP, FTP, and more, into a Network Block Device. It uses
14 a library called libcurl (also known as cURL) to read data from URLs.
15 The exact list of protocols that libcurl can handle depends on how it
16 was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17 more (see: "curl -V").
18 Note: This plugin supports writes. However for HTTP, you may not want
20 nbdkit to issue PUT requests to the remote server (which probably
21 doesn't understand them). To force nbdkit to use a readonly
22 connection, pass the -r flag. Using the -r flag also enables NBD
23 multi-conn, which usually performs much better (if the client supports
24 it).
25 Although this plugin can access SFTP (ie. SSH) servers, it is much
27 better to use nbdkit-ssh-plugin(1). This plugin can be used to access
28 "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
29
31 nbdkit -r curl http://example.com/disk.img
32 serves the remote disk image as NBD on TCP port 10809 (to control ports
34 and protocols used to serve NBD see nbdkit(1)).
35
37 cainfo=FILENAME
38 (nbdkit ≥ 1.18)
39 Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
41 Pass empty string in order to not use the default certificate store
43 that libcurl is compiled with.
44 capath=PATH
46 (nbdkit ≥ 1.18)
47 Set CA certificates directory location for libcurl. See
49 CURLOPT_CAPATH(3) for more information.
50 connections=N
52 (nbdkit ≥ 1.34)
53 Open up to "N" connections to the web server. The default is 16.
55 Connections are shared between all NBD clients, so you may wish to
56 increase this if you expect many simultaneous NBD clients (or a
57 single client using many multi-conn connections).
58 See "NBD CONNECTIONS AND CURL HANDLES" below.
60 cookie=COOKIE
62 cookie=+FILENAME
63 cookie=-
64 cookie=-FD
65 (nbdkit ≥ 1.12)
66 Set a cookie in the request header when connecting to the remote
68 server.
69 A typical example is:
71 cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
73 This option can be used at most once. It only works for HTTP and
75 HTTPS transports. To set multiple cookies you must concatenate
76 them yourself, eg:
77 cookie='name1=content1; name2=content2'
79 See CURLOPT_COOKIE(3) for more information about this. The format
81 is quite strict and must consist of "key=value", each cookie
82 separated by exactly "; " (semicolon and space).
83 If the cookie is used for authentication then passing it on the
85 command line is not secure on shared machines. Use the alternate
86 "+FILENAME" syntax to pass it in a file, "-" to read the cookie
87 interactively, or "-FD" to read it from a file descriptor.
88 cookiefile=
90 (nbdkit ≥ 1.26)
91 Enable cookie processing (eg. when following redirects), starting
93 with an empty list of cookies. This is equivalent to calling
94 CURLOPT_COOKIEFILE(3) with an empty string.
95 cookiefile=FILENAME
97 (nbdkit ≥ 1.26)
98 Enable cookie processing (eg. when following redirects),
100 prepopulating cookies from the given file. The file can contain
101 cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
102 Cookies sent by the server are not saved when nbdkit exits.
103 cookiejar=FILENAME
105 (nbdkit ≥ 1.26)
106 Enable cookie processing (eg. when following redirects),
108 prepopulating cookies from the given file, and writing server
109 cookies back to the file when the NBD handle is closed. The file
110 can contain cookies in any format supported by curl, see
111 CURLOPT_COOKIEJAR(3).
112 There is some advice on the internet telling you to set this to
114 /dev/null, but you should not do this because it can corrupt
115 /dev/null. If you don't want a cookiejar, omit this option. If
116 you want to enable cookie processing without updating a permanent
117 cookiejar, use the "cookiefile=" option instead.
118 cookie-script=SCRIPT
120 cookie-script-renew=SECS
121 (nbdkit ≥ 1.22, not Windows)
122 Run "SCRIPT" (a command or shell script fragment) to generate the
124 HTTP/HTTPS cookies. "cookie-script" cannot be used with "cookie".
125 See "HEADER AND COOKIE SCRIPTS" below.
126 followlocation=false
128 (nbdkit ≥ 1.26)
129 Do not follow redirects from the server. The default is true
131 (follow redirects).
132 You can follow redirects but avoid redirecting to a less secure
134 protocol (eg. HTTPS redirecting to FTP) by using the "protocols"
135 parameter instead.
136 header=HEADER
138 (nbdkit ≥ 1.22)
139 For HTTP/HTTPS, send a custom header, or remove a header that curl
141 has added. To add a custom header:
142 header='X-My-Name: John Doe'
144 To remove a header that curl has added, add the header followed by
146 a colon and no value:
147 header='User-Agent:'
149 To add a custom header that has no value, you have to use a
151 semicolon instead of colon. This adds an "X-Empty:" header with no
152 value:
153 header='X-Empty;'
155 See CURLOPT_HTTPHEADER(3). You can use this option multiple times
157 in order to add several headers. Note this sends the header in all
158 requests, even when following a redirect, which can cause headers
159 (eg. containing sensitive authorization information) to be sent to
160 hosts other than the one originally requested.
161 header-script=SCRIPT
163 header-script-renew=SECS
164 (nbdkit ≥ 1.22, not Windows)
165 Run "SCRIPT" (a command or shell script fragment) to generate the
167 HTTP/HTTPS headers. "header-script" cannot be used with "header".
168 See "HEADER AND COOKIE SCRIPTS" below.
169 http-version=none
171 http-version=1.0
172 http-version=1.1
173 http-version=2.0
174 http-version=2TLS
175 http-version=2-prior-knowledge
176 http-version=3
177 http-version=3only
178 (nbdkit ≥ 1.34)
179 Force curl to use a particular HTTP protocol. The default is
181 "none" meaning curl will negotiate the best protocol with the
182 server. The other settings are mainly for testing. See
183 CURLOPT_HTTP_VERSION(3) for details.
184 ipresolve=any
186 ipresolve=v4
187 ipresolve=v6
188 (nbdkit ≥ 1.36)
189 Force curl to use only IPv4 ("ipresolve=v4"), only IPv6
191 ("ipresolve=v6") or any IP version supported by your system
192 ("ipresolve=any"). The default is "any". See
193 CURLOPT_IPRESOLVE(3).
194 password=PASSWORD
196 Set the password to use when connecting to the remote server.
197 Note that passing this on the command line is not secure on shared
199 machines.
200 password=-
202 Ask for the password (interactively) when nbdkit starts up.
203 password=+FILENAME
205 Read the password from the named file. This is a secure method to
206 supply a password, as long as you set the permissions on the file
207 appropriately.
208 password=-FD
210 Read the password from file descriptor number "FD", inherited from
211 the parent process when nbdkit starts up. This is also a secure
212 method to supply a password.
213 protocols=PROTO,PROTO,...
215 (nbdkit ≥ 1.12)
216 Limit the protocols that are allowed in the URL. Use this option
218 for extra security if the URL comes from an untrusted source and
219 you want to avoid security isues in the more obscure protocols that
220 curl supports. (See qemu CVE-2013-0249 for an example of a
221 security bug introduced by allowing unrestricted protocols).
222 For example if you only intend HTTP and HTTPS URLs to be used, then
224 add this parameter: "protocols=http,https"
225 This restriction also applies if the plugin follows a redirect to
227 another protocol (eg. you start with an "https://" URL which the
228 server redirects to "ftp://"). To prevent redirects altogether see
229 the "followlocation" parameter.
230 The value of this parameter is a comma-separated list of protocols,
232 for example "protocols=https", or "protocols=http,https", or
233 "protocols=file,ftp,gopher". For a list of known protocols, see
234 the libcurl manual (CURLOPT_PROTOCOLS_STR(3)).
235 The default is to allow any protocol.
237 proxy=PROXY
239 (nbdkit ≥ 1.20)
240 Set the proxy. See CURLOPT_PROXY(3).
242 proxy-password=PASSWORD
244 proxy-password=-
245 proxy-password=+FILENAME
246 proxy-password=-FD
247 proxy-user=USERNAME
248 (nbdkit ≥ 1.12)
249 Set the proxy username and password.
251 resolve=HOST:PORT:ADDRESS
253 Provide custom host name to IP address resolution. You can supply
254 this option as many times as needed. See CURLOPT_RESOLVE(3) for
255 the full details of this option.
256 sslverify=false
258 Don't verify the SSL certificate of the remote host.
259 ssl-cipher-list=CIPHER[:CIPHER...]
261 ssl-version=default
262 ssl-version=tlsv1
263 ssl-version=sslv2
264 ssl-version=sslv3
265 ssl-version=tlsv1.0
266 ssl-version=tlsv1.1
267 ssl-version=tlsv1.2
268 ssl-version=tlsv1.3
269 ssl-version=max-default
270 ssl-version=max-tlsv1.0
271 ssl-version=max-tlsv1.1
272 ssl-version=max-tlsv1.2
273 ssl-version=max-tlsv1.3
274 Set the SSL ciphers and TLS version. For further information see
275 CURLOPT_SSL_CIPHER_LIST(3) and CURLOPT_SSLVERSION(3).
276 tcp-keepalive=true
278 (nbdkit ≥ 1.20)
279 Enable TCP keepalives.
281 tcp-nodelay=false
283 (nbdkit ≥ 1.20)
284 Enable Nagle’s algorithm. Small writes on the network socket will
286 not be sent immediately but will be held in a local buffer and
287 coalesced if possible. This is more efficient for the network but
288 can cause increased latency.
289 The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is
291 disabled for better latency at the expense of network efficiency.
292 See CURLOPT_TCP_NODELAY(3).
294 timeout=SECS
296 Set the timeout for requests.
297 timeout=0
299 Use the default libcurl timeout for requests.
300 tls13-ciphers=CIPHER[:CIPHER...]
302 Select TLSv1.3 ciphers available. See CURLOPT_TLS13_CIPHERS(3) and
303 https://curl.se/docs/ssl-ciphers.html
304 unix-socket-path=PATH
306 (nbdkit ≥ 1.10)
307 Instead of using a TCP connection, connect to the server over the
309 named Unix domain socket. See CURLOPT_UNIX_SOCKET_PATH(3).
310 [url=]URL
312 The URL of the remote disk image. This is passed to libcurl
313 directly via CURLOPT_URL(3).
314 This parameter is required.
316 "url=" is a magic config key and may be omitted in most cases. See
318 "Magic parameters" in nbdkit(1).
319 user=USERNAME
321 Set the username to use when connecting to the remote server. This
322 may also be set in the URL (eg. "http://foo@example.com/disk.img")
323 user-agent=USER-AGENT
325 (nbdkit ≥ 1.22)
326 Send user-agent header when using HTTP or HTTPS. The default is no
328 user-agent header.
329
331 nbdkit ≤ 1.32 used a simple model where a new NBD connection would
332 create a new libcurl handle. Since a libcurl handle maintains a small
333 cache of connections, this meant that the number of HTTP connections
334 would be a small multiple of the number of incoming NBD connections and
335 the total would not be limited (assuming http: or https: URL).
336 nbdkit 1.34 changed to using a fixed pool of libcurl handles shared
338 across all NBD connections. You can control the maximum number of curl
339 handles in the pool with the "connections" parameter (default 4).
340 Since each curl handle maintains a small cache of connections, this
341 meant that the number of HTTP connections would be a small multiple of
342 the "connections" parameter. If there are more than 4 incoming NBD
343 connections, they will contend for the libcurl handles, unless you
344 adjust "connections".
345 nbdkit ≥ 1.36 changed again to use a curl multi handle
347 (libcurl-multi(3)). Now the "connections" parameter controls the
348 maximum number of HTTP connections made to the remote server
349 (CURLMOPT_MAX_TOTAL_CONNECTIONS(3)). This is more efficient especially
350 with HTTP/2 and HTTP/3, where each HTTP connection can contain a very
351 large number of streams (typically up to 100) multiplexed over one
352 connection. The default for "connections" was raised to 16.
353
355 While the "header" and "cookie" parameters can be used to specify
356 static headers and cookies which are used in every HTTP/HTTPS request,
357 the alternate "header-script" and "cookie-script" parameters can be
358 used to run an external script or program to generate headers and/or
359 cookies. This is particularly useful to access services which require
360 an authorization token. In addition the "header-script-renew" and
361 "cookie-script-renew" parameters allow you to renew the authorization
362 token by rerunning the script periodically.
363 "header-script" is incompatible with "header", and "cookie-script" is
365 incompatible with "cookie".
366 Header script
368 The header script should print zero or more HTTP headers, each line of
369 output in the same format as the "header" parameter. The headers
370 printed by the script are passed to CURLOPT_HTTPHEADER(3).
371 In the following example, an imaginary web service requires
373 authentication using a token fetched from a separate login server. The
374 token expires after 60 seconds, so we also tell the plugin that it must
375 renew the token (by re-running the script) if more than 50 seconds have
376 elapsed since the last request:
377 nbdkit curl https://service.example.com/disk.img \
379 header-script='
380 printf "Authorization: Bearer "
381 curl -s -X POST https://auth.example.com/login |
382 jq -r .token
383 ' \
384 header-script-renew=50
385 Cookie script
387 The cookie script should print a single line in the same format as the
388 "cookie" parameter. This is passed to CURLOPT_COOKIE(3).
389 Header and cookie script shell variables
391 Within the "header-script" and "cookie-script" the following shell
392 variables are available:
393 $iteration
395 The number of times that the script has been called. The first
396 time the script is called this contains 0.
397 $url
399 The URL as passed to the plugin.
400 Example: VMware ESXi cookies
402 VMware ESXi’s web server can expose both VMDK and raw format disk
403 images, but requires you to log in using HTTP Basic Authentication.
404 While you can use the "user" and "password" parameters to send HTTP
405 Basic Authentication headers in every request, tests have shown that it
406 is faster to accept the cookie which the server returns and send that
407 instead. (It is not clear why it is faster, but one theory is that
408 VMware has to do a more expensive username and password check each
409 time.)
410 The web server can be accessed as below. Since the cookie expires
412 after a certain period of time, we use "cookie-script-renew", and
413 because the server uses a self-signed certificate we must use
414 --insecure and "sslverify=false".
415 SERVER=esx.example.com
417 DCPATH=data
418 DS=datastore1
419 GUEST=guest-name
420 URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
421 nbdkit curl "$URL" \
423 cookie-script='
424 curl --head -s --insecure -u root:password "$url" |
425 sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
426 ' \
427 cookie-script-renew=500 \
428 sslverify=false
429 Example: Docker Hub authorization tokens
431 Accessing objects like container layers from Docker Hub requires that
432 you first fetch an authorization token, even for anonymous access.
433 These tokens expire after about 5 minutes (300 seconds) so must be
434 periodically renewed.
435 You will need this authorization script (/tmp/auth.sh):
437 #!/bin/sh -
439 IMAGE=library/fedora
440 curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
441 jq -r .token
442 You will also need this script to get the blobSum of the layer
444 (/tmp/blobsum.sh):
445 #!/bin/sh -
447 TOKEN=`/tmp/auth.sh`
448 IMAGE=library/fedora
449 curl -s -X GET -H "Authorization: Bearer $TOKEN" \
450 "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
451 jq -r '.fsLayers[0].blobSum'
452 Both scripts must be executable, and both can be run on their own to
454 check they are working. To run nbdkit:
455 IMAGE=library/fedora
457 BLOBSUM=`/tmp/blobsum.sh`
458 URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
459 nbdkit curl "$URL" \
461 header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
462 header-script-renew=200 \
463 --filter=gzip
464 Note that this exposes a tar file over NBD. See also
466 nbdkit-tar-filter(1).
467
469 -D curl.scripts=1
470 This prints out the headers and cookies generated by the
471 "header-script" and "cookie-script" options, which can be useful
472 when debugging these scripts.
473 -D curl.times=1
475 This prints out additional information about the total time taken
476 to do name resolution, connect to the remote server, etc. The
477 information is printed in the debug output before nbdkit exits.
478 The output will look like:
479 nbdkit: debug: times (-D curl.times=1):
481 nbdkit: debug: name resolution : 0.128442 s
482 nbdkit: debug: connection : 4.945213 s
483 nbdkit: debug: SSL negotiation : 4.291362 s
484 nbdkit: debug: pretransfer : 0.104137 s
485 nbdkit: debug: first byte received : 56.115269 s
486 nbdkit: debug: data transfer : 222.633831 s
487 nbdkit: debug: redirection time : 0.000000 s
488 The cumulative time taken to perform each step is shown (summed
490 across all HTTP connections). The redirection time is the total
491 time taken doing HTTP redirections. For further information see
492 "TIMES" in curl_easy_getinfo(3).
493 -D curl.verbose=1
495 This enables very verbose curl debugging. See CURLOPT_VERBOSE(3).
496 This is mainly useful if you suspect there is a bug inside libcurl
497 itself.
498 -D curl.verbose.ids=1
500 This enhances "-D curl.verbose=1" by printing connection and
501 transfer IDs next to each debug message. As this has some overhead
502 it is not enabled by default.
503
505 $plugindir/nbdkit-curl-plugin.so
506 The plugin.
507 Use "nbdkit --dump-config" to find the location of $plugindir.
509
511 "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
512
514 curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
515 CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
516 CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_IPRESOLVE(3),
517 CURLOPT_PROXY(3), CURLOPT_RESOLVE(3), CURLOPT_SSL_CIPHER_LIST(3),
518 CURLOPT_SSLVERSION(3), CURLOPT_TCP_KEEPALIVE(3),
519 CURLOPT_TCP_NODELAY(3), CURLOPT_TLS13_CIPHERS(3), CURLOPT_URL(3),
520 CURLOPT_UNIX_SOCKET_PATH(3), CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3),
521 nbdkit(1), nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
522 nbdkit-retry-filter(1), nbdkit-retry-request-filter(1),
523 nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3),
524 http://curl.haxx.se, https://curl.se/docs/ssl-ciphers.html
525
527 Richard W.M. Jones
528 Parts derived from Alexander Graf's "QEMU Block driver for CURL
530 images".
531
533 Copyright Red Hat
534
536 Redistribution and use in source and binary forms, with or without
537 modification, are permitted provided that the following conditions are
538 met:
539 • Redistributions of source code must retain the above copyright
541 notice, this list of conditions and the following disclaimer.
542 • Redistributions in binary form must reproduce the above copyright
544 notice, this list of conditions and the following disclaimer in the
545 documentation and/or other materials provided with the
546 distribution.
547 • Neither the name of Red Hat nor the names of its contributors may
549 be used to endorse or promote products derived from this software
550 without specific prior written permission.
551 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
553 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
554 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
555 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
556 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
557 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
558 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
559 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
560 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
561 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
562 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
563nbdkit-1.36.2 2023-11-26 nbdkit-curl-plugin(1)