1nbdkit-curl-plugin(1) NBDKIT nbdkit-curl-plugin(1)
2
6 nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7
9 nbdkit -r curl [url=]http://example.com/disk.img
10
12 "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13 served over HTTP, FTP, and more, into a Network Block Device. It uses
14 a library called libcurl (also known as cURL) to read data from URLs.
15 The exact list of protocols that libcurl can handle depends on how it
16 was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17 more (see: "curl -V").
18 Note: This plugin supports writes. However for HTTP, you may not want
20 nbdkit to issue PUT requests to the remote server (which probably
21 doesn't understand them). To force nbdkit to use a readonly
22 connection, pass the -r flag. Using the -r flag also enables NBD
23 multi-conn, which usually performs much better (if the client supports
24 it).
25 Although this plugin can access SFTP (ie. SSH) servers, it is much
27 better to use nbdkit-ssh-plugin(1). This plugin can be used to access
28 "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
29
31 nbdkit -r curl http://example.com/disk.img
32 serves the remote disk image as NBD on TCP port 10809 (to control ports
34 and protocols used to serve NBD see nbdkit(1)).
35
37 cainfo=FILENAME
38 (nbdkit ≥ 1.18)
39 Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
41 Pass empty string in order to not use the default certificate store
43 that libcurl is compiled with.
44 capath=PATH
46 (nbdkit ≥ 1.18)
47 Set CA certificates directory location for libcurl. See
49 CURLOPT_CAPATH(3) for more information.
50 connections=N
52 (nbdkit ≥ 1.34)
53 Open up to "N" curl connections to the web server. The default is
55 4. Curl connections are shared between all NBD clients, so you may
56 wish to increase this if you expect many simultaneous NBD clients
57 (or a single client using many multi-conn connections).
58 See "NBD CONNECTIONS AND CURL HANDLES" below.
60 cookie=COOKIE
62 cookie=+FILENAME
63 cookie=-
64 cookie=-FD
65 (nbdkit ≥ 1.12)
66 Set a cookie in the request header when connecting to the remote
68 server.
69 A typical example is:
71 cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
73 This option can be used at most once. It only works for HTTP and
75 HTTPS transports. To set multiple cookies you must concatenate
76 them yourself, eg:
77 cookie='name1=content1; name2=content2'
79 See CURLOPT_COOKIE(3) for more information about this. The format
81 is quite strict and must consist of "key=value", each cookie
82 separated by exactly "; " (semicolon and space).
83 If the cookie is used for authentication then passing it on the
85 command line is not secure on shared machines. Use the alternate
86 "+FILENAME" syntax to pass it in a file, "-" to read the cookie
87 interactively, or "-FD" to read it from a file descriptor.
88 cookiefile=
90 (nbdkit ≥ 1.26)
91 Enable cookie processing (eg. when following redirects), starting
93 with an empty list of cookies. This is equivalent to calling
94 CURLOPT_COOKIEFILE(3) with an empty string.
95 cookiefile=FILENAME
97 (nbdkit ≥ 1.26)
98 Enable cookie processing (eg. when following redirects),
100 prepopulating cookies from the given file. The file can contain
101 cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
102 Cookies sent by the server are not saved when nbdkit exits.
103 cookiejar=FILENAME
105 (nbdkit ≥ 1.26)
106 Enable cookie processing (eg. when following redirects),
108 prepopulating cookies from the given file, and writing server
109 cookies back to the file when the NBD handle is closed. The file
110 can contain cookies in any format supported by curl, see
111 CURLOPT_COOKIEJAR(3).
112 There is some advice on the internet telling you to set this to
114 /dev/null, but you should not do this because it can corrupt
115 /dev/null. If you don't want a cookiejar, omit this option. If
116 you want to enable cookie processing without updating a permanent
117 cookiejar, use the "cookiefile=" option instead.
118 cookie-script=SCRIPT
120 cookie-script-renew=SECS
121 (nbdkit ≥ 1.22, not Windows)
122 Run "SCRIPT" (a command or shell script fragment) to generate the
124 HTTP/HTTPS cookies. "cookie-script" cannot be used with "cookie".
125 See "HEADER AND COOKIE SCRIPTS" below.
126 followlocation=false
128 (nbdkit ≥ 1.26)
129 Do not follow redirects from the server. The default is true
131 (follow redirects).
132 You can follow redirects but avoid redirecting to a less secure
134 protocol (eg. HTTPS redirecting to FTP) by using the "protocols"
135 parameter instead.
136 header=HEADER
138 (nbdkit ≥ 1.22)
139 For HTTP/HTTPS, send a custom header, or remove a header that curl
141 has added. To add a custom header:
142 header='X-My-Name: John Doe'
144 To remove a header that curl has added, add the header followed by
146 a colon and no value:
147 header='User-Agent:'
149 To add a custom header that has no value, you have to use a
151 semicolon instead of colon. This adds an "X-Empty:" header with no
152 value:
153 header='X-Empty;'
155 See CURLOPT_HTTPHEADER(3). You can use this option multiple times
157 in order to add several headers. Note this sends the header in all
158 requests, even when following a redirect, which can cause headers
159 (eg. containing sensitive authorization information) to be sent to
160 hosts other than the one originally requested.
161 header-script=SCRIPT
163 header-script-renew=SECS
164 (nbdkit ≥ 1.22, not Windows)
165 Run "SCRIPT" (a command or shell script fragment) to generate the
167 HTTP/HTTPS headers. "header-script" cannot be used with "header".
168 See "HEADER AND COOKIE SCRIPTS" below.
169 http-version=none
171 http-version=1.0
172 http-version=1.1
173 http-version=2.0
174 http-version=2TLS
175 http-version=2-prior-knowledge
176 http-version=3
177 http-version=3only
178 (nbdkit ≥ 1.34)
179 Force curl to use a particular HTTP protocol. The default is
181 "none" meaning curl will negotiate the best protocol with the
182 server. The other settings are mainly for testing. See
183 CURLOPT_HTTP_VERSION(3) for details.
184 password=PASSWORD
186 Set the password to use when connecting to the remote server.
187 Note that passing this on the command line is not secure on shared
189 machines.
190 password=-
192 Ask for the password (interactively) when nbdkit starts up.
193 password=+FILENAME
195 Read the password from the named file. This is a secure method to
196 supply a password, as long as you set the permissions on the file
197 appropriately.
198 password=-FD
200 Read the password from file descriptor number "FD", inherited from
201 the parent process when nbdkit starts up. This is also a secure
202 method to supply a password.
203 protocols=PROTO,PROTO,...
205 (nbdkit ≥ 1.12)
206 Limit the protocols that are allowed in the URL. Use this option
208 for extra security if the URL comes from an untrusted source and
209 you want to avoid security isues in the more obscure protocols that
210 curl supports. (See qemu CVE-2013-0249 for an example of a
211 security bug introduced by allowing unrestricted protocols).
212 For example if you only intend HTTP and HTTPS URLs to be used, then
214 add this parameter: "protocols=http,https"
215 This restriction also applies if the plugin follows a redirect to
217 another protocol (eg. you start with an "https://" URL which the
218 server redirects to "ftp://"). To prevent redirects altogether see
219 the "followlocation" parameter.
220 The value of this parameter is a comma-separated list of protocols,
222 for example "protocols=https", or "protocols=http,https", or
223 "protocols=file,ftp,gopher". For a list of known protocols, see
224 the libcurl manual (CURLOPT_PROTOCOLS_STR(3)).
225 The default is to allow any protocol.
227 proxy=PROXY
229 (nbdkit ≥ 1.20)
230 Set the proxy. See CURLOPT_PROXY(3).
232 proxy-password=PASSWORD
234 proxy-password=-
235 proxy-password=+FILENAME
236 proxy-password=-FD
237 proxy-user=USERNAME
238 (nbdkit ≥ 1.12)
239 Set the proxy username and password.
241 sslverify=false
243 Don't verify the SSL certificate of the remote host.
244 ssl-cipher-list=CIPHER[:CIPHER...]
246 ssl-version=default
247 ssl-version=tlsv1
248 ssl-version=sslv2
249 ssl-version=sslv3
250 ssl-version=tlsv1.0
251 ssl-version=tlsv1.1
252 ssl-version=tlsv1.2
253 ssl-version=tlsv1.3
254 ssl-version=max-default
255 ssl-version=max-tlsv1.0
256 ssl-version=max-tlsv1.1
257 ssl-version=max-tlsv1.2
258 ssl-version=max-tlsv1.3
259 Set the SSL ciphers and TLS version. For further information see
260 CURLOPT_SSL_CIPHER_LIST(3) and CURLOPT_SSLVERSION(3).
261 tcp-keepalive=true
263 (nbdkit ≥ 1.20)
264 Enable TCP keepalives.
266 tcp-nodelay=false
268 (nbdkit ≥ 1.20)
269 Enable Nagle’s algorithm. Small writes on the network socket will
271 not be sent immediately but will be held in a local buffer and
272 coalesced if possible. This is more efficient for the network but
273 can cause increased latency.
274 The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is
276 disabled for better latency at the expense of network efficiency.
277 See CURLOPT_TCP_NODELAY(3).
279 timeout=SECS
281 Set the timeout for requests.
282 timeout=0
284 Use the default libcurl timeout for requests.
285 tls13-ciphers=CIPHER[:CIPHER...]
287 Select TLSv1.3 ciphers available. See CURLOPT_TLS13_CIPHERS(3) and
288 https://curl.se/docs/ssl-ciphers.html
289 unix-socket-path=PATH
291 (nbdkit ≥ 1.10)
292 Instead of using a TCP connection, connect to the server over the
294 named Unix domain socket. See CURLOPT_UNIX_SOCKET_PATH(3).
295 [url=]URL
297 The URL of the remote disk image. This is passed to libcurl
298 directly via CURLOPT_URL(3).
299 This parameter is required.
301 "url=" is a magic config key and may be omitted in most cases. See
303 "Magic parameters" in nbdkit(1).
304 user=USERNAME
306 Set the username to use when connecting to the remote server. This
307 may also be set in the URL (eg. "http://foo@example.com/disk.img")
308 user-agent=USER-AGENT
310 (nbdkit ≥ 1.22)
311 Send user-agent header when using HTTP or HTTPS. The default is no
313 user-agent header.
314
316 nbdkit ≤ 1.32 used a simple model where a new NBD connection would
317 create a new libcurl handle. In practice this meant there was a 1-to-1
318 relationship between NBD connections and HTTP connections to the remote
319 web server (assuming http: or https: URL).
320 nbdkit ≥ 1.34 changed to using a fixed pool of libcurl handles shared
322 across all NBD connections. You can control the maximum number of curl
323 handles in the pool with the "connections" parameter (default 4). Note
324 that if there are more than 4 NBD connections, they will share the 4
325 web server connections, unless you adjust "connections".
326
328 While the "header" and "cookie" parameters can be used to specify
329 static headers and cookies which are used in every HTTP/HTTPS request,
330 the alternate "header-script" and "cookie-script" parameters can be
331 used to run an external script or program to generate headers and/or
332 cookies. This is particularly useful to access services which require
333 an authorization token. In addition the "header-script-renew" and
334 "cookie-script-renew" parameters allow you to renew the authorization
335 token by rerunning the script periodically.
336 "header-script" is incompatible with "header", and "cookie-script" is
338 incompatible with "cookie".
339 Header script
341 The header script should print zero or more HTTP headers, each line of
342 output in the same format as the "header" parameter. The headers
343 printed by the script are passed to CURLOPT_HTTPHEADER(3).
344 In the following example, an imaginary web service requires
346 authentication using a token fetched from a separate login server. The
347 token expires after 60 seconds, so we also tell the plugin that it must
348 renew the token (by re-running the script) if more than 50 seconds have
349 elapsed since the last request:
350 nbdkit curl https://service.example.com/disk.img \
352 header-script='
353 printf "Authorization: Bearer "
354 curl -s -X POST https://auth.example.com/login |
355 jq -r .token
356 ' \
357 header-script-renew=50
358 Cookie script
360 The cookie script should print a single line in the same format as the
361 "cookie" parameter. This is passed to CURLOPT_COOKIE(3).
362 Header and cookie script shell variables
364 Within the "header-script" and "cookie-script" the following shell
365 variables are available:
366 $iteration
368 The number of times that the script has been called. The first
369 time the script is called this contains 0.
370 $url
372 The URL as passed to the plugin.
373 Example: VMware ESXi cookies
375 VMware ESXi’s web server can expose both VMDK and raw format disk
376 images, but requires you to log in using HTTP Basic Authentication.
377 While you can use the "user" and "password" parameters to send HTTP
378 Basic Authentication headers in every request, tests have shown that it
379 is faster to accept the cookie which the server returns and send that
380 instead. (It is not clear why it is faster, but one theory is that
381 VMware has to do a more expensive username and password check each
382 time.)
383 The web server can be accessed as below. Since the cookie expires
385 after a certain period of time, we use "cookie-script-renew", and
386 because the server uses a self-signed certificate we must use
387 --insecure and "sslverify=false".
388 SERVER=esx.example.com
390 DCPATH=data
391 DS=datastore1
392 GUEST=guest-name
393 URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
394 nbdkit curl "$URL" \
396 cookie-script='
397 curl --head -s --insecure -u root:password "$url" |
398 sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
399 ' \
400 cookie-script-renew=500 \
401 sslverify=false
402 Example: Docker Hub authorization tokens
404 Accessing objects like container layers from Docker Hub requires that
405 you first fetch an authorization token, even for anonymous access.
406 These tokens expire after about 5 minutes (300 seconds) so must be
407 periodically renewed.
408 You will need this authorization script (/tmp/auth.sh):
410 #!/bin/sh -
412 IMAGE=library/fedora
413 curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
414 jq -r .token
415 You will also need this script to get the blobSum of the layer
417 (/tmp/blobsum.sh):
418 #!/bin/sh -
420 TOKEN=`/tmp/auth.sh`
421 IMAGE=library/fedora
422 curl -s -X GET -H "Authorization: Bearer $TOKEN" \
423 "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
424 jq -r '.fsLayers[0].blobSum'
425 Both scripts must be executable, and both can be run on their own to
427 check they are working. To run nbdkit:
428 IMAGE=library/fedora
430 BLOBSUM=`/tmp/blobsum.sh`
431 URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
432 nbdkit curl "$URL" \
434 header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
435 header-script-renew=200 \
436 --filter=gzip
437 Note that this exposes a tar file over NBD. See also
439 nbdkit-tar-filter(1).
440
442 -D curl.scripts=1
443 This prints out the headers and cookies generated by the
444 "header-script" and "cookie-script" options, which can be useful
445 when debugging these scripts.
446 -D curl.verbose=1
448 This enables very verbose curl debugging. See CURLOPT_VERBOSE(3).
449 This is mainly useful if you suspect there is a bug inside libcurl
450 itself.
451
453 $plugindir/nbdkit-curl-plugin.so
454 The plugin.
455 Use "nbdkit --dump-config" to find the location of $plugindir.
457
459 "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
460
462 curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
463 CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
464 CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3),
465 CURLOPT_SSL_CIPHER_LIST(3), CURLOPT_SSLVERSION(3),
466 CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3),
467 CURLOPT_TLS13_CIPHERS(3), CURLOPT_URL(3), CURLOPT_UNIX_SOCKET_PATH(3),
468 CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3), nbdkit(1),
469 nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
470 nbdkit-retry-filter(1), nbdkit-retry-request-filter(1),
471 nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3),
472 http://curl.haxx.se, https://curl.se/docs/ssl-ciphers.html
473
475 Richard W.M. Jones
476 Parts derived from Alexander Graf's "QEMU Block driver for CURL
478 images".
479
481 Copyright Red Hat
482
484 Redistribution and use in source and binary forms, with or without
485 modification, are permitted provided that the following conditions are
486 met:
487 • Redistributions of source code must retain the above copyright
489 notice, this list of conditions and the following disclaimer.
490 • Redistributions in binary form must reproduce the above copyright
492 notice, this list of conditions and the following disclaimer in the
493 documentation and/or other materials provided with the
494 distribution.
495 • Neither the name of Red Hat nor the names of its contributors may
497 be used to endorse or promote products derived from this software
498 without specific prior written permission.
499 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
501 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
502 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
503 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
504 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
505 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
506 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
507 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
508 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
509 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
510 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
511nbdkit-1.34.4 2023-09-26 nbdkit-curl-plugin(1)