1nbdkit-curl-plugin(1) NBDKIT nbdkit-curl-plugin(1)
2
6 nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7
9 nbdkit -r curl [url=]http://example.com/disk.img
10
12 "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13 served over HTTP, FTP, and more, into a Network Block Device. It uses
14 a library called libcurl (also known as cURL) to read data from URLs.
15 The exact list of protocols that libcurl can handle depends on how it
16 was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17 more (see: "curl -V").
18 Note: This plugin supports writes. However for HTTP, you may not want
20 nbdkit to issue PUT requests to the remote server (which probably
21 doesn't understand them). To force nbdkit to use a readonly
22 connection, pass the -r flag.
23 Although this plugin can access SFTP (ie. SSH) servers, it is much
25 better to use nbdkit-ssh-plugin(1). This plugin can be used to access
26 "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
27
29 nbdkit -r curl http://example.com/disk.img
30 serves the remote disk image as NBD on TCP port 10809 (to control ports
32 and protocols used to serve NBD see nbdkit(1)).
33
35 cainfo=FILENAME
36 (nbdkit ≥ 1.18)
37 Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
39 Pass empty string in order to not use the default certificate store
41 that libcurl is compiled with.
42 capath=PATH
44 (nbdkit ≥ 1.18)
45 Set CA certificates directory location for libcurl. See
47 CURLOPT_CAPATH(3) for more information.
48 cookie=COOKIE
50 cookie=+FILENAME
51 cookie=-
52 cookie=-FD
53 (nbdkit ≥ 1.12)
54 Set a cookie in the request header when connecting to the remote
56 server.
57 A typical example is:
59 cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
61 This option can be used at most once. It only works for HTTP and
63 HTTPS transports. To set multiple cookies you must concatenate
64 them yourself, eg:
65 cookie='name1=content1; name2=content2'
67 See CURLOPT_COOKIE(3) for more information about this. The format
69 is quite strict and must consist of "key=value", each cookie
70 separated by exactly "; " (semicolon and space).
71 If the cookie is used for authentication then passing it on the
73 command line is not secure on shared machines. Use the alternate
74 "+FILENAME" syntax to pass it in a file, "-" to read the cookie
75 interactively, or "-FD" to read it from a file descriptor.
76 cookiefile=
78 (nbdkit ≥ 1.26)
79 Enable cookie processing (eg. when following redirects), starting
81 with an empty list of cookies. This is equivalent to calling
82 CURLOPT_COOKIEFILE(3) with an empty string.
83 cookiefile=FILENAME
85 (nbdkit ≥ 1.26)
86 Enable cookie processing (eg. when following redirects),
88 prepopulating cookies from the given file. The file can contain
89 cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
90 Cookies sent by the server are not saved when nbdkit exits.
91 cookiejar=FILENAME
93 (nbdkit ≥ 1.26)
94 Enable cookie processing (eg. when following redirects),
96 prepopulating cookies from the given file, and writing server
97 cookies back to the file when the NBD handle is closed. The file
98 can contain cookies in any format supported by curl, see
99 CURLOPT_COOKIEJAR(3).
100 There is some advice on the internet telling you to set this to
102 /dev/null, but you should not do this because it can corrupt
103 /dev/null. If you don't want a cookiejar, omit this option. If
104 you want to enable cookie processing without updating a permanent
105 cookiejar, use the "cookiefile=" option instead.
106 cookie-script=SCRIPT
108 cookie-script-renew=SECS
109 (nbdkit ≥ 1.22, not Windows)
110 Run "SCRIPT" (a command or shell script fragment) to generate the
112 HTTP/HTTPS cookies. "cookie-script" cannot be used with "cookie".
113 See "HEADER AND COOKIE SCRIPTS" below.
114 followlocation=false
116 (nbdkit ≥ 1.26)
117 Follow redirects from the server. The default is true (to follow
119 redirects), but you can set this to false to prevent this.
120 header=HEADER
122 (nbdkit ≥ 1.22)
123 For HTTP/HTTPS, send a custom header, or remove a header that curl
125 has added. To add a custom header:
126 header='X-My-Name: John Doe'
128 To remove a header that curl has added, add the header followed by
130 a colon and no value:
131 header='User-Agent:'
133 To add a custom header that has no value, you have to use a
135 semicolon instead of colon. This adds an "X-Empty:" header with no
136 value:
137 header='X-Empty;'
139 See CURLOPT_HTTPHEADER(3). You can use this option multiple times
141 in order to add several headers. Note this sends the header in all
142 requests, even when following a redirect, which can cause headers
143 (eg. containing sensitive authorization information) to be sent to
144 hosts other than the one originally requested.
145 header-script=SCRIPT
147 header-script-renew=SECS
148 (nbdkit ≥ 1.22, not Windows)
149 Run "SCRIPT" (a command or shell script fragment) to generate the
151 HTTP/HTTPS headers. "header-script" cannot be used with "header".
152 See "HEADER AND COOKIE SCRIPTS" below.
153 password=PASSWORD
155 Set the password to use when connecting to the remote server.
156 Note that passing this on the command line is not secure on shared
158 machines.
159 password=-
161 Ask for the password (interactively) when nbdkit starts up.
162 password=+FILENAME
164 Read the password from the named file. This is a secure method to
165 supply a password, as long as you set the permissions on the file
166 appropriately.
167 password=-FD
169 Read the password from file descriptor number "FD", inherited from
170 the parent process when nbdkit starts up. This is also a secure
171 method to supply a password.
172 protocols=PROTO,PROTO,...
174 (nbdkit ≥ 1.12)
175 Limit the protocols that are allowed in the URL. Use this option
177 for extra security if the URL comes from an untrusted source and
178 you want to avoid security isues in the more obscure protocols that
179 curl supports. (See qemu CVE-2013-0249 for an example of a
180 security bug introduced by allowing unrestricted protocols).
181 For example if you only intend HTTP and HTTPS URLs to be used, then
183 add this parameter: "protocols=http,https"
184 This restriction also applies if the plugin follows a redirect to
186 another protocol (eg. you start with an "https://" URL which the
187 server redirects to "ftp://"). To prevent redirects altogether see
188 the "followlocation" parameter.
189 The value of this parameter is a comma-separated list of protocols.
191 The following protocols are known: dict, file, ftp, ftps, gopher,
192 http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp,
193 rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs,
194 smtp, smtps, telnet, tftp.
195 The default is to allow any protocol.
197 proxy=PROXY
199 (nbdkit ≥ 1.20)
200 Set the proxy. See CURLOPT_PROXY(3).
202 proxy-password=PASSWORD
204 proxy-password=-
205 proxy-password=+FILENAME
206 proxy-password=-FD
207 proxy-user=USERNAME
208 (nbdkit ≥ 1.12)
209 Set the proxy username and password.
211 sslverify=false
213 Don't verify the SSL certificate of the remote host.
214 ssl-cipher-list=CIPHER[:CIPHER...]
216 ssl-version=tlsv1
217 ssl-version=sslv2
218 ssl-version=sslv3
219 ssl-version=tlsv1.0
220 ssl-version=tlsv1.1
221 ssl-version=tlsv1.2
222 ssl-version=tlsv1.3
223 Set the SSL ciphers and TLS version. For further information see
224 CURLOPT_SSL_CIPHER_LIST(3) and CURLOPT_SSLVERSION(3).
225 tcp-keepalive=true
227 (nbdkit ≥ 1.20)
228 Enable TCP keepalives.
230 tcp-nodelay=false
232 (nbdkit ≥ 1.20)
233 Enable Nagle’s algorithm. Small writes on the network socket will
235 not be sent immediately but will be held in a local buffer and
236 coalesced if possible. This is more efficient for the network but
237 can cause increased latency.
238 The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is
240 disabled for better latency at the expense of network efficiency.
241 See CURLOPT_TCP_NODELAY(3).
243 timeout=SECS
245 Set the timeout for requests.
246 timeout=0
248 Use the default libcurl timeout for requests.
249 tls13-ciphers=CIPHER[:CIPHER...]
251 Select TLSv1.3 ciphers available. See CURLOPT_TLS13_CIPHERS(3) and
252 https://curl.se/docs/ssl-ciphers.html
253 unix-socket-path=PATH
255 (nbdkit ≥ 1.10)
256 Instead of using a TCP connection, connect to the server over the
258 named Unix domain socket. See CURLOPT_UNIX_SOCKET_PATH(3).
259 [url=]URL
261 The URL of the remote disk image. This is passed to libcurl
262 directly via CURLOPT_URL(3).
263 This parameter is required.
265 "url=" is a magic config key and may be omitted in most cases. See
267 "Magic parameters" in nbdkit(1).
268 user=USERNAME
270 Set the username to use when connecting to the remote server. This
271 may also be set in the URL (eg. "http://foo@example.com/disk.img")
272 user-agent=USER-AGENT
274 (nbdkit ≥ 1.22)
275 Send user-agent header when using HTTP or HTTPS. The default is no
277 user-agent header.
278
280 While the "header" and "cookie" parameters can be used to specify
281 static headers and cookies which are used in every HTTP/HTTPS request,
282 the alternate "header-script" and "cookie-script" parameters can be
283 used to run an external script or program to generate headers and/or
284 cookies. This is particularly useful to access services which require
285 an authorization token. In addition the "header-script-renew" and
286 "cookie-script-renew" parameters allow you to renew the authorization
287 token by rerunning the script periodically.
288 "header-script" is incompatible with "header", and "cookie-script" is
290 incompatible with "cookie".
291 Header script
293 The header script should print zero or more HTTP headers, each line of
294 output in the same format as the "header" parameter. The headers
295 printed by the script are passed to CURLOPT_HTTPHEADER(3).
296 In the following example, an imaginary web service requires
298 authentication using a token fetched from a separate login server. The
299 token expires after 60 seconds, so we also tell the plugin that it must
300 renew the token (by re-running the script) if more than 50 seconds have
301 elapsed since the last request:
302 nbdkit curl https://service.example.com/disk.img \
304 header-script='
305 printf "Authorization: Bearer "
306 curl -s -X POST https://auth.example.com/login |
307 jq -r .token
308 ' \
309 header-script-renew=50
310 Cookie script
312 The cookie script should print a single line in the same format as the
313 "cookie" parameter. This is passed to CURLOPT_COOKIE(3).
314 Header and cookie script shell variables
316 Within the "header-script" and "cookie-script" the following shell
317 variables are available:
318 $iteration
320 The number of times that the script has been called. The first
321 time the script is called this contains 0.
322 $url
324 The URL as passed to the plugin.
325 Example: VMware ESXi cookies
327 VMware ESXi’s web server can expose both VMDK and raw format disk
328 images, but requires you to log in using HTTP Basic Authentication.
329 While you can use the "user" and "password" parameters to send HTTP
330 Basic Authentication headers in every request, tests have shown that it
331 is faster to accept the cookie which the server returns and send that
332 instead. (It is not clear why it is faster, but one theory is that
333 VMware has to do a more expensive username and password check each
334 time.)
335 The web server can be accessed as below. Since the cookie expires
337 after a certain period of time, we use "cookie-script-renew", and
338 because the server uses a self-signed certificate we must use
339 --insecure and "sslverify=false".
340 SERVER=esx.example.com
342 DCPATH=data
343 DS=datastore1
344 GUEST=guest-name
345 URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
346 nbdkit curl "$URL" \
348 cookie-script='
349 curl --head -s --insecure -u root:password "$url" |
350 sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
351 ' \
352 cookie-script-renew=500 \
353 sslverify=false
354 Example: Docker Hub authorization tokens
356 Accessing objects like container layers from Docker Hub requires that
357 you first fetch an authorization token, even for anonymous access.
358 These tokens expire after about 5 minutes (300 seconds) so must be
359 periodically renewed.
360 You will need this authorization script (/tmp/auth.sh):
362 #!/bin/sh -
364 IMAGE=library/fedora
365 curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
366 jq -r .token
367 You will also need this script to get the blobSum of the layer
369 (/tmp/blobsum.sh):
370 #!/bin/sh -
372 TOKEN=`/tmp/auth.sh`
373 IMAGE=library/fedora
374 curl -s -X GET -H "Authorization: Bearer $TOKEN" \
375 "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
376 jq -r '.fsLayers[0].blobSum'
377 Both scripts must be executable, and both can be run on their own to
379 check they are working. To run nbdkit:
380 IMAGE=library/fedora
382 BLOBSUM=`/tmp/blobsum.sh`
383 URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
384 nbdkit curl "$URL" \
386 header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
387 header-script-renew=200 \
388 --filter=gzip
389 Note that this exposes a tar file over NBD. See also
391 nbdkit-tar-filter(1).
392
394 -D curl.scripts=1
395 This prints out the headers and cookies generated by the
396 "header-script" and "cookie-script" options, which can be useful
397 when debugging these scripts.
398 -D curl.verbose=1
400 This enables very verbose curl debugging. See CURLOPT_VERBOSE(3).
401 This is mainly useful if you suspect there is a bug inside libcurl
402 itself.
403
405 $plugindir/nbdkit-curl-plugin.so
406 The plugin.
407 Use "nbdkit --dump-config" to find the location of $plugindir.
409
411 "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
412
414 curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
415 CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
416 CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3),
417 CURLOPT_SSL_CIPHER_LIST(3), CURLOPT_SSLVERSION(3),
418 CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3),
419 CURLOPT_TLS13_CIPHERS(3), CURLOPT_URL(3), CURLOPT_UNIX_SOCKET_PATH(3),
420 CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3), nbdkit(1),
421 nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
422 nbdkit-retry-filter(1), nbdkit-retry-request-filter(1),
423 nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3),
424 http://curl.haxx.se, https://curl.se/docs/ssl-ciphers.html
425
427 Richard W.M. Jones
428 Parts derived from Alexander Graf's "QEMU Block driver for CURL
430 images".
431
433 Copyright (C) 2014-2020 Red Hat Inc.
434
436 Redistribution and use in source and binary forms, with or without
437 modification, are permitted provided that the following conditions are
438 met:
439 • Redistributions of source code must retain the above copyright
441 notice, this list of conditions and the following disclaimer.
442 • Redistributions in binary form must reproduce the above copyright
444 notice, this list of conditions and the following disclaimer in the
445 documentation and/or other materials provided with the
446 distribution.
447 • Neither the name of Red Hat nor the names of its contributors may
449 be used to endorse or promote products derived from this software
450 without specific prior written permission.
451 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
453 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
454 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
455 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
456 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
457 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
458 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
459 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
460 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
461 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
462 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
463nbdkit-1.32.5 2023-01-03 nbdkit-curl-plugin(1)