1nbdkit-curl-plugin(1) NBDKIT nbdkit-curl-plugin(1)
2
6 nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7
9 nbdkit -r curl [url=]http://example.com/disk.img
10
12 "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13 served over HTTP, FTP, and more, into a Network Block Device. It uses
14 a library called libcurl (also known as cURL) to read data from URLs.
15 The exact list of protocols that libcurl can handle depends on how it
16 was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17 more (see: "curl -V").
18 Note: This plugin supports writes. However for HTTP, you may not want
20 nbdkit to issue PUT requests to the remote server (which probably
21 doesn't understand them). To force nbdkit to use a readonly
22 connection, pass the -r flag.
23 Although this plugin can access SFTP (ie. SSH) servers, it is much
25 better to use nbdkit-ssh-plugin(1). This plugin can be used to access
26 "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
27
29 nbdkit -r curl http://example.com/disk.img
30 serves the remote disk image as NBD on TCP port 10809 (to control ports
32 and protocols used to serve NBD see nbdkit(1)).
33
35 cainfo=FILENAME
36 (nbdkit ≥ 1.18)
37 Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
39 capath=PATH
41 (nbdkit ≥ 1.18)
42 Set CA certificates directory location for libcurl. See
44 CURLOPT_CAPATH(3) for more information.
45 cookie=COOKIE
47 cookie=+FILENAME
48 cookie=-
49 cookie=-FD
50 (nbdkit ≥ 1.12)
51 Set a cookie in the request header when connecting to the remote
53 server.
54 A typical example is:
56 cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
58 This option can be used at most once. It only works for HTTP and
60 HTTPS transports. To set multiple cookies you must concatenate
61 them yourself, eg:
62 cookie='name1=content1; name2=content2'
64 See CURLOPT_COOKIE(3) for more information about this. The format
66 is quite strict and must consist of "key=value", each cookie
67 separated by exactly "; " (semicolon and space).
68 If the cookie is used for authentication then passing it on the
70 command line is not secure on shared machines. Use the alternate
71 "+FILENAME" syntax to pass it in a file, "-" to read the cookie
72 interactively, or "-FD" to read it from a file descriptor.
73 cookiefile=
75 (nbdkit ≥ 1.26)
76 Enable cookie processing (eg. when following redirects), starting
78 with an empty list of cookies. This is equivalent to calling
79 CURLOPT_COOKIEFILE(3) with an empty string.
80 cookiefile=FILENAME
82 (nbdkit ≥ 1.26)
83 Enable cookie processing (eg. when following redirects),
85 prepopulating cookies from the given file. The file can contain
86 cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
87 Cookies sent by the server are not saved when nbdkit exits.
88 cookiejar=FILENAME
90 (nbdkit ≥ 1.26)
91 Enable cookie processing (eg. when following redirects),
93 prepopulating cookies from the given file, and writing server
94 cookies back to the file when the NBD handle is closed. The file
95 can contain cookies in any format supported by curl, see
96 CURLOPT_COOKIEJAR(3).
97 There is some advice on the internet telling you to set this to
99 /dev/null, but you should not do this because it can corrupt
100 /dev/null. If you don't want a cookiejar, omit this option. If
101 you want to enable cookie processing without updating a permanent
102 cookiejar, use the "cookiefile=" option instead.
103 cookie-script=SCRIPT
105 cookie-script-renew=SECS
106 (nbdkit ≥ 1.22, not Windows)
107 Run "SCRIPT" (a command or shell script fragment) to generate the
109 HTTP/HTTPS cookies. "cookie-script" cannot be used with "cookie".
110 See "HEADER AND COOKIE SCRIPTS" below.
111 followlocation=false
113 (nbdkit ≥ 1.26)
114 Follow redirects from the server. The default is true (to follow
116 redirects), but you can set this to false to prevent this.
117 header=HEADER
119 (nbdkit ≥ 1.22)
120 For HTTP/HTTPS, send a custom header, or remove a header that curl
122 has added. To add a custom header:
123 header='X-My-Name: John Doe'
125 To remove a header that curl has added, add the header followed by
127 a colon and no value:
128 header='User-Agent:'
130 To add a custom header that has no value, you have to use a
132 semicolon instead of colon. This adds an "X-Empty:" header with no
133 value:
134 header='X-Empty;'
136 See CURLOPT_HTTPHEADER(3). You can use this option multiple times
138 in order to add several headers. Note this sends the header in all
139 requests, even when following a redirect, which can cause headers
140 (eg. containing sensitive authorization information) to be sent to
141 hosts other than the one originally requested.
142 header-script=SCRIPT
144 header-script-renew=SECS
145 (nbdkit ≥ 1.22, not Windows)
146 Run "SCRIPT" (a command or shell script fragment) to generate the
148 HTTP/HTTPS headers. "header-script" cannot be used with "header".
149 See "HEADER AND COOKIE SCRIPTS" below.
150 password=PASSWORD
152 Set the password to use when connecting to the remote server.
153 Note that passing this on the command line is not secure on shared
155 machines.
156 password=-
158 Ask for the password (interactively) when nbdkit starts up.
159 password=+FILENAME
161 Read the password from the named file. This is a secure method to
162 supply a password, as long as you set the permissions on the file
163 appropriately.
164 password=-FD
166 Read the password from file descriptor number "FD", inherited from
167 the parent process when nbdkit starts up. This is also a secure
168 method to supply a password.
169 protocols=PROTO,PROTO,...
171 (nbdkit ≥ 1.12)
172 Limit the protocols that are allowed in the URL. Use this option
174 for extra security if the URL comes from an untrusted source and
175 you want to avoid security isues in the more obscure protocols that
176 curl supports. (See qemu CVE-2013-0249 for an example of a
177 security bug introduced by allowing unrestricted protocols).
178 For example if you only intend HTTP and HTTPS URLs to be used, then
180 add this parameter: "protocols=http,https"
181 This restriction also applies if the plugin follows a redirect to
183 another protocol (eg. you start with an "https://" URL which the
184 server redirects to "ftp://"). To prevent redirects altogether see
185 the "followlocation" parameter.
186 The value of this parameter is a comma-separated list of protocols.
188 The following protocols are known: dict, file, ftp, ftps, gopher,
189 http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp,
190 rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs,
191 smtp, smtps, telnet, tftp.
192 The default is to allow any protocol.
194 proxy=PROXY
196 (nbdkit ≥ 1.20)
197 Set the proxy. See CURLOPT_PROXY(3).
199 proxy-password=PASSWORD
201 proxy-password=-
202 proxy-password=+FILENAME
203 proxy-password=-FD
204 proxy-user=USERNAME
205 (nbdkit ≥ 1.12)
206 Set the proxy username and password.
208 sslverify=false
210 Don't verify the SSL certificate of the remote host.
211 tcp-keepalive=true
213 (nbdkit ≥ 1.20)
214 Enable TCP keepalives.
216 tcp-nodelay=false
218 (nbdkit ≥ 1.20)
219 Disable Nagle’s algorithm.
221 timeout=SECS
223 Set the timeout for requests.
224 timeout=0
226 Use the default libcurl timeout for requests.
227 unix-socket-path=PATH
229 (nbdkit ≥ 1.10)
230 Instead of using a TCP connection, connect to the server over the
232 named Unix domain socket. See CURLOPT_UNIX_SOCKET_PATH(3).
233 [url=]URL
235 The URL of the remote disk image. This is passed to libcurl
236 directly via CURLOPT_URL(3).
237 This parameter is required.
239 "url=" is a magic config key and may be omitted in most cases. See
241 "Magic parameters" in nbdkit(1).
242 user=USERNAME
244 Set the username to use when connecting to the remote server. This
245 may also be set in the URL (eg. "http://foo@example.com/disk.img")
246 user-agent=USER-AGENT
248 (nbdkit ≥ 1.22)
249 Send user-agent header when using HTTP or HTTPS. The default is no
251 user-agent header.
252
254 While the "header" and "cookie" parameters can be used to specify
255 static headers and cookies which are used in every HTTP/HTTPS request,
256 the alternate "header-script" and "cookie-script" parameters can be
257 used to run an external script or program to generate headers and/or
258 cookies. This is particularly useful to access services which require
259 an authorization token. In addition the "header-script-renew" and
260 "cookie-script-renew" parameters allow you to renew the authorization
261 token by rerunning the script periodically.
262 "header-script" is incompatible with "header", and "cookie-script" is
264 incompatible with "cookie".
265 Header script
267 The header script should print zero or more HTTP headers, each line of
268 output in the same format as the "header" parameter. The headers
269 printed by the script are passed to CURLOPT_HTTPHEADER(3).
270 In the following example, an imaginary web service requires
272 authentication using a token fetched from a separate login server. The
273 token expires after 60 seconds, so we also tell the plugin that it must
274 renew the token (by re-running the script) if more than 50 seconds have
275 elapsed since the last request:
276 nbdkit curl https://service.example.com/disk.img \
278 header-script='
279 printf "Authorization: Bearer "
280 curl -s -X POST https://auth.example.com/login |
281 jq -r .token
282 ' \
283 header-script-renew=50
284 Cookie script
286 The cookie script should print a single line in the same format as the
287 "cookie" parameter. This is passed to CURLOPT_COOKIE(3).
288 Header and cookie script shell variables
290 Within the "header-script" and "cookie-script" the following shell
291 variables are available:
292 $iteration
294 The number of times that the script has been called. The first
295 time the script is called this contains 0.
296 $url
298 The URL as passed to the plugin.
299 Example: VMware ESXi cookies
301 VMware ESXi’s web server can expose both VMDK and raw format disk
302 images, but requires you to log in using HTTP Basic Authentication.
303 While you can use the "user" and "password" parameters to send HTTP
304 Basic Authentication headers in every request, tests have shown that it
305 is faster to accept the cookie which the server returns and send that
306 instead. (It is not clear why it is faster, but one theory is that
307 VMware has to do a more expensive username and password check each
308 time.)
309 The web server can be accessed as below. Since the cookie expires
311 after a certain period of time, we use "cookie-script-renew", and
312 because the server uses a self-signed certificate we must use
313 --insecure and "sslverify=false".
314 SERVER=esx.example.com
316 DCPATH=data
317 DS=datastore1
318 GUEST=guest-name
319 URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
320 nbdkit curl "$URL" \
322 cookie-script='
323 curl --head -s --insecure -u root:password "$url" |
324 sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
325 ' \
326 cookie-script-renew=500 \
327 sslverify=false
328 Example: Docker Hub authorization tokens
330 Accessing objects like container layers from Docker Hub requires that
331 you first fetch an authorization token, even for anonymous access.
332 These tokens expire after about 5 minutes (300 seconds) so must be
333 periodically renewed.
334 You will need this authorization script (/tmp/auth.sh):
336 #!/bin/sh -
338 IMAGE=library/fedora
339 curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
340 jq -r .token
341 You will also need this script to get the blobSum of the layer
343 (/tmp/blobsum.sh):
344 #!/bin/sh -
346 TOKEN=`/tmp/auth.sh`
347 IMAGE=library/fedora
348 curl -s -X GET -H "Authorization: Bearer $TOKEN" \
349 "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
350 jq -r '.fsLayers[0].blobSum'
351 Both scripts must be executable, and both can be run on their own to
353 check they are working. To run nbdkit:
354 IMAGE=library/fedora
356 BLOBSUM=`/tmp/blobsum.sh`
357 URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
358 nbdkit curl "$URL" \
360 header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
361 header-script-renew=200 \
362 --filter=gzip
363 Note that this exposes a tar file over NBD. See also
365 nbdkit-tar-filter(1).
366
368 -D curl.scripts=1
369 This prints out the headers and cookies generated by the
370 "header-script" and "cookie-script" options, which can be useful
371 when debugging these scripts.
372 -D curl.verbose=1
374 This enables very verbose curl debugging. See CURLOPT_VERBOSE(3).
375 This is mainly useful if you suspect there is a bug inside libcurl
376 itself.
377
379 $plugindir/nbdkit-curl-plugin.so
380 The plugin.
381 Use "nbdkit --dump-config" to find the location of $plugindir.
383
385 "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
386
388 curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
389 CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
390 CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3),
391 CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3), CURLOPT_URL(3),
392 CURLOPT_UNIX_SOCKET_PATH(3), CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3),
393 nbdkit(1), nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
394 nbdkit-readahead-filter(1), nbdkit-retry-filter(1),
395 nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3),
396 http://curl.haxx.se.
397
399 Richard W.M. Jones
400 Parts derived from Alexander Graf's "QEMU Block driver for CURL
402 images".
403
405 Copyright (C) 2014-2020 Red Hat Inc.
406
408 Redistribution and use in source and binary forms, with or without
409 modification, are permitted provided that the following conditions are
410 met:
411 • Redistributions of source code must retain the above copyright
413 notice, this list of conditions and the following disclaimer.
414 • Redistributions in binary form must reproduce the above copyright
416 notice, this list of conditions and the following disclaimer in the
417 documentation and/or other materials provided with the
418 distribution.
419 • Neither the name of Red Hat nor the names of its contributors may
421 be used to endorse or promote products derived from this software
422 without specific prior written permission.
423 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
425 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
426 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
427 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
428 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
429 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
430 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
431 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
432 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
433 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
434 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
435nbdkit-1.28.2 2021-11-09 nbdkit-curl-plugin(1)