1nbdkit-curl-plugin(1)               NBDKIT               nbdkit-curl-plugin(1)
2
3
4

NAME

6       nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7

SYNOPSIS

9        nbdkit -r curl [url=]http://example.com/disk.img
10

DESCRIPTION

12       "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13       served over HTTP, FTP, and more, into a Network Block Device.  It uses
14       a library called libcurl (also known as cURL) to read data from URLs.
15       The exact list of protocols that libcurl can handle depends on how it
16       was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17       more (see: "curl -V").
18
19       Note: This plugin supports writes.  However for HTTP, you may not want
20       nbdkit to issue PUT requests to the remote server (which probably
21       doesn't understand them).  To force nbdkit to use a readonly
22       connection, pass the -r flag.
23
24       Although this plugin can access SFTP (ie. SSH) servers, it is much
25       better to use nbdkit-ssh-plugin(1).  This plugin can be used to access
26       "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
27

EXAMPLE

29        nbdkit -r curl http://example.com/disk.img
30
31       serves the remote disk image as NBD on TCP port 10809 (to control ports
32       and protocols used to serve NBD see nbdkit(1)).
33

PARAMETERS

35       cainfo=FILENAME
36           (nbdkit ≥ 1.18)
37
38           Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
39
40           Pass empty string in order to not use the default certificate store
41           that libcurl is compiled with.
42
43       capath=PATH
44           (nbdkit ≥ 1.18)
45
46           Set CA certificates directory location for libcurl. See
47           CURLOPT_CAPATH(3) for more information.
48
49       cookie=COOKIE
50       cookie=+FILENAME
51       cookie=-
52       cookie=-FD
53           (nbdkit ≥ 1.12)
54
55           Set a cookie in the request header when connecting to the remote
56           server.
57
58           A typical example is:
59
60            cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
61
62           This option can be used at most once.  It only works for HTTP and
63           HTTPS transports.  To set multiple cookies you must concatenate
64           them yourself, eg:
65
66            cookie='name1=content1; name2=content2'
67
68           See CURLOPT_COOKIE(3) for more information about this.  The format
69           is quite strict and must consist of "key=value", each cookie
70           separated by exactly "; " (semicolon and space).
71
72           If the cookie is used for authentication then passing it on the
73           command line is not secure on shared machines.  Use the alternate
74           "+FILENAME" syntax to pass it in a file, "-" to read the cookie
75           interactively, or "-FD" to read it from a file descriptor.
76
77       cookiefile=
78           (nbdkit ≥ 1.26)
79
80           Enable cookie processing (eg. when following redirects), starting
81           with an empty list of cookies.  This is equivalent to calling
82           CURLOPT_COOKIEFILE(3) with an empty string.
83
84       cookiefile=FILENAME
85           (nbdkit ≥ 1.26)
86
87           Enable cookie processing (eg. when following redirects),
88           prepopulating cookies from the given file.  The file can contain
89           cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
90           Cookies sent by the server are not saved when nbdkit exits.
91
92       cookiejar=FILENAME
93           (nbdkit ≥ 1.26)
94
95           Enable cookie processing (eg. when following redirects),
96           prepopulating cookies from the given file, and writing server
97           cookies back to the file when the NBD handle is closed.  The file
98           can contain cookies in any format supported by curl, see
99           CURLOPT_COOKIEJAR(3).
100
101           There is some advice on the internet telling you to set this to
102           /dev/null, but you should not do this because it can corrupt
103           /dev/null.  If you don't want a cookiejar, omit this option.  If
104           you want to enable cookie processing without updating a permanent
105           cookiejar, use the "cookiefile=" option instead.
106
107       cookie-script=SCRIPT
108       cookie-script-renew=SECS
109           (nbdkit ≥ 1.22, not Windows)
110
111           Run "SCRIPT" (a command or shell script fragment) to generate the
112           HTTP/HTTPS cookies.  "cookie-script" cannot be used with "cookie".
113           See "HEADER AND COOKIE SCRIPTS" below.
114
115       followlocation=false
116           (nbdkit ≥ 1.26)
117
118           Follow redirects from the server.  The default is true (to follow
119           redirects), but you can set this to false to prevent this.
120
121       header=HEADER
122           (nbdkit ≥ 1.22)
123
124           For HTTP/HTTPS, send a custom header, or remove a header that curl
125           has added.  To add a custom header:
126
127            header='X-My-Name: John Doe'
128
129           To remove a header that curl has added, add the header followed by
130           a colon and no value:
131
132            header='User-Agent:'
133
134           To add a custom header that has no value, you have to use a
135           semicolon instead of colon.  This adds an "X-Empty:" header with no
136           value:
137
138            header='X-Empty;'
139
140           See CURLOPT_HTTPHEADER(3).  You can use this option multiple times
141           in order to add several headers.  Note this sends the header in all
142           requests, even when following a redirect, which can cause headers
143           (eg. containing sensitive authorization information) to be sent to
144           hosts other than the one originally requested.
145
146       header-script=SCRIPT
147       header-script-renew=SECS
148           (nbdkit ≥ 1.22, not Windows)
149
150           Run "SCRIPT" (a command or shell script fragment) to generate the
151           HTTP/HTTPS headers.  "header-script" cannot be used with "header".
152           See "HEADER AND COOKIE SCRIPTS" below.
153
154       password=PASSWORD
155           Set the password to use when connecting to the remote server.
156
157           Note that passing this on the command line is not secure on shared
158           machines.
159
160       password=-
161           Ask for the password (interactively) when nbdkit starts up.
162
163       password=+FILENAME
164           Read the password from the named file.  This is a secure method to
165           supply a password, as long as you set the permissions on the file
166           appropriately.
167
168       password=-FD
169           Read the password from file descriptor number "FD", inherited from
170           the parent process when nbdkit starts up.  This is also a secure
171           method to supply a password.
172
173       protocols=PROTO,PROTO,...
174           (nbdkit ≥ 1.12)
175
176           Limit the protocols that are allowed in the URL.  Use this option
177           for extra security if the URL comes from an untrusted source and
178           you want to avoid security isues in the more obscure protocols that
179           curl supports.  (See qemu CVE-2013-0249 for an example of a
180           security bug introduced by allowing unrestricted protocols).
181
182           For example if you only intend HTTP and HTTPS URLs to be used, then
183           add this parameter: "protocols=http,https"
184
185           This restriction also applies if the plugin follows a redirect to
186           another protocol (eg. you start with an "https://" URL which the
187           server redirects to "ftp://").  To prevent redirects altogether see
188           the "followlocation" parameter.
189
190           The value of this parameter is a comma-separated list of protocols.
191           The following protocols are known: dict, file, ftp, ftps, gopher,
192           http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp,
193           rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs,
194           smtp, smtps, telnet, tftp.
195
196           The default is to allow any protocol.
197
198       proxy=PROXY
199           (nbdkit ≥ 1.20)
200
201           Set the proxy.  See CURLOPT_PROXY(3).
202
203       proxy-password=PASSWORD
204       proxy-password=-
205       proxy-password=+FILENAME
206       proxy-password=-FD
207       proxy-user=USERNAME
208           (nbdkit ≥ 1.12)
209
210           Set the proxy username and password.
211
212       sslverify=false
213           Don't verify the SSL certificate of the remote host.
214
215       ssl-cipher-list=CIPHER[:CIPHER...]
216       ssl-version=tlsv1
217       ssl-version=sslv2
218       ssl-version=sslv3
219       ssl-version=tlsv1.0
220       ssl-version=tlsv1.1
221       ssl-version=tlsv1.2
222       ssl-version=tlsv1.3
223           Set the SSL ciphers and TLS version.  For further information see
224           CURLOPT_SSL_CIPHER_LIST(3) and CURLOPT_SSLVERSION(3).
225
226       tcp-keepalive=true
227           (nbdkit ≥ 1.20)
228
229           Enable TCP keepalives.
230
231       tcp-nodelay=false
232           (nbdkit ≥ 1.20)
233
234           Enable Nagle’s algorithm.  Small writes on the network socket will
235           not be sent immediately but will be held in a local buffer and
236           coalesced if possible.  This is more efficient for the network but
237           can cause increased latency.
238
239           The default (in libcurl ≥ 7.50.2) is that Nagle’s algorithm is
240           disabled for better latency at the expense of network efficiency.
241
242           See CURLOPT_TCP_NODELAY(3).
243
244       timeout=SECS
245           Set the timeout for requests.
246
247       timeout=0
248           Use the default libcurl timeout for requests.
249
250       tls13-ciphers=CIPHER[:CIPHER...]
251           Select TLSv1.3 ciphers available.  See CURLOPT_TLS13_CIPHERS(3) and
252           https://curl.se/docs/ssl-ciphers.html
253
254       unix-socket-path=PATH
255           (nbdkit ≥ 1.10)
256
257           Instead of using a TCP connection, connect to the server over the
258           named Unix domain socket.  See CURLOPT_UNIX_SOCKET_PATH(3).
259
260       [url=]URL
261           The URL of the remote disk image.  This is passed to libcurl
262           directly via CURLOPT_URL(3).
263
264           This parameter is required.
265
266           "url=" is a magic config key and may be omitted in most cases.  See
267           "Magic parameters" in nbdkit(1).
268
269       user=USERNAME
270           Set the username to use when connecting to the remote server.  This
271           may also be set in the URL (eg. "http://foo@example.com/disk.img")
272
273       user-agent=USER-AGENT
274           (nbdkit ≥ 1.22)
275
276           Send user-agent header when using HTTP or HTTPS.  The default is no
277           user-agent header.
278
280       While the "header" and "cookie" parameters can be used to specify
281       static headers and cookies which are used in every HTTP/HTTPS request,
282       the alternate "header-script" and "cookie-script" parameters can be
283       used to run an external script or program to generate headers and/or
284       cookies.  This is particularly useful to access services which require
285       an authorization token.  In addition the "header-script-renew" and
286       "cookie-script-renew" parameters allow you to renew the authorization
287       token by rerunning the script periodically.
288
289       "header-script" is incompatible with "header", and "cookie-script" is
290       incompatible with "cookie".
291
292   Header script
293       The header script should print zero or more HTTP headers, each line of
294       output in the same format as the "header" parameter.  The headers
295       printed by the script are passed to CURLOPT_HTTPHEADER(3).
296
297       In the following example, an imaginary web service requires
298       authentication using a token fetched from a separate login server.  The
299       token expires after 60 seconds, so we also tell the plugin that it must
300       renew the token (by re-running the script) if more than 50 seconds have
301       elapsed since the last request:
302
303        nbdkit curl https://service.example.com/disk.img \
304               header-script='
305                 printf "Authorization: Bearer "
306                 curl -s -X POST https://auth.example.com/login |
307                      jq -r .token
308               ' \
309               header-script-renew=50
310
311   Cookie script
312       The cookie script should print a single line in the same format as the
313       "cookie" parameter.  This is passed to CURLOPT_COOKIE(3).
314
315   Header and cookie script shell variables
316       Within the "header-script" and "cookie-script" the following shell
317       variables are available:
318
319       $iteration
320           The number of times that the script has been called.  The first
321           time the script is called this contains 0.
322
323       $url
324           The URL as passed to the plugin.
325
326   Example: VMware ESXi cookies
327       VMware ESXi’s web server can expose both VMDK and raw format disk
328       images, but requires you to log in using HTTP Basic Authentication.
329       While you can use the "user" and "password" parameters to send HTTP
330       Basic Authentication headers in every request, tests have shown that it
331       is faster to accept the cookie which the server returns and send that
332       instead.  (It is not clear why it is faster, but one theory is that
333       VMware has to do a more expensive username and password check each
334       time.)
335
336       The web server can be accessed as below.  Since the cookie expires
337       after a certain period of time, we use "cookie-script-renew", and
338       because the server uses a self-signed certificate we must use
339       --insecure and "sslverify=false".
340
341        SERVER=esx.example.com
342        DCPATH=data
343        DS=datastore1
344        GUEST=guest-name
345        URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
346
347        nbdkit curl "$URL" \
348               cookie-script='
349                   curl --head -s --insecure -u root:password "$url" |
350                        sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
351               ' \
352               cookie-script-renew=500 \
353               sslverify=false
354
355   Example: Docker Hub authorization tokens
356       Accessing objects like container layers from Docker Hub requires that
357       you first fetch an authorization token, even for anonymous access.
358       These tokens expire after about 5 minutes (300 seconds) so must be
359       periodically renewed.
360
361       You will need this authorization script (/tmp/auth.sh):
362
363        #!/bin/sh -
364        IMAGE=library/fedora
365        curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
366             jq -r .token
367
368       You will also need this script to get the blobSum of the layer
369       (/tmp/blobsum.sh):
370
371        #!/bin/sh -
372        TOKEN=`/tmp/auth.sh`
373        IMAGE=library/fedora
374        curl -s -X GET -H "Authorization: Bearer $TOKEN" \
375             "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
376             jq -r '.fsLayers[0].blobSum'
377
378       Both scripts must be executable, and both can be run on their own to
379       check they are working.  To run nbdkit:
380
381        IMAGE=library/fedora
382        BLOBSUM=`/tmp/blobsum.sh`
383        URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
384
385        nbdkit curl "$URL" \
386               header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
387               header-script-renew=200 \
388               --filter=gzip
389
390       Note that this exposes a tar file over NBD.  See also
391       nbdkit-tar-filter(1).
392

DEBUG FLAGS

394       -D curl.scripts=1
395           This prints out the headers and cookies generated by the
396           "header-script" and "cookie-script" options, which can be useful
397           when debugging these scripts.
398
399       -D curl.verbose=1
400           This enables very verbose curl debugging.  See CURLOPT_VERBOSE(3).
401           This is mainly useful if you suspect there is a bug inside libcurl
402           itself.
403

FILES

405       $plugindir/nbdkit-curl-plugin.so
406           The plugin.
407
408           Use "nbdkit --dump-config" to find the location of $plugindir.
409

VERSION

411       "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
412

SEE ALSO

414       curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
415       CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
416       CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3),
417       CURLOPT_SSL_CIPHER_LIST(3), CURLOPT_SSLVERSION(3),
418       CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3),
419       CURLOPT_TLS13_CIPHERS(3), CURLOPT_URL(3), CURLOPT_UNIX_SOCKET_PATH(3),
420       CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3), nbdkit(1),
421       nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
422       nbdkit-readahead-filter(1), nbdkit-retry-filter(1),
423       nbdkit-retry-request-filter(1), nbdkit-ssh-plugin(1),
424       nbdkit-torrent-plugin(1), nbdkit-plugin(3), http://curl.haxx.se,
425       https://curl.se/docs/ssl-ciphers.html
426

AUTHORS

428       Richard W.M. Jones
429
430       Parts derived from Alexander Graf's "QEMU Block driver for CURL
431       images".
432
434       Copyright (C) 2014-2020 Red Hat Inc.
435

LICENSE

437       Redistribution and use in source and binary forms, with or without
438       modification, are permitted provided that the following conditions are
439       met:
440
441       •   Redistributions of source code must retain the above copyright
442           notice, this list of conditions and the following disclaimer.
443
444       •   Redistributions in binary form must reproduce the above copyright
445           notice, this list of conditions and the following disclaimer in the
446           documentation and/or other materials provided with the
447           distribution.
448
449       •   Neither the name of Red Hat nor the names of its contributors may
450           be used to endorse or promote products derived from this software
451           without specific prior written permission.
452
453       THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
454       EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
455       IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
456       PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
457       LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
458       CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
459       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
460       BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
461       WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
462       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
463       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
464
465
466
467nbdkit-1.30.7                     2022-07-10             nbdkit-curl-plugin(1)
Impressum