1nbdkit-curl-plugin(1)               NBDKIT               nbdkit-curl-plugin(1)
2
3
4

NAME

6       nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7

SYNOPSIS

9        nbdkit -r curl [url=]http://example.com/disk.img
10

DESCRIPTION

12       "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13       served over HTTP, FTP, and more, into a Network Block Device.  It uses
14       a library called libcurl (also known as cURL) to read data from URLs.
15       The exact list of protocols that libcurl can handle depends on how it
16       was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17       more (see: "curl -V").
18
19       Note: This plugin supports writes.  However for HTTP, you may not want
20       nbdkit to issue PUT requests to the remote server (which probably
21       doesn't understand them).  To force nbdkit to use a readonly
22       connection, pass the -r flag.
23
24       Although this plugin can access SFTP (ie. SSH) servers, it is much
25       better to use nbdkit-ssh-plugin(1).  This plugin can be used to access
26       "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
27

EXAMPLE

29        nbdkit -r curl http://example.com/disk.img
30
31       serves the remote disk image as NBD on TCP port 10809 (to control ports
32       and protocols used to serve NBD see nbdkit(1)).
33

PARAMETERS

35       cainfo=FILENAME
36           (nbdkit ≥ 1.18)
37
38           Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
39
40       capath=PATH
41           (nbdkit ≥ 1.18)
42
43           Set CA certificates directory location for libcurl. See
44           CURLOPT_CAPATH(3) for more information.
45
46       cookie=COOKIE
47       cookie=+FILENAME
48       cookie=-
49       cookie=-FD
50           (nbdkit ≥ 1.12)
51
52           Set a cookie in the request header when connecting to the remote
53           server.
54
55           A typical example is:
56
57            cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
58
59           This option can be used at most once.  It only works for HTTP and
60           HTTPS transports.  To set multiple cookies you must concatenate
61           them yourself, eg:
62
63            cookie='name1=content1; name2=content2'
64
65           See CURLOPT_COOKIE(3) for more information about this.  The format
66           is quite strict and must consist of "key=value", each cookie
67           separated by exactly "; " (semicolon and space).
68
69           If the cookie is used for authentication then passing it on the
70           command line is not secure on shared machines.  Use the alternate
71           "+FILENAME" syntax to pass it in a file, "-" to read the cookie
72           interactively, or "-FD" to read it from a file descriptor.
73
74       cookiefile=
75           (nbdkit ≥ 1.26)
76
77           Enable cookie processing (eg. when following redirects), starting
78           with an empty list of cookies.  This is equivalent to calling
79           CURLOPT_COOKIEFILE(3) with an empty string.
80
81       cookiefile=FILENAME
82           (nbdkit ≥ 1.26)
83
84           Enable cookie processing (eg. when following redirects),
85           prepopulating cookies from the given file.  The file can contain
86           cookies in any format supported by curl, see CURLOPT_COOKIEFILE(3).
87           Cookies sent by the server are not saved when nbdkit exits.
88
89       cookiejar=FILENAME
90           (nbdkit ≥ 1.26)
91
92           Enable cookie processing (eg. when following redirects),
93           prepopulating cookies from the given file, and writing server
94           cookies back to the file when the NBD handle is closed.  The file
95           can contain cookies in any format supported by curl, see
96           CURLOPT_COOKIEJAR(3).
97
98           There is some advice on the internet telling you to set this to
99           /dev/null, but you should not do this because it can corrupt
100           /dev/null.  If you don't want a cookiejar, omit this option.  If
101           you want to enable cookie processing without updating a permanent
102           cookiejar, use the "cookiefile=" option instead.
103
104       cookie-script=SCRIPT
105       cookie-script-renew=SECS
106           (nbdkit ≥ 1.22, not Windows)
107
108           Run "SCRIPT" (a command or shell script fragment) to generate the
109           HTTP/HTTPS cookies.  "cookie-script" cannot be used with "cookie".
110           See "HEADER AND COOKIE SCRIPTS" below.
111
112       followlocation=false
113           (nbdkit ≥ 1.26)
114
115           Follow redirects from the server.  The default is true (to follow
116           redirects), but you can set this to false to prevent this.
117
118       header=HEADER
119           (nbdkit ≥ 1.22)
120
121           For HTTP/HTTPS, send a custom header, or remove a header that curl
122           has added.  To add a custom header:
123
124            header='X-My-Name: John Doe'
125
126           To remove a header that curl has added, add the header followed by
127           a colon and no value:
128
129            header='User-Agent:'
130
131           To add a custom header that has no value, you have to use a
132           semicolon instead of colon.  This adds an "X-Empty:" header with no
133           value:
134
135            header='X-Empty;'
136
137           See CURLOPT_HTTPHEADER(3).  You can use this option multiple times
138           in order to add several headers.  Note this sends the header in all
139           requests, even when following a redirect, which can cause headers
140           (eg. containing sensitive authorization information) to be sent to
141           hosts other than the one originally requested.
142
143       header-script=SCRIPT
144       header-script-renew=SECS
145           (nbdkit ≥ 1.22, not Windows)
146
147           Run "SCRIPT" (a command or shell script fragment) to generate the
148           HTTP/HTTPS headers.  "header-script" cannot be used with "header".
149           See "HEADER AND COOKIE SCRIPTS" below.
150
151       password=PASSWORD
152           Set the password to use when connecting to the remote server.
153
154           Note that passing this on the command line is not secure on shared
155           machines.
156
157       password=-
158           Ask for the password (interactively) when nbdkit starts up.
159
160       password=+FILENAME
161           Read the password from the named file.  This is a secure method to
162           supply a password, as long as you set the permissions on the file
163           appropriately.
164
165       password=-FD
166           Read the password from file descriptor number "FD", inherited from
167           the parent process when nbdkit starts up.  This is also a secure
168           method to supply a password.
169
170       protocols=PROTO,PROTO,...
171           (nbdkit ≥ 1.12)
172
173           Limit the protocols that are allowed in the URL.  Use this option
174           for extra security if the URL comes from an untrusted source and
175           you want to avoid security isues in the more obscure protocols that
176           curl supports.  (See qemu CVE-2013-0249 for an example of a
177           security bug introduced by allowing unrestricted protocols).
178
179           For example if you only intend HTTP and HTTPS URLs to be used, then
180           add this parameter: "protocols=http,https"
181
182           This restriction also applies if the plugin follows a redirect to
183           another protocol (eg. you start with an "https://" URL which the
184           server redirects to "ftp://").  To prevent redirects altogether see
185           the "followlocation" parameter.
186
187           The value of this parameter is a comma-separated list of protocols.
188           The following protocols are known: dict, file, ftp, ftps, gopher,
189           http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp,
190           rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs,
191           smtp, smtps, telnet, tftp.
192
193           The default is to allow any protocol.
194
195       proxy=PROXY
196           (nbdkit ≥ 1.20)
197
198           Set the proxy.  See CURLOPT_PROXY(3).
199
200       proxy-password=PASSWORD
201       proxy-password=-
202       proxy-password=+FILENAME
203       proxy-password=-FD
204       proxy-user=USERNAME
205           (nbdkit ≥ 1.12)
206
207           Set the proxy username and password.
208
209       sslverify=false
210           Don't verify the SSL certificate of the remote host.
211
212       tcp-keepalive=true
213           (nbdkit ≥ 1.20)
214
215           Enable TCP keepalives.
216
217       tcp-nodelay=false
218           (nbdkit ≥ 1.20)
219
220           Disable Nagle’s algorithm.
221
222       timeout=SECS
223           Set the timeout for requests.
224
225       timeout=0
226           Use the default libcurl timeout for requests.
227
228       unix-socket-path=PATH
229           (nbdkit ≥ 1.10)
230
231           Instead of using a TCP connection, connect to the server over the
232           named Unix domain socket.  See CURLOPT_UNIX_SOCKET_PATH(3).
233
234       [url=]URL
235           The URL of the remote disk image.  This is passed to libcurl
236           directly via CURLOPT_URL(3).
237
238           This parameter is required.
239
240           "url=" is a magic config key and may be omitted in most cases.  See
241           "Magic parameters" in nbdkit(1).
242
243       user=USERNAME
244           Set the username to use when connecting to the remote server.  This
245           may also be set in the URL (eg. "http://foo@example.com/disk.img")
246
247       user-agent=USER-AGENT
248           (nbdkit ≥ 1.22)
249
250           Send user-agent header when using HTTP or HTTPS.  The default is no
251           user-agent header.
252
254       While the "header" and "cookie" parameters can be used to specify
255       static headers and cookies which are used in every HTTP/HTTPS request,
256       the alternate "header-script" and "cookie-script" parameters can be
257       used to run an external script or program to generate headers and/or
258       cookies.  This is particularly useful to access services which require
259       an authorization token.  In addition the "header-script-renew" and
260       "cookie-script-renew" parameters allow you to renew the authorization
261       token by rerunning the script periodically.
262
263       "header-script" is incompatible with "header", and "cookie-script" is
264       incompatible with "cookie".
265
266   Header script
267       The header script should print zero or more HTTP headers, each line of
268       output in the same format as the "header" parameter.  The headers
269       printed by the script are passed to CURLOPT_HTTPHEADER(3).
270
271       In the following example, an imaginary web service requires
272       authentication using a token fetched from a separate login server.  The
273       token expires after 60 seconds, so we also tell the plugin that it must
274       renew the token (by re-running the script) if more than 50 seconds have
275       elapsed since the last request:
276
277        nbdkit curl https://service.example.com/disk.img \
278               header-script='
279                 printf "Authorization: Bearer "
280                 curl -s -X POST https://auth.example.com/login |
281                      jq -r .token
282               ' \
283               header-script-renew=50
284
285   Cookie script
286       The cookie script should print a single line in the same format as the
287       "cookie" parameter.  This is passed to CURLOPT_COOKIE(3).
288
289   Header and cookie script shell variables
290       Within the "header-script" and "cookie-script" the following shell
291       variables are available:
292
293       $iteration
294           The number of times that the script has been called.  The first
295           time the script is called this contains 0.
296
297       $url
298           The URL as passed to the plugin.
299
300   Example: VMware ESXi cookies
301       VMware ESXi’s web server can expose both VMDK and raw format disk
302       images, but requires you to log in using HTTP Basic Authentication.
303       While you can use the "user" and "password" parameters to send HTTP
304       Basic Authentication headers in every request, tests have shown that it
305       is faster to accept the cookie which the server returns and send that
306       instead.  (It is not clear why it is faster, but one theory is that
307       VMware has to do a more expensive username and password check each
308       time.)
309
310       The web server can be accessed as below.  Since the cookie expires
311       after a certain period of time, we use "cookie-script-renew", and
312       because the server uses a self-signed certificate we must use
313       --insecure and "sslverify=false".
314
315        SERVER=esx.example.com
316        DCPATH=data
317        DS=datastore1
318        GUEST=guest-name
319        URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
320
321        nbdkit curl "$URL" \
322               cookie-script='
323                   curl --head -s --insecure -u root:password "$url" |
324                        sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
325               ' \
326               cookie-script-renew=500 \
327               sslverify=false
328
329   Example: Docker Hub authorization tokens
330       Accessing objects like container layers from Docker Hub requires that
331       you first fetch an authorization token, even for anonymous access.
332       These tokens expire after about 5 minutes (300 seconds) so must be
333       periodically renewed.
334
335       You will need this authorization script (/tmp/auth.sh):
336
337        #!/bin/sh -
338        IMAGE=library/fedora
339        curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
340             jq -r .token
341
342       You will also need this script to get the blobSum of the layer
343       (/tmp/blobsum.sh):
344
345        #!/bin/sh -
346        TOKEN=`/tmp/auth.sh`
347        IMAGE=library/fedora
348        curl -s -X GET -H "Authorization: Bearer $TOKEN" \
349             "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
350             jq -r '.fsLayers[0].blobSum'
351
352       Both scripts must be executable, and both can be run on their own to
353       check they are working.  To run nbdkit:
354
355        IMAGE=library/fedora
356        BLOBSUM=`/tmp/blobsum.sh`
357        URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
358
359        nbdkit curl "$URL" \
360               header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
361               header-script-renew=200 \
362               --filter=gzip
363
364       Note that this exposes a tar file over NBD.  See also
365       nbdkit-tar-filter(1).
366

DEBUG FLAGS

368       -D curl.scripts=1
369           This prints out the headers and cookies generated by the
370           "header-script" and "cookie-script" options, which can be useful
371           when debugging these scripts.
372
373       -D curl.verbose=1
374           This enables very verbose curl debugging.  See CURLOPT_VERBOSE(3).
375           This is mainly useful if you suspect there is a bug inside libcurl
376           itself.
377

FILES

379       $plugindir/nbdkit-curl-plugin.so
380           The plugin.
381
382           Use "nbdkit --dump-config" to find the location of $plugindir.
383

VERSION

385       "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
386

SEE ALSO

388       curl(1), libcurl(3), CURLOPT_CAINFO(3), CURLOPT_CAPATH(3),
389       CURLOPT_COOKIE(3), CURLOPT_COOKIEFILE(3), CURLOPT_COOKIEJAR(3),
390       CURLOPT_FOLLOWLOCATION(3), CURLOPT_HTTPHEADER(3), CURLOPT_PROXY(3),
391       CURLOPT_TCP_KEEPALIVE(3), CURLOPT_TCP_NODELAY(3), CURLOPT_URL(3),
392       CURLOPT_UNIX_SOCKET_PATH(3), CURLOPT_USERAGENT(3), CURLOPT_VERBOSE(3),
393       nbdkit(1), nbdkit-extentlist-filter(1), nbdkit-file-plugin(1),
394       nbdkit-readahead-filter(1), nbdkit-retry-filter(1),
395       nbdkit-ssh-plugin(1), nbdkit-torrent-plugin(1), nbdkit-plugin(3),
396       http://curl.haxx.se.
397

AUTHORS

399       Richard W.M. Jones
400
401       Parts derived from Alexander Graf's "QEMU Block driver for CURL
402       images".
403
405       Copyright (C) 2014-2020 Red Hat Inc.
406

LICENSE

408       Redistribution and use in source and binary forms, with or without
409       modification, are permitted provided that the following conditions are
410       met:
411
412       •   Redistributions of source code must retain the above copyright
413           notice, this list of conditions and the following disclaimer.
414
415       •   Redistributions in binary form must reproduce the above copyright
416           notice, this list of conditions and the following disclaimer in the
417           documentation and/or other materials provided with the
418           distribution.
419
420       •   Neither the name of Red Hat nor the names of its contributors may
421           be used to endorse or promote products derived from this software
422           without specific prior written permission.
423
424       THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
425       EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
426       IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
427       PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
428       LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
429       CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
430       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
431       BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
432       WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
433       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
434       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
435
436
437
438nbdkit-1.25.8                     2021-05-25             nbdkit-curl-plugin(1)
Impressum