nbdkit-curl-plugin(1)

1nbdkit-curl-plugin(1)               NBDKIT               nbdkit-curl-plugin(1)
2
3
4

NAME

6       nbdkit-curl-plugin - nbdkit curl plugin (HTTP, FTP and other protocols)
7

SYNOPSIS

9        nbdkit -r curl [url=]http://example.com/disk.img
10

DESCRIPTION

12       "nbdkit-curl-plugin" is a plugin for nbdkit(1) which turns content
13       served over HTTP, FTP, and more, into a Network Block Device.  It uses
14       a library called libcurl (also known as cURL) to read data from URLs.
15       The exact list of protocols that libcurl can handle depends on how it
16       was compiled, but most versions will handle HTTP, HTTPS, FTP, FTPS and
17       more (see: "curl -V").
18
19       Note: This plugin supports writes.  However for HTTP, you may not want
20       nbdkit to issue PUT requests to the remote server (which probably
21       doesn't understand them).  To force nbdkit to use a readonly
22       connection, pass the -r flag.
23
24       Although this plugin can access SFTP (ie. SSH) servers, it is much
25       better to use nbdkit-ssh-plugin(1).  This plugin can be used to access
26       "file:///" URLs, but you should use nbdkit-file-plugin(1) instead.
27

EXAMPLE

29        nbdkit -r curl http://example.com/disk.img
30
31       serves the remote disk image as NBD on TCP port 10809 (to control ports
32       and protocols used to serve NBD see nbdkit(1)).
33

PARAMETERS

35       cainfo=FILENAME
36           (nbdkit ≥ 1.18)
37
38           Configure CA bundle for libcurl. See CURLOPT_CAINFO(3) for details.
39
40       capath=PATH
41           (nbdkit ≥ 1.18)
42
43           Set CA certificates directory location for libcurl. See
44           CURLOPT_CAPATH(3) for more information.
45
46       cookie=COOKIE
47       cookie=+FILENAME
48       cookie=-
49       cookie=-FD
50           (nbdkit ≥ 1.12)
51
52           Set a cookie in the request header when connecting to the remote
53           server.
54
55           A typical example is:
56
57            cookie='vmware_soap_session="52a01262-bf93-ccce-d379-8dabb3e55560"'
58
59           This option can be used at most once.  It only works for HTTP and
60           HTTPS transports.  To set multiple cookies you must concatenate
61           them yourself, eg:
62
63            cookie='name1=content1; name2=content2'
64
65           See CURLOPT_COOKIE(3) for more information about this.  The format
66           is quite strict and must consist of "key=value", each cookie
67           separated by exactly "; " (semicolon and space).
68
69           If the cookie is used for authentication then passing it on the
70           command line is not secure on shared machines.  Use the alternate
71           "+FILENAME" syntax to pass it in a file, "-" to read the cookie
72           interactively, or "-FD" to read it from a file descriptor.
73
74       cookie-script=SCRIPT
75       cookie-script-renew=SECS
76           (nbdkit ≥ 1.22, not Windows)
77
78           Run "SCRIPT" (a command or shell script fragment) to generate the
79           HTTP/HTTPS cookies.  "cookie-script" cannot be used with "cookie".
80           See "HEADER AND COOKIE SCRIPTS" below.
81
82       header=HEADER
83           (nbdkit ≥ 1.22)
84
85           For HTTP/HTTPS, send a custom header, or remove a header that curl
86           has added.  To add a custom header:
87
88            header='X-My-Name: John Doe'
89
90           To remove a header that curl has added, add the header followed by
91           a colon and no value:
92
93            header='User-Agent:'
94
95           To add a custom header that has no value, you have to use a
96           semicolon instead of colon.  This adds an "X-Empty:" header with no
97           value:
98
99            header='X-Empty;'
100
101           See CURLOPT_HTTPHEADER(3).  You can use this option multiple times
102           in order to add several headers.  Note this sends the header in all
103           requests, even when following a redirect, which can cause headers
104           (eg. containing sensitive authorization information) to be sent to
105           hosts other than the one originally requested.
106
107       header-script=SCRIPT
108       header-script-renew=SECS
109           (nbdkit ≥ 1.22, not Windows)
110
111           Run "SCRIPT" (a command or shell script fragment) to generate the
112           HTTP/HTTPS headers.  "header-script" cannot be used with "header".
113           See "HEADER AND COOKIE SCRIPTS" below.
114
115       password=PASSWORD
116           Set the password to use when connecting to the remote server.
117
118           Note that passing this on the command line is not secure on shared
119           machines.
120
121       password=-
122           Ask for the password (interactively) when nbdkit starts up.
123
124       password=+FILENAME
125           Read the password from the named file.  This is a secure method to
126           supply a password, as long as you set the permissions on the file
127           appropriately.
128
129       password=-FD
130           Read the password from file descriptor number "FD", inherited from
131           the parent process when nbdkit starts up.  This is also a secure
132           method to supply a password.
133
134       protocols=PROTO,PROTO,...
135           (nbdkit ≥ 1.12)
136
137           Limit the protocols that are allowed in the URL.  Use this option
138           for extra security if the URL comes from an untrusted source and
139           you want to avoid security isues in the more obscure protocols that
140           curl supports.  (See qemu CVE-2013-0249 for an example of a
141           security bug introduced by allowing unrestricted protocols).
142
143           For example if you only intend HTTP and HTTPS URLs to be used, then
144           add this parameter: "protocols=http,https"
145
146           The value of this parameter is a comma-separated list of protocols.
147           The following protocols are known: dict, file, ftp, ftps, gopher,
148           http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp,
149           rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs,
150           smtp, smtps, telnet, tftp.
151
152           The default is to allow any protocol.
153
154       proxy=PROXY
155           (nbdkit ≥ 1.20)
156
157           Set the proxy.  See CURLOPT_PROXY(3).
158
159       proxy-password=PASSWORD
160       proxy-password=-
161       proxy-password=+FILENAME
162       proxy-password=-FD
163       proxy-user=USERNAME
164           (nbdkit ≥ 1.12)
165
166           Set the proxy username and password.
167
168       sslverify=false
169           Don't verify the SSL certificate of the remote host.
170
171       tcp-keepalive=true
172           (nbdkit ≥ 1.20)
173
174           Enable TCP keepalives.
175
176       tcp-nodelay=false
177           (nbdkit ≥ 1.20)
178
179           Disable Nagle’s algorithm.
180
181       timeout=SECS
182           Set the timeout for requests.
183
184       timeout=0
185           Use the default libcurl timeout for requests.
186
187       unix-socket-path=PATH
188           (nbdkit ≥ 1.10)
189
190           Instead of using a TCP connection, connect to the server over the
191           named Unix domain socket.  See CURLOPT_UNIX_SOCKET_PATH(3).
192
193       [url=]URL
194           The URL of the remote disk image.  This is passed to libcurl
195           directly via CURLOPT_URL(3).
196
197           This parameter is required.
198
199           "url=" is a magic config key and may be omitted in most cases.  See
200           "Magic parameters" in nbdkit(1).
201
202       user=USERNAME
203           Set the username to use when connecting to the remote server.  This
204           may also be set in the URL (eg. "http://foo@example.com/disk.img")
205
206       user-agent=USER-AGENT
207           (nbdkit ≥ 1.22)
208
209           Send user-agent header when using HTTP or HTTPS.  The default is no
210           user-agent header.
211

HEADER AND COOKIE SCRIPTS

213       While the "header" and "cookie" parameters can be used to specify
214       static headers and cookies which are used in every HTTP/HTTPS request,
215       the alternate "header-script" and "cookie-script" parameters can be
216       used to run an external script or program to generate headers and/or
217       cookies.  This is particularly useful to access services which require
218       an authorization token.  In addition the "header-script-renew" and
219       "cookie-script-renew" parameters allow you to renew the authorization
220       token by rerunning the script periodically.
221
222       "header-script" is incompatible with "header", and "cookie-script" is
223       incompatible with "cookie".
224
225   Header script
226       The header script should print zero or more HTTP headers, each line of
227       output in the same format as the "header" parameter.  The headers
228       printed by the script are passed to CURLOPT_HTTPHEADER(3).
229
230       In the following example, an imaginary web service requires
231       authentication using a token fetched from a separate login server.  The
232       token expires after 60 seconds, so we also tell the plugin that it must
233       renew the token (by re-running the script) if more than 50 seconds have
234       elapsed since the last request:
235
236        nbdkit curl https://service.example.com/disk.img \
237               header-script='
238                 printf "Authorization: Bearer "
239                 curl -s -X POST https://auth.example.com/login |
240                      jq -r .token
241               ' \
242               header-script-renew=50
243
244   Cookie script
245       The cookie script should print a single line in the same format as the
246       "cookie" parameter.  This is passed to CURLOPT_COOKIE(3).
247
248   Header and cookie script shell variables
249       Within the "header-script" and "cookie-script" the following shell
250       variables are available:
251
252       $iteration
253           The number of times that the script has been called.  The first
254           time the script is called this contains 0.
255
256       $url
257           The URL as passed to the plugin.
258
259   Example: VMware ESXi cookies
260       VMware ESXi’s web server can expose both VMDK and raw format disk
261       images, but requires you to log in using HTTP Basic Authentication.
262       While you can use the "user" and "password" parameters to send HTTP
263       Basic Authentication headers in every request, tests have shown that it
264       is faster to accept the cookie which the server returns and send that
265       instead.  (It is not clear why it is faster, but one theory is that
266       VMware has to do a more expensive username and password check each
267       time.)
268
269       The web server can be accessed as below.  Since the cookie expires
270       after a certain period of time, we use "cookie-script-renew", and
271       because the server uses a self-signed certificate we must use
272       --insecure and "sslverify=false".
273
274        SERVER=esx.example.com
275        DCPATH=data
276        DS=datastore1
277        GUEST=guest-name
278        URL="https://$SERVER/folder/$GUEST/$GUEST-flat.vmdk?dcPath=$DCPATH&dsName=$DS"
279
280        nbdkit curl "$URL" \
281               cookie-script='
282                   curl --head -s --insecure -u root:password "$url" |
283                        sed -ne "{ s/^Set-Cookie: \([^;]*\);.*/\1/ip }"
284               ' \
285               cookie-script-renew=500 \
286               sslverify=false
287
288   Example: Docker Hub authorization tokens
289       Accessing objects like container layers from Docker Hub requires that
290       you first fetch an authorization token, even for anonymous access.
291       These tokens expire after about 5 minutes (300 seconds) so must be
292       periodically renewed.
293
294       You will need this authorization script (/tmp/auth.sh):
295
296        #!/bin/sh -
297        IMAGE=library/fedora
298        curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:$IMAGE:pull" |
299             jq -r .token
300
301       You will also need this script to get the blobSum of the layer
302       (/tmp/blobsum.sh):
303
304        #!/bin/sh -
305        TOKEN=`/tmp/auth.sh`
306        IMAGE=library/fedora
307        curl -s -X GET -H "Authorization: Bearer $TOKEN" \
308             "https://registry-1.docker.io/v2/$IMAGE/manifests/latest" |
309             jq -r '.fsLayers[0].blobSum'
310
311       Both scripts must be executable, and both can be run on their own to
312       check they are working.  To run nbdkit:
313
314        IMAGE=library/fedora
315        BLOBSUM=`/tmp/blobsum.sh`
316        URL="https://registry-1.docker.io/v2/$IMAGE/blobs/$BLOBSUM"
317
318        nbdkit curl "$URL" \
319               header-script=' printf "Authorization: Bearer "; /tmp/auth.sh ' \
320               header-script-renew=200 \
321               --filter=gzip
322
323       Note that this exposes a tar file over NBD.  See also
324       nbdkit-tar-filter(1).
325

DEBUG FLAGS

327       -D curl.scripts=1
328           This prints out the headers and cookies generated by the
329           "header-script" and "cookie-script" options, which can be useful
330           when debugging these scripts.
331
332       -D curl.verbose=1
333           This enables very verbose curl debugging.  See CURLOPT_VERBOSE(3).
334           This is mainly useful if you suspect there is a bug inside libcurl
335           itself.
336

FILES

338       $plugindir/nbdkit-curl-plugin.so
339           The plugin.
340
341           Use "nbdkit --dump-config" to find the location of $plugindir.
342

VERSION

344       "nbdkit-curl-plugin" first appeared in nbdkit 1.2.
345

AUTHORS

357       Richard W.M. Jones
358
359       Parts derived from Alexander Graf's "QEMU Block driver for CURL
360       images".
361

COPYRIGHT

363       Copyright (C) 2014-2020 Red Hat Inc.
364

LICENSE

366       Redistribution and use in source and binary forms, with or without
367       modification, are permitted provided that the following conditions are
368       met:
369
370       ·   Redistributions of source code must retain the above copyright
371           notice, this list of conditions and the following disclaimer.
372
373       ·   Redistributions in binary form must reproduce the above copyright
374           notice, this list of conditions and the following disclaimer in the
375           documentation and/or other materials provided with the
376           distribution.
377
378       ·   Neither the name of Red Hat nor the names of its contributors may
379           be used to endorse or promote products derived from this software
380           without specific prior written permission.
381
382       THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
383       EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
384       IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
385       PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
386       LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
387       CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
388       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
389       BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
390       WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
391       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
392       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
393
394
395
396nbdkit-1.24.2                     2021-03-02             nbdkit-curl-plugin(1)