1nbdkit-vddk-plugin(1) NBDKIT nbdkit-vddk-plugin(1)
2
3
4
6 nbdkit-vddk-plugin - nbdkit VMware VDDK plugin
7
9 nbdkit vddk [file=]FILENAME
10 [compression=none|zlib|fastlz|skipz]
11 [config=FILENAME] [cookie=COOKIE]
12 [create=true] [create-adapter-type=ide|scsi-buslogic|...]
13 [create-hwversion=workstation4|workstation5|...]
14 [create-size=...] [create-type=monolithic-sparse|...]
15 [libdir=LIBRARY]
16 [nfchostport=PORT] [single-link=true]
17 [password=PASSWORD | password=- | password=+FILENAME |
18 password=-FD]
19 [port=PORT] [server=HOSTNAME] [snapshot=MOREF]
20 [thumbprint=THUMBPRINT] [transports=MODE:MODE:...]
21 [unbuffered=true] [user=USERNAME] [vm=moref=ID]
22 nbdkit vddk --dump-plugin
23
25 "nbdkit-vddk-plugin" is an nbdkit(1) plugin that serves disks from
26 local VMware VMDK files, VMware ESXi servers, VMware VCenter servers,
27 and other sources.
28
29 It requires VMware's proprietary VDDK library that you must download
30 yourself (see "LIBRARY LOCATION" below).
31
32 For an easy-to-use wrapper around this plugin which automates setting
33 things up to connect to a remote VMware server, see:
34 https://gitlab.com/nbdkit/vddk-remote
35
37 Open an existing local VMDK file
38 nbdkit vddk /absolute/path/to/file.vmdk
39
40 When opening local files the filename must be an absolute path.
41
42 Because VDDK needs to lock the file, the file must be on a writable
43 filesystem. You can avoid this by opening the file read-only (the -r
44 option):
45
46 nbdkit -r vddk /absolute/path/to/file.vmdk
47
48 Create a new local VMDK file
49 You can use VDDK to create a VMDK file and fill it with the contents of
50 a disk image. Note the "create-size" parameter is the virtual size of
51 the final VMDK disk image and must be at least as large as the input
52 disk:
53
54 nbdkit vddk \
55 /absolute/path/to/output.vmdk \
56 create=true create-size=100M \
57 --run 'qemu-img convert input.qcow2 $uri'
58
59 Open a file on a remote VMware ESXi hypervisor
60 Connect directly to a VMware ESXi hypervisor and export a particular
61 file:
62
63 nbdkit vddk user=root password=+/tmp/rootpw \
64 server=esxi.example.com thumbprint=xx:xx:xx:... \
65 vm=moref=2 \
66 "[datastore1] Fedora/Fedora.vmdk"
67
68 "user" and "password" must be specified. Use "password=+FILENAME" to
69 provide the password securely in a file.
70
71 "server" is the hostname of the ESXi server.
72
73 "thumbprint" is the thumb print for validating the SSL certificate.
74 How to find the thumb print of a server is described in "THUMBPRINTS"
75 below.
76
77 "vm" is the Managed Object Reference ("moref") of the virtual machine.
78 See "MANAGED OBJECT REFERENCE" below.
79
80 The file parameter is the file you want to open, usually in the form
81 "[datastore] vmname/vmname.vmdk". See "FILE PARAMETER" below.
82
83 Open a file on a remote VMware vCenter server
84 Connect via VMware vCenter and export a particular file:
85
86 nbdkit vddk user=root password=vmware \
87 server=vcenter.example.com thumbprint=xx:xx:xx:... \
88 vm=moref=vm-16 \
89 "[datastore1] Fedora/Fedora.vmdk"
90
91 "user" and "password" must be specified. Use "password=+FILENAME" to
92 provide the password securely in a file.
93
94 "server" is the hostname of the vCenter server.
95
96 "thumbprint" is the thumb print for validating the SSL certificate.
97 How to find the thumb print of a server is described in "THUMBPRINTS"
98 below.
99
100 "vm" is the Managed Object Reference ("moref") of the virtual machine.
101 See "MANAGED OBJECT REFERENCE" below.
102
103 The file parameter is the file you want to open, usually in the form
104 "[datastore] vmname/vmname.vmdk". See "FILE PARAMETER" below.
105
107 For opening a local VMDK file, the "file" parameter is required and
108 must be an absolute path.
109
110 For opening a remote connection, "file", "server", "thumbprint",
111 "user", "password" and "vm" are required.
112
113 All other parameters are optional.
114
115 compression=none
116 compression=zlib
117 compression=fastlz
118 compression=skipz
119 (nbdkit ≥ 1.24, vSphere ≥ 6.5)
120
121 Select the compression type used over the network between VDDK and
122 the VMware server. The default is "none". See VMware document
123 “Best Practices for NBD Transport”.
124
125 config=FILENAME
126 The name of the VDDK configuration file.
127
128 The config file controls rarely adjusted VDDK settings like log
129 level, caching and timeouts. See the VDDK documentation for a full
130 list of settings.
131
132 VDDK itself looks in a few default locations for the configuration
133 file, usually including /etc/vmware/config and
134 $HOME/.vmware/config. Using "config" overrides these defaults.
135
136 cookie=COOKIE
137 (vSphere ≥ 6.7)
138
139 Cookie from existing authenticated session on the host.
140
141 This changes the authentication type from "VIXDISKLIB_CRED_UID" to
142 "VIXDISKLIB_CRED_SESSIONID" which can improve performance. The
143 cookie can be found by connecting to a VCenter Server over HTTPS
144 and retrieving the "vmware_soap_session" cookie.
145
146 create=true
147 (nbdkit ≥ 1.30)
148
149 Create a new, local VMDK file. Instead of opening an existing VMDK
150 file, a new VMDK file is created and opened. The filename is given
151 by the "file" parameter (see below). The file must not exist
152 already. It is not possible to create a remote file using nbdkit.
153
154 If this is used, the "create-size" parameter is required to specify
155 the virtual size of the disk. Other "create-*" parameters (see
156 below) can be used to control the VMDK sub-format.
157
158 create-adapter-type=ide
159 create-adapter-type=scsi-buslogic
160 create-adapter-type=scsi-lsilogic
161 (nbdkit ≥ 1.30)
162
163 Specify the VMDK disk adapter type. The default is
164 "scsi-buslogic".
165
166 create-hwversion=workstation4
167 create-hwversion=workstation5
168 create-hwversion=workstation6
169 create-hwversion=esx30
170 create-hwversion=esx4x
171 create-hwversion=esx50
172 create-hwversion=esx51
173 create-hwversion=esx55
174 create-hwversion=esx60
175 create-hwversion=esx65
176 create-hwversion=N
177 (nbdkit ≥ 1.30)
178
179 Specify the VMDK virtual hardware version. You can give either the
180 named version or the equivalent 16 bit number.
181
182 The default is "workstation5" (N = 4).
183
184 create-size=SIZE
185 (nbdkit ≥ 1.30)
186
187 Specify the virtual size of the created disk. The "SIZE" can use
188 modifiers like "100M" etc. It must be a multiple of 512 bytes
189 because VMware only supports sector sizes.
190
191 If you use "create=true" then this parameter is required.
192
193 create-type=monolithic-sparse
194 create-type=monolithic-flat
195 create-type=split-sparse
196 create-type=split-flat
197 create-type=vmfs-flat
198 create-type=stream-optimized
199 create-type=vmfs-thin
200 create-type=vmfs-sparse
201 (nbdkit ≥ 1.30)
202
203 Specify the VMDK sub-format. The default is "monolithic-sparse".
204
205 Some VMDK sub-formats use multiple files, where the "file"
206 parameter specifies the "Disk Descriptor File" and the disk
207 contents are stored in adjacent files.
208
209 [file=]FILENAME
210 [file=][datastore] vmname/vmname.vmdk
211 Set the name of the VMDK file to serve.
212
213 For local files you must supply an absolute path. For remote files
214 see "FILE PARAMETER" section below.
215
216 If a VM has multiple disks, nbdkit can only serve one at a time.
217 To serve more than one you must run multiple copies of nbdkit.
218 (See "NOTES" below).
219
220 "file=" is a magic config key and may be omitted in most cases.
221 See "Magic parameters" in nbdkit(1).
222
223 libdir=PATHNAME
224 This sets the path of the VMware VDDK distribution.
225
226 VDDK uses this to load its own plugins, if this path is unspecified
227 or wrong then VDDK will work with reduced functionality. See
228 "LIBRARY LOCATION" below.
229
230 nfchostport=PORT
231 Port used to establish an NFC connection to ESXi. Defaults to 902.
232
233 password=PASSWORD
234 Set the password to use when connecting to the remote server.
235
236 Note that passing this on the command line is not secure on shared
237 machines.
238
239 password=-
240 Ask for the password (interactively) when nbdkit starts up.
241
242 password=+FILENAME
243 Read the password from the named file. This is a secure method to
244 supply a password, as long as you set the permissions on the file
245 appropriately.
246
247 password=-FD
248 Read the password from file descriptor number "FD", inherited from
249 the parent process when nbdkit starts up. This is also a secure
250 method to supply a password.
251
252 port=PORT
253 The port on the VCenter/ESXi host. Defaults to 443.
254
255 server=HOSTNAME
256 The hostname or IP address of VCenter or ESXi host.
257
258 single-link=true
259 (nbdkit ≥ 1.12)
260
261 Open the current link, not the entire chain. This corresponds to
262 the "VIXDISKLIB_FLAG_OPEN_SINGLE_LINK" flag.
263
264 snapshot=MOREF
265 The Managed Object Reference of the snapshot. See "MANAGED OBJECT
266 REFERENCE" below.
267
268 thumbprint=THUMBPRINT
269 The SSL (SHA1) thumbprint for validating the SSL certificate.
270
271 The format is
272 "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx" (20
273 hex digit pairs).
274
275 See "THUMBPRINTS" below for how to get this.
276
277 transports=MODE:MODE:...
278 List of one or more transport modes to use. Possible values
279 include ‘nbd’, ‘nbdssl’, ‘san’, ‘hotadd’, ‘file’ (there may be
280 others). If not given, VDDK will try to choose the best transport
281 mode.
282
283 unbuffered=true
284 (nbdkit ≥ 1.12)
285
286 Disable host caching. This corresponds to the
287 "VIXDISKLIB_FLAG_OPEN_UNBUFFERED" flag.
288
289 user=USERNAME
290 The username to connect to the remote server as.
291
292 vm=moref=ID
293 The Managed Object Reference ("moref") of the virtual machine. See
294 "MANAGED OBJECT REFERENCE" below.
295
296 vimapiver=APIVER
297 This parameter is ignored for backwards compatibility.
298
300 The VDDK library should not be placed on a system library path such as
301 /usr/lib. The reason is that the VDDK library ships with recompiled
302 libraries like libcrypto.so and libstdc++.so that conflict with system
303 libraries.
304
305 You have two choices:
306
307 • Place VDDK in the default libdir which is compiled into this
308 plugin, for example:
309
310 $ nbdkit vddk --dump-plugin | grep ^vddk_default_libdir
311 vddk_default_libdir=/usr/lib64/vmware-vix-disklib
312
313 • But the best advice is to unpack the VDDK tarball anywhere you like
314 and set the "libdir=/path/to/vmware-vix-disklib-distrib". For
315 example:
316
317 nbdkit vddk \
318 libdir=/opt/vmware-vix-disklib-distrib \
319 /path/to/file.vmdk
320
321 No need to set "LD_LIBRARY_PATH"
322 In nbdkit ≤ 1.16 you had to set the environment variable
323 "LD_LIBRARY_PATH" when using this plugin. In nbdkit ≥ 1.18 this is not
324 recommended.
325
327 The "file" parameter can either be a local file, in which case it must
328 be the absolute path. Or it can refer to a remote file on the VMware
329 server in the format "[datastore] vmname/vmname.vmdk".
330
331 Finding the remote filename
332 For remote files you can find the path using virsh(1). For ESXi:
333
334 $ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname
335 ...
336 <disk type='file' device='disk'>
337 <source file='[datastore] vmname/vmname.vmdk'/>
338 ...
339
340 For vCenter the command is the same but the URI starts with "vpx://":
341
342 $ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \
343 dumpxml guestname
344
345 See also: https://libvirt.org/drvesx.html
346
347 Optional file= prefix
348 The "file=" part is optional, so these commands are equivalent:
349
350 nbdkit vddk file=/path/to/file.vmdk
351 nbdkit vddk /path/to/file.vmdk
352
354 The thumbprint is a 20 byte string containing the SSL (SHA1)
355 fingerprint of the remote VMware server and it is required when making
356 a remote connection. There are several ways to obtain this.
357
358 Using a web browser
359 Visit "https://SERVER-NAME/folder" and log in. Click the lock icon
360 next to the URL bar and navigate to the SHA-1 fingerprint of the site’s
361 certificate. This 20 hex digit pair string can be directly copied to
362 the "thumbprint=" parameter.
363
364 Using openssl remotely
365 The following command will print the thumbprint of a VMware server
366 called "SERVER-NAME":
367
368 $ openssl s_client -connect SERVER-NAME:443 </dev/null |
369 openssl x509 -in /dev/stdin -fingerprint -sha1 -noout
370
371 By logging in to the ESXi or vCenter server
372 Log in to the ESXi hypervisor shell and run this command:
373
374 # openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout
375
376 For VMware vCenter servers the thumbprint is printed on the text
377 console of the server or is available by logging in to the server and
378 using this command:
379
380 # openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -fingerprint -sha1 -noout
381
382 Trick: Get VDDK to tell you the thumbprint
383 Another way to get the thumbprint of a server is to connect to the
384 server using a bogus thumbprint with debugging enabled:
385
386 nbdkit -fv vddk server=esxi.example.com [...] thumbprint=12 \
387 --run 'qemu-img info "$uri"'
388
389 The nbdkit process will try to connect (and fail because the thumbprint
390 is wrong). However in the debug output will be a message such as this:
391
392 nbdkit: debug: VixDiskLibVim: Failed to verify SSL certificate: actual thumbprint=B2:31:BD:DE:9F:DB:9D:E0:78:EF:30:42:8A:41:B0:28:92:93:C8:DD expected=12
393
394 This gives you the server’s real thumbprint. Of course this method is
395 not secure since it allows a Man-in-the-Middle (MITM) attack.
396
398 Some use cases require you to pass in the Managed Object Reference
399 ("moref") of an object on the VMware server.
400
401 For VMware ESXi hypervisors, the "vm" moref is a number (eg.
402 "vm=moref=2"). For VMware VCenter it is a string beginning with "vm-")
403 (eg. "vm=moref=vm-16"). Across ESXi and vCenter the numbers are
404 different even for the same virtual machine.
405
406 If you have libvirt ≥ 3.7, the moref is available in the virsh(1)
407 "dumpxml" output:
408
409 $ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname
410 ...
411 <vmware:moref>2</vmware:moref>
412 ...
413
414 or:
415
416 $ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \
417 dumpxml guestname
418 ...
419 <vmware:moref>vm-16</vmware:moref>
420 ...
421
422 An alternative way to find the moref of a VM is using the
423 "moRefFinder.pl" script written by William Lam
424 (http://www.virtuallyghetto.com/2011/11/vsphere-moref-managed-object-reference.html
425 https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-2-technical.html).
426
428 To query more information about the plugin (and whether it is working),
429 use:
430
431 nbdkit vddk --dump-plugin
432
433 or:
434
435 nbdkit vddk --dump-plugin libdir=/opt/vmware-vix-disklib-distrib
436
437 (see "LIBRARY LOCATION" above).
438
439 If the plugin is not present or not working you will get an error.
440
441 If it works the output will include:
442
443 "vddk_default_libdir=..."
444 The compiled-in library path. Use "libdir=PATHNAME" to override
445 this at runtime.
446
447 "vddk_has_nfchostport=1"
448 If this is printed then the "nfchostport=PORT" parameter is
449 supported by this build.
450
451 "vddk_library_version=..."
452 The VDDK major library version: 6, 7, 8, ... If this is omitted it
453 means the library could not be loaded.
454
455 "vddk_dll=..."
456 Prints the full path to the VDDK shared library. Since this
457 requires a glibc extension it may not be available in all builds of
458 the plugin.
459
460 "VixDiskLib_...=1"
461 For each VDDK API that the plugin uses and which is present in the
462 VDDK library that was loaded, we print the name of the API (eg.
463 "VixDiskLib_Open=1"). This lets you see which optional APIs are
464 available, such as "VixDiskLib_Flush" and
465 "VixDiskLib_QueryAllocatedBlocks". If the library could not be
466 loaded then these lines are not printed.
467
469 Sector size limitation
470 The VDDK plugin can only answer read/write requests on whole 512 byte
471 sector boundaries. This is because the VDDK Read and Write APIs only
472 take sector numbers. If your client needs finer granularity, you can
473 use nbdkit-blocksize-filter(1):
474
475 nbdkit vddk ... --filter=blocksize minblock=512
476
477 Out of memory errors
478 In the verbose log you may see errors like:
479
480 nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg:
481 received NFC error 5 from server: Failed to allocate the
482 requested 2097176 bytes
483
484 This seems especially common when there are multiple parallel
485 connections open to the VMware server with large NBD reads and writes.
486
487 Increase resource limits on the server
488
489 The error above can be caused by resource limits set on the VMware
490 server. You can increase the limit for the NFC service by editing
491 /etc/vmware/hostd/config.xml and adjusting the "<maxMemory>" setting:
492
493 <nfcsvc>
494 <path>libnfcsvc.so</path>
495 <enabled>true</enabled>
496 <maxMemory>50331648</maxMemory>
497 <maxStreamMemory>10485760</maxStreamMemory>
498 </nfcsvc>
499
500 and restarting the "hostd" service:
501
502 # /etc/init.d/hostd restart
503
504 For more information see https://bugzilla.redhat.com/1614276.
505
506 Limit request sizes
507
508 In addition, or as an alternative to adjusting the server
509 configuration, you can use nbdkit-blocksize-filter(1) to limit the
510 maximum request size. By default this plugin translates NBD requests
511 directly into VDDK requests, and it appears that very large VDDK
512 requests can cause the error seen above.
513
514 Using:
515
516 nbdkit vddk ... --filter=blocksize minblock=512 maxdata=2M
517
518 will cause nbdkit to automatically split and combine requests so that
519 VDDK sees only sizes in the range "[512..2M]".
520
521 Troubleshooting performance problems
522 VDDK has very uneven performance with some operations being very slow.
523 This plugin has options to allow you to debug performance issues. If
524 your application has a debug or diagnostic setting, add the following
525 nbdkit command line options:
526
527 -v -D nbdkit.backend.datapath=0 -D vddk.datapath=0 -D vddk.stats=1
528
529 "-v" enables verbose messages and the two datapath options disable the
530 very verbose per-read/-write messages. "-D vddk.stats=1" enables a
531 summary when nbdkit exits of the cumulative time taken in each VDDK
532 function, the number of times each function was called, and (for read
533 and write) the number of bytes transferred. An example of what those
534 stats look like can be found here:
535 https://gitlab.com/nbdkit/nbdkit/-/commit/5c80f0d290db45a679d55baf37ff39bacb8ce7ec
536
537 You can interpret the stats as follows:
538
539 "Read"
540 The cumulative time spent waiting for VDDK to return from
541 "VixDiskLib_Read" calls, the number of times this function was
542 called, and the total bytes read. You can use this to determine
543 the read bandwidth to the VMware server.
544
545 "Write"
546 "Flush"
547 Same as above, but for writing and flushing writes.
548
549 "ReadAsync"
550 "WriteAsync"
551 Same as above, but for asynchronous read and write calls introduced
552 in nbdkit 1.30. Unfortunately at the moment the amount of time
553 spent in these calls is not accounted for correctly.
554
555 "QueryAllocatedBlocks"
556 This call is used to query information about the sparseness of the
557 remote disk. It is only available in VDDK ≥ 6.7. The call is
558 notably very slow in all versions of VMware we have tested.
559
560 "Open"
561 "Close"
562 "ConnectEx"
563 "Disconnect"
564 "InitEx"
565 "Exit"
566 The cumulative time spent connecting and disconnecting from the
567 VMware server, which can also be very slow.
568
570 This plugin requires VDDK ≥ 6.5 (released Nov 2016). It is only
571 supported on the x64-64 archtecture.
572
573 It has been tested with all versions up to 8.0.0 (but should work with
574 future versions).
575
576 VDDK 6.7 was the first version that supported the
577 "VixDiskLib_QueryAllocatedBlocks" API, required to provide extent
578 information over NBD.
579
581 Debugging messages can be very helpful if you have problems connecting
582 to VMware servers, or to find the list of available transport modes, or
583 to diagnose SAN problems:
584
585 nbdkit -f -v vddk file=FILENAME [...]
586
587 Additional debug flags are available:
588
589 -D vddk.diskinfo=1
590 Debug disk information returned by "GetInfo".
591
592 -D vddk.extents=1
593 Debug extents returned by "QueryAllocatedBlocks".
594
595 -D vddk.datapath=0
596 Suppress debugging of datapath calls ("Read", "ReadAsync", "Write"
597 and "WriteAsync").
598
599 -D vddk.stats=1
600 When the plugin exits print some statistics about each VDDK call.
601
603 $plugindir/nbdkit-vddk-plugin.so
604 The plugin.
605
606 Use "nbdkit --dump-config" to find the location of $plugindir.
607
609 "nbdkit-vddk-plugin" first appeared in nbdkit 1.2.
610
612 nbdkit(1), nbdkit-plugin(3), nbdkit-blocksize-filter(1),
613 nbdkit-readahead-filter(1), nbdkit-retry-filter(1),
614 nbdkit-scan-filter(1), virsh(1), https://gitlab.com/nbdkit/vddk-remote,
615 https://libvirt.org/drvesx.html,
616 https://www.vmware.com/support/developer/vddk/, VMware document “Best
617 Practices for NBD Transport”.
618
620 Richard W.M. Jones
621
623 Copyright Red Hat
624
626 Redistribution and use in source and binary forms, with or without
627 modification, are permitted provided that the following conditions are
628 met:
629
630 • Redistributions of source code must retain the above copyright
631 notice, this list of conditions and the following disclaimer.
632
633 • Redistributions in binary form must reproduce the above copyright
634 notice, this list of conditions and the following disclaimer in the
635 documentation and/or other materials provided with the
636 distribution.
637
638 • Neither the name of Red Hat nor the names of its contributors may
639 be used to endorse or promote products derived from this software
640 without specific prior written permission.
641
642 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
643 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
644 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
645 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
646 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
647 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
648 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
649 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
650 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
651 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
652 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
653
654
655
656nbdkit-1.36.2 2023-11-26 nbdkit-vddk-plugin(1)