1nbdkit-vddk-plugin(1) NBDKIT nbdkit-vddk-plugin(1)
2
6 nbdkit-vddk-plugin - nbdkit VMware VDDK plugin
7
9 nbdkit vddk [file=]FILENAME
10 [compression=none|zlib|fastlz|skipz]
11 [config=FILENAME] [cookie=COOKIE]
12 [create=true] [create-adapter-type=ide|scsi-buslogic|...]
13 [create-hwversion=workstation4|workstation5|...]
14 [create-size=...] [create-type=monolithic-sparse|...]
15 [libdir=LIBRARY]
16 [nfchostport=PORT] [single-link=true]
17 [password=PASSWORD | password=- | password=+FILENAME |
18 password=-FD]
19 [port=PORT] [server=HOSTNAME] [snapshot=MOREF]
20 [thumbprint=THUMBPRINT] [transports=MODE:MODE:...]
21 [unbuffered=true] [user=USERNAME] [vm=moref=ID]
22 nbdkit vddk --dump-plugin
23
25 "nbdkit-vddk-plugin" is an nbdkit(1) plugin that serves disks from
26 local VMware VMDK files, VMware ESXi servers, VMware VCenter servers,
27 and other sources.
28 It requires VMware's proprietary VDDK library that you must download
30 yourself (see "LIBRARY LOCATION" below).
31
33 Open an existing local VMDK file
34 nbdkit vddk /absolute/path/to/file.vmdk
35 When opening local files the filename must be an absolute path.
37 Because VDDK needs to lock the file, the file must be on a writable
39 filesystem. You can avoid this by opening the file read-only (the -r
40 option):
41 nbdkit -r vddk /absolute/path/to/file.vmdk
43 Create a new local VMDK file
45 You can use VDDK to create a VMDK file and fill it with the contents of
46 a disk image. Note the "create-size" parameter is the virtual size of
47 the final VMDK disk image and must be at least as large as the input
48 disk:
49 nbdkit -U - vddk \
51 /absolute/path/to/output.vmdk \
52 create=true create-size=100M \
53 --run 'qemu-img convert input.qcow2 $uri'
54 Open a file on a remote VMware ESXi hypervisor
56 Connect directly to a VMware ESXi hypervisor and export a particular
57 file:
58 nbdkit vddk user=root password=+/tmp/rootpw \
60 server=esxi.example.com thumbprint=xx:xx:xx:... \
61 vm=moref=2 \
62 "[datastore1] Fedora/Fedora.vmdk"
63 "user" and "password" must be specified. Use "password=+FILENAME" to
65 provide the password securely in a file.
66 "server" is the hostname of the ESXi server.
68 "thumbprint" is the thumb print for validating the SSL certificate.
70 How to find the thumb print of a server is described in "THUMBPRINTS"
71 below.
72 "vm" is the Managed Object Reference ("moref") of the virtual machine.
74 See "MANAGED OBJECT REFERENCE" below.
75 The file parameter is the file you want to open, usually in the form
77 "[datastore] vmname/vmname.vmdk". See "FILE PARAMETER" below.
78 Open a file on a remote VMware vCenter server
80 Connect via VMware vCenter and export a particular file:
81 nbdkit vddk user=root password=vmware \
83 server=vcenter.example.com thumbprint=xx:xx:xx:... \
84 vm=moref=vm-16 \
85 "[datastore1] Fedora/Fedora.vmdk"
86 "user" and "password" must be specified. Use "password=+FILENAME" to
88 provide the password securely in a file.
89 "server" is the hostname of the vCenter server.
91 "thumbprint" is the thumb print for validating the SSL certificate.
93 How to find the thumb print of a server is described in "THUMBPRINTS"
94 below.
95 "vm" is the Managed Object Reference ("moref") of the virtual machine.
97 See "MANAGED OBJECT REFERENCE" below.
98 The file parameter is the file you want to open, usually in the form
100 "[datastore] vmname/vmname.vmdk". See "FILE PARAMETER" below.
101
103 For opening a local VMDK file, the "file" parameter is required and
104 must be an absolute path.
105 For opening a remote connection, "file", "server", "thumbprint",
107 "user", "password" and "vm" are required.
108 All other parameters are optional.
110 compression=none
112 compression=zlib
113 compression=fastlz
114 compression=skipz
115 (nbdkit ≥ 1.24, vSphere ≥ 6.5)
116 Select the compression type used over the network between VDDK and
118 the VMware server. The default is "none". See VMware document
119 “Best Practices for NBD Transport”.
120 config=FILENAME
122 The name of the VDDK configuration file.
123 The config file controls rarely adjusted VDDK settings like log
125 level, caching and timeouts. See the VDDK documentation for a full
126 list of settings.
127 VDDK itself looks in a few default locations for the configuration
129 file, usually including /etc/vmware/config and
130 $HOME/.vmware/config. Using "config" overrides these defaults.
131 cookie=COOKIE
133 (vSphere ≥ 6.7)
134 Cookie from existing authenticated session on the host.
136 This changes the authentication type from "VIXDISKLIB_CRED_UID" to
138 "VIXDISKLIB_CRED_SESSIONID" which can improve performance. The
139 cookie can be found by connecting to a VCenter Server over HTTPS
140 and retrieving the "vmware_soap_session" cookie.
141 create=true
143 (nbdkit ≥ 1.30)
144 Create a new, local VMDK file. Instead of opening an existing VMDK
146 file, a new VMDK file is created and opened. The filename is given
147 by the "file" parameter (see below). The file must not exist
148 already. It is not possible to create a remote file using nbdkit.
149 If this is used, the "create-size" parameter is required to specify
151 the virtual size of the disk. Other "create-*" parameters (see
152 below) can be used to control the VMDK sub-format.
153 create-adapter-type=ide
155 create-adapter-type=scsi-buslogic
156 create-adapter-type=scsi-lsilogic
157 (nbdkit ≥ 1.30)
158 Specify the VMDK disk adapter type. The default is
160 "scsi-buslogic".
161 create-hwversion=workstation4
163 create-hwversion=workstation5
164 create-hwversion=workstation6
165 create-hwversion=esx30
166 create-hwversion=esx4x
167 create-hwversion=esx50
168 create-hwversion=esx51
169 create-hwversion=esx55
170 create-hwversion=esx60
171 create-hwversion=esx65
172 create-hwversion=N
173 (nbdkit ≥ 1.30)
174 Specify the VMDK virtual hardware version. You can give either the
176 named version or the equivalent 16 bit number.
177 The default is "workstation5" (N = 4).
179 create-size=SIZE
181 (nbdkit ≥ 1.30)
182 Specify the virtual size of the created disk. The "SIZE" can use
184 modifiers like "100M" etc. It must be a multiple of 512 bytes
185 because VMware only supports sector sizes.
186 If you use "create=true" then this parameter is required.
188 create-type=monolithic-sparse
190 create-type=monolithic-flat
191 create-type=split-sparse
192 create-type=split-flat
193 create-type=vmfs-flat
194 create-type=stream-optimized
195 create-type=vmfs-thin
196 create-type=vmfs-sparse
197 (nbdkit ≥ 1.30)
198 Specify the VMDK sub-format. The default is "monolithic-sparse".
200 Some VMDK sub-formats use multiple files, where the "file"
202 parameter specifies the "Disk Descriptor File" and the disk
203 contents are stored in adjacent files.
204 [file=]FILENAME
206 [file=][datastore] vmname/vmname.vmdk
207 Set the name of the VMDK file to serve.
208 For local files you must supply an absolute path. For remote files
210 see "FILE PARAMETER" section below.
211 If a VM has multiple disks, nbdkit can only serve one at a time.
213 To serve more than one you must run multiple copies of nbdkit.
214 (See "NOTES" below).
215 "file=" is a magic config key and may be omitted in most cases.
217 See "Magic parameters" in nbdkit(1).
218 libdir=PATHNAME
220 This sets the path of the VMware VDDK distribution.
221 VDDK uses this to load its own plugins, if this path is unspecified
223 or wrong then VDDK will work with reduced functionality. See
224 "LIBRARY LOCATION" below.
225 nfchostport=PORT
227 Port used to establish an NFC connection to ESXi. Defaults to 902.
228 password=PASSWORD
230 Set the password to use when connecting to the remote server.
231 Note that passing this on the command line is not secure on shared
233 machines.
234 password=-
236 Ask for the password (interactively) when nbdkit starts up.
237 password=+FILENAME
239 Read the password from the named file. This is a secure method to
240 supply a password, as long as you set the permissions on the file
241 appropriately.
242 password=-FD
244 Read the password from file descriptor number "FD", inherited from
245 the parent process when nbdkit starts up. This is also a secure
246 method to supply a password.
247 port=PORT
249 The port on the VCenter/ESXi host. Defaults to 443.
250 server=HOSTNAME
252 The hostname or IP address of VCenter or ESXi host.
253 single-link=true
255 (nbdkit ≥ 1.12)
256 Open the current link, not the entire chain. This corresponds to
258 the "VIXDISKLIB_FLAG_OPEN_SINGLE_LINK" flag.
259 snapshot=MOREF
261 The Managed Object Reference of the snapshot. See "MANAGED OBJECT
262 REFERENCE" below.
263 thumbprint=THUMBPRINT
265 The SSL (SHA1) thumbprint for validating the SSL certificate.
266 The format is
268 "xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx" (20
269 hex digit pairs).
270 See "THUMBPRINTS" below for how to get this.
272 transports=MODE:MODE:...
274 List of one or more transport modes to use. Possible values
275 include ‘nbd’, ‘nbdssl’, ‘san’, ‘hotadd’, ‘file’ (there may be
276 others). If not given, VDDK will try to choose the best transport
277 mode.
278 unbuffered=true
280 (nbdkit ≥ 1.12)
281 Disable host caching. This corresponds to the
283 "VIXDISKLIB_FLAG_OPEN_UNBUFFERED" flag.
284 user=USERNAME
286 The username to connect to the remote server as.
287 vm=moref=ID
289 The Managed Object Reference ("moref") of the virtual machine. See
290 "MANAGED OBJECT REFERENCE" below.
291 vimapiver=APIVER
293 This parameter is ignored for backwards compatibility.
294
296 The VDDK library should not be placed on a system library path such as
297 /usr/lib. The reason is that the VDDK library ships with recompiled
298 libraries like libcrypto.so and libstdc++.so that conflict with system
299 libraries.
300 You have two choices:
302 • Place VDDK in the default libdir which is compiled into this
304 plugin, for example:
305 $ nbdkit vddk --dump-plugin | grep ^vddk_default_libdir
307 vddk_default_libdir=/usr/lib64/vmware-vix-disklib
308 • But the best advice is to unpack the VDDK tarball anywhere you like
310 and set the "libdir=/path/to/vmware-vix-disklib-distrib". For
311 example:
312 nbdkit vddk \
314 libdir=/opt/vmware-vix-disklib-distrib \
315 /path/to/file.vmdk
316 No need to set "LD_LIBRARY_PATH"
318 In nbdkit ≤ 1.16 you had to set the environment variable
319 "LD_LIBRARY_PATH" when using this plugin. In nbdkit ≥ 1.18 this is not
320 recommended.
321
323 The "file" parameter can either be a local file, in which case it must
324 be the absolute path. Or it can refer to a remote file on the VMware
325 server in the format "[datastore] vmname/vmname.vmdk".
326 Finding the remote filename
328 For remote files you can find the path using virsh(1). For ESXi:
329 $ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname
331 ...
332 <disk type='file' device='disk'>
333 <source file='[datastore] vmname/vmname.vmdk'/>
334 ...
335 For vCenter the command is the same but the URI starts with "vpx://":
337 $ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \
339 dumpxml guestname
340 See also: https://libvirt.org/drvesx.html
342 Optional file= prefix
344 The "file=" part is optional, so these commands are equivalent:
345 nbdkit vddk file=/path/to/file.vmdk
347 nbdkit vddk /path/to/file.vmdk
348
350 The thumbprint is a 20 byte string containing the SSL (SHA1)
351 fingerprint of the remote VMware server and it is required when making
352 a remote connection. There are several ways to obtain this.
353 Using a web browser
355 Visit "https://SERVER-NAME/folder" and log in. Click the lock icon
356 next to the URL bar and navigate to the SHA-1 fingerprint of the site’s
357 certificate. This 20 hex digit pair string can be directly copied to
358 the "thumbprint=" parameter.
359 Using openssl remotely
361 The following command will print the thumbprint of a VMware server
362 called "SERVER-NAME":
363 $ openssl s_client -connect SERVER-NAME:443 </dev/null |
365 openssl x509 -in /dev/stdin -fingerprint -sha1 -noout
366 By logging in to the ESXi or vCenter server
368 Log in to the ESXi hypervisor shell and run this command:
369 # openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout
371 For VMware vCenter servers the thumbprint is printed on the text
373 console of the server or is available by logging in to the server and
374 using this command:
375 # openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -fingerprint -sha1 -noout
377 Trick: Get VDDK to tell you the thumbprint
379 Another way to get the thumbprint of a server is to connect to the
380 server using a bogus thumbprint with debugging enabled:
381 nbdkit -U - -fv vddk server=esxi.example.com [...] thumbprint=12 \
383 --run 'qemu-img info "$uri"'
384 The nbdkit process will try to connect (and fail because the thumbprint
386 is wrong). However in the debug output will be a message such as this:
387 nbdkit: debug: VixDiskLibVim: Failed to verify SSL certificate: actual thumbprint=B2:31:BD:DE:9F:DB:9D:E0:78:EF:30:42:8A:41:B0:28:92:93:C8:DD expected=12
389 This gives you the server’s real thumbprint. Of course this method is
391 not secure since it allows a Man-in-the-Middle (MITM) attack.
392
394 Some use cases require you to pass in the Managed Object Reference
395 ("moref") of an object on the VMware server.
396 For VMware ESXi hypervisors, the "vm" moref is a number (eg.
398 "vm=moref=2"). For VMware VCenter it is a string beginning with "vm-")
399 (eg. "vm=moref=vm-16"). Across ESXi and vCenter the numbers are
400 different even for the same virtual machine.
401 If you have libvirt ≥ 3.7, the moref is available in the virsh(1)
403 "dumpxml" output:
404 $ virsh -c 'esx://esxi.example.com?no_verify=1' dumpxml guestname
406 ...
407 <vmware:moref>2</vmware:moref>
408 ...
409 or:
411 $ virsh -c 'vpx://vcenter.example.com/Datacenter/esxi.example.com?no_verify=1' \
413 dumpxml guestname
414 ...
415 <vmware:moref>vm-16</vmware:moref>
416 ...
417 An alternative way to find the moref of a VM is using the
419 "moRefFinder.pl" script written by William Lam
420 (http://www.virtuallyghetto.com/2011/11/vsphere-moref-managed-object-reference.html
421 https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-2-technical.html).
422
424 To query more information about the plugin (and whether it is working),
425 use:
426 nbdkit vddk --dump-plugin
428 or:
430 nbdkit vddk --dump-plugin libdir=/opt/vmware-vix-disklib-distrib
432 (see "LIBRARY LOCATION" above).
434 If the plugin is not present or not working you will get an error.
436 If it works the output will include:
438 "vddk_default_libdir=..."
440 The compiled-in library path. Use "libdir=PATHNAME" to override
441 this at runtime.
442 "vddk_has_nfchostport=1"
444 If this is printed then the "nfchostport=PORT" parameter is
445 supported by this build.
446 "vddk_library_version=..."
448 The VDDK major library version: 6, 7, ... If this is omitted it
449 means the library could not be loaded.
450 "vddk_dll=..."
452 Prints the full path to the VDDK shared library. Since this
453 requires a glibc extension it may not be available in all builds of
454 the plugin.
455 "VixDiskLib_...=1"
457 For each VDDK API that the plugin uses and which is present in the
458 VDDK library that was loaded, we print the name of the API (eg.
459 "VixDiskLib_Open=1"). This lets you see which optional APIs are
460 available, such as "VixDiskLib_Flush" and
461 "VixDiskLib_QueryAllocatedBlocks". If the library could not be
462 loaded then these lines are not printed.
463
465 Sector size limitation
466 The VDDK plugin can only answer read/write requests on whole 512 byte
467 sector boundaries. This is because the VDDK Read and Write APIs only
468 take sector numbers. If your client needs finer granularity, you can
469 use nbdkit-blocksize-filter(1):
470 nbdkit vddk ... --filter=blocksize minblock=512
472 Out of memory errors
474 In the verbose log you may see errors like:
475 nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg:
477 received NFC error 5 from server: Failed to allocate the
478 requested 2097176 bytes
479 This seems especially common when there are multiple parallel
481 connections open to the VMware server with large NBD reads and writes.
482 Increase resource limits on the server
484 The error above can be caused by resource limits set on the VMware
486 server. You can increase the limit for the NFC service by editing
487 /etc/vmware/hostd/config.xml and adjusting the "<maxMemory>" setting:
488 <nfcsvc>
490 <path>libnfcsvc.so</path>
491 <enabled>true</enabled>
492 <maxMemory>50331648</maxMemory>
493 <maxStreamMemory>10485760</maxStreamMemory>
494 </nfcsvc>
495 and restarting the "hostd" service:
497 # /etc/init.d/hostd restart
499 For more information see https://bugzilla.redhat.com/1614276.
501 Limit request sizes
503 In addition, or as an alternative to adjusting the server
505 configuration, you can use nbdkit-blocksize-filter(1) to limit the
506 maximum request size. By default this plugin translates NBD requests
507 directly into VDDK requests, and it appears that very large VDDK
508 requests can cause the error seen above.
509 Using:
511 nbdkit vddk ... --filter=blocksize minblock=512 maxdata=2M
513 will cause nbdkit to automatically split and combine requests so that
515 VDDK sees only sizes in the range "[512..2M]".
516 Troubleshooting performance problems
518 VDDK has very uneven performance with some operations being very slow.
519 This plugin has options to allow you to debug performance issues. If
520 your application has a debug or diagnostic setting, add the following
521 nbdkit command line options:
522 -v -D nbdkit.backend.datapath=0 -D vddk.datapath=0 -D vddk.stats=1
524 "-v" enables verbose messages and the two datapath options disable the
526 very verbose per-read/-write messages. "-D vddk.stats=1" enables a
527 summary when nbdkit exits of the cumulative time taken in each VDDK
528 function, the number of times each function was called, and (for read
529 and write) the number of bytes transferred. An example of what those
530 stats look like can be found here:
531 https://gitlab.com/nbdkit/nbdkit/-/commit/5c80f0d290db45a679d55baf37ff39bacb8ce7ec
532 You can interpret the stats as follows:
534 "Read"
536 The cumulative time spent waiting for VDDK to return from
537 "VixDiskLib_Read" calls, the number of times this function was
538 called, and the total bytes read. You can use this to determine
539 the read bandwidth to the VMware server.
540 "Write"
542 "Flush"
543 Same as above, but for writing and flushing writes.
544 "ReadAsync"
546 "WriteAsync"
547 Same as above, but for asynchronous read and write calls introduced
548 in nbdkit 1.30. Unfortunately at the moment the amount of time
549 spent in these calls is not accounted for correctly.
550 "QueryAllocatedBlocks"
552 This call is used to query information about the sparseness of the
553 remote disk. It is only available in VDDK ≥ 6.7. The call is
554 notably very slow in all versions of VMware we have tested.
555 "Open"
557 "Close"
558 "ConnectEx"
559 "Disconnect"
560 "InitEx"
561 "Exit"
562 The cumulative time spent connecting and disconnecting from the
563 VMware server, which can also be very slow.
564
566 This plugin requires VDDK ≥ 6.5 (released Nov 2016). It is only
567 supported on the x64-64 archtecture.
568 It has been tested with all versions up to 7.0.3 (but should work with
570 future versions).
571 VDDK 6.7 was the first version that supported the
573 "VixDiskLib_QueryAllocatedBlocks" API, required to provide extent
574 information over NBD.
575
577 Debugging messages can be very helpful if you have problems connecting
578 to VMware servers, or to find the list of available transport modes, or
579 to diagnose SAN problems:
580 nbdkit -f -v vddk file=FILENAME [...]
582 Additional debug flags are available:
584 -D vddk.diskinfo=1
586 Debug disk information returned by "GetInfo".
587 -D vddk.extents=1
589 Debug extents returned by "QueryAllocatedBlocks".
590 -D vddk.datapath=0
592 Suppress debugging of datapath calls ("Read", "ReadAsync", "Write"
593 and "WriteAsync").
594 -D vddk.stats=1
596 When the plugin exits print some statistics about each VDDK call.
597
599 $plugindir/nbdkit-vddk-plugin.so
600 The plugin.
601 Use "nbdkit --dump-config" to find the location of $plugindir.
603
605 "nbdkit-vddk-plugin" first appeared in nbdkit 1.2.
606
608 nbdkit(1), nbdkit-plugin(3), nbdkit-blocksize-filter(1),
609 nbdkit-readahead-filter(1), nbdkit-retry-filter(1), virsh(1),
610 https://libvirt.org/drvesx.html,
611 https://www.vmware.com/support/developer/vddk/, VMware document “Best
612 Practices for NBD Transport”.
613
615 Richard W.M. Jones
616
618 Copyright (C) 2013-2020 Red Hat Inc.
619
621 Redistribution and use in source and binary forms, with or without
622 modification, are permitted provided that the following conditions are
623 met:
624 • Redistributions of source code must retain the above copyright
626 notice, this list of conditions and the following disclaimer.
627 • Redistributions in binary form must reproduce the above copyright
629 notice, this list of conditions and the following disclaimer in the
630 documentation and/or other materials provided with the
631 distribution.
632 • Neither the name of Red Hat nor the names of its contributors may
634 be used to endorse or promote products derived from this software
635 without specific prior written permission.
636 THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
638 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
639 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
640 PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
641 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
642 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
643 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
644 BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
645 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
646 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
647 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
648nbdkit-1.30.7 2022-07-10 nbdkit-vddk-plugin(1)