1guestfs-performance(1) Virtualization Support guestfs-performance(1)
2
6 guestfs-performance - engineering libguestfs for greatest performance
7
9 This page documents how to get the greatest performance out of
10 libguestfs, especially when you expect to use libguestfs to manipulate
11 thousands of virtual machines or disk images.
12 Three main areas are covered. Libguestfs runs an appliance (a small
14 Linux distribution) inside qemu/KVM. The first two areas are:
15 minimizing the time taken to start this appliance, and the number of
16 times the appliance has to be started. The third area is shortening
17 the time taken for inspection of VMs.
18
20 Before making changes to how you use libguestfs, take baseline
21 measurements.
22 Baseline: Starting the appliance
24 On an unloaded machine, time how long it takes to start up the
25 appliance:
26 time guestfish -a /dev/null run
28 Run this command several times in a row and discard the first few runs,
30 so that you are measuring a typical "hot cache" case.
31 Side note for developers: If you are compiling libguestfs from source,
33 there is a program called utils/boot-benchmark/boot-benchmark which
34 does the same thing, but performs multiple runs and prints the mean and
35 standard deviation. To run it, do:
36 make
38 ./run utils/boot-benchmark/boot-benchmark
39 There is a manual page utils/boot-benchmark/boot-benchmark.1
41 Explanation
43 The guestfish command above starts up the libguestfs appliance on a
45 null disk, and then immediately shuts it down. The first time you run
46 the command, it will create an appliance and cache it (usually under
47 /var/tmp/.guestfs-*). Subsequent runs should reuse the cached
48 appliance.
49 Expected results
51 You should expect to be getting times under 6 seconds. If the times
53 you see on an unloaded machine are above this, then see the section
54 "TROUBLESHOOTING POOR PERFORMANCE" below.
55 Baseline: Performing inspection of a guest
57 For this test you will need an unloaded machine and at least one real
58 guest or disk image. If you are planning to use libguestfs against
59 only X guests (eg. X = Windows), then using an X guest here would be
60 most appropriate. If you are planning to run libguestfs against a mix
61 of guests, then use a mix of guests for testing here.
62 Time how long it takes to perform inspection and mount the disks of the
64 guest. Use the first command if you will be using disk images, and the
65 second command if you will be using libvirt.
66 time guestfish --ro -a disk.img -i exit
68 time guestfish --ro -d GuestName -i exit
70 Run the command several times in a row and discard the first few runs,
72 so that you are measuring a typical "hot cache" case.
73 Explanation
75 This command starts up the libguestfs appliance on the named disk image
77 or libvirt guest, performs libguestfs inspection on it (see
78 "INSPECTION" in guestfs(3)), mounts the guest’s disks, then discards
79 all these results and shuts down.
80 The first time you run the command, it will create an appliance and
82 cache it (usually under /var/tmp/.guestfs-*). Subsequent runs should
83 reuse the cached appliance.
84 Expected results
86 You should expect times which are ≤ 5 seconds greater than measured in
88 the first baseline test above. (For example, if the first baseline
89 test ran in 5 seconds, then this test should run in ≤ 10 seconds).
90
92 The first time you use libguestfs, it will build and cache an
93 appliance. This is usually in /var/tmp/.guestfs-*, unless you have set
94 $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that
95 temporary directory.
96 For more information about how the appliance is constructed, see
98 "SUPERMIN APPLIANCES" in supermin(1).
99 Every time libguestfs runs it will check that no host files used by the
101 appliance have changed. If any have, then the appliance is rebuilt.
102 This usually happens when a package is installed or updated on the host
103 (eg. using programs like "yum" or "apt-get"). The reason for
104 reconstructing the appliance is security: the new program that has been
105 installed might contain a security fix, and so we want to include the
106 fixed program in the appliance automatically.
107 These are the performance implications:
109 · The process of building (or rebuilding) the cached appliance is
111 slow, and you can avoid this happening by using a fixed appliance
112 (see below).
113 · If not using a fixed appliance, be aware that updating software on
115 the host will cause a one time rebuild of the appliance.
116 · /var/tmp (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast
118 disk, and have plenty of space for the appliance.
119
121 To fully control when the appliance is built, you can build a fixed
122 appliance. This appliance should be stored on a fast local disk.
123 To build the appliance, run the command:
125 libguestfs-make-fixed-appliance <directory>
127 replacing "<directory>" with the name of a directory where the
129 appliance will be stored (normally you would name a subdirectory, for
130 example: /usr/local/lib/guestfs/appliance or /dev/shm/appliance).
131 Then set $LIBGUESTFS_PATH (and ensure this environment variable is set
133 in your libguestfs program), or modify your program so it calls
134 "guestfs_set_path". For example:
135 export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance
137 Now you can run libguestfs programs, virt tools, guestfish etc. as
139 normal. The programs will use your fixed appliance, and will not ever
140 build, rebuild, or cache their own appliance.
141 (For detailed information on this subject, see:
143 libguestfs-make-fixed-appliance(1)).
144 Performance of the fixed appliance
146 In our testing we did not find that using a fixed appliance gave any
147 measurable performance benefit, even when the appliance was located in
148 memory (ie. on /dev/shm). However there are two points to consider:
149 1. Using a fixed appliance stops libguestfs from ever rebuilding the
151 appliance, meaning that libguestfs will have more predictable
152 start-up times.
153 2. The appliance is loaded on demand. A simple test such as:
155 time guestfish -a /dev/null run
157 does not load very much of the appliance. A real libguestfs
159 program using complicated API calls would demand-load a lot more of
160 the appliance. Being able to store the appliance in a specified
161 location makes the performance more predictable.
162
164 By far the most effective, though not always the simplest way to get
165 good performance is to ensure that the appliance is launched the
166 minimum number of times. This will probably involve changing your
167 libguestfs application.
168 Try to call "guestfs_launch" at most once per target virtual machine or
170 disk image.
171 Instead of using a separate instance of guestfish(1) to make a series
173 of changes to the same guest, use a single instance of guestfish and/or
174 use the guestfish --listen option.
175 Consider writing your program as a daemon which holds a guest open
177 while making a series of changes. Or marshal all the operations you
178 want to perform before opening the guest.
179 You can also try adding disks from multiple guests to a single
181 appliance. Before trying this, note the following points:
182 1. Adding multiple guests to one appliance is a security problem
184 because it may allow one guest to interfere with the disks of
185 another guest. Only do it if you trust all the guests, or if you
186 can group guests by trust.
187 2. There is a hard limit to the number of disks you can add to a
189 single appliance. Call "guestfs_max_disks" in guestfs(3) to get
190 this limit. For further information see "LIMITS" in guestfs(3).
191 3. Using libguestfs this way is complicated. Disks can have
193 unexpected interactions: for example, if two guests use the same
194 UUID for a filesystem (because they were cloned), or have volume
195 groups with the same name (but see "guestfs_lvm_set_filter").
196 virt-df(1) adds multiple disks by default, so the source code for this
198 program would be a good place to start.
199
201 The main advice is obvious: Do not perform inspection (which is
202 expensive) unless you need the results.
203 If you previously performed inspection on the guest, then it may be
205 safe to cache and reuse the results from last time.
206 Some disks don’t need to be inspected at all: for example, if you are
208 creating a disk image, or if the disk image is not a VM, or if the disk
209 image has a known layout.
210 Even when basic inspection ("guestfs_inspect_os") is required,
212 auxiliary inspection operations may be avoided:
213 · Mounting disks is only necessary to get further filesystem
215 information.
216 · Listing applications ("guestfs_inspect_list_applications") is an
218 expensive operation on Linux, but almost free on Windows.
219 · Generating a guest icon ("guestfs_inspect_get_icon") is cheap on
221 Linux but expensive on Windows.
222
224 Libguestfs appliances are mostly I/O bound and you can launch multiple
225 appliances in parallel. Provided there is enough free memory, there
226 should be little difference in launching 1 appliance vs N appliances in
227 parallel.
228 On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not
230 especially realistic) test Perl script below, the following plot shows
231 excellent scalability when running between 1 and 20 appliances in
232 parallel:
233 12 ++---+----+----+----+-----+----+----+----+----+---++
235 + + + + + + + + + + *
236 | |
237 | * |
238 11 ++ ++
239 | |
240 | |
241 | * * |
242 10 ++ ++
243 | * |
244 | |
245 s | |
246 9 ++ ++
247 e | |
248 | * |
249 c | |
250 8 ++ * ++
251 o | * |
252 | |
253 n 7 ++ ++
254 | * |
255 d | * |
256 | |
257 s 6 ++ ++
258 | * * |
259 | * |
260 | |
261 5 ++ ++
262 | |
263 | * |
264 | * * |
265 4 ++ ++
266 | |
267 | |
268 + * * * + + + + + + + +
269 3 ++-*-+----+----+----+-----+----+----+----+----+---++
270 0 2 4 6 8 10 12 14 16 18 20
271 number of parallel appliances
272 It is possible to run many more than 20 appliances in parallel, but if
274 you are using the libvirt backend then you should be aware that out of
275 the box libvirt limits the number of client connections to 20.
276 The simple Perl script below was used to collect the data for the plot
278 above, but there is much more information on this subject, including
279 more advanced test scripts and graphs, available in the following blog
280 postings:
281 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/
283 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/
284 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/
285 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/
286 #!/usr/bin/env perl
288 use strict;
290 use threads;
291 use warnings;
292 use Sys::Guestfs;
293 use Time::HiRes qw(time);
294 sub test {
296 my $g = Sys::Guestfs->new;
297 $g->add_drive_ro ("/dev/null");
298 $g->launch ();
299 # You could add some work for libguestfs to do here.
301 $g->close ();
303 }
304 # Get everything into cache.
306 test (); test (); test ();
307 for my $nr_threads (1..20) {
309 my $start_t = time ();
310 my @threads;
311 foreach (1..$nr_threads) {
312 push @threads, threads->create (\&test)
313 }
314 foreach (@threads) {
315 $_->join ();
316 if (my $err = $_->error ()) {
317 die "launch failed with $nr_threads threads: $err"
318 }
319 }
320 my $end_t = time ();
321 printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
322 }
323
325 Since libguestfs 1.24, it has been possible to use the User-Mode Linux
326 (uml) backend instead of KVM (see "USER-MODE LINUX BACKEND" in
327 guestfs(3)). This section makes some general remarks about this
328 backend, but it is highly advisable to measure your own workload under
329 UML rather than trusting comments or intuition.
330 · UML usually performs the same or slightly slower than KVM, on
332 baremetal.
333 · However UML often performs the same under virtualization as it does
335 on baremetal, whereas KVM can run much slower under virtualization
336 (since hardware virt acceleration is not available).
337 · Upload and download is as much as 10 times slower on UML than KVM.
339 Libguestfs sends this data over the UML emulated serial port, which
340 is far less efficient than KVM’s virtio-serial.
341 · UML lacks some features (eg. qcow2 support), so it may not be
343 applicable at all.
344 For some actual figures, see:
346 http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content
347
349 Ensure hardware virtualization is available
350 Use /proc/cpuinfo to ensure that hardware virtualization is available.
351 Note that you may need to enable it in your BIOS.
352 Hardware virt is not usually available inside VMs, and libguestfs will
354 run slowly inside another virtual machine whatever you do. Nested
355 virtualization does not work well in our experience, and is certainly
356 no substitute for running libguestfs on baremetal.
357 Ensure KVM is available
359 Ensure that KVM is enabled and available to the user that will run
360 libguestfs. It should be safe to set 0666 permissions on /dev/kvm and
361 most distributions now do this.
362 Processors to avoid
364 Avoid processors that don’t have hardware virtualization, and some
365 processors which are simply very slow (AMD Geode being a great
366 example).
367 Xen dom0
369 In Xen, dom0 is a virtual machine, and so hardware virtualization is
370 not available.
371 Use libguestfs ≥ 1.34 and qemu ≥ 2.7
373 During the libguestfs 1.33 development cycle, we spent a large amount
374 of time concentrating on boot performance, and added some patches to
375 libguestfs, qemu and Linux which in some cases can reduce boot times to
376 well under 1 second. You may therefore get much better performance by
377 moving to the versions of libguestfs or qemu mentioned in the heading.
378
380 Boot analysis
381 In the libguestfs source directory, in utils/boot-analysis is a program
382 called "boot-analysis". This program is able to produce a very
383 detailed breakdown of the boot steps (eg. qemu, BIOS, kernel,
384 libguestfs init script), and can measure how long it takes to perform
385 each step.
386 To run this program, do:
388 make
390 ./run utils/boot-analysis/boot-analysis
391 There is a manual page utils/boot-benchmark/boot-analysis.1
393 Detailed timings using ts
395 Use the ts(1) command (from moreutils) to show detailed timings:
396 $ guestfish -a /dev/null run -v |& ts -i '%.s'
398 0.000022 libguestfs: launch: program=guestfish
399 0.000134 libguestfs: launch: version=1.29.31fedora=23,release=2.fc23,libvirt
400 0.000044 libguestfs: launch: backend registered: unix
401 0.000035 libguestfs: launch: backend registered: uml
402 0.000035 libguestfs: launch: backend registered: libvirt
403 0.000032 libguestfs: launch: backend registered: direct
404 0.000030 libguestfs: launch: backend=libvirt
405 0.000031 libguestfs: launch: tmpdir=/tmp/libguestfsw18rBQ
406 0.000029 libguestfs: launch: umask=0002
407 0.000031 libguestfs: launch: euid=1000
408 0.000030 libguestfs: libvirt version = 1002012 (1.2.12)
409 [etc]
410 The timestamps are seconds (incrementally since the previous line).
412 Detailed timings using SystemTap
414 You can use SystemTap (stap(1)) to get detailed timings from libguestfs
415 programs.
416 Save the following script as time.stap:
418 global last;
420 function display_time () {
422 now = gettimeofday_us ();
423 delta = 0;
424 if (last > 0)
425 delta = now - last;
426 last = now;
427 printf ("%d (+%d):", now, delta);
429 }
430 probe begin {
432 last = 0;
433 printf ("ready\n");
434 }
435 /* Display all calls to static markers. */
437 probe process("/usr/lib*/libguestfs.so.0")
438 .provider("guestfs").mark("*") ? {
439 display_time();
440 printf ("\t%s %s\n", $$name, $$parms);
441 }
442 /* Display all calls to guestfs_* functions. */
444 probe process("/usr/lib*/libguestfs.so.0")
445 .function("guestfs_[a-z]*") ? {
446 display_time();
447 printf ("\t%s %s\n", probefunc(), $$parms);
448 }
449 Run it as root in one window:
451 # stap time.stap
453 ready
454 It prints "ready" when SystemTap has loaded the program. Run your
456 libguestfs program, guestfish or a virt tool in another window. For
457 example:
458 $ guestfish -a /dev/null run
460 In the stap window you will see a large amount of output, with the time
462 taken for each step shown (microseconds in parenthesis). For example:
463 xxxx (+0): guestfs_create
465 xxxx (+29): guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
466 xxxx (+9): guestfs_add_drive_opts_argv g=0x17a9de0 [...]
467 xxxx (+8): guestfs_int_safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
468 xxxx (+19): guestfs_int_safe_malloc g=0x17a9de0 nbytes=0x38
469 xxxx (+5): guestfs_int_safe_strdup g=0x17a9de0 str=0x17a9f60
470 xxxx (+10): guestfs_launch g=0x17a9de0
471 xxxx (+4): launch_start
472 [etc]
473 You will need to consult, and even modify, the source to libguestfs to
475 fully understand the output.
476 Detailed debugging using gdb
478 You can attach to the appliance BIOS/kernel using gdb. If you know
479 what you're doing, this can be a useful way to diagnose boot
480 regressions.
481 Firstly, you have to change qemu so it runs with the "-S" and "-s"
483 options. These options cause qemu to pause at boot and allow you to
484 attach a debugger. Read qemu(1) for further information. Libguestfs
485 invokes qemu several times (to scan the help output and so on) and you
486 only want the final invocation of qemu to use these options, so use a
487 qemu wrapper script like this:
488 #!/bin/bash -
490 # Set this to point to the real qemu binary.
492 qemu=/usr/bin/qemu-kvm
493 if [ "$1" != "-global" ]; then
495 # Scanning help output etc.
496 exec $qemu "$@"
497 else
498 # Really running qemu.
499 exec $qemu -S -s "$@"
500 fi
501 Now run guestfish or another libguestfs tool with the qemu wrapper (see
503 "QEMU WRAPPERS" in guestfs(3) to understand what this is doing):
504 LIBGUESTFS_HV=/path/to/qemu-wrapper guestfish -a /dev/null -v run
506 This should pause just after qemu launches. In another window, attach
508 to qemu using gdb:
509 $ gdb
511 (gdb) set architecture i8086
512 The target architecture is assumed to be i8086
513 (gdb) target remote :1234
514 Remote debugging using :1234
515 0x0000fff0 in ?? ()
516 (gdb) cont
517 At this point you can use standard gdb techniques, eg. hitting "^C" to
519 interrupt the boot and "bt" get a stack trace, setting breakpoints,
520 etc. Note that when you are past the BIOS and into the Linux kernel,
521 you'll want to change the architecture back to 32 or 64 bit.
522
524 Sometimes performance regressions happen in other programs (eg. qemu,
525 the kernel) that cause problems for libguestfs.
526 In the libguestfs source, utils/boot-benchmark/boot-benchmark-range.pl
528 is a script which can be used to benchmark libguestfs across a range of
529 git commits in another project to find out if any commit is causing a
530 slowdown (or speedup).
531 To find out how to use this script, consult the manual:
533 ./utils/boot-benchmark/boot-benchmark-range.pl --man
535
537 supermin(1), guestfish(1), guestfs(3), guestfs-examples(3),
538 guestfs-internals(1), libguestfs-make-fixed-appliance(1), stap(1),
539 qemu(1), gdb(1), http://libguestfs.org/.
540
542 Richard W.M. Jones ("rjones at redhat dot com")
543
545 Copyright (C) 2012-2019 Red Hat Inc.
546
548 This library is free software; you can redistribute it and/or modify it
549 under the terms of the GNU Lesser General Public License as published
550 by the Free Software Foundation; either version 2 of the License, or
551 (at your option) any later version.
552 This library is distributed in the hope that it will be useful, but
554 WITHOUT ANY WARRANTY; without even the implied warranty of
555 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
556 Lesser General Public License for more details.
557 You should have received a copy of the GNU Lesser General Public
559 License along with this library; if not, write to the Free Software
560 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
561 02110-1301 USA
562
564 To get a list of bugs against libguestfs, use this link:
565 https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
566 To report a new bug against libguestfs, use this link:
568 https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
569 When reporting a bug, please supply:
571 · The version of libguestfs.
573 · Where you got libguestfs (eg. which Linux distro, compiled from
575 source, etc)
576 · Describe the bug accurately and give a way to reproduce it.
578 · Run libguestfs-test-tool(1) and paste the complete, unedited output
580 into the bug report.
581libguestfs-1.40.2 2019-02-07 guestfs-performance(1)