1guestfs-performance(1) Virtualization Support guestfs-performance(1)
2
6 guestfs-performance - engineering libguestfs for greatest performance
7
9 This page documents how to get the greatest performance out of
10 libguestfs, especially when you expect to use libguestfs to manipulate
11 thousands of virtual machines or disk images.
12 Three main areas are covered. Libguestfs runs an appliance (a small
14 Linux distribution) inside qemu/KVM. The first two areas are:
15 minimizing the time taken to start this appliance, and the number of
16 times the appliance has to be started. The third area is shortening
17 the time taken for inspection of VMs.
18
20 Before making changes to how you use libguestfs, take baseline
21 measurements.
22 Baseline: Starting the appliance
24 On an unloaded machine, time how long it takes to start up the
25 appliance:
26 time guestfish -a /dev/null run
28 Run this command several times in a row and discard the first few runs,
30 so that you are measuring a typical "hot cache" case.
31 Side note for developers: There is a program called boot-benchmark in
33 https://github.com/libguestfs/libguestfs-analysis-tools which does the
34 same thing, but performs multiple runs and prints the mean and standard
35 deviation.
36 Explanation
38 The guestfish command above starts up the libguestfs appliance on a
40 null disk, and then immediately shuts it down. The first time you run
41 the command, it will create an appliance and cache it (usually under
42 /var/tmp/.guestfs-*). Subsequent runs should reuse the cached
43 appliance.
44 Expected results
46 You should expect to be getting times under 6 seconds. If the times
48 you see on an unloaded machine are above this, then see the section
49 "TROUBLESHOOTING POOR PERFORMANCE" below.
50 Baseline: Performing inspection of a guest
52 For this test you will need an unloaded machine and at least one real
53 guest or disk image. If you are planning to use libguestfs against
54 only X guests (eg. X = Windows), then using an X guest here would be
55 most appropriate. If you are planning to run libguestfs against a mix
56 of guests, then use a mix of guests for testing here.
57 Time how long it takes to perform inspection and mount the disks of the
59 guest. Use the first command if you will be using disk images, and the
60 second command if you will be using libvirt.
61 time guestfish --ro -a disk.img -i exit
63 time guestfish --ro -d GuestName -i exit
65 Run the command several times in a row and discard the first few runs,
67 so that you are measuring a typical "hot cache" case.
68 Explanation
70 This command starts up the libguestfs appliance on the named disk image
72 or libvirt guest, performs libguestfs inspection on it (see
73 "INSPECTION" in guestfs(3)), mounts the guest’s disks, then discards
74 all these results and shuts down.
75 The first time you run the command, it will create an appliance and
77 cache it (usually under /var/tmp/.guestfs-*). Subsequent runs should
78 reuse the cached appliance.
79 Expected results
81 You should expect times which are ≤ 5 seconds greater than measured in
83 the first baseline test above. (For example, if the first baseline
84 test ran in 5 seconds, then this test should run in ≤ 10 seconds).
85
87 The first time you use libguestfs, it will build and cache an
88 appliance. This is usually in /var/tmp/.guestfs-*, unless you have set
89 $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that
90 temporary directory.
91 For more information about how the appliance is constructed, see
93 "SUPERMIN APPLIANCES" in supermin(1).
94 Every time libguestfs runs it will check that no host files used by the
96 appliance have changed. If any have, then the appliance is rebuilt.
97 This usually happens when a package is installed or updated on the host
98 (eg. using programs like "yum" or "apt-get"). The reason for
99 reconstructing the appliance is security: the new program that has been
100 installed might contain a security fix, and so we want to include the
101 fixed program in the appliance automatically.
102 These are the performance implications:
104 • The process of building (or rebuilding) the cached appliance is
106 slow, and you can avoid this happening by using a fixed appliance
107 (see below).
108 • If not using a fixed appliance, be aware that updating software on
110 the host will cause a one time rebuild of the appliance.
111 • /var/tmp (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast
113 disk, and have plenty of space for the appliance.
114
116 To fully control when the appliance is built, you can build a fixed
117 appliance. This appliance should be stored on a fast local disk.
118 To build the appliance, run the command:
120 libguestfs-make-fixed-appliance <directory>
122 replacing "<directory>" with the name of a directory where the
124 appliance will be stored (normally you would name a subdirectory, for
125 example: /usr/local/lib/guestfs/appliance or /dev/shm/appliance).
126 Then set $LIBGUESTFS_PATH (and ensure this environment variable is set
128 in your libguestfs program), or modify your program so it calls
129 "guestfs_set_path". For example:
130 export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance
132 Now you can run libguestfs programs, virt tools, guestfish etc. as
134 normal. The programs will use your fixed appliance, and will not ever
135 build, rebuild, or cache their own appliance.
136 (For detailed information on this subject, see:
138 libguestfs-make-fixed-appliance(1)).
139 Performance of the fixed appliance
141 In our testing we did not find that using a fixed appliance gave any
142 measurable performance benefit, even when the appliance was located in
143 memory (ie. on /dev/shm). However there are two points to consider:
144 1. Using a fixed appliance stops libguestfs from ever rebuilding the
146 appliance, meaning that libguestfs will have more predictable
147 start-up times.
148 2. The appliance is loaded on demand. A simple test such as:
150 time guestfish -a /dev/null run
152 does not load very much of the appliance. A real libguestfs
154 program using complicated API calls would demand-load a lot more of
155 the appliance. Being able to store the appliance in a specified
156 location makes the performance more predictable.
157
159 By far the most effective, though not always the simplest way to get
160 good performance is to ensure that the appliance is launched the
161 minimum number of times. This will probably involve changing your
162 libguestfs application.
163 Try to call "guestfs_launch" at most once per target virtual machine or
165 disk image.
166 Instead of using a separate instance of guestfish(1) to make a series
168 of changes to the same guest, use a single instance of guestfish and/or
169 use the guestfish --listen option.
170 Consider writing your program as a daemon which holds a guest open
172 while making a series of changes. Or marshal all the operations you
173 want to perform before opening the guest.
174 You can also try adding disks from multiple guests to a single
176 appliance. Before trying this, note the following points:
177 1. Adding multiple guests to one appliance is a security problem
179 because it may allow one guest to interfere with the disks of
180 another guest. Only do it if you trust all the guests, or if you
181 can group guests by trust.
182 2. There is a hard limit to the number of disks you can add to a
184 single appliance. Call "guestfs_max_disks" in guestfs(3) to get
185 this limit. For further information see "LIMITS" in guestfs(3).
186 3. Using libguestfs this way is complicated. Disks can have
188 unexpected interactions: for example, if two guests use the same
189 UUID for a filesystem (because they were cloned), or have volume
190 groups with the same name (but see "guestfs_lvm_set_filter").
191 virt-df(1) adds multiple disks by default, so the source code for this
193 program would be a good place to start.
194
196 The main advice is obvious: Do not perform inspection (which is
197 expensive) unless you need the results.
198 If you previously performed inspection on the guest, then it may be
200 safe to cache and reuse the results from last time.
201 Some disks don’t need to be inspected at all: for example, if you are
203 creating a disk image, or if the disk image is not a VM, or if the disk
204 image has a known layout.
205 Even when basic inspection ("guestfs_inspect_os") is required,
207 auxiliary inspection operations may be avoided:
208 • Mounting disks is only necessary to get further filesystem
210 information.
211 • Listing applications ("guestfs_inspect_list_applications") is an
213 expensive operation on Linux, but almost free on Windows.
214 • Generating a guest icon ("guestfs_inspect_get_icon") is cheap on
216 Linux but expensive on Windows.
217
219 Libguestfs appliances are mostly I/O bound and you can launch multiple
220 appliances in parallel. Provided there is enough free memory, there
221 should be little difference in launching 1 appliance vs N appliances in
222 parallel.
223 On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not
225 especially realistic) test Perl script below, the following plot shows
226 excellent scalability when running between 1 and 20 appliances in
227 parallel:
228 12 ++---+----+----+----+-----+----+----+----+----+---++
230 + + + + + + + + + + *
231 | |
232 | * |
233 11 ++ ++
234 | |
235 | |
236 | * * |
237 10 ++ ++
238 | * |
239 | |
240 s | |
241 9 ++ ++
242 e | |
243 | * |
244 c | |
245 8 ++ * ++
246 o | * |
247 | |
248 n 7 ++ ++
249 | * |
250 d | * |
251 | |
252 s 6 ++ ++
253 | * * |
254 | * |
255 | |
256 5 ++ ++
257 | |
258 | * |
259 | * * |
260 4 ++ ++
261 | |
262 | |
263 + * * * + + + + + + + +
264 3 ++-*-+----+----+----+-----+----+----+----+----+---++
265 0 2 4 6 8 10 12 14 16 18 20
266 number of parallel appliances
267 It is possible to run many more than 20 appliances in parallel, but if
269 you are using the libvirt backend then you should be aware that out of
270 the box libvirt limits the number of client connections to 20.
271 The simple Perl script below was used to collect the data for the plot
273 above, but there is much more information on this subject, including
274 more advanced test scripts and graphs, available in the following blog
275 postings:
276 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/
278 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/
279 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/
280 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/
281 #!/usr/bin/env perl
283 use strict;
285 use threads;
286 use warnings;
287 use Sys::Guestfs;
288 use Time::HiRes qw(time);
289 sub test {
291 my $g = Sys::Guestfs->new;
292 $g->add_drive_ro ("/dev/null");
293 $g->launch ();
294 # You could add some work for libguestfs to do here.
296 $g->close ();
298 }
299 # Get everything into cache.
301 test (); test (); test ();
302 for my $nr_threads (1..20) {
304 my $start_t = time ();
305 my @threads;
306 foreach (1..$nr_threads) {
307 push @threads, threads->create (\&test)
308 }
309 foreach (@threads) {
310 $_->join ();
311 if (my $err = $_->error ()) {
312 die "launch failed with $nr_threads threads: $err"
313 }
314 }
315 my $end_t = time ();
316 printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
317 }
318
320 Since libguestfs 1.24, it has been possible to use the User-Mode Linux
321 (uml) backend instead of KVM (see "USER-MODE LINUX BACKEND" in
322 guestfs(3)). This section makes some general remarks about this
323 backend, but it is highly advisable to measure your own workload under
324 UML rather than trusting comments or intuition.
325 • UML usually performs the same or slightly slower than KVM, on
327 baremetal.
328 • However UML often performs the same under virtualization as it does
330 on baremetal, whereas KVM can run much slower under virtualization
331 (since hardware virt acceleration is not available).
332 • Upload and download is as much as 10 times slower on UML than KVM.
334 Libguestfs sends this data over the UML emulated serial port, which
335 is far less efficient than KVM’s virtio-serial.
336 • UML lacks some features (eg. qcow2 support), so it may not be
338 applicable at all.
339 For some actual figures, see:
341 http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content
342
344 Ensure hardware virtualization is available
345 Use /proc/cpuinfo to ensure that hardware virtualization is available.
346 Note that you may need to enable it in your BIOS.
347 Hardware virt is not usually available inside VMs, and libguestfs will
349 run slowly inside another virtual machine whatever you do. Nested
350 virtualization does not work well in our experience, and is certainly
351 no substitute for running libguestfs on baremetal.
352 Ensure KVM is available
354 Ensure that KVM is enabled and available to the user that will run
355 libguestfs. It should be safe to set 0666 permissions on /dev/kvm and
356 most distributions now do this.
357 Processors to avoid
359 Avoid processors that don’t have hardware virtualization, and some
360 processors which are simply very slow (AMD Geode being a great
361 example).
362 Xen dom0
364 In Xen, dom0 is a virtual machine, and so hardware virtualization is
365 not available.
366 Use libguestfs ≥ 1.34 and qemu ≥ 2.7
368 During the libguestfs 1.33 development cycle, we spent a large amount
369 of time concentrating on boot performance, and added some patches to
370 libguestfs, qemu and Linux which in some cases can reduce boot times to
371 well under 1 second. You may therefore get much better performance by
372 moving to the versions of libguestfs or qemu mentioned in the heading.
373
375 Boot analysis
376 In https://github.com/libguestfs/libguestfs-analysis-tools is a program
377 called "boot-analysis". This program is able to produce a very
378 detailed breakdown of the boot steps (eg. qemu, BIOS, kernel,
379 libguestfs init script), and can measure how long it takes to perform
380 each step.
381 Detailed timings using ts
383 Use the ts(1) command (from moreutils) to show detailed timings:
384 $ guestfish -a /dev/null run -v |& ts -i '%.s'
386 0.000022 libguestfs: launch: program=guestfish
387 0.000134 libguestfs: launch: version=1.29.31fedora=23,release=2.fc23,libvirt
388 0.000044 libguestfs: launch: backend registered: unix
389 0.000035 libguestfs: launch: backend registered: uml
390 0.000035 libguestfs: launch: backend registered: libvirt
391 0.000032 libguestfs: launch: backend registered: direct
392 0.000030 libguestfs: launch: backend=libvirt
393 0.000031 libguestfs: launch: tmpdir=/tmp/libguestfsw18rBQ
394 0.000029 libguestfs: launch: umask=0002
395 0.000031 libguestfs: launch: euid=1000
396 0.000030 libguestfs: libvirt version = 1002012 (1.2.12)
397 [etc]
398 The timestamps are seconds (incrementally since the previous line).
400 Detailed timings using SystemTap
402 You can use SystemTap (stap(1)) to get detailed timings from libguestfs
403 programs.
404 Save the following script as time.stap:
406 global last;
408 function display_time () {
410 now = gettimeofday_us ();
411 delta = 0;
412 if (last > 0)
413 delta = now - last;
414 last = now;
415 printf ("%d (+%d):", now, delta);
417 }
418 probe begin {
420 last = 0;
421 printf ("ready\n");
422 }
423 /* Display all calls to static markers. */
425 probe process("/usr/lib*/libguestfs.so.0")
426 .provider("guestfs").mark("*") ? {
427 display_time();
428 printf ("\t%s %s\n", $$name, $$parms);
429 }
430 /* Display all calls to guestfs_* functions. */
432 probe process("/usr/lib*/libguestfs.so.0")
433 .function("guestfs_[a-z]*") ? {
434 display_time();
435 printf ("\t%s %s\n", probefunc(), $$parms);
436 }
437 Run it as root in one window:
439 # stap time.stap
441 ready
442 It prints "ready" when SystemTap has loaded the program. Run your
444 libguestfs program, guestfish or a virt tool in another window. For
445 example:
446 $ guestfish -a /dev/null run
448 In the stap window you will see a large amount of output, with the time
450 taken for each step shown (microseconds in parenthesis). For example:
451 xxxx (+0): guestfs_create
453 xxxx (+29): guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
454 xxxx (+9): guestfs_add_drive_opts_argv g=0x17a9de0 [...]
455 xxxx (+8): guestfs_int_safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
456 xxxx (+19): guestfs_int_safe_malloc g=0x17a9de0 nbytes=0x38
457 xxxx (+5): guestfs_int_safe_strdup g=0x17a9de0 str=0x17a9f60
458 xxxx (+10): guestfs_launch g=0x17a9de0
459 xxxx (+4): launch_start
460 [etc]
461 You will need to consult, and even modify, the source to libguestfs to
463 fully understand the output.
464 Detailed debugging using gdb
466 You can attach to the appliance BIOS/kernel using gdb. If you know
467 what you're doing, this can be a useful way to diagnose boot
468 regressions.
469 Firstly, you have to change qemu so it runs with the "-S" and "-s"
471 options. These options cause qemu to pause at boot and allow you to
472 attach a debugger. Read qemu(1) for further information. Libguestfs
473 invokes qemu several times (to scan the help output and so on) and you
474 only want the final invocation of qemu to use these options, so use a
475 qemu wrapper script like this:
476 #!/bin/bash -
478 # Set this to point to the real qemu binary.
480 qemu=/usr/bin/qemu-kvm
481 if [ "$1" != "-global" ]; then
483 # Scanning help output etc.
484 exec $qemu "$@"
485 else
486 # Really running qemu.
487 exec $qemu -S -s "$@"
488 fi
489 Now run guestfish or another libguestfs tool with the qemu wrapper (see
491 "QEMU WRAPPERS" in guestfs(3) to understand what this is doing):
492 LIBGUESTFS_HV=/path/to/qemu-wrapper guestfish -a /dev/null -v run
494 This should pause just after qemu launches. In another window, attach
496 to qemu using gdb:
497 $ gdb
499 (gdb) set architecture i8086
500 The target architecture is assumed to be i8086
501 (gdb) target remote :1234
502 Remote debugging using :1234
503 0x0000fff0 in ?? ()
504 (gdb) cont
505 At this point you can use standard gdb techniques, eg. hitting "^C" to
507 interrupt the boot and "bt" get a stack trace, setting breakpoints,
508 etc. Note that when you are past the BIOS and into the Linux kernel,
509 you'll want to change the architecture back to 32 or 64 bit.
510
512 Sometimes performance regressions happen in other programs (eg. qemu,
513 the kernel) that cause problems for libguestfs.
514 In https://github.com/libguestfs/libguestfs-analysis-tools
516 boot-benchmark/boot-benchmark-range.pl is a script which can be used to
517 benchmark libguestfs across a range of git commits in another project
518 to find out if any commit is causing a slowdown (or speedup).
519 To find out how to use this script, consult the manual:
521 ./boot-benchmark/boot-benchmark-range.pl --man
523
525 supermin(1), guestfish(1), guestfs(3), guestfs-examples(3),
526 guestfs-internals(1), libguestfs-make-fixed-appliance(1), stap(1),
527 qemu(1), gdb(1), http://libguestfs.org/.
528
530 Richard W.M. Jones ("rjones at redhat dot com")
531
533 Copyright (C) 2012-2020 Red Hat Inc.
534
536 This library is free software; you can redistribute it and/or modify it
537 under the terms of the GNU Lesser General Public License as published
538 by the Free Software Foundation; either version 2 of the License, or
539 (at your option) any later version.
540 This library is distributed in the hope that it will be useful, but
542 WITHOUT ANY WARRANTY; without even the implied warranty of
543 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
544 Lesser General Public License for more details.
545 You should have received a copy of the GNU Lesser General Public
547 License along with this library; if not, write to the Free Software
548 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
549 02110-1301 USA
550
552 To get a list of bugs against libguestfs, use this link:
553 https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
554 To report a new bug against libguestfs, use this link:
556 https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
557 When reporting a bug, please supply:
559 • The version of libguestfs.
561 • Where you got libguestfs (eg. which Linux distro, compiled from
563 source, etc)
564 • Describe the bug accurately and give a way to reproduce it.
566 • Run libguestfs-test-tool(1) and paste the complete, unedited output
568 into the bug report.
569libguestfs-1.45.4 2021-04-03 guestfs-performance(1)