1guestfs-performance(1)      Virtualization Support      guestfs-performance(1)
2
3
4

NAME

6       guestfs-performance - engineering libguestfs for greatest performance
7

DESCRIPTION

9       This page documents how to get the greatest performance out of
10       libguestfs, especially when you expect to use libguestfs to manipulate
11       thousands of virtual machines or disk images.
12
13       Three main areas are covered. Libguestfs runs an appliance (a small
14       Linux distribution) inside qemu/KVM.  The first two areas are:
15       minimizing the time taken to start this appliance, and the number of
16       times the appliance has to be started.  The third area is shortening
17       the time taken for inspection of VMs.
18

BASELINE MEASUREMENTS

20       Before making changes to how you use libguestfs, take baseline
21       measurements.
22
23   Baseline: Starting the appliance
24       On an unloaded machine, time how long it takes to start up the
25       appliance:
26
27        time guestfish -a /dev/null run
28
29       Run this command several times in a row and discard the first few runs,
30       so that you are measuring a typical "hot cache" case.
31
32       Side note for developers: There is a program called boot-benchmark in
33       https://github.com/libguestfs/libguestfs-analysis-tools which does the
34       same thing, but performs multiple runs and prints the mean and standard
35       deviation.
36
37       Explanation
38
39       The guestfish command above starts up the libguestfs appliance on a
40       null disk, and then immediately shuts it down.  The first time you run
41       the command, it will create an appliance and cache it (usually under
42       /var/tmp/.guestfs-*).  Subsequent runs should reuse the cached
43       appliance.
44
45       Expected results
46
47       You should expect to be getting times under 6 seconds.  If the times
48       you see on an unloaded machine are above this, then see the section
49       "TROUBLESHOOTING POOR PERFORMANCE" below.
50
51   Baseline: Performing inspection of a guest
52       For this test you will need an unloaded machine and at least one real
53       guest or disk image.  If you are planning to use libguestfs against
54       only X guests (eg. X = Windows), then using an X guest here would be
55       most appropriate.  If you are planning to run libguestfs against a mix
56       of guests, then use a mix of guests for testing here.
57
58       Time how long it takes to perform inspection and mount the disks of the
59       guest.  Use the first command if you will be using disk images, and the
60       second command if you will be using libvirt.
61
62        time guestfish --ro -a disk.img -i exit
63
64        time guestfish --ro -d GuestName -i exit
65
66       Run the command several times in a row and discard the first few runs,
67       so that you are measuring a typical "hot cache" case.
68
69       Explanation
70
71       This command starts up the libguestfs appliance on the named disk image
72       or libvirt guest, performs libguestfs inspection on it (see
73       "INSPECTION" in guestfs(3)), mounts the guest’s disks, then discards
74       all these results and shuts down.
75
76       The first time you run the command, it will create an appliance and
77       cache it (usually under /var/tmp/.guestfs-*).  Subsequent runs should
78       reuse the cached appliance.
79
80       Expected results
81
82       You should expect times which are ≤ 5 seconds greater than measured in
83       the first baseline test above.  (For example, if the first baseline
84       test ran in 5 seconds, then this test should run in ≤ 10 seconds).
85

UNDERSTANDING THE APPLIANCE AND WHEN IT IS BUILT/CACHED

87       The first time you use libguestfs, it will build and cache an
88       appliance.  This is usually in /var/tmp/.guestfs-*, unless you have set
89       $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that
90       temporary directory.
91
92       For more information about how the appliance is constructed, see
93       "SUPERMIN APPLIANCES" in supermin(1).
94
95       Every time libguestfs runs it will check that no host files used by the
96       appliance have changed.  If any have, then the appliance is rebuilt.
97       This usually happens when a package is installed or updated on the host
98       (eg. using programs like "yum" or "apt-get").  The reason for
99       reconstructing the appliance is security: the new program that has been
100       installed might contain a security fix, and so we want to include the
101       fixed program in the appliance automatically.
102
103       These are the performance implications:
104
105       •   The process of building (or rebuilding) the cached appliance is
106           slow, and you can avoid this happening by using a fixed appliance
107           (see below).
108
109       •   If not using a fixed appliance, be aware that updating software on
110           the host will cause a one time rebuild of the appliance.
111
112/var/tmp (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast
113           disk, and have plenty of space for the appliance.
114

USING A FIXED APPLIANCE

116       To fully control when the appliance is built, you can build a fixed
117       appliance.  This appliance should be stored on a fast local disk.
118
119       To build the appliance, run the command:
120
121        libguestfs-make-fixed-appliance <directory>
122
123       replacing "<directory>" with the name of a directory where the
124       appliance will be stored (normally you would name a subdirectory, for
125       example: /usr/local/lib/guestfs/appliance or /dev/shm/appliance).
126
127       Then set $LIBGUESTFS_PATH (and ensure this environment variable is set
128       in your libguestfs program), or modify your program so it calls
129       "guestfs_set_path".  For example:
130
131        export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance
132
133       Now you can run libguestfs programs, virt tools, guestfish etc. as
134       normal.  The programs will use your fixed appliance, and will not ever
135       build, rebuild, or cache their own appliance.
136
137       (For detailed information on this subject, see:
138       libguestfs-make-fixed-appliance(1)).
139
140   Performance of the fixed appliance
141       In our testing we did not find that using a fixed appliance gave any
142       measurable performance benefit, even when the appliance was located in
143       memory (ie. on /dev/shm).  However there are two points to consider:
144
145       1.  Using a fixed appliance stops libguestfs from ever rebuilding the
146           appliance, meaning that libguestfs will have more predictable
147           start-up times.
148
149       2.  The appliance is loaded on demand.  A simple test such as:
150
151            time guestfish -a /dev/null run
152
153           does not load very much of the appliance.  A real libguestfs
154           program using complicated API calls would demand-load a lot more of
155           the appliance.  Being able to store the appliance in a specified
156           location makes the performance more predictable.
157

REDUCING THE NUMBER OF TIMES THE APPLIANCE IS LAUNCHED

159       By far the most effective, though not always the simplest way to get
160       good performance is to ensure that the appliance is launched the
161       minimum number of times.  This will probably involve changing your
162       libguestfs application.
163
164       Try to call "guestfs_launch" at most once per target virtual machine or
165       disk image.
166
167       Instead of using a separate instance of guestfish(1) to make a series
168       of changes to the same guest, use a single instance of guestfish and/or
169       use the guestfish --listen option.
170
171       Consider writing your program as a daemon which holds a guest open
172       while making a series of changes.  Or marshal all the operations you
173       want to perform before opening the guest.
174
175       You can also try adding disks from multiple guests to a single
176       appliance.  Before trying this, note the following points:
177
178       1.  Adding multiple guests to one appliance is a security problem
179           because it may allow one guest to interfere with the disks of
180           another guest.  Only do it if you trust all the guests, or if you
181           can group guests by trust.
182
183       2.  There is a hard limit to the number of disks you can add to a
184           single appliance.  Call "guestfs_max_disks" in guestfs(3) to get
185           this limit.  For further information see "LIMITS" in guestfs(3).
186
187       3.  Using libguestfs this way is complicated.  Disks can have
188           unexpected interactions: for example, if two guests use the same
189           UUID for a filesystem (because they were cloned), or have volume
190           groups with the same name (but see "guestfs_lvm_set_filter").
191
192       virt-df(1) adds multiple disks by default, so the source code for this
193       program would be a good place to start.
194

SHORTENING THE TIME TAKEN FOR INSPECTION OF VMs

196       The main advice is obvious: Do not perform inspection (which is
197       expensive) unless you need the results.
198
199       If you previously performed inspection on the guest, then it may be
200       safe to cache and reuse the results from last time.
201
202       Some disks don’t need to be inspected at all: for example, if you are
203       creating a disk image, or if the disk image is not a VM, or if the disk
204       image has a known layout.
205
206       Even when basic inspection ("guestfs_inspect_os") is required,
207       auxiliary inspection operations may be avoided:
208
209       •   Mounting disks is only necessary to get further filesystem
210           information.
211
212       •   Listing applications ("guestfs_inspect_list_applications") is an
213           expensive operation on Linux, but almost free on Windows.
214
215       •   Generating a guest icon ("guestfs_inspect_get_icon") is cheap on
216           Linux but expensive on Windows.
217

PARALLEL APPLIANCES

219       Libguestfs appliances are mostly I/O bound and you can launch multiple
220       appliances in parallel.  Provided there is enough free memory, there
221       should be little difference in launching 1 appliance vs N appliances in
222       parallel.
223
224       On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not
225       especially realistic) test Perl script below, the following plot shows
226       excellent scalability when running between 1 and 20 appliances in
227       parallel:
228
229         12 ++---+----+----+----+-----+----+----+----+----+---++
230            +    +    +    +    +     +    +    +    +    +    *
231            |                                                  |
232            |                                               *  |
233         11 ++                                                ++
234            |                                                  |
235            |                                                  |
236            |                                          *  *    |
237         10 ++                                                ++
238            |                                        *         |
239            |                                                  |
240        s   |                                                  |
241          9 ++                                                ++
242        e   |                                                  |
243            |                                     *            |
244        c   |                                                  |
245          8 ++                                  *             ++
246        o   |                                *                 |
247            |                                                  |
248        n 7 ++                                                ++
249            |                              *                   |
250        d   |                           *                      |
251            |                                                  |
252        s 6 ++                                                ++
253            |                      *  *                        |
254            |                   *                              |
255            |                                                  |
256          5 ++                                                ++
257            |                                                  |
258            |                 *                                |
259            |            * *                                   |
260          4 ++                                                ++
261            |                                                  |
262            |                                                  |
263            +    *  * *    +    +     +    +    +    +    +    +
264          3 ++-*-+----+----+----+-----+----+----+----+----+---++
265            0    2    4    6    8     10   12   14   16   18   20
266                      number of parallel appliances
267
268       It is possible to run many more than 20 appliances in parallel, but if
269       you are using the libvirt backend then you should be aware that out of
270       the box libvirt limits the number of client connections to 20.
271
272       The simple Perl script below was used to collect the data for the plot
273       above, but there is much more information on this subject, including
274       more advanced test scripts and graphs, available in the following blog
275       postings:
276
277       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/
278       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/
279       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/
280       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/
281
282        #!/usr/bin/env perl
283
284        use strict;
285        use threads;
286        use warnings;
287        use Sys::Guestfs;
288        use Time::HiRes qw(time);
289
290        sub test {
291            my $g = Sys::Guestfs->new;
292            $g->add_drive_ro ("/dev/null");
293            $g->launch ();
294
295            # You could add some work for libguestfs to do here.
296
297            $g->close ();
298        }
299
300        # Get everything into cache.
301        test (); test (); test ();
302
303        for my $nr_threads (1..20) {
304            my $start_t = time ();
305            my @threads;
306            foreach (1..$nr_threads) {
307                push @threads, threads->create (\&test)
308            }
309            foreach (@threads) {
310                $_->join ();
311                if (my $err = $_->error ()) {
312                    die "launch failed with $nr_threads threads: $err"
313                }
314            }
315            my $end_t = time ();
316            printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
317        }
318

USING USER-MODE LINUX

320       Since libguestfs 1.24, it has been possible to use the User-Mode Linux
321       (uml) backend instead of KVM (see "USER-MODE LINUX BACKEND" in
322       guestfs(3)).  This section makes some general remarks about this
323       backend, but it is highly advisable to measure your own workload under
324       UML rather than trusting comments or intuition.
325
326       •   UML usually performs the same or slightly slower than KVM, on
327           baremetal.
328
329       •   However UML often performs the same under virtualization as it does
330           on baremetal, whereas KVM can run much slower under virtualization
331           (since hardware virt acceleration is not available).
332
333       •   Upload and download is as much as 10 times slower on UML than KVM.
334           Libguestfs sends this data over the UML emulated serial port, which
335           is far less efficient than KVM’s virtio-serial.
336
337       •   UML lacks some features (eg. qcow2 support), so it may not be
338           applicable at all.
339
340       For some actual figures, see:
341       http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content
342

TROUBLESHOOTING POOR PERFORMANCE

344   Ensure hardware virtualization is available
345       Use /proc/cpuinfo to ensure that hardware virtualization is available.
346       Note that you may need to enable it in your BIOS.
347
348       Hardware virt is not usually available inside VMs, and libguestfs will
349       run slowly inside another virtual machine whatever you do.  Nested
350       virtualization does not work well in our experience, and is certainly
351       no substitute for running libguestfs on baremetal.
352
353   Ensure KVM is available
354       Ensure that KVM is enabled and available to the user that will run
355       libguestfs.  It should be safe to set 0666 permissions on /dev/kvm and
356       most distributions now do this.
357
358   Processors to avoid
359       Avoid processors that don’t have hardware virtualization, and some
360       processors which are simply very slow (AMD Geode being a great
361       example).
362
363   Xen dom0
364       In Xen, dom0 is a virtual machine, and so hardware virtualization is
365       not available.
366
367   Use libguestfs ≥ 1.34 and qemu ≥ 2.7
368       During the libguestfs 1.33 development cycle, we spent a large amount
369       of time concentrating on boot performance, and added some patches to
370       libguestfs, qemu and Linux which in some cases can reduce boot times to
371       well under 1 second.  You may therefore get much better performance by
372       moving to the versions of libguestfs or qemu mentioned in the heading.
373

DETAILED ANALYSIS

375   Boot analysis
376       In https://github.com/libguestfs/libguestfs-analysis-tools is a program
377       called "boot-analysis".  This program is able to produce a very
378       detailed breakdown of the boot steps (eg. qemu, BIOS, kernel,
379       libguestfs init script), and can measure how long it takes to perform
380       each step.
381
382   Detailed timings using ts
383       Use the ts(1) command (from moreutils) to show detailed timings:
384
385        $ guestfish -a /dev/null run -v |& ts -i '%.s'
386        0.000022 libguestfs: launch: program=guestfish
387        0.000134 libguestfs: launch: version=1.29.31fedora=23,release=2.fc23,libvirt
388        0.000044 libguestfs: launch: backend registered: unix
389        0.000035 libguestfs: launch: backend registered: uml
390        0.000035 libguestfs: launch: backend registered: libvirt
391        0.000032 libguestfs: launch: backend registered: direct
392        0.000030 libguestfs: launch: backend=libvirt
393        0.000031 libguestfs: launch: tmpdir=/tmp/libguestfsw18rBQ
394        0.000029 libguestfs: launch: umask=0002
395        0.000031 libguestfs: launch: euid=1000
396        0.000030 libguestfs: libvirt version = 1002012 (1.2.12)
397        [etc]
398
399       The timestamps are seconds (incrementally since the previous line).
400
401   Detailed timings using SystemTap
402       You can use SystemTap (stap(1)) to get detailed timings from libguestfs
403       programs.
404
405       Save the following script as time.stap:
406
407        global last;
408
409        function display_time () {
410              now = gettimeofday_us ();
411              delta = 0;
412              if (last > 0)
413                    delta = now - last;
414              last = now;
415
416              printf ("%d (+%d):", now, delta);
417        }
418
419        probe begin {
420              last = 0;
421              printf ("ready\n");
422        }
423
424        /* Display all calls to static markers. */
425        probe process("/usr/lib*/libguestfs.so.0")
426                  .provider("guestfs").mark("*") ? {
427              display_time();
428              printf ("\t%s %s\n", $$name, $$parms);
429        }
430
431        /* Display all calls to guestfs_* functions. */
432        probe process("/usr/lib*/libguestfs.so.0")
433                  .function("guestfs_[a-z]*") ? {
434              display_time();
435              printf ("\t%s %s\n", probefunc(), $$parms);
436        }
437
438       Run it as root in one window:
439
440        # stap time.stap
441        ready
442
443       It prints "ready" when SystemTap has loaded the program.  Run your
444       libguestfs program, guestfish or a virt tool in another window.  For
445       example:
446
447        $ guestfish -a /dev/null run
448
449       In the stap window you will see a large amount of output, with the time
450       taken for each step shown (microseconds in parenthesis).  For example:
451
452        xxxx (+0):     guestfs_create
453        xxxx (+29):    guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
454        xxxx (+9):     guestfs_add_drive_opts_argv g=0x17a9de0 [...]
455        xxxx (+8):     guestfs_int_safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
456        xxxx (+19):    guestfs_int_safe_malloc g=0x17a9de0 nbytes=0x38
457        xxxx (+5):     guestfs_int_safe_strdup g=0x17a9de0 str=0x17a9f60
458        xxxx (+10):    guestfs_launch g=0x17a9de0
459        xxxx (+4):     launch_start
460        [etc]
461
462       You will need to consult, and even modify, the source to libguestfs to
463       fully understand the output.
464
465   Detailed debugging using gdb
466       You can attach to the appliance BIOS/kernel using gdb.  If you know
467       what you're doing, this can be a useful way to diagnose boot
468       regressions.
469
470       Firstly, you have to change qemu so it runs with the "-S" and "-s"
471       options.  These options cause qemu to pause at boot and allow you to
472       attach a debugger.  Read qemu(1) for further information.  Libguestfs
473       invokes qemu several times (to scan the help output and so on) and you
474       only want the final invocation of qemu to use these options, so use a
475       qemu wrapper script like this:
476
477        #!/bin/bash -
478
479        # Set this to point to the real qemu binary.
480        qemu=/usr/bin/qemu-kvm
481
482        if [ "$1" != "-global" ]; then
483            # Scanning help output etc.
484            exec $qemu "$@"
485        else
486            # Really running qemu.
487            exec $qemu -S -s "$@"
488        fi
489
490       Now run guestfish or another libguestfs tool with the qemu wrapper (see
491       "QEMU WRAPPERS" in guestfs(3) to understand what this is doing):
492
493        LIBGUESTFS_HV=/path/to/qemu-wrapper guestfish -a /dev/null -v run
494
495       This should pause just after qemu launches.  In another window, attach
496       to qemu using gdb:
497
498        $ gdb
499        (gdb) set architecture i8086
500        The target architecture is assumed to be i8086
501        (gdb) target remote :1234
502        Remote debugging using :1234
503        0x0000fff0 in ?? ()
504        (gdb) cont
505
506       At this point you can use standard gdb techniques, eg. hitting "^C" to
507       interrupt the boot and "bt" get a stack trace, setting breakpoints,
508       etc.  Note that when you are past the BIOS and into the Linux kernel,
509       you'll want to change the architecture back to 32 or 64 bit.
510

PERFORMANCE REGRESSIONS IN OTHER PROGRAMS

512       Sometimes performance regressions happen in other programs (eg. qemu,
513       the kernel) that cause problems for libguestfs.
514
515       In https://github.com/libguestfs/libguestfs-analysis-tools
516       boot-benchmark/boot-benchmark-range.pl is a script which can be used to
517       benchmark libguestfs across a range of git commits in another project
518       to find out if any commit is causing a slowdown (or speedup).
519
520       To find out how to use this script, consult the manual:
521
522        ./boot-benchmark/boot-benchmark-range.pl --man
523

SEE ALSO

525       supermin(1), guestfish(1), guestfs(3), guestfs-examples(3),
526       guestfs-internals(1), libguestfs-make-fixed-appliance(1), stap(1),
527       qemu(1), gdb(1), http://libguestfs.org/.
528

AUTHORS

530       Richard W.M. Jones ("rjones at redhat dot com")
531
533       Copyright (C) 2012-2020 Red Hat Inc.
534

LICENSE

536       This library is free software; you can redistribute it and/or modify it
537       under the terms of the GNU Lesser General Public License as published
538       by the Free Software Foundation; either version 2 of the License, or
539       (at your option) any later version.
540
541       This library is distributed in the hope that it will be useful, but
542       WITHOUT ANY WARRANTY; without even the implied warranty of
543       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
544       Lesser General Public License for more details.
545
546       You should have received a copy of the GNU Lesser General Public
547       License along with this library; if not, write to the Free Software
548       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
549       02110-1301 USA
550

BUGS

552       To get a list of bugs against libguestfs, use this link:
553       https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
554
555       To report a new bug against libguestfs, use this link:
556       https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
557
558       When reporting a bug, please supply:
559
560       •   The version of libguestfs.
561
562       •   Where you got libguestfs (eg. which Linux distro, compiled from
563           source, etc)
564
565       •   Describe the bug accurately and give a way to reproduce it.
566
567       •   Run libguestfs-test-tool(1) and paste the complete, unedited output
568           into the bug report.
569
570
571
572libguestfs-1.45.4                 2021-04-03            guestfs-performance(1)
Impressum