1guestfs-performance(1)      Virtualization Support      guestfs-performance(1)
2
3
4

NAME

6       guestfs-performance - engineering libguestfs for greatest performance
7

DESCRIPTION

9       This page documents how to get the greatest performance out of
10       libguestfs, especially when you expect to use libguestfs to manipulate
11       thousands of virtual machines or disk images.
12
13       Three main areas are covered. Libguestfs runs an appliance (a small
14       Linux distribution) inside qemu/KVM.  The first two areas are:
15       minimizing the time taken to start this appliance, and the number of
16       times the appliance has to be started.  The third area is shortening
17       the time taken for inspection of VMs.
18

BASELINE MEASUREMENTS

20       Before making changes to how you use libguestfs, take baseline
21       measurements.
22
23   Baseline: Starting the appliance
24       On an unloaded machine, time how long it takes to start up the
25       appliance:
26
27        time guestfish -a /dev/null run
28
29       Run this command several times in a row and discard the first few runs,
30       so that you are measuring a typical "hot cache" case.
31
32       Side note for developers: If you are compiling libguestfs from source,
33       there is a program called utils/boot-benchmark/boot-benchmark which
34       does the same thing, but performs multiple runs and prints the mean and
35       standard deviation.  To run it, do:
36
37        make
38        ./run utils/boot-benchmark/boot-benchmark
39
40       There is a manual page utils/boot-benchmark/boot-benchmark.1
41
42       Explanation
43
44       The guestfish command above starts up the libguestfs appliance on a
45       null disk, and then immediately shuts it down.  The first time you run
46       the command, it will create an appliance and cache it (usually under
47       /var/tmp/.guestfs-*).  Subsequent runs should reuse the cached
48       appliance.
49
50       Expected results
51
52       You should expect to be getting times under 6 seconds.  If the times
53       you see on an unloaded machine are above this, then see the section
54       "TROUBLESHOOTING POOR PERFORMANCE" below.
55
56   Baseline: Performing inspection of a guest
57       For this test you will need an unloaded machine and at least one real
58       guest or disk image.  If you are planning to use libguestfs against
59       only X guests (eg. X = Windows), then using an X guest here would be
60       most appropriate.  If you are planning to run libguestfs against a mix
61       of guests, then use a mix of guests for testing here.
62
63       Time how long it takes to perform inspection and mount the disks of the
64       guest.  Use the first command if you will be using disk images, and the
65       second command if you will be using libvirt.
66
67        time guestfish --ro -a disk.img -i exit
68
69        time guestfish --ro -d GuestName -i exit
70
71       Run the command several times in a row and discard the first few runs,
72       so that you are measuring a typical "hot cache" case.
73
74       Explanation
75
76       This command starts up the libguestfs appliance on the named disk image
77       or libvirt guest, performs libguestfs inspection on it (see
78       "INSPECTION" in guestfs(3)), mounts the guest’s disks, then discards
79       all these results and shuts down.
80
81       The first time you run the command, it will create an appliance and
82       cache it (usually under /var/tmp/.guestfs-*).  Subsequent runs should
83       reuse the cached appliance.
84
85       Expected results
86
87       You should expect times which are ≤ 5 seconds greater than measured in
88       the first baseline test above.  (For example, if the first baseline
89       test ran in 5 seconds, then this test should run in ≤ 10 seconds).
90

UNDERSTANDING THE APPLIANCE AND WHEN IT IS BUILT/CACHED

92       The first time you use libguestfs, it will build and cache an
93       appliance.  This is usually in /var/tmp/.guestfs-*, unless you have set
94       $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that
95       temporary directory.
96
97       For more information about how the appliance is constructed, see
98       "SUPERMIN APPLIANCES" in supermin(1).
99
100       Every time libguestfs runs it will check that no host files used by the
101       appliance have changed.  If any have, then the appliance is rebuilt.
102       This usually happens when a package is installed or updated on the host
103       (eg. using programs like "yum" or "apt-get").  The reason for
104       reconstructing the appliance is security: the new program that has been
105       installed might contain a security fix, and so we want to include the
106       fixed program in the appliance automatically.
107
108       These are the performance implications:
109
110       ·   The process of building (or rebuilding) the cached appliance is
111           slow, and you can avoid this happening by using a fixed appliance
112           (see below).
113
114       ·   If not using a fixed appliance, be aware that updating software on
115           the host will cause a one time rebuild of the appliance.
116
117       ·   /var/tmp (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast
118           disk, and have plenty of space for the appliance.
119

USING A FIXED APPLIANCE

121       To fully control when the appliance is built, you can build a fixed
122       appliance.  This appliance should be stored on a fast local disk.
123
124       To build the appliance, run the command:
125
126        libguestfs-make-fixed-appliance <directory>
127
128       replacing "<directory>" with the name of a directory where the
129       appliance will be stored (normally you would name a subdirectory, for
130       example: /usr/local/lib/guestfs/appliance or /dev/shm/appliance).
131
132       Then set $LIBGUESTFS_PATH (and ensure this environment variable is set
133       in your libguestfs program), or modify your program so it calls
134       "guestfs_set_path".  For example:
135
136        export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance
137
138       Now you can run libguestfs programs, virt tools, guestfish etc. as
139       normal.  The programs will use your fixed appliance, and will not ever
140       build, rebuild, or cache their own appliance.
141
142       (For detailed information on this subject, see:
143       libguestfs-make-fixed-appliance(1)).
144
145   Performance of the fixed appliance
146       In our testing we did not find that using a fixed appliance gave any
147       measurable performance benefit, even when the appliance was located in
148       memory (ie. on /dev/shm).  However there are two points to consider:
149
150       1.  Using a fixed appliance stops libguestfs from ever rebuilding the
151           appliance, meaning that libguestfs will have more predictable
152           start-up times.
153
154       2.  The appliance is loaded on demand.  A simple test such as:
155
156            time guestfish -a /dev/null run
157
158           does not load very much of the appliance.  A real libguestfs
159           program using complicated API calls would demand-load a lot more of
160           the appliance.  Being able to store the appliance in a specified
161           location makes the performance more predictable.
162

REDUCING THE NUMBER OF TIMES THE APPLIANCE IS LAUNCHED

164       By far the most effective, though not always the simplest way to get
165       good performance is to ensure that the appliance is launched the
166       minimum number of times.  This will probably involve changing your
167       libguestfs application.
168
169       Try to call "guestfs_launch" at most once per target virtual machine or
170       disk image.
171
172       Instead of using a separate instance of guestfish(1) to make a series
173       of changes to the same guest, use a single instance of guestfish and/or
174       use the guestfish --listen option.
175
176       Consider writing your program as a daemon which holds a guest open
177       while making a series of changes.  Or marshal all the operations you
178       want to perform before opening the guest.
179
180       You can also try adding disks from multiple guests to a single
181       appliance.  Before trying this, note the following points:
182
183       1.  Adding multiple guests to one appliance is a security problem
184           because it may allow one guest to interfere with the disks of
185           another guest.  Only do it if you trust all the guests, or if you
186           can group guests by trust.
187
188       2.  There is a hard limit to the number of disks you can add to a
189           single appliance.  Call "guestfs_max_disks" in guestfs(3) to get
190           this limit.  For further information see "LIMITS" in guestfs(3).
191
192       3.  Using libguestfs this way is complicated.  Disks can have
193           unexpected interactions: for example, if two guests use the same
194           UUID for a filesystem (because they were cloned), or have volume
195           groups with the same name (but see "guestfs_lvm_set_filter").
196
197       virt-df(1) adds multiple disks by default, so the source code for this
198       program would be a good place to start.
199

SHORTENING THE TIME TAKEN FOR INSPECTION OF VMs

201       The main advice is obvious: Do not perform inspection (which is
202       expensive) unless you need the results.
203
204       If you previously performed inspection on the guest, then it may be
205       safe to cache and reuse the results from last time.
206
207       Some disks don’t need to be inspected at all: for example, if you are
208       creating a disk image, or if the disk image is not a VM, or if the disk
209       image has a known layout.
210
211       Even when basic inspection ("guestfs_inspect_os") is required,
212       auxiliary inspection operations may be avoided:
213
214       ·   Mounting disks is only necessary to get further filesystem
215           information.
216
217       ·   Listing applications ("guestfs_inspect_list_applications") is an
218           expensive operation on Linux, but almost free on Windows.
219
220       ·   Generating a guest icon ("guestfs_inspect_get_icon") is cheap on
221           Linux but expensive on Windows.
222

PARALLEL APPLIANCES

224       Libguestfs appliances are mostly I/O bound and you can launch multiple
225       appliances in parallel.  Provided there is enough free memory, there
226       should be little difference in launching 1 appliance vs N appliances in
227       parallel.
228
229       On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not
230       especially realistic) test Perl script below, the following plot shows
231       excellent scalability when running between 1 and 20 appliances in
232       parallel:
233
234         12 ++---+----+----+----+-----+----+----+----+----+---++
235            +    +    +    +    +     +    +    +    +    +    *
236            |                                                  |
237            |                                               *  |
238         11 ++                                                ++
239            |                                                  |
240            |                                                  |
241            |                                          *  *    |
242         10 ++                                                ++
243            |                                        *         |
244            |                                                  |
245        s   |                                                  |
246          9 ++                                                ++
247        e   |                                                  |
248            |                                     *            |
249        c   |                                                  |
250          8 ++                                  *             ++
251        o   |                                *                 |
252            |                                                  |
253        n 7 ++                                                ++
254            |                              *                   |
255        d   |                           *                      |
256            |                                                  |
257        s 6 ++                                                ++
258            |                      *  *                        |
259            |                   *                              |
260            |                                                  |
261          5 ++                                                ++
262            |                                                  |
263            |                 *                                |
264            |            * *                                   |
265          4 ++                                                ++
266            |                                                  |
267            |                                                  |
268            +    *  * *    +    +     +    +    +    +    +    +
269          3 ++-*-+----+----+----+-----+----+----+----+----+---++
270            0    2    4    6    8     10   12   14   16   18   20
271                      number of parallel appliances
272
273       It is possible to run many more than 20 appliances in parallel, but if
274       you are using the libvirt backend then you should be aware that out of
275       the box libvirt limits the number of client connections to 20.
276
277       The simple Perl script below was used to collect the data for the plot
278       above, but there is much more information on this subject, including
279       more advanced test scripts and graphs, available in the following blog
280       postings:
281
282       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/
283       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/
284       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/
285       http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/
286
287        #!/usr/bin/env perl
288
289        use strict;
290        use threads;
291        use warnings;
292        use Sys::Guestfs;
293        use Time::HiRes qw(time);
294
295        sub test {
296            my $g = Sys::Guestfs->new;
297            $g->add_drive_ro ("/dev/null");
298            $g->launch ();
299
300            # You could add some work for libguestfs to do here.
301
302            $g->close ();
303        }
304
305        # Get everything into cache.
306        test (); test (); test ();
307
308        for my $nr_threads (1..20) {
309            my $start_t = time ();
310            my @threads;
311            foreach (1..$nr_threads) {
312                push @threads, threads->create (\&test)
313            }
314            foreach (@threads) {
315                $_->join ();
316                if (my $err = $_->error ()) {
317                    die "launch failed with $nr_threads threads: $err"
318                }
319            }
320            my $end_t = time ();
321            printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
322        }
323

USING USER-MODE LINUX

325       Since libguestfs 1.24, it has been possible to use the User-Mode Linux
326       (uml) backend instead of KVM (see "USER-MODE LINUX BACKEND" in
327       guestfs(3)).  This section makes some general remarks about this
328       backend, but it is highly advisable to measure your own workload under
329       UML rather than trusting comments or intuition.
330
331       ·   UML usually performs the same or slightly slower than KVM, on
332           baremetal.
333
334       ·   However UML often performs the same under virtualization as it does
335           on baremetal, whereas KVM can run much slower under virtualization
336           (since hardware virt acceleration is not available).
337
338       ·   Upload and download is as much as 10 times slower on UML than KVM.
339           Libguestfs sends this data over the UML emulated serial port, which
340           is far less efficient than KVM’s virtio-serial.
341
342       ·   UML lacks some features (eg. qcow2 support), so it may not be
343           applicable at all.
344
345       For some actual figures, see:
346       http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content
347

TROUBLESHOOTING POOR PERFORMANCE

349   Ensure hardware virtualization is available
350       Use /proc/cpuinfo to ensure that hardware virtualization is available.
351       Note that you may need to enable it in your BIOS.
352
353       Hardware virt is not usually available inside VMs, and libguestfs will
354       run slowly inside another virtual machine whatever you do.  Nested
355       virtualization does not work well in our experience, and is certainly
356       no substitute for running libguestfs on baremetal.
357
358   Ensure KVM is available
359       Ensure that KVM is enabled and available to the user that will run
360       libguestfs.  It should be safe to set 0666 permissions on /dev/kvm and
361       most distributions now do this.
362
363   Processors to avoid
364       Avoid processors that don’t have hardware virtualization, and some
365       processors which are simply very slow (AMD Geode being a great
366       example).
367
368   Xen dom0
369       In Xen, dom0 is a virtual machine, and so hardware virtualization is
370       not available.
371
372   Use libguestfs ≥ 1.34 and qemu ≥ 2.7
373       During the libguestfs 1.33 development cycle, we spent a large amount
374       of time concentrating on boot performance, and added some patches to
375       libguestfs, qemu and Linux which in some cases can reduce boot times to
376       well under 1 second.  You may therefore get much better performance by
377       moving to the versions of libguestfs or qemu mentioned in the heading.
378

DETAILED ANALYSIS

380   Boot analysis
381       In the libguestfs source directory, in utils/boot-analysis is a program
382       called "boot-analysis".  This program is able to produce a very
383       detailed breakdown of the boot steps (eg. qemu, BIOS, kernel,
384       libguestfs init script), and can measure how long it takes to perform
385       each step.
386
387       To run this program, do:
388
389        make
390        ./run utils/boot-analysis/boot-analysis
391
392       There is a manual page utils/boot-benchmark/boot-analysis.1
393
394   Detailed timings using ts
395       Use the ts(1) command (from moreutils) to show detailed timings:
396
397        $ guestfish -a /dev/null run -v |& ts -i '%.s'
398        0.000022 libguestfs: launch: program=guestfish
399        0.000134 libguestfs: launch: version=1.29.31fedora=23,release=2.fc23,libvirt
400        0.000044 libguestfs: launch: backend registered: unix
401        0.000035 libguestfs: launch: backend registered: uml
402        0.000035 libguestfs: launch: backend registered: libvirt
403        0.000032 libguestfs: launch: backend registered: direct
404        0.000030 libguestfs: launch: backend=libvirt
405        0.000031 libguestfs: launch: tmpdir=/tmp/libguestfsw18rBQ
406        0.000029 libguestfs: launch: umask=0002
407        0.000031 libguestfs: launch: euid=1000
408        0.000030 libguestfs: libvirt version = 1002012 (1.2.12)
409        [etc]
410
411       The timestamps are seconds (incrementally since the previous line).
412
413   Detailed timings using SystemTap
414       You can use SystemTap (stap(1)) to get detailed timings from libguestfs
415       programs.
416
417       Save the following script as time.stap:
418
419        global last;
420
421        function display_time () {
422              now = gettimeofday_us ();
423              delta = 0;
424              if (last > 0)
425                    delta = now - last;
426              last = now;
427
428              printf ("%d (+%d):", now, delta);
429        }
430
431        probe begin {
432              last = 0;
433              printf ("ready\n");
434        }
435
436        /* Display all calls to static markers. */
437        probe process("/usr/lib*/libguestfs.so.0")
438                  .provider("guestfs").mark("*") ? {
439              display_time();
440              printf ("\t%s %s\n", $$name, $$parms);
441        }
442
443        /* Display all calls to guestfs_* functions. */
444        probe process("/usr/lib*/libguestfs.so.0")
445                  .function("guestfs_[a-z]*") ? {
446              display_time();
447              printf ("\t%s %s\n", probefunc(), $$parms);
448        }
449
450       Run it as root in one window:
451
452        # stap time.stap
453        ready
454
455       It prints "ready" when SystemTap has loaded the program.  Run your
456       libguestfs program, guestfish or a virt tool in another window.  For
457       example:
458
459        $ guestfish -a /dev/null run
460
461       In the stap window you will see a large amount of output, with the time
462       taken for each step shown (microseconds in parenthesis).  For example:
463
464        xxxx (+0):     guestfs_create
465        xxxx (+29):    guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
466        xxxx (+9):     guestfs_add_drive_opts_argv g=0x17a9de0 [...]
467        xxxx (+8):     guestfs_int_safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
468        xxxx (+19):    guestfs_int_safe_malloc g=0x17a9de0 nbytes=0x38
469        xxxx (+5):     guestfs_int_safe_strdup g=0x17a9de0 str=0x17a9f60
470        xxxx (+10):    guestfs_launch g=0x17a9de0
471        xxxx (+4):     launch_start
472        [etc]
473
474       You will need to consult, and even modify, the source to libguestfs to
475       fully understand the output.
476
477   Detailed debugging using gdb
478       You can attach to the appliance BIOS/kernel using gdb.  If you know
479       what you're doing, this can be a useful way to diagnose boot
480       regressions.
481
482       Firstly, you have to change qemu so it runs with the "-S" and "-s"
483       options.  These options cause qemu to pause at boot and allow you to
484       attach a debugger.  Read qemu(1) for further information.  Libguestfs
485       invokes qemu several times (to scan the help output and so on) and you
486       only want the final invocation of qemu to use these options, so use a
487       qemu wrapper script like this:
488
489        #!/bin/bash -
490
491        # Set this to point to the real qemu binary.
492        qemu=/usr/bin/qemu-kvm
493
494        if [ "$1" != "-global" ]; then
495            # Scanning help output etc.
496            exec $qemu "$@"
497        else
498            # Really running qemu.
499            exec $qemu -S -s "$@"
500        fi
501
502       Now run guestfish or another libguestfs tool with the qemu wrapper (see
503       "QEMU WRAPPERS" in guestfs(3) to understand what this is doing):
504
505        LIBGUESTFS_HV=/path/to/qemu-wrapper guestfish -a /dev/null -v run
506
507       This should pause just after qemu launches.  In another window, attach
508       to qemu using gdb:
509
510        $ gdb
511        (gdb) set architecture i8086
512        The target architecture is assumed to be i8086
513        (gdb) target remote :1234
514        Remote debugging using :1234
515        0x0000fff0 in ?? ()
516        (gdb) cont
517
518       At this point you can use standard gdb techniques, eg. hitting "^C" to
519       interrupt the boot and "bt" get a stack trace, setting breakpoints,
520       etc.  Note that when you are past the BIOS and into the Linux kernel,
521       you'll want to change the architecture back to 32 or 64 bit.
522

PERFORMANCE REGRESSIONS IN OTHER PROGRAMS

524       Sometimes performance regressions happen in other programs (eg. qemu,
525       the kernel) that cause problems for libguestfs.
526
527       In the libguestfs source, utils/boot-benchmark/boot-benchmark-range.pl
528       is a script which can be used to benchmark libguestfs across a range of
529       git commits in another project to find out if any commit is causing a
530       slowdown (or speedup).
531
532       To find out how to use this script, consult the manual:
533
534        ./utils/boot-benchmark/boot-benchmark-range.pl --man
535

SEE ALSO

537       supermin(1), guestfish(1), guestfs(3), guestfs-examples(3),
538       guestfs-internals(1), libguestfs-make-fixed-appliance(1), stap(1),
539       qemu(1), gdb(1), http://libguestfs.org/.
540

AUTHORS

542       Richard W.M. Jones ("rjones at redhat dot com")
543
545       Copyright (C) 2012-2019 Red Hat Inc.
546

LICENSE

548       This library is free software; you can redistribute it and/or modify it
549       under the terms of the GNU Lesser General Public License as published
550       by the Free Software Foundation; either version 2 of the License, or
551       (at your option) any later version.
552
553       This library is distributed in the hope that it will be useful, but
554       WITHOUT ANY WARRANTY; without even the implied warranty of
555       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
556       Lesser General Public License for more details.
557
558       You should have received a copy of the GNU Lesser General Public
559       License along with this library; if not, write to the Free Software
560       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
561       02110-1301 USA
562

BUGS

564       To get a list of bugs against libguestfs, use this link:
565       https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
566
567       To report a new bug against libguestfs, use this link:
568       https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
569
570       When reporting a bug, please supply:
571
572       ·   The version of libguestfs.
573
574       ·   Where you got libguestfs (eg. which Linux distro, compiled from
575           source, etc)
576
577       ·   Describe the bug accurately and give a way to reproduce it.
578
579       ·   Run libguestfs-test-tool(1) and paste the complete, unedited output
580           into the bug report.
581
582
583
584libguestfs-1.40.2                 2019-02-07            guestfs-performance(1)
Impressum