1guestfs-performance(1) Virtualization Support guestfs-performance(1)
2
6 guestfs-performance - engineering libguestfs for greatest performance
7
9 This page documents how to get the greatest performance out of
10 libguestfs, especially when you expect to use libguestfs to manipulate
11 thousands of virtual machines or disk images.
12 Three main areas are covered. Libguestfs runs an appliance (a small
14 Linux distribution) inside qemu/KVM. The first two areas are:
15 minimizing the time taken to start this appliance, and the number of
16 times the appliance has to be started. The third area is shortening
17 the time taken for inspection of VMs.
18
20 Before making changes to how you use libguestfs, take baseline
21 measurements.
22 BASELINE: STARTING THE APPLIANCE
24 On an unloaded machine, time how long it takes to start up the
25 appliance:
26 time guestfish -a /dev/null run
28 Run this command several times in a row and discard the first few runs,
30 so that you are measuring a typical "hot cache" case.
31 Explanation
33 This command starts up the libguestfs appliance on a null disk, and
35 then immediately shuts it down. The first time you run the command, it
36 will create an appliance and cache it (usually under
37 "/var/tmp/.guestfs-*"). Subsequent runs should reuse the cached
38 appliance.
39 Expected results
41 You should expect to be getting times under 6 seconds. If the times
43 you see on an unloaded machine are above this, then see the section
44 "TROUBLESHOOTING POOR PERFORMANCE" below.
45 BASELINE: PERFORMING INSPECTION OF A GUEST
47 For this test you will need an unloaded machine and at least one real
48 guest or disk image. If you are planning to use libguestfs against
49 only X guests (eg. X = Windows), then using an X guest here would be
50 most appropriate. If you are planning to run libguestfs against a mix
51 of guests, then use a mix of guests for testing here.
52 Time how long it takes to perform inspection and mount the disks of the
54 guest. Use the first command if you will be using disk images, and the
55 second command if you will be using libvirt.
56 time guestfish --ro -a disk.img -i exit
58 time guestfish --ro -d GuestName -i exit
60 Run the command several times in a row and discard the first few runs,
62 so that you are measuring a typical "hot cache" case.
63 Explanation
65 This command starts up the libguestfs appliance on the named disk image
67 or libvirt guest, performs libguestfs inspection on it (see
68 "INSPECTION" in guestfs(3)), mounts the guest's disks, then discards
69 all these results and shuts down.
70 The first time you run the command, it will create an appliance and
72 cache it (usually under "/var/tmp/.guestfs-*"). Subsequent runs should
73 reuse the cached appliance.
74 Expected results
76 You should expect times which are ≤ 5 seconds greater than measured in
78 the first baseline test above. (For example, if the first baseline
79 test ran in 5 seconds, then this test should run in ≤ 10 seconds).
80
82 The first time you use libguestfs, it will build and cache an
83 appliance. This is usually in "/var/tmp/.guestfs-*", unless you have
84 set $TMPDIR or $LIBGUESTFS_CACHEDIR in which case it will be under that
85 temporary directory.
86 For more information about how the appliance is constructed, see
88 "SUPERMIN APPLIANCES" in supermin(8).
89 Every time libguestfs runs it will check that no host files used by the
91 appliance have changed. If any have, then the appliance is rebuilt.
92 This usually happens when a package is installed or updated on the host
93 (eg. using programs like "yum" or "apt-get"). The reason for
94 reconstructing the appliance is security: the new program that has been
95 installed might contain a security fix, and so we want to include the
96 fixed program in the appliance automatically.
97 These are the performance implications:
99 · The process of building (or rebuilding) the cached appliance is
101 slow, and you can avoid this happening by using a fixed appliance
102 (see below).
103 · If not using a fixed appliance, be aware that updating software on
105 the host will cause a one time rebuild of the appliance.
106 · "/var/tmp" (or $TMPDIR, $LIBGUESTFS_CACHEDIR) should be on a fast
108 disk, and have plenty of space for the appliance.
109
111 To fully control when the appliance is built, you can build a fixed
112 appliance. This appliance should be stored on a fast local disk.
113 To build the appliance, run the command:
115 libguestfs-make-fixed-appliance <directory>
117 replacing "<directory>" with the name of a directory where the
119 appliance will be stored (normally you would name a subdirectory, for
120 example: "/usr/local/lib/guestfs/appliance" or "/dev/shm/appliance").
121 Then set $LIBGUESTFS_PATH (and ensure this environment variable is set
123 in your libguestfs program), or modify your program so it calls
124 "guestfs_set_path". For example:
125 export LIBGUESTFS_PATH=/usr/local/lib/guestfs/appliance
127 Now you can run libguestfs programs, virt tools, guestfish etc. as
129 normal. The programs will use your fixed appliance, and will not ever
130 build, rebuild, or cache their own appliance.
131 (For detailed information on this subject, see:
133 libguestfs-make-fixed-appliance(1)).
134 PERFORMANCE OF THE FIXED APPLIANCE
136 In our testing we did not find that using a fixed appliance gave any
137 measurable performance benefit, even when the appliance was located in
138 memory (ie. on "/dev/shm"). However there are three points to
139 consider:
140 1. Using a fixed appliance stops libguestfs from ever rebuilding the
142 appliance, meaning that libguestfs will have more predictable
143 start-up times.
144 2. By default libguestfs (or rather, supermin-helper(8)) searches over
146 the root filesystem to find out if any host files have changed and
147 if it needs to rebuild the appliance. If these files are not
148 cached and the root filesystem is on an HDD, then this generates
149 lots of seeks. Using a fixed appliance avoids this.
150 3. The appliance is loaded on demand. A simple test such as:
152 time guestfish -a /dev/null run
154 does not load very much of the appliance. A real libguestfs
156 program using complicated API calls would demand-load a lot more of
157 the appliance. Being able to store the appliance in a specified
158 location makes the performance more predictable.
159
161 By far the most effective, though not always the simplest way to get
162 good performance is to ensure that the appliance is launched the
163 minimum number of times. This will probably involve changing your
164 libguestfs application.
165 Try to call "guestfs_launch" at most once per target virtual machine or
167 disk image.
168 Instead of using a separate instance of guestfish(1) to make a series
170 of changes to the same guest, use a single instance of guestfish and/or
171 use the guestfish --listen option.
172 Consider writing your program as a daemon which holds a guest open
174 while making a series of changes. Or marshal all the operations you
175 want to perform before opening the guest.
176 You can also try adding disks from multiple guests to a single
178 appliance. Before trying this, note the following points:
179 1. Adding multiple guests to one appliance is a security problem
181 because it may allow one guest to interfere with the disks of
182 another guest. Only do it if you trust all the guests, or if you
183 can group guests by trust.
184 2. There is a hard limit to the number of disks you can add to a
186 single appliance. Call "guestfs_max_disks" in guestfs(3) to get
187 this limit. For further information see "LIMITS" in guestfs(3).
188 3. Using libguestfs this way is complicated. Disks can have
190 unexpected interactions: for example, if two guests use the same
191 UUID for a filesystem (because they were cloned), or have volume
192 groups with the same name (but see "guestfs_lvm_set_filter").
193 virt-df(1) adds multiple disks by default, so the source code for this
195 program would be a good place to start.
196
198 The main advice is obvious: Do not perform inspection (which is
199 expensive) unless you need the results.
200 If you previously performed inspection on the guest, then it may be
202 safe to cache and reuse the results from last time.
203 Some disks don't need to be inspected at all: for example, if you are
205 creating a disk image, or if the disk image is not a VM, or if the disk
206 image has a known layout.
207 Even when basic inspection ("guestfs_inspect_os") is required,
209 auxiliary inspection operations may be avoided:
210 · Mounting disks is only necessary to get further filesystem
212 information.
213 · Listing applications ("guestfs_inspect_list_applications") is an
215 expensive operation on Linux, but almost free on Windows.
216 · Generating a guest icon ("guestfs_inspect_get_icon") is cheap on
218 Linux but expensive on Windows.
219
221 Libguestfs appliances are mostly I/O bound and you can launch multiple
222 appliances in parallel. Provided there is enough free memory, there
223 should be little difference in launching 1 appliance vs N appliances in
224 parallel.
225 On a 2-core (4-thread) laptop with 16 GB of RAM, using the (not
227 especially realistic) test Perl script below, the following plot shows
228 excellent scalability when running between 1 and 20 appliances in
229 parallel:
230 12 ++---+----+----+----+-----+----+----+----+----+---++
232 + + + + + + + + + + *
233 | |
234 | * |
235 11 ++ ++
236 | |
237 | |
238 | * * |
239 10 ++ ++
240 | * |
241 | |
242 s | |
243 9 ++ ++
244 e | |
245 | * |
246 c | |
247 8 ++ * ++
248 o | * |
249 | |
250 n 7 ++ ++
251 | * |
252 d | * |
253 | |
254 s 6 ++ ++
255 | * * |
256 | * |
257 | |
258 5 ++ ++
259 | |
260 | * |
261 | * * |
262 4 ++ ++
263 | |
264 | |
265 + * * * + + + + + + + +
266 3 ++-*-+----+----+----+-----+----+----+----+----+---++
267 0 2 4 6 8 10 12 14 16 18 20
268 number of parallel appliances
269 It is possible to run many more than 20 appliances in parallel, but if
271 you are using the libvirt backend then you should be aware that out of
272 the box libvirt limits the number of client connections to 20.
273 The simple Perl script below was used to collect the data for the plot
275 above, but there is much more information on this subject, including
276 more advanced test scripts and graphs, available in the following blog
277 postings:
278 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-1/
280 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-2/
281 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-3/
282 http://rwmj.wordpress.com/2013/02/25/multiple-libguestfs-appliances-in-parallel-part-4/
283 #!/usr/bin/perl -w
285 use strict;
287 use threads;
288 use Sys::Guestfs;
289 use Time::HiRes qw(time);
290 sub test {
292 my $g = Sys::Guestfs->new;
293 $g->add_drive_ro ("/dev/null");
294 $g->launch ();
295 # You could add some work for libguestfs to do here.
297 $g->close ();
299 }
300 # Get everything into cache.
302 test (); test (); test ();
303 for my $nr_threads (1..20) {
305 my $start_t = time ();
306 my @threads;
307 foreach (1..$nr_threads) {
308 push @threads, threads->create (\&test)
309 }
310 foreach (@threads) {
311 $_->join ();
312 if (my $err = $_->error ()) {
313 die "launch failed with $nr_threads threads: $err"
314 }
315 }
316 my $end_t = time ();
317 printf ("%d %.2f\n", $nr_threads, $end_t - $start_t);
318 }
319
321 ENSURE HARDWARE VIRTUALIZATION IS AVAILABLE
322 Use "/proc/cpuinfo" and this page:
323 http://virt-tools.org/learning/check-hardware-virt/
325 to ensure that hardware virtualization is available. Note that you may
327 need to enable it in your BIOS.
328 Hardware virt is not usually available inside VMs, and libguestfs will
330 run slowly inside another virtual machine whatever you do. Nested
331 virtualization does not work well in our experience, and is certainly
332 no substitute for running libguestfs on baremetal.
333 ENSURE KVM IS AVAILABLE
335 Ensure that KVM is enabled and available to the user that will run
336 libguestfs. It should be safe to set 0666 permissions on "/dev/kvm"
337 and most distributions now do this.
338 PROCESSORS TO AVOID
340 Avoid processors that don't have hardware virtualization, and some
341 processors which are simply very slow (AMD Geode being a great
342 example).
343
345 Use the annotate(1)/annotate-output(1) command to show detailed
346 timings:
347 $ annotate-output +'%T.%N' guestfish -a /dev/null run -v
349 22:17:53.215784625 I: Started guestfish -a /dev/null run -v
350 22:17:53.240335409 E: libguestfs: [00000ms] supermin-helper --verbose -f checksum '/usr/lib64/guestfs/supermin.d' x86_64
351 22:17:53.266857866 E: supermin helper [00000ms] whitelist = (not specified), host_cpu = x86_64, kernel = (null), initrd = (null), appliance = (null)
352 22:17:53.272704072 E: supermin helper [00000ms] inputs[0] = /usr/lib64/guestfs/supermin.d
353 22:17:53.276528651 E: checking modpath /lib/modules/3.4.0-1.fc17.x86_64.debug is a directory
354 [etc]
355 The timestamps are "hours:minutes:seconds.nanoseconds". By comparing
357 the timestamps you can see exactly how long each operation in the boot
358 sequence takes.
359
361 You can use SystemTap (stap(1)) to get detailed timings from libguestfs
362 programs.
363 Save the following script as "time.stap":
365 global last;
367 function display_time () {
369 now = gettimeofday_us ();
370 delta = 0;
371 if (last > 0)
372 delta = now - last;
373 last = now;
374 printf ("%d (+%d):", now, delta);
376 }
377 probe begin {
379 last = 0;
380 printf ("ready\n");
381 }
382 /* Display all calls to static markers. */
384 probe process("/usr/lib*/libguestfs.so.0")
385 .provider("guestfs").mark("*") ? {
386 display_time();
387 printf ("\t%s %s\n", $$name, $$parms);
388 }
389 /* Display all calls to guestfs_* functions. */
391 probe process("/usr/lib*/libguestfs.so.0")
392 .function("guestfs_[a-z]*") ? {
393 display_time();
394 printf ("\t%s %s\n", probefunc(), $$parms);
395 }
396 Run it as root in one window:
398 # stap time.stap
400 ready
401 It prints "ready" when SystemTap has loaded the program. Run your
403 libguestfs program, guestfish or a virt tool in another window. For
404 example:
405 $ guestfish -a /dev/null run
407 In the stap window you will see a large amount of output, with the time
409 taken for each step shown (microseconds in parenthesis). For example:
410 xxxx (+0): guestfs_create
412 xxxx (+29): guestfs_set_pgroup g=0x17a9de0 pgroup=0x1
413 xxxx (+9): guestfs_add_drive_opts_argv g=0x17a9de0 [...]
414 xxxx (+8): guestfs___safe_strdup g=0x17a9de0 str=0x7f8a153bed5d
415 xxxx (+19): guestfs___safe_malloc g=0x17a9de0 nbytes=0x38
416 xxxx (+5): guestfs___safe_strdup g=0x17a9de0 str=0x17a9f60
417 xxxx (+10): guestfs_launch g=0x17a9de0
418 xxxx (+4): launch_start
419 [etc]
420 You will need to consult, and even modify, the source to libguestfs to
422 fully understand the output.
423
425 You can attach to the appliance BIOS/kernel using gdb. If you know
426 what you're doing, this can be a useful way to diagnose boot
427 regressions.
428 Firstly, you have to change qemu so it runs with the "-S" and "-s"
430 options. These options cause qemu to pause at boot and allow you to
431 attach a debugger. Read qemu(1) for further information. Libguestfs
432 invokes qemu several times (to scan the help output and so on) and you
433 only want the final invocation of qemu to use these options, so use a
434 qemu wrapper script like this:
435 #!/bin/bash -
437 # Set this to point to the real qemu binary.
439 qemu=/usr/bin/qemu-kvm
440 if [ "$1" != "-global" ]; then
442 # Scanning help output etc.
443 exec $qemu "$@"
444 else
445 # Really running qemu.
446 exec $qemu -S -s "$@"
447 fi
448 Now run guestfish or another libguestfs tool with the qemu wrapper (see
450 "QEMU WRAPPERS" in guestfs(3) to understand what this is doing):
451 LIBGUESTFS_QEMU=/path/to/qemu-wrapper guestfish -a /dev/null -v run
453 This should pause just after qemu launches. In another window, attach
455 to qemu using gdb:
456 $ gdb
458 (gdb) set architecture i8086
459 The target architecture is assumed to be i8086
460 (gdb) target remote :1234
461 Remote debugging using :1234
462 0x0000fff0 in ?? ()
463 (gdb) cont
464 At this point you can use standard gdb techniques, eg. hitting "^C" to
466 interrupt the boot and "bt" get a stack trace, setting breakpoints,
467 etc. Note that when you are past the BIOS and into the Linux kernel,
468 you'll want to change the architecture back to 32 or 64 bit.
469
471 supermin(8), supermin-helper(8), guestfish(1), guestfs(3),
472 guestfs-examples(3), libguestfs-make-fixed-appliance(1), stap(1),
473 qemu(1), gdb(1), http://libguestfs.org/.
474
476 Richard W.M. Jones ("rjones at redhat dot com")
477
479 Copyright (C) 2012 Red Hat Inc.
480
482 This library is free software; you can redistribute it and/or modify it
483 under the terms of the GNU Lesser General Public License as published
484 by the Free Software Foundation; either version 2 of the License, or
485 (at your option) any later version.
486 This library is distributed in the hope that it will be useful, but
488 WITHOUT ANY WARRANTY; without even the implied warranty of
489 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
490 Lesser General Public License for more details.
491 You should have received a copy of the GNU Lesser General Public
493 License along with this library; if not, write to the Free Software
494 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
495 02110-1301 USA
496
498 To get a list of bugs against libguestfs, use this link:
499 https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
500 To report a new bug against libguestfs, use this link:
502 https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
503 When reporting a bug, please supply:
505 · The version of libguestfs.
507 · Where you got libguestfs (eg. which Linux distro, compiled from
509 source, etc)
510 · Describe the bug accurately and give a way to reproduce it.
512 · Run libguestfs-test-tool(1) and paste the complete, unedited output
514 into the bug report.
515libguestfs-1.20.11 2013-08-27 guestfs-performance(1)