1guestfs-internals(1) Virtualization Support guestfs-internals(1)
2
6 guestfs-internals - architecture and internals of libguestfs
7
9 This manual page is for hackers who want to understand how libguestfs
10 works internally. This is just a description of how libguestfs works
11 now, and it may change at any time in the future.
12
14 Internally, libguestfs is implemented by running an appliance (a
15 special type of small virtual machine) using qemu(1). Qemu runs as a
16 child process of the main program.
17 ┌───────────────────┐
19 │ main program │
20 │ │
21 │ │ child process / appliance
22 │ │ ┌──────────────────────────┐
23 │ │ │ qemu │
24 ├───────────────────┤ RPC │ ┌─────────────────┐ │
25 │ libguestfs ◀╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍▶ guestfsd │ │
26 │ │ │ ├─────────────────┤ │
27 └───────────────────┘ │ │ Linux kernel │ │
28 │ └────────┬────────┘ │
29 └───────────────│──────────┘
30 │
31 │ virtio-scsi
32 ┌──────┴──────┐
33 │ Device or │
34 │ disk image │
35 └─────────────┘
36 The library, linked to the main program, creates the child process and
38 hence the appliance in the "guestfs_launch" in guestfs(3) function.
39 Inside the appliance is a Linux kernel and a complete stack of
41 userspace tools (such as LVM and ext2 programs) and a small controlling
42 daemon called "guestfsd". The library talks to "guestfsd" using remote
43 procedure calls (RPC). There is a mostly one-to-one correspondence
44 between libguestfs API calls and RPC calls to the daemon. Lastly the
45 disk image(s) are attached to the qemu process which translates device
46 access by the appliance’s Linux kernel into accesses to the image.
47 A common misunderstanding is that the appliance "is" the virtual
49 machine. Although the disk image you are attached to might also be
50 used by some virtual machine, libguestfs doesn't know or care about
51 this. (But you will care if both libguestfs’s qemu process and your
52 virtual machine are trying to update the disk image at the same time,
53 since these usually results in massive disk corruption).
54
56 libguestfs uses a state machine to model the child process:
57 |
59 guestfs_create / guestfs_create_flags
60 |
61 |
62 ____V_____
63 / \
64 | CONFIG |
65 \__________/
66 ^ ^ \
67 | \ \ guestfs_launch
68 | _\__V______
69 | / \
70 | | LAUNCHING |
71 | \___________/
72 | /
73 | guestfs_launch
74 | /
75 __|____V
76 / \
77 | READY |
78 \________/
79 The normal transitions are (1) CONFIG (when the handle is created, but
81 there is no child process), (2) LAUNCHING (when the child process is
82 booting up), (3) READY meaning the appliance is up, actions can be
83 issued to, and carried out by, the child process.
84 The guest may be killed by "guestfs_kill_subprocess" in guestfs(3), or
86 may die asynchronously at any time (eg. due to some internal error),
87 and that causes the state to transition back to CONFIG.
88 Configuration commands for qemu such as "guestfs_set_path" in
90 guestfs(3) can only be issued when in the CONFIG state.
91 The API offers one call that goes from CONFIG through LAUNCHING to
93 READY. "guestfs_launch" in guestfs(3) blocks until the child process
94 is READY to accept commands (or until some failure or timeout).
95 "guestfs_launch" in guestfs(3) internally moves the state from CONFIG
96 to LAUNCHING while it is running.
97 API actions such as "guestfs_mount" in guestfs(3) can only be issued
99 when in the READY state. These API calls block waiting for the command
100 to be carried out. There are no non-blocking versions, and no way to
101 issue more than one command per handle at the same time.
102 Finally, the child process sends asynchronous messages back to the main
104 program, such as kernel log messages. You can register a callback to
105 receive these messages.
106
108 APPLIANCE BOOT PROCESS
109 This process has evolved and continues to evolve. The description here
110 corresponds only to the current version of libguestfs and is provided
111 for information only.
112 In order to follow the stages involved below, enable libguestfs
114 debugging (set the environment variable "LIBGUESTFS_DEBUG=1").
115 Create the appliance
117 "supermin --build" is invoked to create the kernel, a small initrd
118 and the appliance.
119 The appliance is cached in /var/tmp/.guestfs-<UID> (or in another
121 directory if "LIBGUESTFS_CACHEDIR" or "TMPDIR" are set).
122 For a complete description of how the appliance is created and
124 cached, read the supermin(1) man page.
125 Start qemu and boot the kernel
127 qemu is invoked to boot the kernel.
128 Run the initrd
130 "supermin --build" builds a small initrd. The initrd is not the
131 appliance. The purpose of the initrd is to load enough kernel
132 modules in order that the appliance itself can be mounted and
133 started.
134 The initrd is a cpio archive called
136 /var/tmp/.guestfs-<UID>/appliance.d/initrd.
137 When the initrd has started you will see messages showing that
139 kernel modules are being loaded, similar to this:
140 supermin: ext2 mini initrd starting up
142 supermin: mounting /sys
143 supermin: internal insmod libcrc32c.ko
144 supermin: internal insmod crc32c-intel.ko
145 Find and mount the appliance device
147 The appliance is a sparse file containing an ext2 filesystem which
148 contains a familiar (although reduced in size) Linux operating
149 system. It would normally be called
150 /var/tmp/.guestfs-<UID>/appliance.d/root.
151 The regular disks being inspected by libguestfs are the first
153 devices exposed by qemu (eg. as /dev/vda).
154 The last disk added to qemu is the appliance itself (eg. /dev/vdb
156 if there was only one regular disk).
157 Thus the final job of the initrd is to locate the appliance disk,
159 mount it, and switch root into the appliance, and run /init from
160 the appliance.
161 If this works successfully you will see messages such as:
163 supermin: picked /sys/block/vdb/dev as root device
165 supermin: creating /dev/root as block special 252:16
166 supermin: mounting new root on /root
167 supermin: chroot
168 Starting /init script ...
169 Note that "Starting /init script ..." indicates that the
171 appliance's init script is now running.
172 Initialize the appliance
174 The appliance itself now initializes itself. This involves
175 starting certain processes like "udev", possibly printing some
176 debug information, and finally running the daemon ("guestfsd").
177 The daemon
179 Finally the daemon ("guestfsd") runs inside the appliance. If it
180 runs you should see:
181 verbose daemon enabled
183 The daemon expects to see a named virtio-serial port exposed by
185 qemu and connected on the other end to the library.
186 The daemon connects to this port (and hence to the library) and
188 sends a four byte message "GUESTFS_LAUNCH_FLAG", which initiates
189 the communication protocol (see below).
190 COMMUNICATION PROTOCOL
192 Don’t rely on using this protocol directly. This section documents how
193 it currently works, but it may change at any time.
194 The protocol used to talk between the library and the daemon running
196 inside the qemu virtual machine is a simple RPC mechanism built on top
197 of XDR (RFC 1014, RFC 1832, RFC 4506).
198 The detailed format of structures is in
200 common/protocol/guestfs_protocol.x (note: this file is automatically
201 generated).
202 There are two broad cases, ordinary functions that don’t have any
204 "FileIn" and "FileOut" parameters, which are handled with very simple
205 request/reply messages. Then there are functions that have any
206 "FileIn" or "FileOut" parameters, which use the same request and reply
207 messages, but they may also be followed by files sent using a chunked
208 encoding.
209 ORDINARY FUNCTIONS (NO FILEIN/FILEOUT PARAMS)
211 For ordinary functions, the request message is:
213 total length (header + arguments,
215 but not including the length word itself)
216 struct guestfs_message_header (encoded as XDR)
217 struct guestfs_<foo>_args (encoded as XDR)
218 The total length field allows the daemon to allocate a fixed size
220 buffer into which it slurps the rest of the message. As a result, the
221 total length is limited to "GUESTFS_MESSAGE_MAX" bytes (currently 4MB),
222 which means the effective size of any request is limited to somewhere
223 under this size.
224 Note also that many functions don’t take any arguments, in which case
226 the "guestfs_foo_args" is completely omitted.
227 The header contains the procedure number ("guestfs_proc") which is how
229 the receiver knows what type of args structure to expect, or none at
230 all.
231 For functions that take optional arguments, the optional arguments are
233 encoded in the "guestfs_foo_args" structure in the same way as ordinary
234 arguments. A bitmask in the header indicates which optional arguments
235 are meaningful. The bitmask is also checked to see if it contains bits
236 set which the daemon does not know about (eg. if more optional
237 arguments were added in a later version of the library), and this
238 causes the call to be rejected.
239 The reply message for ordinary functions is:
241 total length (header + ret,
243 but not including the length word itself)
244 struct guestfs_message_header (encoded as XDR)
245 struct guestfs_<foo>_ret (encoded as XDR)
246 As above the "guestfs_foo_ret" structure may be completely omitted for
248 functions that return no formal return values.
249 As above the total length of the reply is limited to
251 "GUESTFS_MESSAGE_MAX".
252 In the case of an error, a flag is set in the header, and the reply
254 message is slightly changed:
255 total length (header + error,
257 but not including the length word itself)
258 struct guestfs_message_header (encoded as XDR)
259 struct guestfs_message_error (encoded as XDR)
260 The "guestfs_message_error" structure contains the error message as a
262 string.
263 FUNCTIONS THAT HAVE FILEIN PARAMETERS
265 A "FileIn" parameter indicates that we transfer a file into the guest.
267 The normal request message is sent (see above). However this is
268 followed by a sequence of file chunks.
269 total length (header + arguments,
271 but not including the length word itself,
272 and not including the chunks)
273 struct guestfs_message_header (encoded as XDR)
274 struct guestfs_<foo>_args (encoded as XDR)
275 sequence of chunks for FileIn param #0
276 sequence of chunks for FileIn param #1 etc.
277 The "sequence of chunks" is:
279 length of chunk (not including length word itself)
281 struct guestfs_chunk (encoded as XDR)
282 length of chunk
283 struct guestfs_chunk (encoded as XDR)
284 ...
285 length of chunk
286 struct guestfs_chunk (with data.data_len == 0)
287 The final chunk has the "data_len" field set to zero. Additionally a
289 flag is set in the final chunk to indicate either successful completion
290 or early cancellation.
291 At time of writing there are no functions that have more than one
293 FileIn parameter. However this is (theoretically) supported, by
294 sending the sequence of chunks for each FileIn parameter one after
295 another (from left to right).
296 Both the library (sender) and the daemon (receiver) may cancel the
298 transfer. The library does this by sending a chunk with a special flag
299 set to indicate cancellation. When the daemon sees this, it cancels
300 the whole RPC, does not send any reply, and goes back to reading the
301 next request.
302 The daemon may also cancel. It does this by writing a special word
304 "GUESTFS_CANCEL_FLAG" to the socket. The library listens for this
305 during the transfer, and if it gets it, it will cancel the transfer (it
306 sends a cancel chunk). The special word is chosen so that even if
307 cancellation happens right at the end of the transfer (after the
308 library has finished writing and has started listening for the reply),
309 the "spurious" cancel flag will not be confused with the reply message.
310 This protocol allows the transfer of arbitrary sized files (no 32 bit
312 limit), and also files where the size is not known in advance (eg. from
313 pipes or sockets). However the chunks are rather small
314 ("GUESTFS_MAX_CHUNK_SIZE"), so that neither the library nor the daemon
315 need to keep much in memory.
316 FUNCTIONS THAT HAVE FILEOUT PARAMETERS
318 The protocol for FileOut parameters is exactly the same as for FileIn
320 parameters, but with the roles of daemon and library reversed.
321 total length (header + ret,
323 but not including the length word itself,
324 and not including the chunks)
325 struct guestfs_message_header (encoded as XDR)
326 struct guestfs_<foo>_ret (encoded as XDR)
327 sequence of chunks for FileOut param #0
328 sequence of chunks for FileOut param #1 etc.
329 INITIAL MESSAGE
331 When the daemon launches it sends an initial word
333 ("GUESTFS_LAUNCH_FLAG") which indicates that the guest and daemon is
334 alive. This is what "guestfs_launch" in guestfs(3) waits for.
335 PROGRESS NOTIFICATION MESSAGES
337 The daemon may send progress notification messages at any time. These
339 are distinguished by the normal length word being replaced by
340 "GUESTFS_PROGRESS_FLAG", followed by a fixed size progress message.
341 The library turns them into progress callbacks (see
343 "GUESTFS_EVENT_PROGRESS" in guestfs(3)) if there is a callback
344 registered, or discards them if not.
345 The daemon self-limits the frequency of progress messages it sends (see
347 "daemon/proto.c:notify_progress"). Not all calls generate progress
348 messages.
349 FIXED APPLIANCE
351 When libguestfs (or libguestfs tools) are run, they search a path
352 looking for an appliance. The path is built into libguestfs, or can be
353 set using the "LIBGUESTFS_PATH" environment variable.
354 Normally a supermin appliance is located on this path (see "SUPERMIN
356 APPLIANCE" in supermin(1)). libguestfs reconstructs this into a full
357 appliance by running "supermin --build".
358 However, a simpler "fixed appliance" can also be used. libguestfs
360 detects this by looking for a directory on the path containing all the
361 following files:
362 • kernel
364 • initrd
366 • root
368 • README.fixed (note that it must be present as well)
370 If the fixed appliance is found, libguestfs skips supermin entirely and
372 just runs the virtual machine (using qemu or the current backend, see
373 "BACKEND" in guestfs(3)) with the kernel, initrd and root disk from the
374 fixed appliance.
375 Thus the fixed appliance can be used when a platform or a Linux
377 distribution does not support supermin. You build the fixed appliance
378 on a platform that does support supermin using
379 libguestfs-make-fixed-appliance(1), copy it over, and use that to run
380 libguestfs.
381
383 guestfs(3), guestfs-hacking(1), guestfs-examples(3),
384 libguestfs-test-tool(1), libguestfs-make-fixed-appliance(1),
385 http://libguestfs.org/.
386
388 Richard W.M. Jones ("rjones at redhat dot com")
389
391 Copyright (C) 2009-2020 Red Hat Inc.
392
394 This library is free software; you can redistribute it and/or modify it
395 under the terms of the GNU Lesser General Public License as published
396 by the Free Software Foundation; either version 2 of the License, or
397 (at your option) any later version.
398 This library is distributed in the hope that it will be useful, but
400 WITHOUT ANY WARRANTY; without even the implied warranty of
401 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
402 Lesser General Public License for more details.
403 You should have received a copy of the GNU Lesser General Public
405 License along with this library; if not, write to the Free Software
406 Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
407 02110-1301 USA
408
410 To get a list of bugs against libguestfs, use this link:
411 https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
412 To report a new bug against libguestfs, use this link:
414 https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
415 When reporting a bug, please supply:
417 • The version of libguestfs.
419 • Where you got libguestfs (eg. which Linux distro, compiled from
421 source, etc)
422 • Describe the bug accurately and give a way to reproduce it.
424 • Run libguestfs-test-tool(1) and paste the complete, unedited output
426 into the bug report.
427libguestfs-1.49.9 2023-01-19 guestfs-internals(1)