1guestfs-internals(1)        Virtualization Support        guestfs-internals(1)
2
3
4

NAME

6       guestfs-internals - architecture and internals of libguestfs
7

DESCRIPTION

9       This manual page is for hackers who want to understand how libguestfs
10       works internally.  This is just a description of how libguestfs works
11       now, and it may change at any time in the future.
12

ARCHITECTURE

14       Internally, libguestfs is implemented by running an appliance (a
15       special type of small virtual machine) using qemu(1).  Qemu runs as a
16       child process of the main program.
17
18        ┌───────────────────┐
19        │ main program      │
20        │                   │
21        │                   │           child process / appliance
22        │                   │          ┌──────────────────────────┐
23        │                   │          │ qemu                     │
24        ├───────────────────┤   RPC    │      ┌─────────────────┐ │
25        │ libguestfs  ◀╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍▶ guestfsd        │ │
26        │                   │          │      ├─────────────────┤ │
27        └───────────────────┘          │      │ Linux kernel    │ │
28                                       │      └────────┬────────┘ │
29                                       └───────────────│──────────┘
30
31                                                       │ virtio-scsi
32                                                ┌──────┴──────┐
33                                                │  Device or  │
34                                                │  disk image │
35                                                └─────────────┘
36
37       The library, linked to the main program, creates the child process and
38       hence the appliance in the "guestfs_launch" in guestfs(3) function.
39
40       Inside the appliance is a Linux kernel and a complete stack of
41       userspace tools (such as LVM and ext2 programs) and a small controlling
42       daemon called "guestfsd".  The library talks to "guestfsd" using remote
43       procedure calls (RPC).  There is a mostly one-to-one correspondence
44       between libguestfs API calls and RPC calls to the daemon.  Lastly the
45       disk image(s) are attached to the qemu process which translates device
46       access by the appliance’s Linux kernel into accesses to the image.
47
48       A common misunderstanding is that the appliance "is" the virtual
49       machine.  Although the disk image you are attached to might also be
50       used by some virtual machine, libguestfs doesn't know or care about
51       this.  (But you will care if both libguestfs’s qemu process and your
52       virtual machine are trying to update the disk image at the same time,
53       since these usually results in massive disk corruption).
54

STATE MACHINE

56       libguestfs uses a state machine to model the child process:
57
58                                |
59                 guestfs_create / guestfs_create_flags
60                                |
61                                |
62                            ____V_____
63                           /          \
64                           |  CONFIG  |
65                           \__________/
66                              ^   ^  \
67                              |    \  \ guestfs_launch
68                              |    _\__V______
69                              |   /           \
70                              |   | LAUNCHING |
71                              |   \___________/
72                              |       /
73                              |  guestfs_launch
74                              |     /
75                            __|____V
76                           /        \
77                           | READY  |
78                           \________/
79
80       The normal transitions are (1) CONFIG (when the handle is created, but
81       there is no child process), (2) LAUNCHING (when the child process is
82       booting up), (3) READY meaning the appliance is up, actions can be
83       issued to, and carried out by, the child process.
84
85       The guest may be killed by "guestfs_kill_subprocess" in guestfs(3), or
86       may die asynchronously at any time (eg. due to some internal error),
87       and that causes the state to transition back to CONFIG.
88
89       Configuration commands for qemu such as "guestfs_set_path" in
90       guestfs(3) can only be issued when in the CONFIG state.
91
92       The API offers one call that goes from CONFIG through LAUNCHING to
93       READY.  "guestfs_launch" in guestfs(3) blocks until the child process
94       is READY to accept commands (or until some failure or timeout).
95       "guestfs_launch" in guestfs(3) internally moves the state from CONFIG
96       to LAUNCHING while it is running.
97
98       API actions such as "guestfs_mount" in guestfs(3) can only be issued
99       when in the READY state.  These API calls block waiting for the command
100       to be carried out.  There are no non-blocking versions, and no way to
101       issue more than one command per handle at the same time.
102
103       Finally, the child process sends asynchronous messages back to the main
104       program, such as kernel log messages.  You can register a callback to
105       receive these messages.
106

INTERNALS

108   APPLIANCE BOOT PROCESS
109       This process has evolved and continues to evolve.  The description here
110       corresponds only to the current version of libguestfs and is provided
111       for information only.
112
113       In order to follow the stages involved below, enable libguestfs
114       debugging (set the environment variable "LIBGUESTFS_DEBUG=1").
115
116       Create the appliance
117           "supermin --build" is invoked to create the kernel, a small initrd
118           and the appliance.
119
120           The appliance is cached in /var/tmp/.guestfs-<UID> (or in another
121           directory if "LIBGUESTFS_CACHEDIR" or "TMPDIR" are set).
122
123           For a complete description of how the appliance is created and
124           cached, read the supermin(1) man page.
125
126       Start qemu and boot the kernel
127           qemu is invoked to boot the kernel.
128
129       Run the initrd
130           "supermin --build" builds a small initrd.  The initrd is not the
131           appliance.  The purpose of the initrd is to load enough kernel
132           modules in order that the appliance itself can be mounted and
133           started.
134
135           The initrd is a cpio archive called
136           /var/tmp/.guestfs-<UID>/appliance.d/initrd.
137
138           When the initrd has started you will see messages showing that
139           kernel modules are being loaded, similar to this:
140
141            supermin: ext2 mini initrd starting up
142            supermin: mounting /sys
143            supermin: internal insmod libcrc32c.ko
144            supermin: internal insmod crc32c-intel.ko
145
146       Find and mount the appliance device
147           The appliance is a sparse file containing an ext2 filesystem which
148           contains a familiar (although reduced in size) Linux operating
149           system.  It would normally be called
150           /var/tmp/.guestfs-<UID>/appliance.d/root.
151
152           The regular disks being inspected by libguestfs are the first
153           devices exposed by qemu (eg. as /dev/vda).
154
155           The last disk added to qemu is the appliance itself (eg. /dev/vdb
156           if there was only one regular disk).
157
158           Thus the final job of the initrd is to locate the appliance disk,
159           mount it, and switch root into the appliance, and run /init from
160           the appliance.
161
162           If this works successfully you will see messages such as:
163
164            supermin: picked /sys/block/vdb/dev as root device
165            supermin: creating /dev/root as block special 252:16
166            supermin: mounting new root on /root
167            supermin: chroot
168            Starting /init script ...
169
170           Note that "Starting /init script ..." indicates that the
171           appliance's init script is now running.
172
173       Initialize the appliance
174           The appliance itself now initializes itself.  This involves
175           starting certain processes like "udev", possibly printing some
176           debug information, and finally running the daemon ("guestfsd").
177
178       The daemon
179           Finally the daemon ("guestfsd") runs inside the appliance.  If it
180           runs you should see:
181
182            verbose daemon enabled
183
184           The daemon expects to see a named virtio-serial port exposed by
185           qemu and connected on the other end to the library.
186
187           The daemon connects to this port (and hence to the library) and
188           sends a four byte message "GUESTFS_LAUNCH_FLAG", which initiates
189           the communication protocol (see below).
190
191   COMMUNICATION PROTOCOL
192       Don’t rely on using this protocol directly.  This section documents how
193       it currently works, but it may change at any time.
194
195       The protocol used to talk between the library and the daemon running
196       inside the qemu virtual machine is a simple RPC mechanism built on top
197       of XDR (RFC 1014, RFC 1832, RFC 4506).
198
199       The detailed format of structures is in
200       common/protocol/guestfs_protocol.x (note: this file is automatically
201       generated).
202
203       There are two broad cases, ordinary functions that don’t have any
204       "FileIn" and "FileOut" parameters, which are handled with very simple
205       request/reply messages.  Then there are functions that have any
206       "FileIn" or "FileOut" parameters, which use the same request and reply
207       messages, but they may also be followed by files sent using a chunked
208       encoding.
209
210       ORDINARY FUNCTIONS (NO FILEIN/FILEOUT PARAMS)
211
212       For ordinary functions, the request message is:
213
214        total length (header + arguments,
215             but not including the length word itself)
216        struct guestfs_message_header (encoded as XDR)
217        struct guestfs_<foo>_args (encoded as XDR)
218
219       The total length field allows the daemon to allocate a fixed size
220       buffer into which it slurps the rest of the message.  As a result, the
221       total length is limited to "GUESTFS_MESSAGE_MAX" bytes (currently 4MB),
222       which means the effective size of any request is limited to somewhere
223       under this size.
224
225       Note also that many functions don’t take any arguments, in which case
226       the "guestfs_foo_args" is completely omitted.
227
228       The header contains the procedure number ("guestfs_proc") which is how
229       the receiver knows what type of args structure to expect, or none at
230       all.
231
232       For functions that take optional arguments, the optional arguments are
233       encoded in the "guestfs_foo_args" structure in the same way as ordinary
234       arguments.  A bitmask in the header indicates which optional arguments
235       are meaningful.  The bitmask is also checked to see if it contains bits
236       set which the daemon does not know about (eg. if more optional
237       arguments were added in a later version of the library), and this
238       causes the call to be rejected.
239
240       The reply message for ordinary functions is:
241
242        total length (header + ret,
243             but not including the length word itself)
244        struct guestfs_message_header (encoded as XDR)
245        struct guestfs_<foo>_ret (encoded as XDR)
246
247       As above the "guestfs_foo_ret" structure may be completely omitted for
248       functions that return no formal return values.
249
250       As above the total length of the reply is limited to
251       "GUESTFS_MESSAGE_MAX".
252
253       In the case of an error, a flag is set in the header, and the reply
254       message is slightly changed:
255
256        total length (header + error,
257             but not including the length word itself)
258        struct guestfs_message_header (encoded as XDR)
259        struct guestfs_message_error (encoded as XDR)
260
261       The "guestfs_message_error" structure contains the error message as a
262       string.
263
264       FUNCTIONS THAT HAVE FILEIN PARAMETERS
265
266       A "FileIn" parameter indicates that we transfer a file into the guest.
267       The normal request message is sent (see above).  However this is
268       followed by a sequence of file chunks.
269
270        total length (header + arguments,
271             but not including the length word itself,
272             and not including the chunks)
273        struct guestfs_message_header (encoded as XDR)
274        struct guestfs_<foo>_args (encoded as XDR)
275        sequence of chunks for FileIn param #0
276        sequence of chunks for FileIn param #1 etc.
277
278       The "sequence of chunks" is:
279
280        length of chunk (not including length word itself)
281        struct guestfs_chunk (encoded as XDR)
282        length of chunk
283        struct guestfs_chunk (encoded as XDR)
284          ...
285        length of chunk
286        struct guestfs_chunk (with data.data_len == 0)
287
288       The final chunk has the "data_len" field set to zero.  Additionally a
289       flag is set in the final chunk to indicate either successful completion
290       or early cancellation.
291
292       At time of writing there are no functions that have more than one
293       FileIn parameter.  However this is (theoretically) supported, by
294       sending the sequence of chunks for each FileIn parameter one after
295       another (from left to right).
296
297       Both the library (sender) and the daemon (receiver) may cancel the
298       transfer.  The library does this by sending a chunk with a special flag
299       set to indicate cancellation.  When the daemon sees this, it cancels
300       the whole RPC, does not send any reply, and goes back to reading the
301       next request.
302
303       The daemon may also cancel.  It does this by writing a special word
304       "GUESTFS_CANCEL_FLAG" to the socket.  The library listens for this
305       during the transfer, and if it gets it, it will cancel the transfer (it
306       sends a cancel chunk).  The special word is chosen so that even if
307       cancellation happens right at the end of the transfer (after the
308       library has finished writing and has started listening for the reply),
309       the "spurious" cancel flag will not be confused with the reply message.
310
311       This protocol allows the transfer of arbitrary sized files (no 32 bit
312       limit), and also files where the size is not known in advance (eg. from
313       pipes or sockets).  However the chunks are rather small
314       ("GUESTFS_MAX_CHUNK_SIZE"), so that neither the library nor the daemon
315       need to keep much in memory.
316
317       FUNCTIONS THAT HAVE FILEOUT PARAMETERS
318
319       The protocol for FileOut parameters is exactly the same as for FileIn
320       parameters, but with the roles of daemon and library reversed.
321
322        total length (header + ret,
323             but not including the length word itself,
324             and not including the chunks)
325        struct guestfs_message_header (encoded as XDR)
326        struct guestfs_<foo>_ret (encoded as XDR)
327        sequence of chunks for FileOut param #0
328        sequence of chunks for FileOut param #1 etc.
329
330       INITIAL MESSAGE
331
332       When the daemon launches it sends an initial word
333       ("GUESTFS_LAUNCH_FLAG") which indicates that the guest and daemon is
334       alive.  This is what "guestfs_launch" in guestfs(3) waits for.
335
336       PROGRESS NOTIFICATION MESSAGES
337
338       The daemon may send progress notification messages at any time.  These
339       are distinguished by the normal length word being replaced by
340       "GUESTFS_PROGRESS_FLAG", followed by a fixed size progress message.
341
342       The library turns them into progress callbacks (see
343       "GUESTFS_EVENT_PROGRESS" in guestfs(3)) if there is a callback
344       registered, or discards them if not.
345
346       The daemon self-limits the frequency of progress messages it sends (see
347       "daemon/proto.c:notify_progress").  Not all calls generate progress
348       messages.
349
350   FIXED APPLIANCE
351       When libguestfs (or libguestfs tools) are run, they search a path
352       looking for an appliance.  The path is built into libguestfs, or can be
353       set using the "LIBGUESTFS_PATH" environment variable.
354
355       Normally a supermin appliance is located on this path (see "SUPERMIN
356       APPLIANCE" in supermin(1)).  libguestfs reconstructs this into a full
357       appliance by running "supermin --build".
358
359       However, a simpler "fixed appliance" can also be used.  libguestfs
360       detects this by looking for a directory on the path containing all the
361       following files:
362
363kernel
364
365initrd
366
367root
368
369README.fixed (note that it must be present as well)
370
371       If the fixed appliance is found, libguestfs skips supermin entirely and
372       just runs the virtual machine (using qemu or the current backend, see
373       "BACKEND" in guestfs(3)) with the kernel, initrd and root disk from the
374       fixed appliance.
375
376       Thus the fixed appliance can be used when a platform or a Linux
377       distribution does not support supermin.  You build the fixed appliance
378       on a platform that does support supermin using
379       libguestfs-make-fixed-appliance(1), copy it over, and use that to run
380       libguestfs.
381

SEE ALSO

383       guestfs(3), guestfs-hacking(1), guestfs-examples(3),
384       libguestfs-test-tool(1), libguestfs-make-fixed-appliance(1),
385       http://libguestfs.org/.
386

AUTHORS

388       Richard W.M. Jones ("rjones at redhat dot com")
389
391       Copyright (C) 2009-2020 Red Hat Inc.
392

LICENSE

394       This library is free software; you can redistribute it and/or modify it
395       under the terms of the GNU Lesser General Public License as published
396       by the Free Software Foundation; either version 2 of the License, or
397       (at your option) any later version.
398
399       This library is distributed in the hope that it will be useful, but
400       WITHOUT ANY WARRANTY; without even the implied warranty of
401       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
402       Lesser General Public License for more details.
403
404       You should have received a copy of the GNU Lesser General Public
405       License along with this library; if not, write to the Free Software
406       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
407       02110-1301 USA
408

BUGS

410       To get a list of bugs against libguestfs, use this link:
411       https://bugzilla.redhat.com/buglist.cgi?component=libguestfs&product=Virtualization+Tools
412
413       To report a new bug against libguestfs, use this link:
414       https://bugzilla.redhat.com/enter_bug.cgi?component=libguestfs&product=Virtualization+Tools
415
416       When reporting a bug, please supply:
417
418       •   The version of libguestfs.
419
420       •   Where you got libguestfs (eg. which Linux distro, compiled from
421           source, etc)
422
423       •   Describe the bug accurately and give a way to reproduce it.
424
425       •   Run libguestfs-test-tool(1) and paste the complete, unedited output
426           into the bug report.
427
428
429
430libguestfs-1.49.9                 2023-01-19              guestfs-internals(1)
Impressum