1DEBUGINFOD(8) System Manager's Manual DEBUGINFOD(8)
3
7 debuginfod - debuginfo-related http file-server daemon
8
11 debuginfod [OPTION]... [PATH]...
12
15 debuginfod serves debuginfo-related artifacts over HTTP. It periodi‐
16 cally scans a set of directories for ELF/DWARF files and their associ‐
17 ated source code, as well as archive files containing the above, to
18 build an index by their buildid. This index is used when remote
19 clients use the HTTP webapi, to fetch these files by the same buildid.
20 If a debuginfod cannot service a given buildid artifact request itself,
22 and it is configured with information about upstream debuginfod
23 servers, it queries them for the same information, just as debuginfod-
24 find would. If successful, it locally caches then relays the file con‐
25 tent to the original requester.
26 Indexing the given PATHs proceeds using multiple threads. One thread
28 periodically traverses all the given PATHs logically or physically (see
29 the -L option). Duplicate PATHs are ignored. You may use a file name
30 for a PATH, but source code indexing may be incomplete; prefer using a
31 directory that contains the binaries. The traversal thread enumerates
32 all matching files (see the -I and -X options) into a work queue. A
33 collection of scanner threads (see the -c option) wait at the work
34 queue to analyze files in parallel.
35 If the -F option is given, each file is scanned as an ELF/DWARF file.
37 Source files are matched with DWARF files based on the AT_comp_dir
38 (compilation directory) attributes inside it. Caution: source files
39 listed in the DWARF may be a path anywhere in the file system, and de‐
40 buginfod will readily serve their content on demand. (Imagine a doc‐
41 tored DWARF file that lists /etc/passwd as a source file.) If this is
42 a concern, audit your binaries with tools such as:
43 % eu-readelf -wline BINARY | sed -n '/^Directory.table/,/^File.name.table/p'
45 or
46 % eu-readelf -wline BINARY | sed -n '/^Directory.table/,/^Line.number/p'
47 or even use debuginfod itself:
48 % debuginfod -vvv -d :memory: -F BINARY 2>&1 | grep 'recorded.*source'
49 ^C
50 If any of the -R, -U, or -Z options is given, each file is scanned as
52 an archive file that may contain ELF/DWARF/source files. Archive files
53 are recognized by extension. If -R is given, ".rpm" files are scanned;
54 if -U is given, ".deb" and ".ddeb" files are scanned; if -Z is given,
55 the listed extensions are scanned. Because of complications such as
56 DWZ-compressed debuginfo, may require two traversal passes to identify
57 all source code. Source files for RPMs are only served from other
58 RPMs, so the caution for -F does not apply. Note that due to De‐
59 bian/Ubuntu packaging policies & mechanisms, debuginfod cannot resolve
60 source files for DEB/DDEB at all.
61 If no PATH is listed, or none of the scanning options is given, then
63 debuginfod will simply serve content that it accumulated into its index
64 in all previous runs, periodically groom the database, and federate to
65 any upstream debuginfod servers. In passive mode, debuginfod will only
66 serve content from a read-only index and federated upstream servers,
67 but will not scan or groom.
68
71 -F Activate ELF/DWARF file scanning. The default is off.
72 -Z EXT -Z EXT=CMD
75 Activate an additional pattern in archive scanning. Files with
76 name extension EXT (include the dot) will be processed. If CMD
77 is given, it is invoked with the file name added to its argument
78 list, and should produce a common archive on its standard out‐
79 put. Otherwise, the file is read as if CMD were "cat". Since
80 debuginfod internally uses libarchive to read archive files, it
81 can accept a wide range of archive formats and compression
82 modes. The default is no additional patterns. This option may
83 be repeated.
84 -R Activate RPM patterns in archive scanning. The default is off.
87 Equivalent to -Z .rpm=cat, since libarchive can natively process
88 RPM archives. If your version of libarchive is much older than
89 2020, be aware that some distributions have switched to an in‐
90 compatible zstd compression for their payload. You may experi‐
91 ment with -Z .rpm='(rpm2cpio|zstdcat)<' instead of -R.
92 -U Activate DEB/DDEB patterns in archive scanning. The default is
95 off. Equivalent to -Z .deb='dpkg-deb --fsys-tarfile'
96 -Z .ddeb='dpkg-deb --fsys-tarfile'.
97 -d FILE --database=FILE
100 Set the path of the sqlite database used to store the index.
101 This file is disposable in the sense that a later rescan will
102 repopulate data. It will contain absolute file path names, so
103 it may not be portable across machines. It may be frequently
104 read/written, so it should be on a fast filesystem. It should
105 not be shared across machines or users, to maximize sqlite lock‐
106 ing performance. For quick testing the magic string ":memory:"
107 can be used to use an one-time memory-only database. The de‐
108 fault database file is $HOME/.debuginfod.sqlite.
109 --passive
112 Set the server to passive mode, where it only services webapi
113 requests, including participating in federation. It performs no
114 scanning, no grooming, and so only opens the sqlite database
115 read-only. This way a database can be safely shared between a
116 active scanner/groomer server and multiple passive ones, thereby
117 sharing service load. Archive pattern options must still be
118 given, so debuginfod can recognize file name extensions for un‐
119 packing.
120 -D SQL --ddl=SQL
123 Execute given sqlite statement after the database is opened and
124 initialized as extra DDL (SQL data definition language). This
125 may be useful to tune performance-related pragmas or indexes.
126 May be repeated. The default is nothing extra.
127 -p NUM --port=NUM
130 Set the TCP port number (0 < NUM < 65536) on which debuginfod
131 should listen, to service HTTP requests. Both IPv4 and IPV6
132 sockets are opened, if possible. The webapi is documented be‐
133 low. The default port number is 8002.
134 -I REGEX --include=REGEX -X REGEX --exclude=REGEX
137 Govern the inclusion and exclusion of file names under the
138 search paths. The regular expressions are interpreted as unan‐
139 chored POSIX extended REs, thus may include alternation. They
140 are evaluated against the full path of each file, based on its
141 realpath(3) canonicalization. By default, all files are includ‐
142 ed and none are excluded. A file that matches both include and
143 exclude REGEX is excluded. (The contents of archive files are
144 not subject to inclusion or exclusion filtering: they are all
145 processed.) Only the last of each type of regular expression
146 given is used.
147 -t SECONDS --rescan-time=SECONDS
150 Set the rescan time for the file and archive directories. This
151 is the amount of time the traversal thread will wait after fin‐
152 ishing a scan, before doing it again. A rescan for unchanged
153 files is fast (because the index also stores the file mtimes).
154 A time of zero is acceptable, and means that only one initial
155 scan should performed. The default rescan time is 300 seconds.
156 Receiving a SIGUSR1 signal triggers a new scan, independent of
157 the rescan time (including if it was zero), interrupting a groom
158 pass (if any).
159 -r Apply the -I and -X during groom cycles, so that most content
162 related to files excluded by the regexes are removed from the
163 index. Not all content can be practically removed, so eventual‐
164 ly a "-G" "maximal-groom" operation may be needed.
165 -g SECONDS --groom-time=SECONDS
168 Set the groom time for the index database. This is the amount
169 of time the grooming thread will wait after finishing a grooming
170 pass before doing it again. A groom operation quickly rescans
171 all previously scanned files, only to see if they are still
172 present and current, so it can deindex obsolete files. See also
173 the DATA MANAGEMENT section. The default groom time is 86400
174 seconds (1 day). A time of zero is acceptable, and means that
175 only one initial groom should be performed. Receiving a SIGUSR2
176 signal triggers a new grooming pass, independent of the groom
177 time (including if it was zero), interrupting a rescan pass (if
178 any)..
179 -G Run an extraordinary maximal-grooming pass at debuginfod start‐
182 up. This pass can take considerable time, because it tries to
183 remove any debuginfo-unrelated content from the archive-related
184 parts of the index. It should not be run if any recent archive-
185 related indexing operations were aborted early. It can take
186 considerable space, because it finishes up with an sqlite "vacu‐
187 um" operation, which repacks the database file by triplicating
188 it temporarily. The default is not to do maximal-grooming. See
189 also the DATA MANAGEMENT section.
190 -c NUM --concurrency=NUM
193 Set the concurrency limit for the scanning queue threads, which
194 work together to process archives & files located by the traver‐
195 sal thread. This important for controlling CPU-intensive opera‐
196 tions like parsing an ELF file and especially decompressing ar‐
197 chives. The default is related to the number of processors on
198 the system and other constraints; the minimum is 1.
199 -C -C=NUM --connection-pool --connection-pool=NUM
202 Set the size of the pool of threads serving webapi queries. The
203 following table summarizes the interpretaton of this option and
204 its optional NUM parameter.
205 no option, -C use a fixed thread pool sized automatically
207 -C=NUM use a fixed thread pool sized NUM, minimum 2
208 The first mode is a simple and safe configuration related to the
210 number of processors and other constraints. The second mode is
211 suitable for tuned load-limiting configurations facing unruly
212 traffic.
213 -L Traverse symbolic links encountered during traversal of the
216 PATHs, including across devices - as in find -L. The default is
217 to traverse the physical directory structure only, stay on the
218 same device, and ignore symlinks - as in find -P -xdev. Cau‐
219 tion: a loops in the symbolic directory tree might lead to infi‐
220 nite traversal.
221 --fdcache-fds=NUM --fdcache-mbs=MB
224 Configure limits on a cache that keeps recently extracted files
225 from archives. Up to NUM requested files and up to a total of
226 MB megabytes will be kept extracted, in order to avoid having to
227 decompress their archives over and over again. The default NUM
228 and MB values depend on the concurrency of the system, and on
229 the available disk space on the $TMPDIR or /tmp filesystem.
230 This is because that is where the most recently used extracted
231 files are kept. Grooming cleans out this cache.
232 --fdcache-prefetch-fds=NUM --fdcache-prefetch-mbs=MB
235 --fdcache-prefetch=NUM2
236 In addition to the main fdcache, up to NUM2 other files from an
238 archive may be prefetched into another cache before they are
239 even requested. Configure how many file descriptors (fds) and
240 megabytes (mbs) are allocated to the prefetch fdcache. If un‐
241 specified, these values depend on concurrency of the system and
242 on the available disk space on the $TMPDIR. Allocating more to
243 the prefetch cache will improve performance in environments
244 where different parts of several large archives are being ac‐
245 cessed. This cache is also cleaned out during grooming.
246 --fdcache-mintmp=NUM
249 Configure a disk space threshold for emergency flushing of the
250 caches. The filesystem holding the caches is checked periodi‐
251 cally. If the available space falls below the given percentage,
252 the caches are flushed, and the fdcaches will stay disabled un‐
253 til the next groom cycle. This mechanism, along a few associat‐
254 ed /metrics on the webapi, are intended to give an operator no‐
255 tice about storage scarcity - which can translate to RAM scarci‐
256 ty if the disk happens to be on a RAM virtual disk. The default
257 threshold is 25%.
258 --forwarded-ttl-limit=NUM
261 Configure limits of X-Forwarded-For hops. if X-Forwarded-For ex‐
262 ceeds N hops, it will not delegate a local lookup miss to up‐
263 stream debuginfods. The default limit is 8.
264 --disable-source-scan
267 Disable scan of the dwarf source info of debuginfo sections. If
268 a setup has no access to source code, the source info is not re‐
269 quired.
270 --scan-checkpoint=NUM
273 Run a synchronized SQLITE WAL checkpoint operation after every
274 NUM completed archive or file scans. This may slow down paral‐
275 lel scanning phase somewhat, but generate much smaller "-wal"
276 temporary files on busy servers. The default is 256. Disabled
277 if 0.
278 -v Increase verbosity of logging to the standard error file de‐
281 scriptor. May be repeated to increase details. The default
282 verbosity is 0.
283
286 debuginfod's webapi resembles ordinary file service, where a GET re‐
287 quest with a path containing a known buildid results in a file. Un‐
288 known buildid / request combinations result in HTTP error codes. This
289 file service resemblance is intentional, so that an installation can
290 take advantage of standard HTTP management infrastructure.
291 For most queries, some custom http headers are added to the response,
293 providing additional metadata about the buildid-related response. For
294 example:
295 % debuginfod-find -v debuginfo /bin/ls |& grep -i x-debuginfo
297 x-debuginfod-size: 502024
298 x-debuginfod-archive: /mnt/fedora_koji_prod/koji/packages/coreutils/9.3/4.fc39/x86_64/coreutils-debuginfo-9.3-4.fc39.x86_64.rpm
299 x-debuginfod-file: /usr/lib/debug/usr/bin/ls-9.3-4.fc39.x86_64.debug
300 X-DEBUGINFOD-SIZE
303 The size of the file, in bytes. This may differ from the http
304 Content-Length: field (if present), due to compression in tran‐
305 sit.
306 X-DEBUGINFOD-FILE
309 The full path name of the file related to the given buildid.
310 X-DEBUGINFOD-ARCHIVE
313 The full path name of the archive that contained the above file,
314 if any.
315 There are a handful of buildid-related requests. In each case,
320 the buildid is encoded as a lowercase hexadecimal string. For
321 example, for a program /bin/ls, look at the ELF note
322 GNU_BUILD_ID:
323 % readelf -n /bin/ls | grep -A4 build.id
325 Note section [ 4] '.note.gnu.buildid' of 36 bytes at offset 0x340:
326 Owner Data size Type
327 GNU 20 GNU_BUILD_ID
328 Build ID: 8713b9c3fb8a720137a4a08b325905c7aaf8429d
329 Then the hexadecimal BUILDID is simply:
331 8713b9c3fb8a720137a4a08b325905c7aaf8429d
333 /buildid/BUILDID/debuginfo
336 If the given buildid is known to the server, this request will result
337 in a binary object that contains the customary .*debug_* sections.
338 This may be a split debuginfo file as created by strip, or it may be an
339 original unstripped executable.
340 /buildid/BUILDID/executable
343 If the given buildid is known to the server, this request will result
344 in a binary object that contains the normal executable segments. This
345 may be a executable stripped by strip, or it may be an original un‐
346 stripped executable. ET_DYN shared libraries are considered to be a
347 type of executable.
348 /buildid/BUILDID/source/SOURCE/FILE
351 If the given buildid is known to the server, this request will result
352 in a binary object that contains the source file mentioned. The path
353 should be absolute. Relative path names commonly appear in the DWARF
354 file's source directory, but these paths are relative to individual
355 compilation unit AT_comp_dir paths, and yet an executable is made up of
356 multiple CUs. Therefore, to disambiguate, debuginfod expects source
357 queries to prefix relative path names with the CU compilation-directo‐
358 ry, followed by a mandatory "/".
359 Note: the caller may or may not elide ../ or /./ or extraneous ///
361 sorts of path components in the directory names. debuginfod accepts
362 both forms. Specifically, debuginfod canonicalizes path names accord‐
363 ing to RFC3986 section 5.2.4 (Remove Dot Segments), plus reducing any
364 // to / in the path.
365 For example:
367 #include <stdio.h> /buildid/BUILDID/source/usr/include/stdio.h
369 /path/to/foo.c /buildid/BUILDID/source/path/to/foo.c
370 ../bar/foo.c AT_comp_dir=/zoo/ /buildid/BUILDID/source/zoo//../bar/foo.c
371 Note: the client should %-escape characters in /SOURCE/FILE that are
373 not shown as "unreserved" in section 2.3 of RFC3986. Some characters
374 that will be escaped include "+", "\", "$", "!", the 'space' character,
375 and ";". RFC3986 includes a more comprehensive list of these charac‐
376 ters.
377 /buildid/BUILDID/section/SECTION
380 If the given buildid is known to the server, the server will attempt to
381 extract the contents of an ELF/DWARF section named SECTION from the de‐
382 buginfo file matching BUILDID. If the debuginfo file can't be found or
383 the section has type SHT_NOBITS, then the server will attempt to ex‐
384 tract the section from the executable matching BUILDID. If the section
385 is successfully extracted then this request results in a binary object
386 of the section's contents. Note that this result is the raw binary
387 contents of the section, not an ELF file.
388 /metrics
391 This endpoint returns a Prometheus formatted text/plain dump of a vari‐
392 ety of statistics about the operation of the debuginfod server. The
393 exact set of metrics and their meanings may change in future versions.
394
397 debuginfod stores its index in an sqlite database in a densely packed
398 set of interlinked tables. While the representation is as efficient as
399 we have been able to make it, it still takes a considerable amount of
400 data to record all debuginfo-related data of potentially a great many
401 files. This section offers some advice about the implications.
402 As a general explanation for size, consider that debuginfod indexes
404 ELF/DWARF files, it stores their names and referenced source file
405 names, and buildids will be stored. When indexing archives, it stores
406 every file name of or in an archive, every buildid, plus every source
407 file name referenced from a DWARF file. (Indexing archives takes more
408 space because the source files often reside in separate subpackages
409 that may not be indexed at the same pass, so extra metadata has to be
410 kept.)
411 Getting down to numbers, in the case of Fedora RPMs (essentially, gzip-
413 compressed cpio files), the sqlite index database tends to be from 0.5%
414 to 3% of their size. It's larger for binaries that are assembled out
415 of a great many source files, or packages that carry much debuginfo-un‐
416 related content. It may be even larger during the indexing phase due
417 to temporary sqlite write-ahead-logging files; these are checkpointed
418 (cleaned out and removed) at shutdown. It may be helpful to apply
419 tight -I or -X regular-expression constraints to exclude files from
420 scanning that you know have no debuginfo-relevant content.
421 As debuginfod runs in normal active mode, it periodically rescans its
423 target directories, and any new content found is added to the database.
424 Old content, such as data for files that have disappeared or that have
425 been replaced with newer versions is removed at a periodic grooming
426 pass. This means that the sqlite files grow fast during initial index‐
427 ing, slowly during index rescans, and periodically shrink during groom‐
428 ing. There is also an optional one-shot maximal grooming pass is
429 available. It removes information debuginfo-unrelated data from the
430 archive content index such as file names found in archives ("archive
431 sdef" records) that are not referred to as source files from any bina‐
432 ries find in archives ("archive sref" records). This can save consid‐
433 erable disk space. However, it is slow and temporarily requires up to
434 twice the database size as free space. Worse: it may result in missing
435 source-code info if the archive traversals were interrupted, so that
436 not all source file references were known. Use it rarely to polish a
437 complete index.
438 You should ensure that ample disk space remains available. (The flood
440 of error messages on -ENOSPC is ugly and nagging. But, like for most
441 other errors, debuginfod will resume when resources permit.) If neces‐
442 sary, debuginfod can be stopped, the database file moved or removed,
443 and debuginfod restarted.
444 sqlite offers several performance-related options in the form of prag‐
446 mas. Some may be useful to fine-tune the defaults plus the debuginfod
447 extras. The -D option may be useful to tell debuginfod to execute the
448 given bits of SQL after the basic schema creation commands. For exam‐
449 ple, the "synchronous", "cache_size", "auto_vacuum", "threads", "jour‐
450 nal_mode" pragmas may be fun to tweak via -D, if you're searching for
451 peak performance. The "optimize", "wal_checkpoint" pragmas may be use‐
452 ful to run periodically, outside debuginfod. The default settings are
453 performance- rather than reliability-oriented, so a hardware crash
454 might corrupt the database. In these cases, it may be necessary to
455 manually delete the sqlite database and start over.
456 As debuginfod changes in the future, we may have no choice but to
458 change the database schema in an incompatible manner. If this happens,
459 new versions of debuginfod will issue SQL statements to drop all prior
460 schema & data, and start over. So, disk space will not be wasted for
461 retaining a no-longer-useable dataset.
462 In summary, if your system can bear a 0.5%-3% index-to-archive-dataset
464 size ratio, and slow growth afterwards, you should not need to worry
465 about disk space. If a system crash corrupts the database, or you want
466 to force debuginfod to reset and start over, simply erase the sqlite
467 file before restarting debuginfod.
468 In contrast, in passive mode, all scanning and grooming is disabled,
470 and the index database remains read-only. This makes the database more
471 suitable for sharing between servers or sites with simple one-way
472 replication, and data management considerations are generally moot.
473
476 debuginfod does not include any particular security features. While it
477 is robust with respect to inputs, some abuse is possible. It forks a
478 new thread for each incoming HTTP request, which could lead to a de‐
479 nial-of-service in terms of RAM, CPU, disk I/O, or network I/O. If
480 this is a problem, users are advised to install debuginfod with a HTTPS
481 reverse-proxy front-end that enforces site policies for firewalling,
482 authentication, integrity, authorization, and load control.
483 Front-end proxies may elide sensitive path name components in X-DEBUG‐
485 INFOD-FILE/ARCHIVE response headers. For example, using Apache httpd's
486 mod_headers, you can remove the entire directory name prefix:
487 Header edit x-debuginfod-archive ".*/" ""
489 When relaying queries to upstream debuginfods, debuginfod does not in‐
491 clude any particular security features. It trusts that the binaries
492 returned by the debuginfods are accurate. Therefore, the list of
493 servers should include only trustworthy ones. If accessed across HTTP
494 rather than HTTPS, the network should be trustworthy. Authentication
495 information through the internal libcurl library is not currently en‐
496 abled.
497
501 $HOME/.debuginfod.sqlite
502 Default database file.
503
507 debuginfod-find(1) sqlite3(1)
508 https://prometheus.io/docs/instrumenting/exporters/
509 DEBUGINFOD(8)