1DEBUGINFOD(8) System Manager's Manual DEBUGINFOD(8)
3
7 debuginfod - debuginfo-related http file-server daemon
8
11 debuginfod [OPTION]... [PATH]...
12
15 debuginfod serves debuginfo-related artifacts over HTTP. It periodi‐
16 cally scans a set of directories for ELF/DWARF files and their associ‐
17 ated source code, as well as archive files containing the above, to
18 build an index by their buildid. This index is used when remote
19 clients use the HTTP webapi, to fetch these files by the same buildid.
20 If a debuginfod cannot service a given buildid artifact request itself,
22 and it is configured with information about upstream debuginfod
23 servers, it queries them for the same information, just as debuginfod-
24 find would. If successful, it locally caches then relays the file con‐
25 tent to the original requester.
26 Indexing the given PATHs proceeds using multiple threads. One thread
28 periodically traverses all the given PATHs logically or physically (see
29 the -L option). Duplicate PATHs are ignored. You may use a file name
30 for a PATH, but source code indexing may be incomplete; prefer using a
31 directory that contains the binaries. The traversal thread enumerates
32 all matching files (see the -I and -X options) into a work queue. A
33 collection of scanner threads (see the -c option) wait at the work
34 queue to analyze files in parallel.
35 If the -F option is given, each file is scanned as an ELF/DWARF file.
37 Source files are matched with DWARF files based on the AT_comp_dir
38 (compilation directory) attributes inside it. Caution: source files
39 listed in the DWARF may be a path anywhere in the file system, and de‐
40 buginfod will readily serve their content on demand. (Imagine a doc‐
41 tored DWARF file that lists /etc/passwd as a source file.) If this is
42 a concern, audit your binaries with tools such as:
43 % eu-readelf -wline BINARY | sed -n '/^Directory.table/,/^File.name.table/p'
45 or
46 % eu-readelf -wline BINARY | sed -n '/^Directory.table/,/^Line.number/p'
47 or even use debuginfod itself:
48 % debuginfod -vvv -d :memory: -F BINARY 2>&1 | grep 'recorded.*source'
49 ^C
50 If any of the -R, -U, or -Z options is given, each file is scanned as
52 an archive file that may contain ELF/DWARF/source files. Archive files
53 are recognized by extension. If -R is given, ".rpm" files are scanned;
54 if -U is given, ".deb" and ".ddeb" files are scanned; if -Z is given,
55 the listed extensions are scanned. Because of complications such as
56 DWZ-compressed debuginfo, may require two traversal passes to identify
57 all source code. Source files for RPMs are only served from other
58 RPMs, so the caution for -F does not apply. Note that due to De‐
59 bian/Ubuntu packaging policies & mechanisms, debuginfod cannot resolve
60 source files for DEB/DDEB at all.
61 If no PATH is listed, or none of the scanning options is given, then
63 debuginfod will simply serve content that it accumulated into its index
64 in all previous runs, periodically groom the database, and federate to
65 any upstream debuginfod servers. In passive mode, debuginfod will only
66 serve content from a read-only index and federated upstream servers,
67 but will not scan or groom.
68
71 -F Activate ELF/DWARF file scanning. The default is off.
72 -Z EXT -Z EXT=CMD
75 Activate an additional pattern in archive scanning. Files with
76 name extension EXT (include the dot) will be processed. If CMD
77 is given, it is invoked with the file name added to its argument
78 list, and should produce a common archive on its standard out‐
79 put. Otherwise, the file is read as if CMD were "cat". Since
80 debuginfod internally uses libarchive to read archive files, it
81 can accept a wide range of archive formats and compression
82 modes. The default is no additional patterns. This option may
83 be repeated.
84 -R Activate RPM patterns in archive scanning. The default is off.
87 Equivalent to -Z .rpm=cat, since libarchive can natively process
88 RPM archives. If your version of libarchive is much older than
89 2020, be aware that some distributions have switched to an in‐
90 compatible zstd compression for their payload. You may experi‐
91 ment with -Z .rpm='(rpm2cpio|zstdcat)<' instead of -R.
92 -U Activate DEB/DDEB patterns in archive scanning. The default is
95 off. Equivalent to -Z .deb='dpkg-deb --fsys-tarfile'
96 -Z .ddeb='dpkg-deb --fsys-tarfile'.
97 -d FILE --database=FILE
100 Set the path of the sqlite database used to store the index.
101 This file is disposable in the sense that a later rescan will
102 repopulate data. It will contain absolute file path names, so
103 it may not be portable across machines. It may be frequently
104 read/written, so it should be on a fast filesystem. It should
105 not be shared across machines or users, to maximize sqlite lock‐
106 ing performance. For quick testing the magic string ":memory:"
107 can be used to use an one-time memory-only database. The de‐
108 fault database file is $HOME/.debuginfod.sqlite.
109 --passive
112 Set the server to passive mode, where it only services webapi
113 requests, including participating in federation. It performs no
114 scanning, no grooming, and so only opens the sqlite database
115 read-only. This way a database can be safely shared between a
116 active scanner/groomer server and multiple passive ones, thereby
117 sharing service load. Archive pattern options must still be
118 given, so debuginfod can recognize file name extensions for un‐
119 packing.
120 -D SQL --ddl=SQL
123 Execute given sqlite statement after the database is opened and
124 initialized as extra DDL (SQL data definition language). This
125 may be useful to tune performance-related pragmas or indexes.
126 May be repeated. The default is nothing extra.
127 -p NUM --port=NUM
130 Set the TCP port number (0 < NUM < 65536) on which debuginfod
131 should listen, to service HTTP requests. Both IPv4 and IPV6
132 sockets are opened, if possible. The webapi is documented be‐
133 low. The default port number is 8002.
134 -I REGEX --include=REGEX -X REGEX --exclude=REGEX
137 Govern the inclusion and exclusion of file names under the
138 search paths. The regular expressions are interpreted as unan‐
139 chored POSIX extended REs, thus may include alternation. They
140 are evaluated against the full path of each file, based on its
141 realpath(3) canonicalization. By default, all files are includ‐
142 ed and none are excluded. A file that matches both include and
143 exclude REGEX is excluded. (The contents of archive files are
144 not subject to inclusion or exclusion filtering: they are all
145 processed.) Only the last of each type of regular expression
146 given is used.
147 -t SECONDS --rescan-time=SECONDS
150 Set the rescan time for the file and archive directories. This
151 is the amount of time the traversal thread will wait after fin‐
152 ishing a scan, before doing it again. A rescan for unchanged
153 files is fast (because the index also stores the file mtimes).
154 A time of zero is acceptable, and means that only one initial
155 scan should performed. The default rescan time is 300 seconds.
156 Receiving a SIGUSR1 signal triggers a new scan, independent of
157 the rescan time (including if it was zero), interrupting a groom
158 pass (if any).
159 -r Apply the -I and -X during groom cycles, so that files excluded
162 by the regexes are removed from the index. These parameters are
163 in addition to what normally qualifies a file for grooming, not
164 a replacement.
165 -g SECONDS --groom-time=SECONDS Set the groom time for the index
167 database. This is the amount of time the grooming thread will
168 wait after finishing a grooming pass before doing it again. A
169 groom operation quickly rescans all previously scanned files,
170 only to see if they are still present and current, so it can
171 deindex obsolete files. See also the DATA MANAGEMENT section.
172 The default groom time is 86400 seconds (1 day). A time of zero
173 is acceptable, and means that only one initial groom should be
174 performed. Receiving a SIGUSR2 signal triggers a new grooming
175 pass, independent of the groom time (including if it was zero),
176 interrupting a rescan pass (if any)..
177 -G Run an extraordinary maximal-grooming pass at debuginfod start‐
180 up. This pass can take considerable time, because it tries to
181 remove any debuginfo-unrelated content from the archive-related
182 parts of the index. It should not be run if any recent archive-
183 related indexing operations were aborted early. It can take
184 considerable space, because it finishes up with an sqlite "vacu‐
185 um" operation, which repacks the database file by triplicating
186 it temporarily. The default is not to do maximal-grooming. See
187 also the DATA MANAGEMENT section.
188 -c NUM --concurrency=NUM
191 Set the concurrency limit for the scanning queue threads, which
192 work together to process archives & files located by the traver‐
193 sal thread. This important for controlling CPU-intensive opera‐
194 tions like parsing an ELF file and especially decompressing ar‐
195 chives. The default is the number of processors on the system;
196 the minimum is 1.
197 -C -C=NUM --connection-pool --connection-pool=NUM
200 Set the size of the pool of threads serving webapi queries. The
201 following table summarizes the interpretaton of this option and
202 its optional NUM parameter.
203 no option, -C use a fixed thread pool sized automatically
205 -C=NUM use a fixed thread pool sized NUM, minimum 2
206 The first mode is a simple and safe configuration based on the
208 number of processors. The second mode is suitable for tuned
209 load-limiting configurations facing unruly traffic.
210 -L Traverse symbolic links encountered during traversal of the
213 PATHs, including across devices - as in find -L. The default is
214 to traverse the physical directory structure only, stay on the
215 same device, and ignore symlinks - as in find -P -xdev. Cau‐
216 tion: a loops in the symbolic directory tree might lead to infi‐
217 nite traversal.
218 --fdcache-fds=NUM --fdcache-mbs=MB
221 Configure limits on a cache that keeps recently extracted files
222 from archives. Up to NUM requested files and up to a total of
223 MB megabytes will be kept extracted, in order to avoid having to
224 decompress their archives over and over again. The default NUM
225 and MB values depend on the concurrency of the system, and on
226 the available disk space on the $TMPDIR or /tmp filesystem.
227 This is because that is where the most recently used extracted
228 files are kept. Grooming cleans out this cache.
229 --fdcache-prefetch-fds=NUM --fdcache-prefetch-mbs=MB
232 --fdcache-prefetch=NUM2
233 In addition to the main fdcache, up to NUM2 other files from an
235 archive may be prefetched into another cache before they are
236 even requested. Configure how many file descriptors (fds) and
237 megabytes (mbs) are allocated to the prefetch fdcache. If un‐
238 specified, these values depend on concurrency of the system and
239 on the available disk space on the $TMPDIR. Allocating more to
240 the prefetch cache will improve performance in environments
241 where different parts of several large archives are being ac‐
242 cessed. This cache is also cleaned out during grooming.
243 --fdcache-mintmp=NUM
246 Configure a disk space threshold for emergency flushing of the
247 caches. The filesystem holding the caches is checked periodi‐
248 cally. If the available space falls below the given percentage,
249 the caches are flushed, and the fdcaches will stay disabled un‐
250 til the next groom cycle. This mechanism, along a few associat‐
251 ed /metrics on the webapi, are intended to give an operator no‐
252 tice about storage scarcity - which can translate to RAM scarci‐
253 ty if the disk happens to be on a RAM virtual disk. The default
254 threshold is 25%.
255 --forwarded-ttl-limit=NUM
258 Configure limits of X-Forwarded-For hops. if X-Forwarded-For ex‐
259 ceeds N hops, it will not delegate a local lookup miss to up‐
260 stream debuginfods. The default limit is 8.
261 --disable-source-scan
264 Disable scan of the dwarf source info of debuginfo sections. If
265 a setup has no access to source code, the source info is not re‐
266 quired.
267 -v Increase verbosity of logging to the standard error file de‐
270 scriptor. May be repeated to increase details. The default
271 verbosity is 0.
272
275 debuginfod's webapi resembles ordinary file service, where a GET re‐
276 quest with a path containing a known buildid results in a file. Un‐
277 known buildid / request combinations result in HTTP error codes. This
278 file service resemblance is intentional, so that an installation can
279 take advantage of standard HTTP management infrastructure.
280 Upon finding a file in an archive or simply in the database, some cus‐
282 tom http headers are added to the response. For files in the database
283 X-DEBUGINFOD-FILE and X-DEBUGINFOD-SIZE are added. X-DEBUGINFOD-FILE
284 is simply the unescaped filename and X-DEBUGINFOD-SIZE is the size of
285 the file. For files found in archives, in addition to X-DEBUGINFOD-FILE
286 and X-DEBUGINFOD-SIZE, X-DEBUGINFOD-ARCHIVE is added. X-DEBUGINFOD-AR‐
287 CHIVE is the name of the archive the file was found in.
288 There are three requests. In each case, the buildid is encoded as a
290 lowercase hexadecimal string. For example, for a program /bin/ls, look
291 at the ELF note GNU_BUILD_ID:
292 % readelf -n /bin/ls | grep -A4 build.id
294 Note section [ 4] '.note.gnu.buildid' of 36 bytes at offset 0x340:
295 Owner Data size Type
296 GNU 20 GNU_BUILD_ID
297 Build ID: 8713b9c3fb8a720137a4a08b325905c7aaf8429d
298 Then the hexadecimal BUILDID is simply:
300 8713b9c3fb8a720137a4a08b325905c7aaf8429d
302 /buildid/BUILDID/debuginfo
305 If the given buildid is known to the server, this request will result
306 in a binary object that contains the customary .*debug_* sections.
307 This may be a split debuginfo file as created by strip, or it may be an
308 original unstripped executable.
309 /buildid/BUILDID/executable
312 If the given buildid is known to the server, this request will result
313 in a binary object that contains the normal executable segments. This
314 may be a executable stripped by strip, or it may be an original un‐
315 stripped executable. ET_DYN shared libraries are considered to be a
316 type of executable.
317 /buildid/BUILDID/source/SOURCE/FILE
320 If the given buildid is known to the server, this request will result
321 in a binary object that contains the source file mentioned. The path
322 should be absolute. Relative path names commonly appear in the DWARF
323 file's source directory, but these paths are relative to individual
324 compilation unit AT_comp_dir paths, and yet an executable is made up of
325 multiple CUs. Therefore, to disambiguate, debuginfod expects source
326 queries to prefix relative path names with the CU compilation-directo‐
327 ry, followed by a mandatory "/".
328 Note: the caller may or may not elide ../ or /./ or extraneous ///
330 sorts of path components in the directory names. debuginfod accepts
331 both forms. Specifically, debuginfod canonicalizes path names accord‐
332 ing to RFC3986 section 5.2.4 (Remove Dot Segments), plus reducing any
333 // to / in the path.
334 For example:
336 #include <stdio.h> /buildid/BUILDID/source/usr/include/stdio.h
338 /path/to/foo.c /buildid/BUILDID/source/path/to/foo.c
339 ../bar/foo.c AT_comp_dir=/zoo/ /buildid/BUILDID/source/zoo//../bar/foo.c
340 Note: the client should %-escape characters in /SOURCE/FILE that are
342 not shown as "unreserved" in section 2.3 of RFC3986. Some characters
343 that will be escaped include "+", "\", "$", "!", the 'space' character,
344 and ";". RFC3986 includes a more comprehensive list of these charac‐
345 ters.
346 /buildid/BUILDID/section/SECTION
349 If the given buildid is known to the server, the server will attempt to
350 extract the contents of an ELF/DWARF section named SECTION from the de‐
351 buginfo file matching BUILDID. If the debuginfo file can't be found or
352 the section has type SHT_NOBITS, then the server will attempt to ex‐
353 tract the section from the executable matching BUILDID. If the section
354 is succesfully extracted then this request results in a binary object
355 of the section's contents. Note that this result is the raw binary
356 contents of the section, not an ELF file.
357 /metrics
360 This endpoint returns a Prometheus formatted text/plain dump of a vari‐
361 ety of statistics about the operation of the debuginfod server. The
362 exact set of metrics and their meanings may change in future versions.
363 Caution: configuration information (path names, versions) may be dis‐
364 closed.
365
368 debuginfod stores its index in an sqlite database in a densely packed
369 set of interlinked tables. While the representation is as efficient as
370 we have been able to make it, it still takes a considerable amount of
371 data to record all debuginfo-related data of potentially a great many
372 files. This section offers some advice about the implications.
373 As a general explanation for size, consider that debuginfod indexes
375 ELF/DWARF files, it stores their names and referenced source file
376 names, and buildids will be stored. When indexing archives, it stores
377 every file name of or in an archive, every buildid, plus every source
378 file name referenced from a DWARF file. (Indexing archives takes more
379 space because the source files often reside in separate subpackages
380 that may not be indexed at the same pass, so extra metadata has to be
381 kept.)
382 Getting down to numbers, in the case of Fedora RPMs (essentially, gzip-
384 compressed cpio files), the sqlite index database tends to be from 0.5%
385 to 3% of their size. It's larger for binaries that are assembled out
386 of a great many source files, or packages that carry much debuginfo-un‐
387 related content. It may be even larger during the indexing phase due
388 to temporary sqlite write-ahead-logging files; these are checkpointed
389 (cleaned out and removed) at shutdown. It may be helpful to apply
390 tight -I or -X regular-expression constraints to exclude files from
391 scanning that you know have no debuginfo-relevant content.
392 As debuginfod runs in normal active mode, it periodically rescans its
394 target directories, and any new content found is added to the database.
395 Old content, such as data for files that have disappeared or that have
396 been replaced with newer versions is removed at a periodic grooming
397 pass. This means that the sqlite files grow fast during initial index‐
398 ing, slowly during index rescans, and periodically shrink during groom‐
399 ing. There is also an optional one-shot maximal grooming pass is
400 available. It removes information debuginfo-unrelated data from the
401 archive content index such as file names found in archives ("archive
402 sdef" records) that are not referred to as source files from any bina‐
403 ries find in archives ("archive sref" records). This can save consid‐
404 erable disk space. However, it is slow and temporarily requires up to
405 twice the database size as free space. Worse: it may result in missing
406 source-code info if the archive traversals were interrupted, so that
407 not all source file references were known. Use it rarely to polish a
408 complete index.
409 You should ensure that ample disk space remains available. (The flood
411 of error messages on -ENOSPC is ugly and nagging. But, like for most
412 other errors, debuginfod will resume when resources permit.) If neces‐
413 sary, debuginfod can be stopped, the database file moved or removed,
414 and debuginfod restarted.
415 sqlite offers several performance-related options in the form of prag‐
417 mas. Some may be useful to fine-tune the defaults plus the debuginfod
418 extras. The -D option may be useful to tell debuginfod to execute the
419 given bits of SQL after the basic schema creation commands. For exam‐
420 ple, the "synchronous", "cache_size", "auto_vacuum", "threads", "jour‐
421 nal_mode" pragmas may be fun to tweak via -D, if you're searching for
422 peak performance. The "optimize", "wal_checkpoint" pragmas may be use‐
423 ful to run periodically, outside debuginfod. The default settings are
424 performance- rather than reliability-oriented, so a hardware crash
425 might corrupt the database. In these cases, it may be necessary to
426 manually delete the sqlite database and start over.
427 As debuginfod changes in the future, we may have no choice but to
429 change the database schema in an incompatible manner. If this happens,
430 new versions of debuginfod will issue SQL statements to drop all prior
431 schema & data, and start over. So, disk space will not be wasted for
432 retaining a no-longer-useable dataset.
433 In summary, if your system can bear a 0.5%-3% index-to-archive-dataset
435 size ratio, and slow growth afterwards, you should not need to worry
436 about disk space. If a system crash corrupts the database, or you want
437 to force debuginfod to reset and start over, simply erase the sqlite
438 file before restarting debuginfod.
439 In contrast, in passive mode, all scanning and grooming is disabled,
441 and the index database remains read-only. This makes the database more
442 suitable for sharing between servers or sites with simple one-way
443 replication, and data management considerations are generally moot.
444
447 debuginfod does not include any particular security features. While it
448 is robust with respect to inputs, some abuse is possible. It forks a
449 new thread for each incoming HTTP request, which could lead to a de‐
450 nial-of-service in terms of RAM, CPU, disk I/O, or network I/O. If
451 this is a problem, users are advised to install debuginfod with a HTTPS
452 reverse-proxy front-end that enforces site policies for firewalling,
453 authentication, integrity, authorization, and load control. The /met‐
454 rics webapi endpoint is probably not appropriate for disclosure to the
455 public.
456 When relaying queries to upstream debuginfods, debuginfod does not in‐
458 clude any particular security features. It trusts that the binaries
459 returned by the debuginfods are accurate. Therefore, the list of
460 servers should include only trustworthy ones. If accessed across HTTP
461 rather than HTTPS, the network should be trustworthy. Authentication
462 information through the internal libcurl library is not currently en‐
463 abled.
464
469 $TMPDIR
470 This environment variable points to a file system to be used for
471 temporary files. The default is /tmp.
472 $DEBUGINFOD_URLS
475 This environment variable contains a list of URL prefixes for
476 trusted debuginfod instances. Alternate URL prefixes are sepa‐
477 rated by space. This environment variable may be set by
478 /etc/profile.d scripts reading /etc/debuginfod/*.urls files.
479 $DEBUGINFOD_CACHE_PATH
482 This environment variable governs the location of the cache
483 where downloaded files and cache-control files are kept. The
484 default directory is chosen based on other environment vari‐
485 ables, see below.
486 $DEBUGINFOD_PROGRESS
489 This environment variable governs the default progress function.
490 If set, and if a progressfn is not explicitly set, then the li‐
491 brary will configure a default progressfn. This function will
492 append a simple progress message periodically to stderr. The
493 default is no progress function output.
494 $DEBUGINFOD_VERBOSE
497 This environment variable governs the default file descriptor
498 for verbose output. If set, and if a verbose fd is not explic‐
499 itly set, then the verbose output will be produced on
500 STDERR_FILENO.
501 $DEBUGINFOD_RETRY_LIMIT
504 This environment variable governs the default limit of retry at‐
505 tempts. If a query failed with errno other than ENOENT, will
506 initiate several attempts within the limit.
507 $DEBUGINFOD_TIMEOUT
510 This environment variable governs the download commencing time‐
511 out for each debuginfod HTTP connection. A server that fails to
512 provide at least 100K of data within this many seconds is
513 skipped. The default is 90 seconds. (Zero or negative means "no
514 timeout".)
515 $DEBUGINFOD_MAXTIME
518 This environment variable dictates how long the client will wait
519 to complete the download a file found on a server in seconds. It
520 is best used to ensure that a file is downloaded quickly or be
521 rejected. The default is 0 (infinite time).
522 $DEBUGINFOD_MAXSIZE
525 This environment variable dictates the maximum size of a file to
526 download in bytes. This is best used if the user would like to
527 ensure only small files are downloaded. A value of 0 causes no
528 consideration for size, and the client may attempt to download a
529 file of any size. The default is 0 (infinite size).
530 $DEBUGINFOD_HEADERS_FILE
533 This environment variable points to a file that supplies headers
534 to outbound HTTP requests, one per line. The header lines
535 shouldn't end with CRLF, unless that's the system newline con‐
536 vention. Whitespace-only lines are skipped.
537
540 Before each query, the debuginfod client library checks for a need to
541 clean the cache. If it's time to clean, the library traverses the
542 cache directory and removes downloaded debuginfo-related artifacts and
543 newly empty directories, if they have not been accessed recently.
544 Control files are located directly under the cache directory. They
546 contain simple decimal numbers to set cache-related configuration pa‐
547 rameters. If the files do not exist, the client library creates the
548 files with the default parameter values as content.
549 After each query, the debuginfod client library deposits newly received
551 files into a directory & file that is named based on the build-id. A
552 failed query is also cached by a special file. The naming convention
553 used for these artifacts is deliberately undocumented.
554 $XDG_CACHE_HOME/debuginfod_client/
557 Default cache directory, if $XDG_CACHE_HOME is set.
558 $HOME/.cache/debuginfod_client/
561 Default cache directory, if $XDG_CACHE_HOME is not set.
562 $HOME/.debuginfod_client_cache/
565 Deprecated cache directory, used only if preexisting.
566 cache_clean_interval_s
569 This control file gives the interval between cache cleaning
570 rounds, in seconds. The default is 86400, one day. 0 means
571 "immediately".
572 max_unused_age_s
575 This control file sets how long unaccessed debuginfo-related
576 files are retained, in seconds. The default is 604800, one
577 week. 0 means "immediately".
578 cache_miss_s
581 This control file sets how long to remember a query failure, in
582 seconds. New queries for the same artifacts within this time
583 window are short-circuited (returning an immediate failure in‐
584 stead of sending a new query to servers). This accelerates
585 queries that probably would still fail. The default is 600, 10
586 minutes. 0 means "forget immediately".
587
590 $HOME/.debuginfod.sqlite
591 Default database file.
592
596 debuginfod-find(1) sqlite3(1)
597 https://prometheus.io/docs/instrumenting/exporters/
598 DEBUGINFOD(8)