1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user in‐
15 terface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially concurrently) and
22 once all tests have been run lit will print summary information on the
23 number of tests which passed or failed (see TEST STATUS RESULTS). The
24 lit program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are ex‐
31 ecuted (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 lit parses options from the environment variable LIT_OPTS after parsing
39 options from the command line. LIT_OPTS is primarily useful for sup‐
40 plementing or overriding the command-line options supplied to lit by
41 check targets defined by a project's build system.
42 lit can also read options from response files which are specified as
44 inputs using the @path/to/file.rsp syntax. Arguments read from a file
45 must be one per line and are treated as if they were in the same place
46 as the original file referencing argument on the command line. A re‐
47 sponse file can reference other response files.
48 Users interested in the lit architecture or designing a lit testing im‐
50 plementation should see LIT INFRASTRUCTURE.
51
53 -h, --help
54 Show the lit help message.
55 -j N, --workers=N
57 Run N tests in parallel. By default, this is automatically cho‐
58 sen to match the number of detected available CPUs.
59 --config-prefix=NAME
61 Search for NAME.cfg and NAME.site.cfg when searching for test
62 suites, instead of lit.cfg and lit.site.cfg.
63 -D NAME[=VALUE], --param NAME[=VALUE]
65 Add a user defined parameter NAME with the given VALUE (or the
66 empty string if not given). The meaning and use of these param‐
67 eters is test suite dependent.
68
70 -q, --quiet
71 Suppress any output except for test failures.
72 -s, --succinct
74 Show less output, for example don't show information on tests
75 that pass. Also show a progress bar, unless --no-progress-bar
76 is specified.
77 -v, --verbose
79 Show more information on test failures, for example the entire
80 test output instead of just the test result.
81 -vv, --echo-all-commands
83 On test failure, echo all commands to stdout as they are being
84 executed. This can be valuable for debugging test failures, as
85 the last echoed command will be the one which has failed. lit
86 normally inserts a no-op command (: in the case of bash) with
87 argument 'RUN: at line N' before each command pipeline, and this
88 option also causes those no-op commands to be echoed to stdout
89 to help you locate the source line of the failed command. This
90 option implies --verbose.
91 -a, --show-all
93 Show more information about all tests, for example the entire
94 test commandline and output.
95 --no-progress-bar
97 Do not use curses based progress bar.
98 --show-unsupported
100 Show the names of unsupported tests.
101 --show-xfail
103 Show the names of tests that were expected to fail.
104
106 --path=PATH
107 Specify an additional PATH to use when searching for executables
108 in tests.
109 --vg Run individual tests under valgrind (using the memcheck tool).
111 The --error-exitcode argument for valgrind is used so that val‐
112 grind failures will cause the program to exit with a non-zero
113 status.
114 When this option is enabled, lit will also automatically provide
116 a "valgrind" feature that can be used to conditionally disable
117 (or expect failure in) certain tests.
118 --vg-arg=ARG
120 When --vg is used, specify an additional argument to pass to
121 valgrind itself.
122 --vg-leak
124 When --vg is used, enable memory leak checks. When this option
125 is enabled, lit will also automatically provide a "vg_leak" fea‐
126 ture that can be used to conditionally disable (or expect fail‐
127 ure in) certain tests.
128 --time-tests
130 Track the wall time individual tests take to execute and in‐
131 cludes the results in the summary output. This is useful for
132 determining which tests in a test suite take the most time to
133 execute.
134 --ignore-fail
136 Exit with status zero even if some tests fail.
137
139 By default, lit will run failing tests first, then run tests in de‐
140 scending execution time order to optimize concurrency. The execution
141 order can be changed using the --order option.
142 The timing data is stored in the test_exec_root in a file named
144 .lit_test_times.txt. If this file does not exist, then lit checks the
145 test_source_root for the file to optionally accelerate clean builds.
146 --shuffle
148 Run the tests in a random order, not failing/slowest first. Dep‐
149 recated, use --order instead.
150 --max-failures N
152 Stop execution after the given number N of failures. An integer
153 argument should be passed on the command line prior to execu‐
154 tion.
155 --max-tests=N
157 Run at most N tests and then terminate.
158 --max-time=N
160 Spend at most N seconds (approximately) running tests and then
161 terminate. Note that this is not an alias for --timeout; the
162 two are different kinds of maximums.
163 --num-shards=M
165 Divide the set of selected tests into M equal-sized subsets or
166 "shards", and run only one of them. Must be used with the
167 --run-shard=N option, which selects the shard to run. The envi‐
168 ronment variable LIT_NUM_SHARDS can also be used in place of
169 this option. These two options provide a coarse mechanism for
170 partitioning large testsuites, for parallel execution on sepa‐
171 rate machines (say in a large testing farm).
172 --order={lexical,random,smart}
174 Define the order in which tests are run. The supported values
175 are:
176 • lexical - tests will be run in lexical order according to the
178 test file path. This option is useful when predictable test
179 order is desired.
180 • random - tests will be run in random order.
182 • smart - tests that failed previously will be run first, then
184 the remaining tests, all in descending execution time order.
185 This is the default as it optimizes concurrency.
186 --run-shard=N
188 Select which shard to run, assuming the --num-shards=M option
189 was provided. The two options must be used together, and the
190 value of N must be in the range 1..M. The environment variable
191 LIT_RUN_SHARD can also be used in place of this option.
192 --timeout=N
194 Spend at most N seconds (approximately) running each individual
195 test. 0 means no time limit, and 0 is the default. Note that
196 this is not an alias for --max-time; the two are different kinds
197 of maximums.
198 --filter=REGEXP
200 Run only those tests whose name matches the regular expression
201 specified in REGEXP. The environment variable LIT_FILTER can be
202 also used in place of this option, which is especially useful in
203 environments where the call to lit is issued indirectly.
204 --filter-out=REGEXP
206 Filter out those tests whose name matches the regular expression
207 specified in REGEXP. The environment variable LIT_FILTER_OUT can
208 be also used in place of this option, which is especially useful
209 in environments where the call to lit is issued indirectly.
210 --xfail=LIST
212 Treat those tests whose name is in the semicolon separated list
213 LIST as XFAIL. This can be helpful when one does not want to
214 modify the test suite. The environment variable LIT_XFAIL can be
215 also used in place of this option, which is especially useful in
216 environments where the call to lit is issued indirectly.
217 A test name can specified as a file name relative to the test
219 suite directory. For example:
220 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
222 In this case, all of the following tests are treated as XFAIL:
224 libomp :: affinity/kmp-hw-subset.c
226 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
227 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
228 Alternatively, a test name can be specified as the full test
230 name reported in LIT output. For example, we can adjust the
231 previous example not to treat the nvptx64-nvidia-cuda version of
232 offloading/memory_manager.cpp as XFAIL:
233 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
235 --xfail-not=LIST
237 Do not treat the specified tests as XFAIL. The environment
238 variable LIT_XFAIL_NOT can also be used in place of this option.
239 The syntax is the same as for --xfail and LIT_XFAIL.
240 --xfail-not and LIT_XFAIL_NOT always override all other XFAIL
241 specifications, including an --xfail appearing later on the com‐
242 mand line. The primary purpose is to suppress an XPASS result
243 without modifying a test case that uses the XFAIL directive.
244
246 --debug
247 Run lit in debug mode, for debugging configuration issues and
248 lit itself.
249 --show-suites
251 List the discovered test suites and exit.
252 --show-tests
254 List all of the discovered tests and exit.
255
257 lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
258 sults. Otherwise, it will exit with the status 0. Other exit codes
259 are used for non-test related failures (for example a user error or an
260 internal program error).
261
263 The inputs passed to lit can be either individual tests, or entire di‐
264 rectories or hierarchies of tests to run. When lit starts up, the
265 first thing it does is convert the inputs into a complete list of tests
266 to run as part of test discovery.
267 In the lit model, every test must exist inside some test suite. lit
269 resolves the inputs specified on the command line to test suites by
270 searching upwards from the input path until it finds a lit.cfg or
271 lit.site.cfg file. These files serve as both a marker of test suites
272 and as configuration files which lit loads in order to understand how
273 to find and run the tests inside the test suite.
274 Once lit has mapped the inputs into test suites it traverses the list
276 of inputs adding tests for individual files and recursively searching
277 for tests in directories.
278 This behavior makes it easy to specify a subset of tests to run, while
280 still allowing the test suite configuration to control exactly how
281 tests are interpreted. In addition, lit always identifies tests by the
282 test suite they are in, and their relative path inside the test suite.
283 For appropriately configured projects, this allows lit to provide con‐
284 venient and flexible support for out-of-tree builds.
285
287 Each test ultimately produces one of the following eight results:
288 PASS
290 The test succeeded.
291 FLAKYPASS
293 The test succeeded after being re-run more than once. This only ap‐
294 plies to tests containing an ALLOW_RETRIES: annotation.
295 XFAIL
297 The test failed, but that is expected. This is used for test for‐
298 mats which allow specifying that a test does not currently work, but
299 wish to leave it in the test suite.
300 XPASS
302 The test succeeded, but it was expected to fail. This is used for
303 tests which were specified as expected to fail, but are now succeed‐
304 ing (generally because the feature they test was broken and has been
305 fixed).
306 FAIL
308 The test failed.
309 UNRESOLVED
311 The test result could not be determined. For example, this occurs
312 when the test could not be run, the test itself is invalid, or the
313 test was interrupted.
314 UNSUPPORTED
316 The test is not supported in this environment. This is used by test
317 formats which can report unsupported tests.
318 TIMEOUT
320 The test was run, but it timed out before it was able to complete.
321 This is considered a failure.
322 Depending on the test format tests may produce additional information
324 about their status (generally only for failures). See the OUTPUT OP‐
325 TIONS section for more information.
326
328 This section describes the lit testing architecture for users inter‐
329 ested in creating a new lit testing implementation, or extending an ex‐
330 isting one.
331 lit proper is primarily an infrastructure for discovering and running
333 arbitrary tests, and to expose a single convenient interface to these
334 tests. lit itself doesn't know how to run tests, rather this logic is
335 defined by test suites.
336 TEST SUITES
338 As described in TEST DISCOVERY, tests are always located inside a test
339 suite. Test suites serve to define the format of the tests they con‐
340 tain, the logic for finding those tests, and any additional information
341 to run the tests.
342 lit identifies test suites as directories containing lit.cfg or
344 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
345 tially discovered by recursively searching up the directory hierarchy
346 for all the input files passed on the command line. You can use
347 --show-suites to display the discovered test suites at startup.
348 Once a test suite is discovered, its config file is loaded. Config
350 files themselves are Python modules which will be executed. When the
351 config file is executed, two important global variables are predefined:
352 lit_config
354 The global lit configuration object (a LitConfig instance), which
355 defines the builtin test formats, global configuration parameters,
356 and other helper routines for implementing test configurations.
357 config
359 This is the config object (a TestingConfig instance) for the test
360 suite, which the config file is expected to populate. The following
361 variables are also available on the config object, some of which
362 must be set by the config and others are optional or predefined:
363 name [required] The name of the test suite, for use in reports and
365 diagnostics.
366 test_format [required] The test format object which will be used to
368 discover and run tests in the test suite. Generally this will be a
369 builtin test format available from the lit.formats module.
370 test_source_root The filesystem path to the test suite root. For
372 out-of-dir builds this is the directory that will be scanned for
373 tests.
374 test_exec_root For out-of-dir builds, the path to the test suite
376 root inside the object directory. This is where tests will be run
377 and temporary output files placed.
378 environment A dictionary representing the environment to use when
380 executing tests in the suite.
381 standalone_tests When true, mark a directory with tests expected to
383 be run standalone. Test discovery is disabled for that directory.
384 lit.suffixes and lit.excludes must be empty when this variable is
385 true.
386 suffixes For lit test formats which scan directories for tests, this
388 variable is a list of suffixes to identify test files. Used by:
389 ShTest.
390 substitutions For lit test formats which substitute variables into a
392 test script, the list of substitutions to perform. Used by: ShTest.
393 unsupported Mark an unsupported directory, all tests within it will
395 be reported as unsupported. Used by: ShTest.
396 parent The parent configuration, this is the config object for the
398 directory containing the test suite, or None.
399 root The root configuration. This is the top-most lit configuration
401 in the project.
402 pipefail Normally a test using a shell pipe fails if any of the com‐
404 mands on the pipe fail. If this is not desired, setting this vari‐
405 able to false makes the test fail only if the last command in the
406 pipe fails.
407 available_features A set of features that can be used in XFAIL, RE‐
409 QUIRES, and UNSUPPORTED directives.
410 TEST DISCOVERY
412 Once test suites are located, lit recursively traverses the source di‐
413 rectory (following test_source_root) looking for tests. When lit en‐
414 ters a sub-directory, it first checks to see if a nested test suite is
415 defined in that directory. If so, it loads that test suite recur‐
416 sively, otherwise it instantiates a local test config for the directory
417 (see LOCAL CONFIGURATION FILES).
418 Tests are identified by the test suite they are contained within, and
420 the relative path inside that suite. Note that the relative path may
421 not refer to an actual file on disk; some test formats (such as
422 GoogleTest) define "virtual tests" which have a path that contains both
423 the path to the actual test file and a subpath to identify the virtual
424 test.
425 LOCAL CONFIGURATION FILES
427 When lit loads a subdirectory in a test suite, it instantiates a local
428 test configuration by cloning the configuration for the parent direc‐
429 tory --- the root of this configuration chain will always be a test
430 suite. Once the test configuration is cloned lit checks for a lit.lo‐
431 cal.cfg file in the subdirectory. If present, this file will be loaded
432 and can be used to specialize the configuration for each individual di‐
433 rectory. This facility can be used to define subdirectories of op‐
434 tional tests, or to change other configuration parameters --- for exam‐
435 ple, to change the test format, or the suffixes which identify test
436 files.
437 SUBSTITUTIONS
439 lit allows patterns to be substituted inside RUN commands. It also pro‐
440 vides the following base set of substitutions, which are defined in
441 TestRunner.py:
442 ┌────────────────────────┬────────────────────────────┐
444 │Macro │ Substitution │
445 ├────────────────────────┼────────────────────────────┤
446 │%s │ source path (path to the │
447 │ │ file currently being run) │
448 ├────────────────────────┼────────────────────────────┤
449 │%S │ source dir (directory of │
450 │ │ the file currently being │
451 │ │ run) │
452 ├────────────────────────┼────────────────────────────┤
453 │%p │ same as %S │
454 ├────────────────────────┼────────────────────────────┤
455 │%{pathsep} │ path separator │
456 ├────────────────────────┼────────────────────────────┤
457 │%{fs-src-root} │ root component of file │
458 │ │ system paths pointing to │
459 │ │ the LLVM checkout │
460 └────────────────────────┴────────────────────────────┘
461 │%{fs-tmp-root} │ root component of file │
464 │ │ system paths pointing to │
465 │ │ the test's temporary di‐ │
466 │ │ rectory │
467 ├────────────────────────┼────────────────────────────┤
468 │%{fs-sep} │ file system path separator │
469 ├────────────────────────┼────────────────────────────┤
470 │%t │ temporary file name unique │
471 │ │ to the test │
472 ├────────────────────────┼────────────────────────────┤
473 │%basename_t │ The last path component of │
474 │ │ %t but without the .tmp │
475 │ │ extension │
476 ├────────────────────────┼────────────────────────────┤
477 │%T │ parent directory of %t │
478 │ │ (not unique, deprecated, │
479 │ │ do not use) │
480 ├────────────────────────┼────────────────────────────┤
481 │%% │ % │
482 ├────────────────────────┼────────────────────────────┤
483 │%/s │ %s but \ is replaced by / │
484 ├────────────────────────┼────────────────────────────┤
485 │%/S │ %S but \ is replaced by / │
486 ├────────────────────────┼────────────────────────────┤
487 │%/p │ %p but \ is replaced by / │
488 ├────────────────────────┼────────────────────────────┤
489 │%/t │ %t but \ is replaced by / │
490 ├────────────────────────┼────────────────────────────┤
491 │%/T │ %T but \ is replaced by / │
492 ├────────────────────────┼────────────────────────────┤
493 │%{/s:regex_replacement} │ %/s but escaped for use in │
494 │ │ the replacement of a s@@@ │
495 │ │ command in sed │
496 ├────────────────────────┼────────────────────────────┤
497 │%{/S:regex_replacement} │ %/S but escaped for use in │
498 │ │ the replacement of a s@@@ │
499 │ │ command in sed │
500 ├────────────────────────┼────────────────────────────┤
501 │%{/p:regex_replacement} │ %/p but escaped for use in │
502 │ │ the replacement of a s@@@ │
503 │ │ command in sed │
504 ├────────────────────────┼────────────────────────────┤
505 │%{/t:regex_replacement} │ %/t but escaped for use in │
506 │ │ the replacement of a s@@@ │
507 │ │ command in sed │
508 ├────────────────────────┼────────────────────────────┤
509 │%{/T:regex_replacement} │ %/T but escaped for use in │
510 │ │ the replacement of a s@@@ │
511 │ │ command in sed │
512 ├────────────────────────┼────────────────────────────┤
513 │%:s │ On Windows, %/s but a : is │
514 │ │ removed if its the second │
515 │ │ character. Otherwise, %s │
516 │ │ but with a single leading │
517 │ │ / removed. │
518 ├────────────────────────┼────────────────────────────┤
519 │%:S │ On Windows, %/S but a : is │
520 │ │ removed if its the second │
521 │ │ character. Otherwise, %S │
522 │ │ but with a single leading │
523 │ │ / removed. │
524 └────────────────────────┴────────────────────────────┘
525 │%:p │ On Windows, %/p but a : is │
530 │ │ removed if its the second │
531 │ │ character. Otherwise, %p │
532 │ │ but with a single leading │
533 │ │ / removed. │
534 ├────────────────────────┼────────────────────────────┤
535 │%:t │ On Windows, %/t but a : is │
536 │ │ removed if its the second │
537 │ │ character. Otherwise, %t │
538 │ │ but with a single leading │
539 │ │ / removed. │
540 ├────────────────────────┼────────────────────────────┤
541 │%:T │ On Windows, %/T but a : is │
542 │ │ removed if its the second │
543 │ │ character. Otherwise, %T │
544 │ │ but with a single leading │
545 │ │ / removed. │
546 └────────────────────────┴────────────────────────────┘
547 Other substitutions are provided that are variations on this base set
549 and further substitution patterns can be defined by each test module.
550 See the modules LOCAL CONFIGURATION FILES.
551 More detailed information on substitutions can be found in the LLVM
553 Testing Infrastructure Guide.
554 TEST RUN OUTPUT FORMAT
556 The lit output for a test run conforms to the following schema, in both
557 short and verbose modes (although in short mode no PASS lines will be
558 shown). This schema has been chosen to be relatively easy to reliably
559 parse by a machine (for example in buildbot log scraping), and for
560 other tools to generate.
561 Each test result is expected to appear on a line that matches:
563 <result code>: <test name> (<progress info>)
565 where <result-code> is a standard test result such as PASS, FAIL,
567 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
568 of IMPROVED and REGRESSED are also allowed.
569 The <test name> field can consist of an arbitrary string containing no
571 newline.
572 The <progress info> field can be used to report progress information
574 such as (1/300) or can be empty, but even when empty the parentheses
575 are required.
576 Each test result may include additional (multiline) log information in
578 the following format:
579 <log delineator> TEST '(<test name>)' <trailing delineator>
581 ... log message ...
582 <log delineator>
583 where <test name> should be the name of a preceding reported test, <log
585 delineator> is a string of "*" characters at least four characters long
586 (the recommended length is 20), and <trailing delineator> is an arbi‐
587 trary (unparsed) string.
588 The following is an example of a test run output which consists of four
590 tests A, B, C, and D, and a log message for the failing test C:
591 PASS: A (1 of 4)
593 PASS: B (2 of 4)
594 FAIL: C (3 of 4)
595 ******************** TEST 'C' FAILED ********************
596 Test 'C' failed as a result of exit code 1.
597 ********************
598 PASS: D (4 of 4)
599 LIT EXAMPLE TESTS
601 The lit distribution contains several example implementations of test
602 suites in the ExampleTests directory.
603
605 valgrind(1)
606
608 Maintained by the LLVM Team (https://llvm.org/).
609
611 2003-2023, LLVM Project
61217 2023-11-28 LIT(1)