1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user in‐
15 terface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially concurrently) and
22 once all tests have been run lit will print summary information on the
23 number of tests which passed or failed (see TEST STATUS RESULTS). The
24 lit program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are ex‐
31 ecuted (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 lit parses options from the environment variable LIT_OPTS after parsing
39 options from the command line. LIT_OPTS is primarily useful for sup‐
40 plementing or overriding the command-line options supplied to lit by
41 check targets defined by a project's build system.
42 lit can also read options from response files which are specified as
44 inputs using the @path/to/file.rsp syntax.
45 Users interested in the lit architecture or designing a lit testing im‐
47 plementation should see LIT INFRASTRUCTURE.
48
50 -h, --help
51 Show the lit help message.
52 -j N, --workers=N
54 Run N tests in parallel. By default, this is automatically cho‐
55 sen to match the number of detected available CPUs.
56 --config-prefix=NAME
58 Search for NAME.cfg and NAME.site.cfg when searching for test
59 suites, instead of lit.cfg and lit.site.cfg.
60 -D NAME[=VALUE], --param NAME[=VALUE]
62 Add a user defined parameter NAME with the given VALUE (or the
63 empty string if not given). The meaning and use of these param‐
64 eters is test suite dependent.
65
67 -q, --quiet
68 Suppress any output except for test failures.
69 -s, --succinct
71 Show less output, for example don't show information on tests
72 that pass. Also show a progress bar, unless --no-progress-bar
73 is specified.
74 -v, --verbose
76 Show more information on test failures, for example the entire
77 test output instead of just the test result.
78 -vv, --echo-all-commands
80 Echo all commands to stdout, as they are being executed. This
81 can be valuable for debugging test failures, as the last echoed
82 command will be the one which has failed. lit normally inserts
83 a no-op command (: in the case of bash) with argument 'RUN: at
84 line N' before each command pipeline, and this option also
85 causes those no-op commands to be echoed to stdout to help you
86 locate the source line of the failed command. This option im‐
87 plies --verbose.
88 -a, --show-all
90 Show more information about all tests, for example the entire
91 test commandline and output.
92 --no-progress-bar
94 Do not use curses based progress bar.
95 --show-unsupported
97 Show the names of unsupported tests.
98 --show-xfail
100 Show the names of tests that were expected to fail.
101
103 --path=PATH
104 Specify an additional PATH to use when searching for executables
105 in tests.
106 --vg Run individual tests under valgrind (using the memcheck tool).
108 The --error-exitcode argument for valgrind is used so that val‐
109 grind failures will cause the program to exit with a non-zero
110 status.
111 When this option is enabled, lit will also automatically provide
113 a "valgrind" feature that can be used to conditionally disable
114 (or expect failure in) certain tests.
115 --vg-arg=ARG
117 When --vg is used, specify an additional argument to pass to
118 valgrind itself.
119 --vg-leak
121 When --vg is used, enable memory leak checks. When this option
122 is enabled, lit will also automatically provide a "vg_leak" fea‐
123 ture that can be used to conditionally disable (or expect fail‐
124 ure in) certain tests.
125 --time-tests
127 Track the wall time individual tests take to execute and in‐
128 cludes the results in the summary output. This is useful for
129 determining which tests in a test suite take the most time to
130 execute.
131 --ignore-fail
133 Exit with status zero even if some tests fail.
134 --no-indirectly-run-check
136 Do not error if a test would not be run if the user had speci‐
137 fied the containing directory instead of naming the test di‐
138 rectly.
139
141 By default, lit will run failing tests first, then run tests in de‐
142 scending execution time order to optimize concurrency. The execution
143 order can be changed using the --order option.
144 The timing data is stored in the test_exec_root in a file named
146 .lit_test_times.txt. If this file does not exist, then lit checks the
147 test_source_root for the file to optionally accelerate clean builds.
148 --shuffle
150 Run the tests in a random order, not failing/slowest first. Dep‐
151 recated, use --order instead.
152 --max-failures N
154 Stop execution after the given number N of failures. An integer
155 argument should be passed on the command line prior to execu‐
156 tion.
157 --max-tests=N
159 Run at most N tests and then terminate.
160 --max-time=N
162 Spend at most N seconds (approximately) running tests and then
163 terminate. Note that this is not an alias for --timeout; the
164 two are different kinds of maximums.
165 --num-shards=M
167 Divide the set of selected tests into M equal-sized subsets or
168 "shards", and run only one of them. Must be used with the
169 --run-shard=N option, which selects the shard to run. The envi‐
170 ronment variable LIT_NUM_SHARDS can also be used in place of
171 this option. These two options provide a coarse mechanism for
172 partitioning large testsuites, for parallel execution on sepa‐
173 rate machines (say in a large testing farm).
174 --order={lexical,random,smart}
176 Define the order in which tests are run. The supported values
177 are:
178 • lexical - tests will be run in lexical order according to the
180 test file path. This option is useful when predictable test
181 order is desired.
182 • random - tests will be run in random order.
184 • smart - tests that failed previously will be run first, then
186 the remaining tests, all in descending execution time order.
187 This is the default as it optimizes concurrency.
188 --run-shard=N
190 Select which shard to run, assuming the --num-shards=M option
191 was provided. The two options must be used together, and the
192 value of N must be in the range 1..M. The environment variable
193 LIT_RUN_SHARD can also be used in place of this option.
194 --timeout=N
196 Spend at most N seconds (approximately) running each individual
197 test. 0 means no time limit, and 0 is the default. Note that
198 this is not an alias for --max-time; the two are different kinds
199 of maximums.
200 --filter=REGEXP
202 Run only those tests whose name matches the regular expression
203 specified in REGEXP. The environment variable LIT_FILTER can be
204 also used in place of this option, which is especially useful in
205 environments where the call to lit is issued indirectly.
206 --filter-out=REGEXP
208 Filter out those tests whose name matches the regular expression
209 specified in REGEXP. The environment variable LIT_FILTER_OUT can
210 be also used in place of this option, which is especially useful
211 in environments where the call to lit is issued indirectly.
212 --xfail=LIST
214 Treat those tests whose name is in the semicolon separated list
215 LIST as XFAIL. This can be helpful when one does not want to
216 modify the test suite. The environment variable LIT_XFAIL can be
217 also used in place of this option, which is especially useful in
218 environments where the call to lit is issued indirectly.
219 A test name can specified as a file name relative to the test
221 suite directory. For example:
222 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
224 In this case, all of the following tests are treated as XFAIL:
226 libomp :: affinity/kmp-hw-subset.c
228 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
229 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
230 Alternatively, a test name can be specified as the full test
232 name reported in LIT output. For example, we can adjust the
233 previous example not to treat the nvptx64-nvidia-cuda version of
234 offloading/memory_manager.cpp as XFAIL:
235 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
237 --xfail-not=LIST
239 Do not treat the specified tests as XFAIL. The environment
240 variable LIT_XFAIL_NOT can also be used in place of this option.
241 The syntax is the same as for --xfail and LIT_XFAIL.
242 --xfail-not and LIT_XFAIL_NOT always override all other XFAIL
243 specifications, including an --xfail appearing later on the com‐
244 mand line. The primary purpose is to suppress an XPASS result
245 without modifying a test case that uses the XFAIL directive.
246
248 --debug
249 Run lit in debug mode, for debugging configuration issues and
250 lit itself.
251 --show-suites
253 List the discovered test suites and exit.
254 --show-tests
256 List all of the discovered tests and exit.
257
259 lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
260 sults. Otherwise, it will exit with the status 0. Other exit codes
261 are used for non-test related failures (for example a user error or an
262 internal program error).
263
265 The inputs passed to lit can be either individual tests, or entire di‐
266 rectories or hierarchies of tests to run. When lit starts up, the
267 first thing it does is convert the inputs into a complete list of tests
268 to run as part of test discovery.
269 In the lit model, every test must exist inside some test suite. lit
271 resolves the inputs specified on the command line to test suites by
272 searching upwards from the input path until it finds a lit.cfg or
273 lit.site.cfg file. These files serve as both a marker of test suites
274 and as configuration files which lit loads in order to understand how
275 to find and run the tests inside the test suite.
276 Once lit has mapped the inputs into test suites it traverses the list
278 of inputs adding tests for individual files and recursively searching
279 for tests in directories.
280 This behavior makes it easy to specify a subset of tests to run, while
282 still allowing the test suite configuration to control exactly how
283 tests are interpreted. In addition, lit always identifies tests by the
284 test suite they are in, and their relative path inside the test suite.
285 For appropriately configured projects, this allows lit to provide con‐
286 venient and flexible support for out-of-tree builds.
287
289 Each test ultimately produces one of the following eight results:
290 PASS
292 The test succeeded.
293 FLAKYPASS
295 The test succeeded after being re-run more than once. This only ap‐
296 plies to tests containing an ALLOW_RETRIES: annotation.
297 XFAIL
299 The test failed, but that is expected. This is used for test for‐
300 mats which allow specifying that a test does not currently work, but
301 wish to leave it in the test suite.
302 XPASS
304 The test succeeded, but it was expected to fail. This is used for
305 tests which were specified as expected to fail, but are now succeed‐
306 ing (generally because the feature they test was broken and has been
307 fixed).
308 FAIL
310 The test failed.
311 UNRESOLVED
313 The test result could not be determined. For example, this occurs
314 when the test could not be run, the test itself is invalid, or the
315 test was interrupted.
316 UNSUPPORTED
318 The test is not supported in this environment. This is used by test
319 formats which can report unsupported tests.
320 TIMEOUT
322 The test was run, but it timed out before it was able to complete.
323 This is considered a failure.
324 Depending on the test format tests may produce additional information
326 about their status (generally only for failures). See the OUTPUT OP‐
327 TIONS section for more information.
328
330 This section describes the lit testing architecture for users inter‐
331 ested in creating a new lit testing implementation, or extending an ex‐
332 isting one.
333 lit proper is primarily an infrastructure for discovering and running
335 arbitrary tests, and to expose a single convenient interface to these
336 tests. lit itself doesn't know how to run tests, rather this logic is
337 defined by test suites.
338 TEST SUITES
340 As described in TEST DISCOVERY, tests are always located inside a test
341 suite. Test suites serve to define the format of the tests they con‐
342 tain, the logic for finding those tests, and any additional information
343 to run the tests.
344 lit identifies test suites as directories containing lit.cfg or
346 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
347 tially discovered by recursively searching up the directory hierarchy
348 for all the input files passed on the command line. You can use
349 --show-suites to display the discovered test suites at startup.
350 Once a test suite is discovered, its config file is loaded. Config
352 files themselves are Python modules which will be executed. When the
353 config file is executed, two important global variables are predefined:
354 lit_config
356 The global lit configuration object (a LitConfig instance), which
357 defines the builtin test formats, global configuration parameters,
358 and other helper routines for implementing test configurations.
359 config
361 This is the config object (a TestingConfig instance) for the test
362 suite, which the config file is expected to populate. The following
363 variables are also available on the config object, some of which
364 must be set by the config and others are optional or predefined:
365 name [required] The name of the test suite, for use in reports and
367 diagnostics.
368 test_format [required] The test format object which will be used to
370 discover and run tests in the test suite. Generally this will be a
371 builtin test format available from the lit.formats module.
372 test_source_root The filesystem path to the test suite root. For
374 out-of-dir builds this is the directory that will be scanned for
375 tests.
376 test_exec_root For out-of-dir builds, the path to the test suite
378 root inside the object directory. This is where tests will be run
379 and temporary output files placed.
380 environment A dictionary representing the environment to use when
382 executing tests in the suite.
383 standalone_tests When true, mark a directory with tests expected to
385 be run standalone. Test discovery is disabled for that directory and
386 --no-indirectly-run-check is in effect. lit.suffixes and lit.ex‐
387 cludes must be empty when this variable is true.
388 suffixes For lit test formats which scan directories for tests, this
390 variable is a list of suffixes to identify test files. Used by:
391 ShTest.
392 substitutions For lit test formats which substitute variables into a
394 test script, the list of substitutions to perform. Used by: ShTest.
395 unsupported Mark an unsupported directory, all tests within it will
397 be reported as unsupported. Used by: ShTest.
398 parent The parent configuration, this is the config object for the
400 directory containing the test suite, or None.
401 root The root configuration. This is the top-most lit configuration
403 in the project.
404 pipefail Normally a test using a shell pipe fails if any of the com‐
406 mands on the pipe fail. If this is not desired, setting this vari‐
407 able to false makes the test fail only if the last command in the
408 pipe fails.
409 available_features A set of features that can be used in XFAIL, RE‐
411 QUIRES, and UNSUPPORTED directives.
412 TEST DISCOVERY
414 Once test suites are located, lit recursively traverses the source di‐
415 rectory (following test_source_root) looking for tests. When lit en‐
416 ters a sub-directory, it first checks to see if a nested test suite is
417 defined in that directory. If so, it loads that test suite recur‐
418 sively, otherwise it instantiates a local test config for the directory
419 (see LOCAL CONFIGURATION FILES).
420 Tests are identified by the test suite they are contained within, and
422 the relative path inside that suite. Note that the relative path may
423 not refer to an actual file on disk; some test formats (such as
424 GoogleTest) define "virtual tests" which have a path that contains both
425 the path to the actual test file and a subpath to identify the virtual
426 test.
427 LOCAL CONFIGURATION FILES
429 When lit loads a subdirectory in a test suite, it instantiates a local
430 test configuration by cloning the configuration for the parent direc‐
431 tory --- the root of this configuration chain will always be a test
432 suite. Once the test configuration is cloned lit checks for a lit.lo‐
433 cal.cfg file in the subdirectory. If present, this file will be loaded
434 and can be used to specialize the configuration for each individual di‐
435 rectory. This facility can be used to define subdirectories of op‐
436 tional tests, or to change other configuration parameters --- for exam‐
437 ple, to change the test format, or the suffixes which identify test
438 files.
439 SUBSTITUTIONS
441 lit allows patterns to be substituted inside RUN commands. It also pro‐
442 vides the following base set of substitutions, which are defined in
443 TestRunner.py:
444 ┌────────────────────────┬────────────────────────────┐
446 │Macro │ Substitution │
447 ├────────────────────────┼────────────────────────────┤
448 │%s │ source path (path to the │
449 │ │ file currently being run) │
450 ├────────────────────────┼────────────────────────────┤
451 │%S │ source dir (directory of │
452 │ │ the file currently being │
453 │ │ run) │
454 ├────────────────────────┼────────────────────────────┤
455 │%p │ same as %S │
456 ├────────────────────────┼────────────────────────────┤
457 │%{pathsep} │ path separator │
458 ├────────────────────────┼────────────────────────────┤
459 │%{fs-src-root} │ root component of file │
460 │ │ system paths pointing to │
461 │ │ the LLVM checkout │
462 ├────────────────────────┼────────────────────────────┤
463 │%{fs-tmp-root} │ root component of file │
464 │ │ system paths pointing to │
465 │ │ the test's temporary di‐ │
466 │ │ rectory │
467 ├────────────────────────┼────────────────────────────┤
468 │%{fs-sep} │ file system path separator │
469 ├────────────────────────┼────────────────────────────┤
470 │%t │ temporary file name unique │
471 │ │ to the test │
472 ├────────────────────────┼────────────────────────────┤
473 │%basename_t │ The last path component of │
474 │ │ %t but without the .tmp │
475 │ │ extension │
476 ├────────────────────────┼────────────────────────────┤
477 │%T │ parent directory of %t │
478 │ │ (not unique, deprecated, │
479 │ │ do not use) │
480 ├────────────────────────┼────────────────────────────┤
481 │%% │ % │
482 ├────────────────────────┼────────────────────────────┤
483 │%/s │ %s but \ is replaced by / │
484 ├────────────────────────┼────────────────────────────┤
485 │%/S │ %S but \ is replaced by / │
486 ├────────────────────────┼────────────────────────────┤
487 │%/p │ %p but \ is replaced by / │
488 ├────────────────────────┼────────────────────────────┤
489 │%/t │ %t but \ is replaced by / │
490 ├────────────────────────┼────────────────────────────┤
491 │%/T │ %T but \ is replaced by / │
492 └────────────────────────┴────────────────────────────┘
493 │%{/s:regex_replacement} │ %/s but escaped for use in │
495 │ │ the replacement of a s@@@ │
496 │ │ command in sed │
497 ├────────────────────────┼────────────────────────────┤
498 │%{/S:regex_replacement} │ %/S but escaped for use in │
499 │ │ the replacement of a s@@@ │
500 │ │ command in sed │
501 ├────────────────────────┼────────────────────────────┤
502 │%{/p:regex_replacement} │ %/p but escaped for use in │
503 │ │ the replacement of a s@@@ │
504 │ │ command in sed │
505 ├────────────────────────┼────────────────────────────┤
506 │%{/t:regex_replacement} │ %/t but escaped for use in │
507 │ │ the replacement of a s@@@ │
508 │ │ command in sed │
509 ├────────────────────────┼────────────────────────────┤
510 │%{/T:regex_replacement} │ %/T but escaped for use in │
511 │ │ the replacement of a s@@@ │
512 │ │ command in sed │
513 ├────────────────────────┼────────────────────────────┤
514 │%:s │ On Windows, %/s but a : is │
515 │ │ removed if its the second │
516 │ │ character. Otherwise, %s │
517 │ │ but with a single leading │
518 │ │ / removed. │
519 ├────────────────────────┼────────────────────────────┤
520 │%:S │ On Windows, %/S but a : is │
521 │ │ removed if its the second │
522 │ │ character. Otherwise, %S │
523 │ │ but with a single leading │
524 │ │ / removed. │
525 ├────────────────────────┼────────────────────────────┤
526 │%:p │ On Windows, %/p but a : is │
527 │ │ removed if its the second │
528 │ │ character. Otherwise, %p │
529 │ │ but with a single leading │
530 │ │ / removed. │
531 ├────────────────────────┼────────────────────────────┤
532 │%:t │ On Windows, %/t but a : is │
533 │ │ removed if its the second │
534 │ │ character. Otherwise, %t │
535 │ │ but with a single leading │
536 │ │ / removed. │
537 ├────────────────────────┼────────────────────────────┤
538 │%:T │ On Windows, %/T but a : is │
539 │ │ removed if its the second │
540 │ │ character. Otherwise, %T │
541 │ │ but with a single leading │
542 │ │ / removed. │
543 └────────────────────────┴────────────────────────────┘
544 Other substitutions are provided that are variations on this base set
546 and further substitution patterns can be defined by each test module.
547 See the modules LOCAL CONFIGURATION FILES.
548 By default, substitutions are expanded exactly once, so that if e.g. a
550 substitution %build is defined in top of another substitution %cxx,
551 %build will expand to %cxx textually, not to what %cxx expands to.
552 However, if the recursiveExpansionLimit property of the TestingConfig
553 is set to a non-negative integer, substitutions will be expanded recur‐
554 sively until that limit is reached. It is an error if the limit is
555 reached and expanding substitutions again would yield a different re‐
556 sult.
557 More detailed information on substitutions can be found in the LLVM
559 Testing Infrastructure Guide.
560 TEST RUN OUTPUT FORMAT
562 The lit output for a test run conforms to the following schema, in both
563 short and verbose modes (although in short mode no PASS lines will be
564 shown). This schema has been chosen to be relatively easy to reliably
565 parse by a machine (for example in buildbot log scraping), and for
566 other tools to generate.
567 Each test result is expected to appear on a line that matches:
569 <result code>: <test name> (<progress info>)
571 where <result-code> is a standard test result such as PASS, FAIL,
573 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
574 of IMPROVED and REGRESSED are also allowed.
575 The <test name> field can consist of an arbitrary string containing no
577 newline.
578 The <progress info> field can be used to report progress information
580 such as (1/300) or can be empty, but even when empty the parentheses
581 are required.
582 Each test result may include additional (multiline) log information in
584 the following format:
585 <log delineator> TEST '(<test name>)' <trailing delineator>
587 ... log message ...
588 <log delineator>
589 where <test name> should be the name of a preceding reported test, <log
591 delineator> is a string of "*" characters at least four characters long
592 (the recommended length is 20), and <trailing delineator> is an arbi‐
593 trary (unparsed) string.
594 The following is an example of a test run output which consists of four
596 tests A, B, C, and D, and a log message for the failing test C:
597 PASS: A (1 of 4)
599 PASS: B (2 of 4)
600 FAIL: C (3 of 4)
601 ******************** TEST 'C' FAILED ********************
602 Test 'C' failed as a result of exit code 1.
603 ********************
604 PASS: D (4 of 4)
605 LIT EXAMPLE TESTS
607 The lit distribution contains several example implementations of test
608 suites in the ExampleTests directory.
609
611 valgrind(1)
612
614 Maintained by the LLVM Team (https://llvm.org/).
615
617 2003-2023, LLVM Project
61815 2023-01-12 LIT(1)