1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user in‐
15 terface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially in parallel) and once
22 all tests have been run lit will print summary information on the num‐
23 ber of tests which passed or failed (see TEST STATUS RESULTS). The lit
24 program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are ex‐
31 ecuted (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 lit parses options from the environment variable LIT_OPTS after parsing
39 options from the command line. LIT_OPTS is primarily useful for sup‐
40 plementing or overriding the command-line options supplied to lit by
41 check targets defined by a project's build system.
42 Users interested in the lit architecture or designing a lit testing im‐
44 plementation should see LIT INFRASTRUCTURE.
45
47 -h, --help
48 Show the lit help message.
49 -j N, --workers=N
51 Run N tests in parallel. By default, this is automatically cho‐
52 sen to match the number of detected available CPUs.
53 --config-prefix=NAME
55 Search for NAME.cfg and NAME.site.cfg when searching for test
56 suites, instead of lit.cfg and lit.site.cfg.
57 -D NAME[=VALUE], --param NAME[=VALUE]
59 Add a user defined parameter NAME with the given VALUE (or the
60 empty string if not given). The meaning and use of these param‐
61 eters is test suite dependent.
62
64 -q, --quiet
65 Suppress any output except for test failures.
66 -s, --succinct
68 Show less output, for example don't show information on tests
69 that pass.
70 -v, --verbose
72 Show more information on test failures, for example the entire
73 test output instead of just the test result.
74 -vv, --echo-all-commands
76 Echo all commands to stdout, as they are being executed. This
77 can be valuable for debugging test failures, as the last echoed
78 command will be the one which has failed. lit normally inserts
79 a no-op command (: in the case of bash) with argument 'RUN: at
80 line N' before each command pipeline, and this option also
81 causes those no-op commands to be echoed to stdout to help you
82 locate the source line of the failed command. This option im‐
83 plies --verbose.
84 -a, --show-all
86 Show more information about all tests, for example the entire
87 test commandline and output.
88 --no-progress-bar
90 Do not use curses based progress bar.
91 --show-unsupported
93 Show the names of unsupported tests.
94 --show-xfail
96 Show the names of tests that were expected to fail.
97
99 --path=PATH
100 Specify an additional PATH to use when searching for executables
101 in tests.
102 --vg Run individual tests under valgrind (using the memcheck tool).
104 The --error-exitcode argument for valgrind is used so that val‐
105 grind failures will cause the program to exit with a non-zero
106 status.
107 When this option is enabled, lit will also automatically provide
109 a "valgrind" feature that can be used to conditionally disable
110 (or expect failure in) certain tests.
111 --vg-arg=ARG
113 When --vg is used, specify an additional argument to pass to
114 valgrind itself.
115 --vg-leak
117 When --vg is used, enable memory leak checks. When this option
118 is enabled, lit will also automatically provide a "vg_leak" fea‐
119 ture that can be used to conditionally disable (or expect fail‐
120 ure in) certain tests.
121 --time-tests
123 Track the wall time individual tests take to execute and in‐
124 cludes the results in the summary output. This is useful for
125 determining which tests in a test suite take the most time to
126 execute. Note that this option is most useful with -j 1.
127
129 --max-failures N
130 Stop execution after the given number N of failures. An integer
131 argument should be passed on the command line prior to execu‐
132 tion.
133 --max-tests=N
135 Run at most N tests and then terminate.
136 --max-time=N
138 Spend at most N seconds (approximately) running tests and then
139 terminate. Note that this is not an alias for --timeout; the
140 two are different kinds of maximums.
141 --num-shards=M
143 Divide the set of selected tests into M equal-sized subsets or
144 "shards", and run only one of them. Must be used with the
145 --run-shard=N option, which selects the shard to run. The envi‐
146 ronment variable LIT_NUM_SHARDS can also be used in place of
147 this option. These two options provide a coarse mechanism for
148 partitioning large testsuites, for parallel execution on sepa‐
149 rate machines (say in a large testing farm).
150 --run-shard=N
152 Select which shard to run, assuming the --num-shards=M option
153 was provided. The two options must be used together, and the
154 value of N must be in the range 1..M. The environment variable
155 LIT_RUN_SHARD can also be used in place of this option.
156 --shuffle
158 Run the tests in a random order.
159 --timeout=N
161 Spend at most N seconds (approximately) running each individual
162 test. 0 means no time limit, and 0 is the default. Note that
163 this is not an alias for --max-time; the two are different kinds
164 of maximums.
165 --filter=REGEXP
167 Run only those tests whose name matches the regular expression
168 specified in REGEXP. The environment variable LIT_FILTER can be
169 also used in place of this option, which is especially useful in
170 environments where the call to lit is issued indirectly.
171
173 --debug
174 Run lit in debug mode, for debugging configuration issues and
175 lit itself.
176 --show-suites
178 List the discovered test suites and exit.
179 --show-tests
181 List all of the discovered tests and exit.
182
184 lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
185 sults. Otherwise, it will exit with the status 0. Other exit codes
186 are used for non-test related failures (for example a user error or an
187 internal program error).
188
190 The inputs passed to lit can be either individual tests, or entire di‐
191 rectories or hierarchies of tests to run. When lit starts up, the
192 first thing it does is convert the inputs into a complete list of tests
193 to run as part of test discovery.
194 In the lit model, every test must exist inside some test suite. lit
196 resolves the inputs specified on the command line to test suites by
197 searching upwards from the input path until it finds a lit.cfg or
198 lit.site.cfg file. These files serve as both a marker of test suites
199 and as configuration files which lit loads in order to understand how
200 to find and run the tests inside the test suite.
201 Once lit has mapped the inputs into test suites it traverses the list
203 of inputs adding tests for individual files and recursively searching
204 for tests in directories.
205 This behavior makes it easy to specify a subset of tests to run, while
207 still allowing the test suite configuration to control exactly how
208 tests are interpreted. In addition, lit always identifies tests by the
209 test suite they are in, and their relative path inside the test suite.
210 For appropriately configured projects, this allows lit to provide con‐
211 venient and flexible support for out-of-tree builds.
212
214 Each test ultimately produces one of the following eight results:
215 PASS
217 The test succeeded.
218 FLAKYPASS
220 The test succeeded after being re-run more than once. This only ap‐
221 plies to tests containing an ALLOW_RETRIES: annotation.
222 XFAIL
224 The test failed, but that is expected. This is used for test for‐
225 mats which allow specifying that a test does not currently work, but
226 wish to leave it in the test suite.
227 XPASS
229 The test succeeded, but it was expected to fail. This is used for
230 tests which were specified as expected to fail, but are now succeed‐
231 ing (generally because the feature they test was broken and has been
232 fixed).
233 FAIL
235 The test failed.
236 UNRESOLVED
238 The test result could not be determined. For example, this occurs
239 when the test could not be run, the test itself is invalid, or the
240 test was interrupted.
241 UNSUPPORTED
243 The test is not supported in this environment. This is used by test
244 formats which can report unsupported tests.
245 TIMEOUT
247 The test was run, but it timed out before it was able to complete.
248 This is considered a failure.
249 Depending on the test format tests may produce additional information
251 about their status (generally only for failures). See the OUTPUT OP‐
252 TIONS section for more information.
253
255 This section describes the lit testing architecture for users inter‐
256 ested in creating a new lit testing implementation, or extending an ex‐
257 isting one.
258 lit proper is primarily an infrastructure for discovering and running
260 arbitrary tests, and to expose a single convenient interface to these
261 tests. lit itself doesn't know how to run tests, rather this logic is
262 defined by test suites.
263 TEST SUITES
265 As described in TEST DISCOVERY, tests are always located inside a test
266 suite. Test suites serve to define the format of the tests they con‐
267 tain, the logic for finding those tests, and any additional information
268 to run the tests.
269 lit identifies test suites as directories containing lit.cfg or
271 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
272 tially discovered by recursively searching up the directory hierarchy
273 for all the input files passed on the command line. You can use
274 --show-suites to display the discovered test suites at startup.
275 Once a test suite is discovered, its config file is loaded. Config
277 files themselves are Python modules which will be executed. When the
278 config file is executed, two important global variables are predefined:
279 lit_config
281 The global lit configuration object (a LitConfig instance), which
282 defines the builtin test formats, global configuration parameters,
283 and other helper routines for implementing test configurations.
284 config
286 This is the config object (a TestingConfig instance) for the test
287 suite, which the config file is expected to populate. The following
288 variables are also available on the config object, some of which
289 must be set by the config and others are optional or predefined:
290 name [required] The name of the test suite, for use in reports and
292 diagnostics.
293 test_format [required] The test format object which will be used to
295 discover and run tests in the test suite. Generally this will be a
296 builtin test format available from the lit.formats module.
297 test_source_root The filesystem path to the test suite root. For
299 out-of-dir builds this is the directory that will be scanned for
300 tests.
301 test_exec_root For out-of-dir builds, the path to the test suite
303 root inside the object directory. This is where tests will be run
304 and temporary output files placed.
305 environment A dictionary representing the environment to use when
307 executing tests in the suite.
308 suffixes For lit test formats which scan directories for tests, this
310 variable is a list of suffixes to identify test files. Used by:
311 ShTest.
312 substitutions For lit test formats which substitute variables into a
314 test script, the list of substitutions to perform. Used by: ShTest.
315 unsupported Mark an unsupported directory, all tests within it will
317 be reported as unsupported. Used by: ShTest.
318 parent The parent configuration, this is the config object for the
320 directory containing the test suite, or None.
321 root The root configuration. This is the top-most lit configuration
323 in the project.
324 pipefail Normally a test using a shell pipe fails if any of the com‐
326 mands on the pipe fail. If this is not desired, setting this vari‐
327 able to false makes the test fail only if the last command in the
328 pipe fails.
329 available_features A set of features that can be used in XFAIL, RE‐
331 QUIRES, and UNSUPPORTED directives.
332 TEST DISCOVERY
334 Once test suites are located, lit recursively traverses the source di‐
335 rectory (following test_source_root) looking for tests. When lit en‐
336 ters a sub-directory, it first checks to see if a nested test suite is
337 defined in that directory. If so, it loads that test suite recur‐
338 sively, otherwise it instantiates a local test config for the directory
339 (see LOCAL CONFIGURATION FILES).
340 Tests are identified by the test suite they are contained within, and
342 the relative path inside that suite. Note that the relative path may
343 not refer to an actual file on disk; some test formats (such as
344 GoogleTest) define "virtual tests" which have a path that contains both
345 the path to the actual test file and a subpath to identify the virtual
346 test.
347 LOCAL CONFIGURATION FILES
349 When lit loads a subdirectory in a test suite, it instantiates a local
350 test configuration by cloning the configuration for the parent direc‐
351 tory --- the root of this configuration chain will always be a test
352 suite. Once the test configuration is cloned lit checks for a lit.lo‐
353 cal.cfg file in the subdirectory. If present, this file will be loaded
354 and can be used to specialize the configuration for each individual di‐
355 rectory. This facility can be used to define subdirectories of op‐
356 tional tests, or to change other configuration parameters --- for exam‐
357 ple, to change the test format, or the suffixes which identify test
358 files.
359 SUBSTITUTIONS
361 lit allows patterns to be substituted inside RUN commands. It also pro‐
362 vides the following base set of substitutions, which are defined in
363 TestRunner.py:
364 ┌────────────────────────┬────────────────────────────┐
366 │Macro │ Substitution │
367 ├────────────────────────┼────────────────────────────┤
368 │%s │ source path (path to the │
369 │ │ file currently being run) │
370 ├────────────────────────┼────────────────────────────┤
371 │%S │ source dir (directory of │
372 │ │ the file currently being │
373 │ │ run) │
374 ├────────────────────────┼────────────────────────────┤
375 │%p │ same as %S │
376 ├────────────────────────┼────────────────────────────┤
377 │%{pathsep} │ path separator │
378 ├────────────────────────┼────────────────────────────┤
379 │%t │ temporary file name unique │
380 │ │ to the test │
381 ├────────────────────────┼────────────────────────────┤
382 │%basename_t │ The last path component of │
383 │ │ %t but without the .tmp │
384 │ │ extension │
385 ├────────────────────────┼────────────────────────────┤
386 │%T │ parent directory of %t │
387 │ │ (not unique, deprecated, │
388 │ │ do not use) │
389 ├────────────────────────┼────────────────────────────┤
390 │%% │ % │
391 ├────────────────────────┼────────────────────────────┤
392 │%/s │ %s but \ is replaced by / │
393 ├────────────────────────┼────────────────────────────┤
394 │%/S │ %S but \ is replaced by / │
395 ├────────────────────────┼────────────────────────────┤
396 │%/p │ %p but \ is replaced by / │
397 ├────────────────────────┼────────────────────────────┤
398 │%/t │ %t but \ is replaced by / │
399 ├────────────────────────┼────────────────────────────┤
400 │%/T │ %T but \ is replaced by / │
401 └────────────────────────┴────────────────────────────┘
402 │%{/s:regex_replacement} │ %/s but escaped for use in │
406 │ │ the replacement of a s@@@ │
407 │ │ command in sed │
408 ├────────────────────────┼────────────────────────────┤
409 │%{/S:regex_replacement} │ %/S but escaped for use in │
410 │ │ the replacement of a s@@@ │
411 │ │ command in sed │
412 ├────────────────────────┼────────────────────────────┤
413 │%{/p:regex_replacement} │ %/p but escaped for use in │
414 │ │ the replacement of a s@@@ │
415 │ │ command in sed │
416 ├────────────────────────┼────────────────────────────┤
417 │%{/t:regex_replacement} │ %/t but escaped for use in │
418 │ │ the replacement of a s@@@ │
419 │ │ command in sed │
420 ├────────────────────────┼────────────────────────────┤
421 │%{/T:regex_replacement} │ %/T but escaped for use in │
422 │ │ the replacement of a s@@@ │
423 │ │ command in sed │
424 ├────────────────────────┼────────────────────────────┤
425 │%:s │ On Windows, %/s but a : is │
426 │ │ removed if its the second │
427 │ │ character. Otherwise, %s │
428 │ │ but with a single leading │
429 │ │ / removed. │
430 ├────────────────────────┼────────────────────────────┤
431 │%:S │ On Windows, %/S but a : is │
432 │ │ removed if its the second │
433 │ │ character. Otherwise, %S │
434 │ │ but with a single leading │
435 │ │ / removed. │
436 ├────────────────────────┼────────────────────────────┤
437 │%:p │ On Windows, %/p but a : is │
438 │ │ removed if its the second │
439 │ │ character. Otherwise, %p │
440 │ │ but with a single leading │
441 │ │ / removed. │
442 ├────────────────────────┼────────────────────────────┤
443 │%:t │ On Windows, %/t but a : is │
444 │ │ removed if its the second │
445 │ │ character. Otherwise, %t │
446 │ │ but with a single leading │
447 │ │ / removed. │
448 ├────────────────────────┼────────────────────────────┤
449 │%:T │ On Windows, %/T but a : is │
450 │ │ removed if its the second │
451 │ │ character. Otherwise, %T │
452 │ │ but with a single leading │
453 │ │ / removed. │
454 └────────────────────────┴────────────────────────────┘
455 Other substitutions are provided that are variations on this base set
457 and further substitution patterns can be defined by each test module.
458 See the modules LOCAL CONFIGURATION FILES.
459 By default, substitutions are expanded exactly once, so that if e.g. a
461 substitution %build is defined in top of another substitution %cxx,
462 %build will expand to %cxx textually, not to what %cxx expands to.
463 However, if the recursiveExpansionLimit property of the TestingConfig
464 is set to a non-negative integer, substitutions will be expanded recur‐
465 sively until that limit is reached. It is an error if the limit is
466 reached and expanding substitutions again would yield a different re‐
467 sult.
468 More detailed information on substitutions can be found in the ../Test‐
470 ingGuide.
471 TEST RUN OUTPUT FORMAT
473 The lit output for a test run conforms to the following schema, in both
474 short and verbose modes (although in short mode no PASS lines will be
475 shown). This schema has been chosen to be relatively easy to reliably
476 parse by a machine (for example in buildbot log scraping), and for
477 other tools to generate.
478 Each test result is expected to appear on a line that matches:
480 <result code>: <test name> (<progress info>)
482 where <result-code> is a standard test result such as PASS, FAIL,
484 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
485 of IMPROVED and REGRESSED are also allowed.
486 The <test name> field can consist of an arbitrary string containing no
488 newline.
489 The <progress info> field can be used to report progress information
491 such as (1/300) or can be empty, but even when empty the parentheses
492 are required.
493 Each test result may include additional (multiline) log information in
495 the following format:
496 <log delineator> TEST '(<test name>)' <trailing delineator>
498 ... log message ...
499 <log delineator>
500 where <test name> should be the name of a preceding reported test, <log
502 delineator> is a string of "*" characters at least four characters long
503 (the recommended length is 20), and <trailing delineator> is an arbi‐
504 trary (unparsed) string.
505 The following is an example of a test run output which consists of four
507 tests A, B, C, and D, and a log message for the failing test C:
508 PASS: A (1 of 4)
510 PASS: B (2 of 4)
511 FAIL: C (3 of 4)
512 ******************** TEST 'C' FAILED ********************
513 Test 'C' failed as a result of exit code 1.
514 ********************
515 PASS: D (4 of 4)
516 LIT EXAMPLE TESTS
518 The lit distribution contains several example implementations of test
519 suites in the ExampleTests directory.
520
522 valgrind(1)
523
525 Maintained by the LLVM Team (https://llvm.org/).
526
528 2003-2021, LLVM Project
52911 2021-03-23 LIT(1)