1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user
15 interface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially in parallel) and once
22 all tests have been run lit will print summary information on the num‐
23 ber of tests which passed or failed (see TEST STATUS RESULTS). The lit
24 program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are
31 executed (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 lit parses options from the environment variable LIT_OPTS after parsing
39 options from the command line. LIT_OPTS is primarily useful for sup‐
40 plementing or overriding the command-line options supplied to lit by
41 check targets defined by a project's build system.
42 Users interested in the lit architecture or designing a lit testing
44 implementation should see LIT INFRASTRUCTURE.
45
47 -h, --help
48 Show the lit help message.
49 -j N, --workers=N
51 Run N tests in parallel. By default, this is automatically cho‐
52 sen to match the number of detected available CPUs.
53 --config-prefix=NAME
55 Search for NAME.cfg and NAME.site.cfg when searching for test
56 suites, instead of lit.cfg and lit.site.cfg.
57 -D NAME[=VALUE], --param NAME[=VALUE]
59 Add a user defined parameter NAME with the given VALUE (or the
60 empty string if not given). The meaning and use of these param‐
61 eters is test suite dependent.
62
64 -q, --quiet
65 Suppress any output except for test failures.
66 -s, --succinct
68 Show less output, for example don't show information on tests
69 that pass.
70 -v, --verbose
72 Show more information on test failures, for example the entire
73 test output instead of just the test result.
74 -vv, --echo-all-commands
76 Echo all commands to stdout, as they are being executed. This
77 can be valuable for debugging test failures, as the last echoed
78 command will be the one which has failed. lit normally inserts
79 a no-op command (: in the case of bash) with argument 'RUN: at
80 line N' before each command pipeline, and this option also
81 causes those no-op commands to be echoed to stdout to help you
82 locate the source line of the failed command. This option
83 implies --verbose.
84 -a, --show-all
86 Show more information about all tests, for example the entire
87 test commandline and output.
88 --no-progress-bar
90 Do not use curses based progress bar.
91 --show-unsupported
93 Show the names of unsupported tests.
94 --show-xfail
96 Show the names of tests that were expected to fail.
97
99 --path=PATH
100 Specify an additional PATH to use when searching for executables
101 in tests.
102 --vg Run individual tests under valgrind (using the memcheck tool).
104 The --error-exitcode argument for valgrind is used so that val‐
105 grind failures will cause the program to exit with a non-zero
106 status.
107 When this option is enabled, lit will also automatically provide
109 a "valgrind" feature that can be used to conditionally disable
110 (or expect failure in) certain tests.
111 --vg-arg=ARG
113 When --vg is used, specify an additional argument to pass to
114 valgrind itself.
115 --vg-leak
117 When --vg is used, enable memory leak checks. When this option
118 is enabled, lit will also automatically provide a "vg_leak" fea‐
119 ture that can be used to conditionally disable (or expect fail‐
120 ure in) certain tests.
121 --time-tests
123 Track the wall time individual tests take to execute and
124 includes the results in the summary output. This is useful for
125 determining which tests in a test suite take the most time to
126 execute. Note that this option is most useful with -j 1.
127
129 --max-tests=N
130 Run at most N tests and then terminate.
131 --max-time=N
133 Spend at most N seconds (approximately) running tests and then
134 terminate.
135 --shuffle
137 Run the tests in a random order.
138 --num-shards=M
140 Divide the set of selected tests into M equal-sized subsets or
141 "shards", and run only one of them. Must be used with the
142 --run-shard=N option, which selects the shard to run. The envi‐
143 ronment variable LIT_NUM_SHARDS can also be used in place of
144 this option. These two options provide a coarse mechanism for
145 paritioning large testsuites, for parallel execution on separate
146 machines (say in a large testing farm).
147 --run-shard=N
149 Select which shard to run, assuming the --num-shards=M option
150 was provided. The two options must be used together, and the
151 value of N must be in the range 1..M. The environment variable
152 LIT_RUN_SHARD can also be used in place of this option.
153 --filter=REGEXP
155 Run only those tests whose name matches the regular expression
156 specified in REGEXP. The environment variable LIT_FILTER can be
157 also used in place of this option, which is especially useful in
158 environments where the call to lit is issued indirectly.
159
161 --debug
162 Run lit in debug mode, for debugging configuration issues and
163 lit itself.
164 --show-suites
166 List the discovered test suites and exit.
167 --show-tests
169 List all of the discovered tests and exit.
170
172 lit will exit with an exit code of 1 if there are any FAIL or XPASS
173 results. Otherwise, it will exit with the status 0. Other exit codes
174 are used for non-test related failures (for example a user error or an
175 internal program error).
176
178 The inputs passed to lit can be either individual tests, or entire
179 directories or hierarchies of tests to run. When lit starts up, the
180 first thing it does is convert the inputs into a complete list of tests
181 to run as part of test discovery.
182 In the lit model, every test must exist inside some test suite. lit
184 resolves the inputs specified on the command line to test suites by
185 searching upwards from the input path until it finds a lit.cfg or
186 lit.site.cfg file. These files serve as both a marker of test suites
187 and as configuration files which lit loads in order to understand how
188 to find and run the tests inside the test suite.
189 Once lit has mapped the inputs into test suites it traverses the list
191 of inputs adding tests for individual files and recursively searching
192 for tests in directories.
193 This behavior makes it easy to specify a subset of tests to run, while
195 still allowing the test suite configuration to control exactly how
196 tests are interpreted. In addition, lit always identifies tests by the
197 test suite they are in, and their relative path inside the test suite.
198 For appropriately configured projects, this allows lit to provide con‐
199 venient and flexible support for out-of-tree builds.
200
202 Each test ultimately produces one of the following six results:
203 PASS
205 The test succeeded.
206 XFAIL
208 The test failed, but that is expected. This is used for test for‐
209 mats which allow specifying that a test does not currently work, but
210 wish to leave it in the test suite.
211 XPASS
213 The test succeeded, but it was expected to fail. This is used for
214 tests which were specified as expected to fail, but are now succeed‐
215 ing (generally because the feature they test was broken and has been
216 fixed).
217 FAIL
219 The test failed.
220 UNRESOLVED
222 The test result could not be determined. For example, this occurs
223 when the test could not be run, the test itself is invalid, or the
224 test was interrupted.
225 UNSUPPORTED
227 The test is not supported in this environment. This is used by test
228 formats which can report unsupported tests.
229 Depending on the test format tests may produce additional information
231 about their status (generally only for failures). See the OUTPUT
232 OPTIONS section for more information.
233
235 This section describes the lit testing architecture for users inter‐
236 ested in creating a new lit testing implementation, or extending an
237 existing one.
238 lit proper is primarily an infrastructure for discovering and running
240 arbitrary tests, and to expose a single convenient interface to these
241 tests. lit itself doesn't know how to run tests, rather this logic is
242 defined by test suites.
243 TEST SUITES
245 As described in TEST DISCOVERY, tests are always located inside a test
246 suite. Test suites serve to define the format of the tests they con‐
247 tain, the logic for finding those tests, and any additional information
248 to run the tests.
249 lit identifies test suites as directories containing lit.cfg or
251 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
252 tially discovered by recursively searching up the directory hierarchy
253 for all the input files passed on the command line. You can use
254 --show-suites to display the discovered test suites at startup.
255 Once a test suite is discovered, its config file is loaded. Config
257 files themselves are Python modules which will be executed. When the
258 config file is executed, two important global variables are predefined:
259 lit_config
261 The global lit configuration object (a LitConfig instance), which
262 defines the builtin test formats, global configuration parameters,
263 and other helper routines for implementing test configurations.
264 config
266 This is the config object (a TestingConfig instance) for the test
267 suite, which the config file is expected to populate. The following
268 variables are also available on the config object, some of which
269 must be set by the config and others are optional or predefined:
270 name [required] The name of the test suite, for use in reports and
272 diagnostics.
273 test_format [required] The test format object which will be used to
275 discover and run tests in the test suite. Generally this will be a
276 builtin test format available from the lit.formats module.
277 test_source_root The filesystem path to the test suite root. For
279 out-of-dir builds this is the directory that will be scanned for
280 tests.
281 test_exec_root For out-of-dir builds, the path to the test suite
283 root inside the object directory. This is where tests will be run
284 and temporary output files placed.
285 environment A dictionary representing the environment to use when
287 executing tests in the suite.
288 suffixes For lit test formats which scan directories for tests, this
290 variable is a list of suffixes to identify test files. Used by:
291 ShTest.
292 substitutions For lit test formats which substitute variables into a
294 test script, the list of substitutions to perform. Used by: ShTest.
295 unsupported Mark an unsupported directory, all tests within it will
297 be reported as unsupported. Used by: ShTest.
298 parent The parent configuration, this is the config object for the
300 directory containing the test suite, or None.
301 root The root configuration. This is the top-most lit configuration
303 in the project.
304 pipefail Normally a test using a shell pipe fails if any of the com‐
306 mands on the pipe fail. If this is not desired, setting this vari‐
307 able to false makes the test fail only if the last command in the
308 pipe fails.
309 available_features A set of features that can be used in XFAIL,
311 REQUIRES, and UNSUPPORTED directives.
312 TEST DISCOVERY
314 Once test suites are located, lit recursively traverses the source
315 directory (following test_source_root) looking for tests. When lit
316 enters a sub-directory, it first checks to see if a nested test suite
317 is defined in that directory. If so, it loads that test suite recur‐
318 sively, otherwise it instantiates a local test config for the directory
319 (see LOCAL CONFIGURATION FILES).
320 Tests are identified by the test suite they are contained within, and
322 the relative path inside that suite. Note that the relative path may
323 not refer to an actual file on disk; some test formats (such as
324 GoogleTest) define "virtual tests" which have a path that contains both
325 the path to the actual test file and a subpath to identify the virtual
326 test.
327 LOCAL CONFIGURATION FILES
329 When lit loads a subdirectory in a test suite, it instantiates a local
330 test configuration by cloning the configuration for the parent direc‐
331 tory --- the root of this configuration chain will always be a test
332 suite. Once the test configuration is cloned lit checks for a
333 lit.local.cfg file in the subdirectory. If present, this file will be
334 loaded and can be used to specialize the configuration for each indi‐
335 vidual directory. This facility can be used to define subdirectories
336 of optional tests, or to change other configuration parameters --- for
337 example, to change the test format, or the suffixes which identify test
338 files.
339 PRE-DEFINED SUBSTITUTIONS
341 lit provides various patterns that can be used with the RUN command.
342 These are defined in TestRunner.py. The base set of substitutions are:
343 ┌────────────────────────┬────────────────────────────┐
345 │Macro │ Substitution │
346 ├────────────────────────┼────────────────────────────┤
347 │%s │ source path (path to the │
348 │ │ file currently being run) │
349 ├────────────────────────┼────────────────────────────┤
350 │%S │ source dir (directory of │
351 │ │ the file currently being │
352 │ │ run) │
353 ├────────────────────────┼────────────────────────────┤
354 │%p │ same as %S │
355 ├────────────────────────┼────────────────────────────┤
356 │%{pathsep} │ path separator │
357 ├────────────────────────┼────────────────────────────┤
358 │%t │ temporary file name unique │
359 │ │ to the test │
360 ├────────────────────────┼────────────────────────────┤
361 │%basename_t │ The last path component of │
362 │ │ %t but without the .tmp │
363 │ │ extension │
364 ├────────────────────────┼────────────────────────────┤
365 │%T │ parent directory of %t │
366 │ │ (not unique, deprecated, │
367 │ │ do not use) │
368 ├────────────────────────┼────────────────────────────┤
369 │%% │ % │
370 ├────────────────────────┼────────────────────────────┤
371 │%/s │ %s but \ is replaced by / │
372 ├────────────────────────┼────────────────────────────┤
373 │%/S │ %S but \ is replaced by / │
374 ├────────────────────────┼────────────────────────────┤
375 │%/p │ %p but \ is replaced by / │
376 ├────────────────────────┼────────────────────────────┤
377 │%/t │ %t but \ is replaced by / │
378 ├────────────────────────┼────────────────────────────┤
379 │%/T │ %T but \ is replaced by / │
380 ├────────────────────────┼────────────────────────────┤
381 │%{/s:regex_replacement} │ %/s but escaped for use in │
382 │ │ the replacement of a s@@@ │
383 │ │ command in sed │
384 ├────────────────────────┼────────────────────────────┤
385 │%{/S:regex_replacement} │ %/S but escaped for use in │
386 │ │ the replacement of a s@@@ │
387 │ │ command in sed │
388 ├────────────────────────┼────────────────────────────┤
389 │%{/p:regex_replacement} │ %/p but escaped for use in │
390 │ │ the replacement of a s@@@ │
391 │ │ command in sed │
392 ├────────────────────────┼────────────────────────────┤
393 │%{/t:regex_replacement} │ %/t but escaped for use in │
394 │ │ the replacement of a s@@@ │
395 │ │ command in sed │
396 ├────────────────────────┼────────────────────────────┤
397 │%{/T:regex_replacement} │ %/T but escaped for use in │
398 │ │ the replacement of a s@@@ │
399 │ │ command in sed │
400 ├────────────────────────┼────────────────────────────┤
401 │%:s │ On Windows, %/s but a : is │
402 │ │ removed if its the second │
403 │ │ character. Otherwise, %s │
404 │ │ but with a single leading │
405 │ │ / removed. │
406 └────────────────────────┴────────────────────────────┘
407 │%:S │ On Windows, %/S but a : is │
414 │ │ removed if its the second │
415 │ │ character. Otherwise, %S │
416 │ │ but with a single leading │
417 │ │ / removed. │
418 ├────────────────────────┼────────────────────────────┤
419 │%:p │ On Windows, %/p but a : is │
420 │ │ removed if its the second │
421 │ │ character. Otherwise, %p │
422 │ │ but with a single leading │
423 │ │ / removed. │
424 ├────────────────────────┼────────────────────────────┤
425 │%:t │ On Windows, %/t but a : is │
426 │ │ removed if its the second │
427 │ │ character. Otherwise, %t │
428 │ │ but with a single leading │
429 │ │ / removed. │
430 ├────────────────────────┼────────────────────────────┤
431 │%:T │ On Windows, %/T but a : is │
432 │ │ removed if its the second │
433 │ │ character. Otherwise, %T │
434 │ │ but with a single leading │
435 │ │ / removed. │
436 └────────────────────────┴────────────────────────────┘
437 Other substitutions are provided that are variations on this base set
439 and further substitution patterns can be defined by each test module.
440 See the modules LOCAL CONFIGURATION FILES.
441 More detailed information on substitutions can be found in the ../Test‐
443 ingGuide.
444 TEST RUN OUTPUT FORMAT
446 The lit output for a test run conforms to the following schema, in both
447 short and verbose modes (although in short mode no PASS lines will be
448 shown). This schema has been chosen to be relatively easy to reliably
449 parse by a machine (for example in buildbot log scraping), and for
450 other tools to generate.
451 Each test result is expected to appear on a line that matches:
453 <result code>: <test name> (<progress info>)
455 where <result-code> is a standard test result such as PASS, FAIL,
457 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
458 of IMPROVED and REGRESSED are also allowed.
459 The <test name> field can consist of an arbitrary string containing no
461 newline.
462 The <progress info> field can be used to report progress information
464 such as (1/300) or can be empty, but even when empty the parentheses
465 are required.
466 Each test result may include additional (multiline) log information in
468 the following format:
469 <log delineator> TEST '(<test name>)' <trailing delineator>
471 ... log message ...
472 <log delineator>
473 where <test name> should be the name of a preceding reported test, <log
475 delineator> is a string of "*" characters at least four characters long
476 (the recommended length is 20), and <trailing delineator> is an arbi‐
477 trary (unparsed) string.
478 The following is an example of a test run output which consists of four
480 tests A, B, C, and D, and a log message for the failing test C:
481 PASS: A (1 of 4)
483 PASS: B (2 of 4)
484 FAIL: C (3 of 4)
485 ******************** TEST 'C' FAILED ********************
486 Test 'C' failed as a result of exit code 1.
487 ********************
488 PASS: D (4 of 4)
489 LIT EXAMPLE TESTS
491 The lit distribution contains several example implementations of test
492 suites in the ExampleTests directory.
493
495 valgrind(1)
496
498 Maintained by the LLVM Team (https://llvm.org/).
499
501 2003-2020, LLVM Project
50210 2020-04-01 LIT(1)