1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user in‐
15 terface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially in parallel) and once
22 all tests have been run lit will print summary information on the num‐
23 ber of tests which passed or failed (see TEST STATUS RESULTS). The lit
24 program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are ex‐
31 ecuted (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 Users interested in the lit architecture or designing a lit testing im‐
39 plementation should see LIT INFRASTRUCTURE.
40
42 -h, --help
43 Show the lit help message.
44 -j N, --threads=N
46 Run N tests in parallel. By default, this is automatically cho‐
47 sen to match the number of detected available CPUs.
48 --config-prefix=NAME
50 Search for NAME.cfg and NAME.site.cfg when searching for test
51 suites, instead of lit.cfg and lit.site.cfg.
52 -D NAME[=VALUE], --param NAME[=VALUE]
54 Add a user defined parameter NAME with the given VALUE (or the
55 empty string if not given). The meaning and use of these param‐
56 eters is test suite dependent.
57
59 -q, --quiet
60 Suppress any output except for test failures.
61 -s, --succinct
63 Show less output, for example don't show information on tests
64 that pass.
65 -v, --verbose
67 Show more information on test failures, for example the entire
68 test output instead of just the test result.
69 -vv, --echo-all-commands
71 Echo all commands to stdout, as they are being executed. This
72 can be valuable for debugging test failures, as the last echoed
73 command will be the one which has failed. lit normally inserts
74 a no-op command (: in the case of bash) with argument 'RUN: at
75 line N' before each command pipeline, and this option also
76 causes those no-op commands to be echoed to stdout to help you
77 locate the source line of the failed command. This option im‐
78 plies --verbose.
79 -a, --show-all
81 Show more information about all tests, for example the entire
82 test commandline and output.
83 --no-progress-bar
85 Do not use curses based progress bar.
86 --show-unsupported
88 Show the names of unsupported tests.
89 --show-xfail
91 Show the names of tests that were expected to fail.
92
94 --path=PATH
95 Specify an additional PATH to use when searching for executables
96 in tests.
97 --vg Run individual tests under valgrind (using the memcheck tool).
99 The --error-exitcode argument for valgrind is used so that val‐
100 grind failures will cause the program to exit with a non-zero
101 status.
102 When this option is enabled, lit will also automatically provide
104 a "valgrind" feature that can be used to conditionally disable
105 (or expect failure in) certain tests.
106 --vg-arg=ARG
108 When --vg is used, specify an additional argument to pass to
109 valgrind itself.
110 --vg-leak
112 When --vg is used, enable memory leak checks. When this option
113 is enabled, lit will also automatically provide a "vg_leak" fea‐
114 ture that can be used to conditionally disable (or expect fail‐
115 ure in) certain tests.
116 --time-tests
118 Track the wall time individual tests take to execute and in‐
119 cludes the results in the summary output. This is useful for
120 determining which tests in a test suite take the most time to
121 execute. Note that this option is most useful with -j 1.
122
124 --max-tests=N
125 Run at most N tests and then terminate.
126 --max-time=N
128 Spend at most N seconds (approximately) running tests and then
129 terminate.
130 --shuffle
132 Run the tests in a random order.
133 --num-shards=M
135 Divide the set of selected tests into M equal-sized subsets or
136 "shards", and run only one of them. Must be used with the
137 --run-shard=N option, which selects the shard to run. The envi‐
138 ronment variable LIT_NUM_SHARDS can also be used in place of
139 this option. These two options provide a coarse mechanism for
140 paritioning large testsuites, for parallel execution on separate
141 machines (say in a large testing farm).
142 --run-shard=N
144 Select which shard to run, assuming the --num-shards=M option
145 was provided. The two options must be used together, and the
146 value of N must be in the range 1..M. The environment variable
147 LIT_RUN_SHARD can also be used in place of this option.
148 --filter=REGEXP
150 Run only those tests whose name matches the regular expression
151 specified in REGEXP. The environment variable LIT_FILTER can be
152 also used in place of this option, which is especially useful in
153 environments where the call to lit is issued indirectly.
154
156 --debug
157 Run lit in debug mode, for debugging configuration issues and
158 lit itself.
159 --show-suites
161 List the discovered test suites and exit.
162 --show-tests
164 List all of the discovered tests and exit.
165
167 lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
168 sults. Otherwise, it will exit with the status 0. Other exit codes
169 are used for non-test related failures (for example a user error or an
170 internal program error).
171
173 The inputs passed to lit can be either individual tests, or entire di‐
174 rectories or hierarchies of tests to run. When lit starts up, the
175 first thing it does is convert the inputs into a complete list of tests
176 to run as part of test discovery.
177 In the lit model, every test must exist inside some test suite. lit
179 resolves the inputs specified on the command line to test suites by
180 searching upwards from the input path until it finds a lit.cfg or
181 lit.site.cfg file. These files serve as both a marker of test suites
182 and as configuration files which lit loads in order to understand how
183 to find and run the tests inside the test suite.
184 Once lit has mapped the inputs into test suites it traverses the list
186 of inputs adding tests for individual files and recursively searching
187 for tests in directories.
188 This behavior makes it easy to specify a subset of tests to run, while
190 still allowing the test suite configuration to control exactly how
191 tests are interpreted. In addition, lit always identifies tests by the
192 test suite they are in, and their relative path inside the test suite.
193 For appropriately configured projects, this allows lit to provide con‐
194 venient and flexible support for out-of-tree builds.
195
197 Each test ultimately produces one of the following six results:
198 PASS
200 The test succeeded.
201 XFAIL
203 The test failed, but that is expected. This is used for test for‐
204 mats which allow specifying that a test does not currently work, but
205 wish to leave it in the test suite.
206 XPASS
208 The test succeeded, but it was expected to fail. This is used for
209 tests which were specified as expected to fail, but are now succeed‐
210 ing (generally because the feature they test was broken and has been
211 fixed).
212 FAIL
214 The test failed.
215 UNRESOLVED
217 The test result could not be determined. For example, this occurs
218 when the test could not be run, the test itself is invalid, or the
219 test was interrupted.
220 UNSUPPORTED
222 The test is not supported in this environment. This is used by test
223 formats which can report unsupported tests.
224 Depending on the test format tests may produce additional information
226 about their status (generally only for failures). See the OUTPUT OP‐
227 TIONS section for more information.
228
230 This section describes the lit testing architecture for users inter‐
231 ested in creating a new lit testing implementation, or extending an ex‐
232 isting one.
233 lit proper is primarily an infrastructure for discovering and running
235 arbitrary tests, and to expose a single convenient interface to these
236 tests. lit itself doesn't know how to run tests, rather this logic is
237 defined by test suites.
238 TEST SUITES
240 As described in TEST DISCOVERY, tests are always located inside a test
241 suite. Test suites serve to define the format of the tests they con‐
242 tain, the logic for finding those tests, and any additional information
243 to run the tests.
244 lit identifies test suites as directories containing lit.cfg or
246 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
247 tially discovered by recursively searching up the directory hierarchy
248 for all the input files passed on the command line. You can use
249 --show-suites to display the discovered test suites at startup.
250 Once a test suite is discovered, its config file is loaded. Config
252 files themselves are Python modules which will be executed. When the
253 config file is executed, two important global variables are predefined:
254 lit_config
256 The global lit configuration object (a LitConfig instance), which
257 defines the builtin test formats, global configuration parameters,
258 and other helper routines for implementing test configurations.
259 config
261 This is the config object (a TestingConfig instance) for the test
262 suite, which the config file is expected to populate. The following
263 variables are also available on the config object, some of which
264 must be set by the config and others are optional or predefined:
265 name [required] The name of the test suite, for use in reports and
267 diagnostics.
268 test_format [required] The test format object which will be used to
270 discover and run tests in the test suite. Generally this will be a
271 builtin test format available from the lit.formats module.
272 test_source_root The filesystem path to the test suite root. For
274 out-of-dir builds this is the directory that will be scanned for
275 tests.
276 test_exec_root For out-of-dir builds, the path to the test suite
278 root inside the object directory. This is where tests will be run
279 and temporary output files placed.
280 environment A dictionary representing the environment to use when
282 executing tests in the suite.
283 suffixes For lit test formats which scan directories for tests, this
285 variable is a list of suffixes to identify test files. Used by:
286 ShTest.
287 substitutions For lit test formats which substitute variables into a
289 test script, the list of substitutions to perform. Used by: ShTest.
290 unsupported Mark an unsupported directory, all tests within it will
292 be reported as unsupported. Used by: ShTest.
293 parent The parent configuration, this is the config object for the
295 directory containing the test suite, or None.
296 root The root configuration. This is the top-most lit configuration
298 in the project.
299 pipefail Normally a test using a shell pipe fails if any of the com‐
301 mands on the pipe fail. If this is not desired, setting this vari‐
302 able to false makes the test fail only if the last command in the
303 pipe fails.
304 available_features A set of features that can be used in XFAIL, RE‐
306 QUIRES, and UNSUPPORTED directives.
307 TEST DISCOVERY
309 Once test suites are located, lit recursively traverses the source di‐
310 rectory (following test_source_root) looking for tests. When lit en‐
311 ters a sub-directory, it first checks to see if a nested test suite is
312 defined in that directory. If so, it loads that test suite recur‐
313 sively, otherwise it instantiates a local test config for the directory
314 (see LOCAL CONFIGURATION FILES).
315 Tests are identified by the test suite they are contained within, and
317 the relative path inside that suite. Note that the relative path may
318 not refer to an actual file on disk; some test formats (such as
319 GoogleTest) define "virtual tests" which have a path that contains both
320 the path to the actual test file and a subpath to identify the virtual
321 test.
322 LOCAL CONFIGURATION FILES
324 When lit loads a subdirectory in a test suite, it instantiates a local
325 test configuration by cloning the configuration for the parent direc‐
326 tory --- the root of this configuration chain will always be a test
327 suite. Once the test configuration is cloned lit checks for a lit.lo‐
328 cal.cfg file in the subdirectory. If present, this file will be loaded
329 and can be used to specialize the configuration for each individual di‐
330 rectory. This facility can be used to define subdirectories of op‐
331 tional tests, or to change other configuration parameters --- for exam‐
332 ple, to change the test format, or the suffixes which identify test
333 files.
334 PRE-DEFINED SUBSTITUTIONS
336 lit provides various patterns that can be used with the RUN command.
337 These are defined in TestRunner.py. The base set of substitutions are:
338 ┌───────────┬────────────────────────────┐
342 │Macro │ Substitution │
343 ├───────────┼────────────────────────────┤
344 │%s │ source path (path to the │
345 │ │ file currently being run) │
346 ├───────────┼────────────────────────────┤
347 │%S │ source dir (directory of │
348 │ │ the file currently being │
349 │ │ run) │
350 ├───────────┼────────────────────────────┤
351 │%p │ same as %S │
352 ├───────────┼────────────────────────────┤
353 │%{pathsep} │ path separator │
354 ├───────────┼────────────────────────────┤
355 │%t │ temporary file name unique │
356 │ │ to the test │
357 ├───────────┼────────────────────────────┤
358 │%T │ temporary directory unique │
359 │ │ to the test │
360 ├───────────┼────────────────────────────┤
361 │%% │ % │
362 └───────────┴────────────────────────────┘
363 Other substitutions are provided that are variations on this base set
365 and further substitution patterns can be defined by each test module.
366 See the modules LOCAL CONFIGURATION FILES.
367 More detailed information on substitutions can be found in the LLVM
369 Testing Infrastructure Guide.
370 TEST RUN OUTPUT FORMAT
372 The lit output for a test run conforms to the following schema, in both
373 short and verbose modes (although in short mode no PASS lines will be
374 shown). This schema has been chosen to be relatively easy to reliably
375 parse by a machine (for example in buildbot log scraping), and for
376 other tools to generate.
377 Each test result is expected to appear on a line that matches:
379 <result code>: <test name> (<progress info>)
381 where <result-code> is a standard test result such as PASS, FAIL,
383 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
384 of IMPROVED and REGRESSED are also allowed.
385 The <test name> field can consist of an arbitrary string containing no
387 newline.
388 The <progress info> field can be used to report progress information
390 such as (1/300) or can be empty, but even when empty the parentheses
391 are required.
392 Each test result may include additional (multiline) log information in
394 the following format:
395 <log delineator> TEST '(<test name>)' <trailing delineator>
397 ... log message ...
398 <log delineator>
399 where <test name> should be the name of a preceding reported test, <log
401 delineator> is a string of "*" characters at least four characters long
402 (the recommended length is 20), and <trailing delineator> is an arbi‐
403 trary (unparsed) string.
404 The following is an example of a test run output which consists of four
406 tests A, B, C, and D, and a log message for the failing test C:
407 PASS: A (1 of 4)
409 PASS: B (2 of 4)
410 FAIL: C (3 of 4)
411 ******************** TEST 'C' FAILED ********************
412 Test 'C' failed as a result of exit code 1.
413 ********************
414 PASS: D (4 of 4)
415 LIT EXAMPLE TESTS
417 The lit distribution contains several example implementations of test
418 suites in the ExampleTests directory.
419
421 valgrind(1)
422
424 Maintained by The LLVM Team (http://llvm.org/).
425
427 2003-2023, LLVM Project
4287 2023-01-19 LIT(1)