1LIT(1) LLVM LIT(1)
2
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user
15 interface as possible.
16 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20 Each specified test will be executed (potentially in parallel) and once
22 all tests have been run lit will print summary information on the num‐
23 ber of tests which passed or failed (see TEST STATUS RESULTS). The lit
24 program will execute with a non-zero exit code if any tests fail.
25 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29 lit also includes a number of options for controlling how tests are
31 executed (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37 Users interested in the lit architecture or designing a lit testing
39 implementation should see LIT INFRASTRUCTURE.
40
42 -h, --help
43 Show the lit help message.
44 -j N, --threads=N
46 Run N tests in parallel. By default, this is automatically cho‐
47 sen to match the number of detected available CPUs.
48 --config-prefix=NAME
50 Search for NAME.cfg and NAME.site.cfg when searching for test
51 suites, instead of lit.cfg and lit.site.cfg.
52 -D NAME, -D NAME=VALUE, --param NAME, --param NAME=VALUE
54 Add a user defined parameter NAME with the given VALUE (or the
55 empty string if not given). The meaning and use of these param‐
56 eters is test suite dependent.
57
59 -q, --quiet
60 Suppress any output except for test failures.
61 -s, --succinct
63 Show less output, for example don't show information on tests
64 that pass.
65 -v, --verbose
67 Show more information on test failures, for example the entire
68 test output instead of just the test result.
69 -a, --show-all
71 Show more information about all tests, for example the entire
72 test commandline and output.
73 --no-progress-bar
75 Do not use curses based progress bar.
76 --show-unsupported
78 Show the names of unsupported tests.
79 --show-xfail
81 Show the names of tests that were expected to fail.
82
84 --path=PATH
85 Specify an additional PATH to use when searching for executables
86 in tests.
87 --vg Run individual tests under valgrind (using the memcheck tool).
89 The --error-exitcode argument for valgrind is used so that val‐
90 grind failures will cause the program to exit with a non-zero
91 status.
92 When this option is enabled, lit will also automatically provide
94 a "valgrind" feature that can be used to conditionally disable
95 (or expect failure in) certain tests.
96 --vg-arg=ARG
98 When --vg is used, specify an additional argument to pass to
99 valgrind itself.
100 --vg-leak
102 When --vg is used, enable memory leak checks. When this option
103 is enabled, lit will also automatically provide a "vg_leak" fea‐
104 ture that can be used to conditionally disable (or expect fail‐
105 ure in) certain tests.
106 --time-tests
108 Track the wall time individual tests take to execute and
109 includes the results in the summary output. This is useful for
110 determining which tests in a test suite take the most time to
111 execute. Note that this option is most useful with -j 1.
112
114 --max-tests=N
115 Run at most N tests and then terminate.
116 --max-time=N
118 Spend at most N seconds (approximately) running tests and then
119 terminate.
120 --shuffle
122 Run the tests in a random order.
123
125 --debug
126 Run lit in debug mode, for debugging configuration issues and
127 lit itself.
128 --show-suites
130 List the discovered test suites and exit.
131 --show-tests
133 List all of the discovered tests and exit.
134
136 lit will exit with an exit code of 1 if there are any FAIL or XPASS
137 results. Otherwise, it will exit with the status 0. Other exit codes
138 are used for non-test related failures (for example a user error or an
139 internal program error).
140
142 The inputs passed to lit can be either individual tests, or entire
143 directories or hierarchies of tests to run. When lit starts up, the
144 first thing it does is convert the inputs into a complete list of tests
145 to run as part of test discovery.
146 In the lit model, every test must exist inside some test suite. lit
148 resolves the inputs specified on the command line to test suites by
149 searching upwards from the input path until it finds a lit.cfg or
150 lit.site.cfg file. These files serve as both a marker of test suites
151 and as configuration files which lit loads in order to understand how
152 to find and run the tests inside the test suite.
153 Once lit has mapped the inputs into test suites it traverses the list
155 of inputs adding tests for individual files and recursively searching
156 for tests in directories.
157 This behavior makes it easy to specify a subset of tests to run, while
159 still allowing the test suite configuration to control exactly how
160 tests are interpreted. In addition, lit always identifies tests by the
161 test suite they are in, and their relative path inside the test suite.
162 For appropriately configured projects, this allows lit to provide con‐
163 venient and flexible support for out-of-tree builds.
164
166 Each test ultimately produces one of the following six results:
167 PASS
169 The test succeeded.
170 XFAIL
172 The test failed, but that is expected. This is used for test for‐
173 mats which allow specifying that a test does not currently work, but
174 wish to leave it in the test suite.
175 XPASS
177 The test succeeded, but it was expected to fail. This is used for
178 tests which were specified as expected to fail, but are now succeed‐
179 ing (generally because the feature they test was broken and has been
180 fixed).
181 FAIL
183 The test failed.
184 UNRESOLVED
186 The test result could not be determined. For example, this occurs
187 when the test could not be run, the test itself is invalid, or the
188 test was interrupted.
189 UNSUPPORTED
191 The test is not supported in this environment. This is used by test
192 formats which can report unsupported tests.
193 Depending on the test format tests may produce additional information
195 about their status (generally only for failures). See the OUTPUT
196 OPTIONS section for more information.
197
199 This section describes the lit testing architecture for users inter‐
200 ested in creating a new lit testing implementation, or extending an
201 existing one.
202 lit proper is primarily an infrastructure for discovering and running
204 arbitrary tests, and to expose a single convenient interface to these
205 tests. lit itself doesn't know how to run tests, rather this logic is
206 defined by test suites.
207 TEST SUITES
209 As described in TEST DISCOVERY, tests are always located inside a test
210 suite. Test suites serve to define the format of the tests they con‐
211 tain, the logic for finding those tests, and any additional information
212 to run the tests.
213 lit identifies test suites as directories containing lit.cfg or
215 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
216 tially discovered by recursively searching up the directory hierarchy
217 for all the input files passed on the command line. You can use
218 --show-suites to display the discovered test suites at startup.
219 Once a test suite is discovered, its config file is loaded. Config
221 files themselves are Python modules which will be executed. When the
222 config file is executed, two important global variables are predefined:
223 lit_config
225 The global lit configuration object (a LitConfig instance), which
226 defines the builtin test formats, global configuration parameters,
227 and other helper routines for implementing test configurations.
228 config
230 This is the config object (a TestingConfig instance) for the test
231 suite, which the config file is expected to populate. The following
232 variables are also available on the config object, some of which
233 must be set by the config and others are optional or predefined:
234 name [required] The name of the test suite, for use in reports and
236 diagnostics.
237 test_format [required] The test format object which will be used to
239 discover and run tests in the test suite. Generally this will be a
240 builtin test format available from the lit.formats module.
241 test_source_root The filesystem path to the test suite root. For
243 out-of-dir builds this is the directory that will be scanned for
244 tests.
245 test_exec_root For out-of-dir builds, the path to the test suite
247 root inside the object directory. This is where tests will be run
248 and temporary output files placed.
249 environment A dictionary representing the environment to use when
251 executing tests in the suite.
252 suffixes For lit test formats which scan directories for tests, this
254 variable is a list of suffixes to identify test files. Used by:
255 ShTest.
256 substitutions For lit test formats which substitute variables into a
258 test script, the list of substitutions to perform. Used by: ShTest.
259 unsupported Mark an unsupported directory, all tests within it will
261 be reported as unsupported. Used by: ShTest.
262 parent The parent configuration, this is the config object for the
264 directory containing the test suite, or None.
265 root The root configuration. This is the top-most lit configuration
267 in the project.
268 pipefail Normally a test using a shell pipe fails if any of the com‐
270 mands on the pipe fail. If this is not desired, setting this vari‐
271 able to false makes the test fail only if the last command in the
272 pipe fails.
273 TEST DISCOVERY
275 Once test suites are located, lit recursively traverses the source
276 directory (following test_source_root) looking for tests. When lit
277 enters a sub-directory, it first checks to see if a nested test suite
278 is defined in that directory. If so, it loads that test suite recur‐
279 sively, otherwise it instantiates a local test config for the directory
280 (see LOCAL CONFIGURATION FILES).
281 Tests are identified by the test suite they are contained within, and
283 the relative path inside that suite. Note that the relative path may
284 not refer to an actual file on disk; some test formats (such as
285 GoogleTest) define "virtual tests" which have a path that contains both
286 the path to the actual test file and a subpath to identify the virtual
287 test.
288 LOCAL CONFIGURATION FILES
290 When lit loads a subdirectory in a test suite, it instantiates a local
291 test configuration by cloning the configuration for the parent direc‐
292 tory --- the root of this configuration chain will always be a test
293 suite. Once the test configuration is cloned lit checks for a
294 lit.local.cfg file in the subdirectory. If present, this file will be
295 loaded and can be used to specialize the configuration for each indi‐
296 vidual directory. This facility can be used to define subdirectories
297 of optional tests, or to change other configuration parameters --- for
298 example, to change the test format, or the suffixes which identify test
299 files.
300 PRE-DEFINED SUBSTITUTIONS
302 lit provides various patterns that can be used with the RUN command.
303 These are defined in TestRunner.py.
304 ┌───────────┬────────────────────────────┐
306 │Macro │ Substitution │
307 ├───────────┼────────────────────────────┤
308 │%s │ source path (path to the │
309 │ │ file currently being run) │
310 ├───────────┼────────────────────────────┤
311 │%S │ source dir (directory of │
312 │ │ the file currently being │
313 │ │ run) │
314 ├───────────┼────────────────────────────┤
315 │%p │ same as %S │
316 ├───────────┼────────────────────────────┤
317 │%{pathsep} │ path separator │
318 ├───────────┼────────────────────────────┤
319 │%t │ temporary file name unique │
320 │ │ to the test │
321 ├───────────┼────────────────────────────┤
322 │%T │ temporary directory unique │
323 │ │ to the test │
324 ├───────────┼────────────────────────────┤
325 │%% │ % │
326 ├───────────┼────────────────────────────┤
327 │%/s │ same as %s but replace all │
328 │ │ / with \ │
329 ├───────────┼────────────────────────────┤
330 │%/S │ same as %S but replace all │
331 │ │ / with \ │
332 ├───────────┼────────────────────────────┤
333 │%/p │ same as %p but replace all │
334 │ │ / with \ │
335 ├───────────┼────────────────────────────┤
336 │%/t │ same as %t but replace all │
337 │ │ / with \ │
338 ├───────────┼────────────────────────────┤
339 │%/T │ same as %T but replace all │
340 │ │ / with \ │
341 └───────────┴────────────────────────────┘
342 Further substitution patterns might be defined by each test module.
344 See the modules LOCAL CONFIGURATION FILES.
345 More information on the testing infrastucture can be found in the
347 ../TestingGuide.
348 TEST RUN OUTPUT FORMAT
350 The lit output for a test run conforms to the following schema, in both
351 short and verbose modes (although in short mode no PASS lines will be
352 shown). This schema has been chosen to be relatively easy to reliably
353 parse by a machine (for example in buildbot log scraping), and for
354 other tools to generate.
355 Each test result is expected to appear on a line that matches:
357 <result code>: <test name> (<progress info>)
359 where <result-code> is a standard test result such as PASS, FAIL,
361 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
362 of IMPROVED and REGRESSED are also allowed.
363 The <test name> field can consist of an arbitrary string containing no
365 newline.
366 The <progress info> field can be used to report progress information
368 such as (1/300) or can be empty, but even when empty the parentheses
369 are required.
370 Each test result may include additional (multiline) log information in
372 the following format:
373 <log delineator> TEST '(<test name>)' <trailing delineator>
375 ... log message ...
376 <log delineator>
377 where <test name> should be the name of a preceding reported test, <log
379 delineator> is a string of "*" characters at least four characters long
380 (the recommended length is 20), and <trailing delineator> is an arbi‐
381 trary (unparsed) string.
382 The following is an example of a test run output which consists of four
384 tests A, B, C, and D, and a log message for the failing test C:
385 PASS: A (1 of 4)
387 PASS: B (2 of 4)
388 FAIL: C (3 of 4)
389 ******************** TEST 'C' FAILED ********************
390 Test 'C' failed as a result of exit code 1.
391 ********************
392 PASS: D (4 of 4)
393 LIT EXAMPLE TESTS
395 The lit distribution contains several example implementations of test
396 suites in the ExampleTests directory.
397
399 valgrind(1)
400
402 Maintained by The LLVM Team (http://llvm.org/).
403
405 2003-2019, LLVM Project
4063.9 2019-02-01 LIT(1)