1LIT(1) LLVM LIT(1)
2
3
4
6 lit - LLVM Integrated Tester
7
9 lit [options] [tests]
10
12 lit is a portable tool for executing LLVM and Clang style test suites,
13 summarizing their results, and providing indication of failures. lit
14 is designed to be a lightweight testing tool with as simple a user
15 interface as possible.
16
17 lit should be run with one or more tests to run specified on the com‐
18 mand line. Tests can be either individual test files or directories to
19 search for tests (see TEST DISCOVERY).
20
21 Each specified test will be executed (potentially in parallel) and once
22 all tests have been run lit will print summary information on the num‐
23 ber of tests which passed or failed (see TEST STATUS RESULTS). The lit
24 program will execute with a non-zero exit code if any tests fail.
25
26 By default lit will use a succinct progress display and will only print
27 summary information for test failures. See OUTPUT OPTIONS for options
28 controlling the lit progress display and output.
29
30 lit also includes a number of options for controlling how tests are
31 executed (specific features may depend on the particular test format).
32 See EXECUTION OPTIONS for more information.
33
34 Finally, lit also supports additional options for only running a subset
35 of the options specified on the command line, see SELECTION OPTIONS for
36 more information.
37
38 Users interested in the lit architecture or designing a lit testing
39 implementation should see LIT INFRASTRUCTURE.
40
42 -h, --help
43 Show the lit help message.
44
45 -j N, --threads=N
46 Run N tests in parallel. By default, this is automatically cho‐
47 sen to match the number of detected available CPUs.
48
49 --config-prefix=NAME
50 Search for NAME.cfg and NAME.site.cfg when searching for test
51 suites, instead of lit.cfg and lit.site.cfg.
52
53 -D NAME[=VALUE], --param NAME[=VALUE]
54 Add a user defined parameter NAME with the given VALUE (or the
55 empty string if not given). The meaning and use of these param‐
56 eters is test suite dependent.
57
59 -q, --quiet
60 Suppress any output except for test failures.
61
62 -s, --succinct
63 Show less output, for example don't show information on tests
64 that pass.
65
66 -v, --verbose
67 Show more information on test failures, for example the entire
68 test output instead of just the test result.
69
70 -vv, --echo-all-commands
71 Echo all commands to stdout, as they are being executed. This
72 can be valuable for debugging test failures, as the last echoed
73 command will be the one which has failed. This option implies
74 --verbose.
75
76 -a, --show-all
77 Show more information about all tests, for example the entire
78 test commandline and output.
79
80 --no-progress-bar
81 Do not use curses based progress bar.
82
83 --show-unsupported
84 Show the names of unsupported tests.
85
86 --show-xfail
87 Show the names of tests that were expected to fail.
88
90 --path=PATH
91 Specify an additional PATH to use when searching for executables
92 in tests.
93
94 --vg Run individual tests under valgrind (using the memcheck tool).
95 The --error-exitcode argument for valgrind is used so that val‐
96 grind failures will cause the program to exit with a non-zero
97 status.
98
99 When this option is enabled, lit will also automatically provide
100 a "valgrind" feature that can be used to conditionally disable
101 (or expect failure in) certain tests.
102
103 --vg-arg=ARG
104 When --vg is used, specify an additional argument to pass to
105 valgrind itself.
106
107 --vg-leak
108 When --vg is used, enable memory leak checks. When this option
109 is enabled, lit will also automatically provide a "vg_leak" fea‐
110 ture that can be used to conditionally disable (or expect fail‐
111 ure in) certain tests.
112
113 --time-tests
114 Track the wall time individual tests take to execute and
115 includes the results in the summary output. This is useful for
116 determining which tests in a test suite take the most time to
117 execute. Note that this option is most useful with -j 1.
118
120 --max-tests=N
121 Run at most N tests and then terminate.
122
123 --max-time=N
124 Spend at most N seconds (approximately) running tests and then
125 terminate.
126
127 --shuffle
128 Run the tests in a random order.
129
130 --num-shards=M
131 Divide the set of selected tests into M equal-sized subsets or
132 "shards", and run only one of them. Must be used with the
133 --run-shard=N option, which selects the shard to run. The envi‐
134 ronment variable LIT_NUM_SHARDS can also be used in place of
135 this option. These two options provide a coarse mechanism for
136 paritioning large testsuites, for parallel execution on separate
137 machines (say in a large testing farm).
138
139 --run-shard=N
140 Select which shard to run, assuming the --num-shards=M option
141 was provided. The two options must be used together, and the
142 value of N must be in the range 1..M. The environment variable
143 LIT_RUN_SHARD can also be used in place of this option.
144
145 --filter=REGEXP
146 Run only those tests whose name matches the regular expression
147 specified in REGEXP. The environment variable LIT_FILTER can be
148 also used in place of this option, which is especially useful in
149 environments where the call to lit is issued indirectly.
150
152 --debug
153 Run lit in debug mode, for debugging configuration issues and
154 lit itself.
155
156 --show-suites
157 List the discovered test suites and exit.
158
159 --show-tests
160 List all of the discovered tests and exit.
161
163 lit will exit with an exit code of 1 if there are any FAIL or XPASS
164 results. Otherwise, it will exit with the status 0. Other exit codes
165 are used for non-test related failures (for example a user error or an
166 internal program error).
167
169 The inputs passed to lit can be either individual tests, or entire
170 directories or hierarchies of tests to run. When lit starts up, the
171 first thing it does is convert the inputs into a complete list of tests
172 to run as part of test discovery.
173
174 In the lit model, every test must exist inside some test suite. lit
175 resolves the inputs specified on the command line to test suites by
176 searching upwards from the input path until it finds a lit.cfg or
177 lit.site.cfg file. These files serve as both a marker of test suites
178 and as configuration files which lit loads in order to understand how
179 to find and run the tests inside the test suite.
180
181 Once lit has mapped the inputs into test suites it traverses the list
182 of inputs adding tests for individual files and recursively searching
183 for tests in directories.
184
185 This behavior makes it easy to specify a subset of tests to run, while
186 still allowing the test suite configuration to control exactly how
187 tests are interpreted. In addition, lit always identifies tests by the
188 test suite they are in, and their relative path inside the test suite.
189 For appropriately configured projects, this allows lit to provide con‐
190 venient and flexible support for out-of-tree builds.
191
193 Each test ultimately produces one of the following six results:
194
195 PASS
196 The test succeeded.
197
198 XFAIL
199 The test failed, but that is expected. This is used for test for‐
200 mats which allow specifying that a test does not currently work, but
201 wish to leave it in the test suite.
202
203 XPASS
204 The test succeeded, but it was expected to fail. This is used for
205 tests which were specified as expected to fail, but are now succeed‐
206 ing (generally because the feature they test was broken and has been
207 fixed).
208
209 FAIL
210 The test failed.
211
212 UNRESOLVED
213 The test result could not be determined. For example, this occurs
214 when the test could not be run, the test itself is invalid, or the
215 test was interrupted.
216
217 UNSUPPORTED
218 The test is not supported in this environment. This is used by test
219 formats which can report unsupported tests.
220
221 Depending on the test format tests may produce additional information
222 about their status (generally only for failures). See the OUTPUT
223 OPTIONS section for more information.
224
226 This section describes the lit testing architecture for users inter‐
227 ested in creating a new lit testing implementation, or extending an
228 existing one.
229
230 lit proper is primarily an infrastructure for discovering and running
231 arbitrary tests, and to expose a single convenient interface to these
232 tests. lit itself doesn't know how to run tests, rather this logic is
233 defined by test suites.
234
235 TEST SUITES
236 As described in TEST DISCOVERY, tests are always located inside a test
237 suite. Test suites serve to define the format of the tests they con‐
238 tain, the logic for finding those tests, and any additional information
239 to run the tests.
240
241 lit identifies test suites as directories containing lit.cfg or
242 lit.site.cfg files (see also --config-prefix). Test suites are ini‐
243 tially discovered by recursively searching up the directory hierarchy
244 for all the input files passed on the command line. You can use
245 --show-suites to display the discovered test suites at startup.
246
247 Once a test suite is discovered, its config file is loaded. Config
248 files themselves are Python modules which will be executed. When the
249 config file is executed, two important global variables are predefined:
250
251 lit_config
252 The global lit configuration object (a LitConfig instance), which
253 defines the builtin test formats, global configuration parameters,
254 and other helper routines for implementing test configurations.
255
256 config
257 This is the config object (a TestingConfig instance) for the test
258 suite, which the config file is expected to populate. The following
259 variables are also available on the config object, some of which
260 must be set by the config and others are optional or predefined:
261
262 name [required] The name of the test suite, for use in reports and
263 diagnostics.
264
265 test_format [required] The test format object which will be used to
266 discover and run tests in the test suite. Generally this will be a
267 builtin test format available from the lit.formats module.
268
269 test_source_root The filesystem path to the test suite root. For
270 out-of-dir builds this is the directory that will be scanned for
271 tests.
272
273 test_exec_root For out-of-dir builds, the path to the test suite
274 root inside the object directory. This is where tests will be run
275 and temporary output files placed.
276
277 environment A dictionary representing the environment to use when
278 executing tests in the suite.
279
280 suffixes For lit test formats which scan directories for tests, this
281 variable is a list of suffixes to identify test files. Used by:
282 ShTest.
283
284 substitutions For lit test formats which substitute variables into a
285 test script, the list of substitutions to perform. Used by: ShTest.
286
287 unsupported Mark an unsupported directory, all tests within it will
288 be reported as unsupported. Used by: ShTest.
289
290 parent The parent configuration, this is the config object for the
291 directory containing the test suite, or None.
292
293 root The root configuration. This is the top-most lit configuration
294 in the project.
295
296 pipefail Normally a test using a shell pipe fails if any of the com‐
297 mands on the pipe fail. If this is not desired, setting this vari‐
298 able to false makes the test fail only if the last command in the
299 pipe fails.
300
301 available_features A set of features that can be used in XFAIL,
302 REQUIRES, and UNSUPPORTED directives.
303
304 TEST DISCOVERY
305 Once test suites are located, lit recursively traverses the source
306 directory (following test_source_root) looking for tests. When lit
307 enters a sub-directory, it first checks to see if a nested test suite
308 is defined in that directory. If so, it loads that test suite recur‐
309 sively, otherwise it instantiates a local test config for the directory
310 (see LOCAL CONFIGURATION FILES).
311
312 Tests are identified by the test suite they are contained within, and
313 the relative path inside that suite. Note that the relative path may
314 not refer to an actual file on disk; some test formats (such as
315 GoogleTest) define "virtual tests" which have a path that contains both
316 the path to the actual test file and a subpath to identify the virtual
317 test.
318
319 LOCAL CONFIGURATION FILES
320 When lit loads a subdirectory in a test suite, it instantiates a local
321 test configuration by cloning the configuration for the parent direc‐
322 tory --- the root of this configuration chain will always be a test
323 suite. Once the test configuration is cloned lit checks for a
324 lit.local.cfg file in the subdirectory. If present, this file will be
325 loaded and can be used to specialize the configuration for each indi‐
326 vidual directory. This facility can be used to define subdirectories
327 of optional tests, or to change other configuration parameters --- for
328 example, to change the test format, or the suffixes which identify test
329 files.
330
331 PRE-DEFINED SUBSTITUTIONS
332 lit provides various patterns that can be used with the RUN command.
333 These are defined in TestRunner.py. The base set of substitutions are:
334
335 ┌───────────┬────────────────────────────┐
336 │Macro │ Substitution │
337 └───────────┴────────────────────────────┘
338
339
340
341 │%s │ source path (path to the │
342 │ │ file currently being run) │
343 ├───────────┼────────────────────────────┤
344 │%S │ source dir (directory of │
345 │ │ the file currently being │
346 │ │ run) │
347 ├───────────┼────────────────────────────┤
348 │%p │ same as %S │
349 ├───────────┼────────────────────────────┤
350 │%{pathsep} │ path separator │
351 ├───────────┼────────────────────────────┤
352 │%t │ temporary file name unique │
353 │ │ to the test │
354 ├───────────┼────────────────────────────┤
355 │%T │ temporary directory unique │
356 │ │ to the test │
357 ├───────────┼────────────────────────────┤
358 │%% │ % │
359 └───────────┴────────────────────────────┘
360
361 Other substitutions are provided that are variations on this base set
362 and further substitution patterns can be defined by each test module.
363 See the modules LOCAL CONFIGURATION FILES.
364
365 More detailed information on substitutions can be found in the ../Test‐
366 ingGuide.
367
368 TEST RUN OUTPUT FORMAT
369 The lit output for a test run conforms to the following schema, in both
370 short and verbose modes (although in short mode no PASS lines will be
371 shown). This schema has been chosen to be relatively easy to reliably
372 parse by a machine (for example in buildbot log scraping), and for
373 other tools to generate.
374
375 Each test result is expected to appear on a line that matches:
376
377 <result code>: <test name> (<progress info>)
378
379 where <result-code> is a standard test result such as PASS, FAIL,
380 XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes
381 of IMPROVED and REGRESSED are also allowed.
382
383 The <test name> field can consist of an arbitrary string containing no
384 newline.
385
386 The <progress info> field can be used to report progress information
387 such as (1/300) or can be empty, but even when empty the parentheses
388 are required.
389
390 Each test result may include additional (multiline) log information in
391 the following format:
392
393 <log delineator> TEST '(<test name>)' <trailing delineator>
394 ... log message ...
395 <log delineator>
396
397 where <test name> should be the name of a preceding reported test, <log
398 delineator> is a string of "*" characters at least four characters long
399 (the recommended length is 20), and <trailing delineator> is an arbi‐
400 trary (unparsed) string.
401
402 The following is an example of a test run output which consists of four
403 tests A, B, C, and D, and a log message for the failing test C:
404
405 PASS: A (1 of 4)
406 PASS: B (2 of 4)
407 FAIL: C (3 of 4)
408 ******************** TEST 'C' FAILED ********************
409 Test 'C' failed as a result of exit code 1.
410 ********************
411 PASS: D (4 of 4)
412
413 LIT EXAMPLE TESTS
414 The lit distribution contains several example implementations of test
415 suites in the ExampleTests directory.
416
418 valgrind(1)
419
421 Maintained by The LLVM Team (http://llvm.org/).
422
424 2003-2019, LLVM Project
425
426
427
428
4295 2019-02-01 LIT(1)