1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be executed (potentially in parallel) and once
22       all  tests have been run lit will print summary information on the num‐
23       ber of tests which passed or failed (see TEST STATUS RESULTS).  The lit
24       program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       Users interested in the lit architecture or designing a lit testing im‐
39       plementation should see LIT INFRASTRUCTURE.
40

GENERAL OPTIONS

42       -h, --help
43              Show the lit help message.
44
45       -j N, --threads=N
46              Run N tests in parallel.  By default, this is automatically cho‐
47              sen to match the number of detected available CPUs.
48
49       --config-prefix=NAME
50              Search  for  NAME.cfg  and NAME.site.cfg when searching for test
51              suites, instead of lit.cfg and lit.site.cfg.
52
53       -D NAME[=VALUE], --param NAME[=VALUE]
54              Add a user defined parameter NAME with the given VALUE  (or  the
55              empty string if not given).  The meaning and use of these param‐
56              eters is test suite dependent.
57

OUTPUT OPTIONS

59       -q, --quiet
60              Suppress any output except for test failures.
61
62       -s, --succinct
63              Show less output, for example don't show  information  on  tests
64              that pass.
65
66       -v, --verbose
67              Show  more  information on test failures, for example the entire
68              test output instead of just the test result.
69
70       -vv, --echo-all-commands
71              Echo all commands to stdout, as they are being  executed.   This
72              can  be valuable for debugging test failures, as the last echoed
73              command will be the one which has failed.  lit normally  inserts
74              a  no-op  command (: in the case of bash) with argument 'RUN: at
75              line N' before each  command  pipeline,  and  this  option  also
76              causes  those  no-op commands to be echoed to stdout to help you
77              locate the source line of the failed command.  This  option  im‐
78              plies --verbose.
79
80       -a, --show-all
81              Show  more  information  about all tests, for example the entire
82              test commandline and output.
83
84       --no-progress-bar
85              Do not use curses based progress bar.
86
87       --show-unsupported
88              Show the names of unsupported tests.
89
90       --show-xfail
91              Show the names of tests that were expected to fail.
92

EXECUTION OPTIONS

94       --path=PATH
95              Specify an additional PATH to use when searching for executables
96              in tests.
97
98       --vg   Run  individual  tests under valgrind (using the memcheck tool).
99              The --error-exitcode argument for valgrind is used so that  val‐
100              grind  failures  will  cause the program to exit with a non-zero
101              status.
102
103              When this option is enabled, lit will also automatically provide
104              a  "valgrind"  feature that can be used to conditionally disable
105              (or expect failure in) certain tests.
106
107       --vg-arg=ARG
108              When --vg is used, specify an additional  argument  to  pass  to
109              valgrind itself.
110
111       --vg-leak
112              When  --vg is used, enable memory leak checks.  When this option
113              is enabled, lit will also automatically provide a "vg_leak" fea‐
114              ture  that can be used to conditionally disable (or expect fail‐
115              ure in) certain tests.
116
117       --time-tests
118              Track the wall time individual tests take  to  execute  and  in‐
119              cludes  the  results  in the summary output.  This is useful for
120              determining which tests in a test suite take the  most  time  to
121              execute.  Note that this option is most useful with -j 1.
122

SELECTION OPTIONS

124       --max-tests=N
125              Run at most N tests and then terminate.
126
127       --max-time=N
128              Spend  at  most N seconds (approximately) running tests and then
129              terminate.
130
131       --shuffle
132              Run the tests in a random order.
133
134       --num-shards=M
135              Divide the set of selected tests into M equal-sized  subsets  or
136              "shards",  and  run  only  one  of  them.  Must be used with the
137              --run-shard=N option, which selects the shard to run. The  envi‐
138              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
139              this option. These two options provide a  coarse  mechanism  for
140              paritioning large testsuites, for parallel execution on separate
141              machines (say in a large testing farm).
142
143       --run-shard=N
144              Select which shard to run, assuming  the  --num-shards=M  option
145              was  provided.  The  two  options must be used together, and the
146              value of N must be in the range 1..M. The  environment  variable
147              LIT_RUN_SHARD can also be used in place of this option.
148
149       --filter=REGEXP
150              Run  only  those tests whose name matches the regular expression
151              specified in REGEXP. The environment variable LIT_FILTER can  be
152              also used in place of this option, which is especially useful in
153              environments where the call to lit is issued indirectly.
154

ADDITIONAL OPTIONS

156       --debug
157              Run lit in debug mode, for debugging  configuration  issues  and
158              lit itself.
159
160       --show-suites
161              List the discovered test suites and exit.
162
163       --show-tests
164              List all of the discovered tests and exit.
165

EXIT STATUS

167       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
168       sults.  Otherwise, it will exit with the status 0.   Other  exit  codes
169       are  used for non-test related failures (for example a user error or an
170       internal program error).
171

TEST DISCOVERY

173       The inputs passed to lit can be either individual tests, or entire  di‐
174       rectories  or  hierarchies  of  tests  to run.  When lit starts up, the
175       first thing it does is convert the inputs into a complete list of tests
176       to run as part of test discovery.
177
178       In  the  lit  model, every test must exist inside some test suite.  lit
179       resolves the inputs specified on the command line  to  test  suites  by
180       searching  upwards  from  the  input  path  until it finds a lit.cfg or
181       lit.site.cfg file.  These files serve as both a marker of  test  suites
182       and  as  configuration files which lit loads in order to understand how
183       to find and run the tests inside the test suite.
184
185       Once lit has mapped the inputs into test suites it traverses  the  list
186       of  inputs  adding tests for individual files and recursively searching
187       for tests in directories.
188
189       This behavior makes it easy to specify a subset of tests to run,  while
190       still  allowing  the  test  suite  configuration to control exactly how
191       tests are interpreted.  In addition, lit always identifies tests by the
192       test  suite they are in, and their relative path inside the test suite.
193       For appropriately configured projects, this allows lit to provide  con‐
194       venient and flexible support for out-of-tree builds.
195

TEST STATUS RESULTS

197       Each test ultimately produces one of the following six results:
198
199       PASS
200          The test succeeded.
201
202       XFAIL
203          The  test  failed, but that is expected.  This is used for test for‐
204          mats which allow specifying that a test does not currently work, but
205          wish to leave it in the test suite.
206
207       XPASS
208          The  test  succeeded, but it was expected to fail.  This is used for
209          tests which were specified as expected to fail, but are now succeed‐
210          ing (generally because the feature they test was broken and has been
211          fixed).
212
213       FAIL
214          The test failed.
215
216       UNRESOLVED
217          The test result could not be determined.  For example,  this  occurs
218          when  the  test could not be run, the test itself is invalid, or the
219          test was interrupted.
220
221       UNSUPPORTED
222          The test is not supported in this environment.  This is used by test
223          formats which can report unsupported tests.
224
225       Depending  on  the test format tests may produce additional information
226       about their status (generally only for failures).  See the  OUTPUT  OP‐
227       TIONS section for more information.
228

LIT INFRASTRUCTURE

230       This  section  describes  the lit testing architecture for users inter‐
231       ested in creating a new lit testing implementation, or extending an ex‐
232       isting one.
233
234       lit  proper  is primarily an infrastructure for discovering and running
235       arbitrary tests, and to expose a single convenient interface  to  these
236       tests.  lit  itself doesn't know how to run tests, rather this logic is
237       defined by test suites.
238
239   TEST SUITES
240       As described in TEST DISCOVERY, tests are always located inside a  test
241       suite.   Test  suites serve to define the format of the tests they con‐
242       tain, the logic for finding those tests, and any additional information
243       to run the tests.
244
245       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
246       lit.site.cfg files (see also --config-prefix).  Test  suites  are  ini‐
247       tially  discovered  by recursively searching up the directory hierarchy
248       for all the input files passed  on  the  command  line.   You  can  use
249       --show-suites to display the discovered test suites at startup.
250
251       Once  a  test  suite  is discovered, its config file is loaded.  Config
252       files themselves are Python modules which will be executed.   When  the
253       config file is executed, two important global variables are predefined:
254
255       lit_config
256          The  global  lit  configuration object (a LitConfig instance), which
257          defines the builtin test formats, global  configuration  parameters,
258          and other helper routines for implementing test configurations.
259
260       config
261          This  is  the  config object (a TestingConfig instance) for the test
262          suite, which the config file is expected to populate.  The following
263          variables  are  also  available  on the config object, some of which
264          must be set by the config and others are optional or predefined:
265
266          name [required] The name of the test suite, for use in  reports  and
267          diagnostics.
268
269          test_format  [required] The test format object which will be used to
270          discover and run tests in the test suite.  Generally this will be  a
271          builtin test format available from the lit.formats module.
272
273          test_source_root  The  filesystem  path to the test suite root.  For
274          out-of-dir builds this is the directory that  will  be  scanned  for
275          tests.
276
277          test_exec_root  For  out-of-dir  builds,  the path to the test suite
278          root inside the object directory.  This is where tests will  be  run
279          and temporary output files placed.
280
281          environment  A  dictionary  representing the environment to use when
282          executing tests in the suite.
283
284          suffixes For lit test formats which scan directories for tests, this
285          variable  is  a  list  of suffixes to identify test files.  Used by:
286          ShTest.
287
288          substitutions For lit test formats which substitute variables into a
289          test script, the list of substitutions to perform.  Used by: ShTest.
290
291          unsupported  Mark an unsupported directory, all tests within it will
292          be reported as unsupported.  Used by: ShTest.
293
294          parent The parent configuration, this is the config object  for  the
295          directory containing the test suite, or None.
296
297          root The root configuration.  This is the top-most lit configuration
298          in the project.
299
300          pipefail Normally a test using a shell pipe fails if any of the com‐
301          mands  on  the pipe fail. If this is not desired, setting this vari‐
302          able to false makes the test fail only if the last  command  in  the
303          pipe fails.
304
305          available_features  A set of features that can be used in XFAIL, RE‐
306          QUIRES, and UNSUPPORTED directives.
307
308   TEST DISCOVERY
309       Once test suites are located, lit recursively traverses the source  di‐
310       rectory  (following  test_source_root) looking for tests.  When lit en‐
311       ters a sub-directory, it first checks to see if a nested test suite  is
312       defined  in  that  directory.   If  so, it loads that test suite recur‐
313       sively, otherwise it instantiates a local test config for the directory
314       (see LOCAL CONFIGURATION FILES).
315
316       Tests  are  identified by the test suite they are contained within, and
317       the relative path inside that suite.  Note that the relative  path  may
318       not  refer  to  an  actual  file  on  disk;  some test formats (such as
319       GoogleTest) define "virtual tests" which have a path that contains both
320       the  path to the actual test file and a subpath to identify the virtual
321       test.
322
323   LOCAL CONFIGURATION FILES
324       When lit loads a subdirectory in a test suite, it instantiates a  local
325       test  configuration  by cloning the configuration for the parent direc‐
326       tory --- the root of this configuration chain will  always  be  a  test
327       suite.   Once the test configuration is cloned lit checks for a lit.lo‐
328       cal.cfg file in the subdirectory.  If present, this file will be loaded
329       and can be used to specialize the configuration for each individual di‐
330       rectory.  This facility can be used to  define  subdirectories  of  op‐
331       tional tests, or to change other configuration parameters --- for exam‐
332       ple, to change the test format, or the  suffixes  which  identify  test
333       files.
334
335   PRE-DEFINED SUBSTITUTIONS
336       lit  provides  various  patterns that can be used with the RUN command.
337       These are defined in TestRunner.py. The base set of substitutions are:
338
339
340
341                       ┌───────────┬────────────────────────────┐
342                       │Macro      │ Substitution               │
343                       ├───────────┼────────────────────────────┤
344                       │%s         │ source path (path  to  the │
345                       │           │ file currently being run)  │
346                       ├───────────┼────────────────────────────┤
347                       │%S         │ source  dir  (directory of │
348                       │           │ the file  currently  being │
349                       │           │ run)                       │
350                       ├───────────┼────────────────────────────┤
351                       │%p         │ same as %S                 │
352                       ├───────────┼────────────────────────────┤
353                       │%{pathsep} │ path separator             │
354                       ├───────────┼────────────────────────────┤
355                       │%t         │ temporary file name unique │
356                       │           │ to the test                │
357                       ├───────────┼────────────────────────────┤
358                       │%T         │ temporary directory unique │
359                       │           │ to the test                │
360                       ├───────────┼────────────────────────────┤
361                       │%%         │ %                          │
362                       └───────────┴────────────────────────────┘
363
364       Other  substitutions  are provided that are variations on this base set
365       and further substitution patterns can be defined by each  test  module.
366       See the modules LOCAL CONFIGURATION FILES.
367
368       More detailed information on substitutions can be found in the ../Test‐
369       ingGuide.
370
371   TEST RUN OUTPUT FORMAT
372       The lit output for a test run conforms to the following schema, in both
373       short  and  verbose modes (although in short mode no PASS lines will be
374       shown).  This schema has been chosen to be relatively easy to  reliably
375       parse  by  a  machine  (for  example in buildbot log scraping), and for
376       other tools to generate.
377
378       Each test result is expected to appear on a line that matches:
379
380          <result code>: <test name> (<progress info>)
381
382       where <result-code> is a standard  test  result  such  as  PASS,  FAIL,
383       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
384       of IMPROVED and REGRESSED are also allowed.
385
386       The <test name> field can consist of an arbitrary string containing  no
387       newline.
388
389       The  <progress  info>  field can be used to report progress information
390       such as (1/300) or can be empty, but even when  empty  the  parentheses
391       are required.
392
393       Each  test result may include additional (multiline) log information in
394       the following format:
395
396          <log delineator> TEST '(<test name>)' <trailing delineator>
397          ... log message ...
398          <log delineator>
399
400       where <test name> should be the name of a preceding reported test, <log
401       delineator> is a string of "*" characters at least four characters long
402       (the recommended length is 20), and <trailing delineator> is  an  arbi‐
403       trary (unparsed) string.
404
405       The following is an example of a test run output which consists of four
406       tests A, B, C, and D, and a log message for the failing test C:
407
408          PASS: A (1 of 4)
409          PASS: B (2 of 4)
410          FAIL: C (3 of 4)
411          ******************** TEST 'C' FAILED ********************
412          Test 'C' failed as a result of exit code 1.
413          ********************
414          PASS: D (4 of 4)
415
416   LIT EXAMPLE TESTS
417       The lit distribution contains several example implementations  of  test
418       suites in the ExampleTests directory.
419

SEE ALSO

421       valgrind(1)
422

AUTHOR

424       Maintained by The LLVM Team (http://llvm.org/).
425
427       2003-2021, LLVM Project
428
429
430
431
4327                                 2021-03-30                            LIT(1)
Impressum