1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed  to  be  a  lightweight testing tool with as simple a user
15       interface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be executed (potentially in parallel) and once
22       all  tests have been run lit will print summary information on the num‐
23       ber of tests which passed or failed (see TEST STATUS RESULTS).  The lit
24       program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit  also  includes  a  number of options for controlling how tests are
31       executed (specific features may depend on the particular test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       Users interested in the lit architecture or  designing  a  lit  testing
44       implementation should see LIT INFRASTRUCTURE.
45

GENERAL OPTIONS

47       -h, --help
48              Show the lit help message.
49
50       -j N, --threads=N
51              Run N tests in parallel.  By default, this is automatically cho‐
52              sen to match the number of detected available CPUs.
53
54       --config-prefix=NAME
55              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
56              suites, instead of lit.cfg and lit.site.cfg.
57
58       -D NAME[=VALUE], --param NAME[=VALUE]
59              Add  a  user defined parameter NAME with the given VALUE (or the
60              empty string if not given).  The meaning and use of these param‐
61              eters is test suite dependent.
62

OUTPUT OPTIONS

64       -q, --quiet
65              Suppress any output except for test failures.
66
67       -s, --succinct
68              Show  less  output,  for example don't show information on tests
69              that pass.
70
71       -v, --verbose
72              Show more information on test failures, for example  the  entire
73              test output instead of just the test result.
74
75       -vv, --echo-all-commands
76              Echo  all  commands to stdout, as they are being executed.  This
77              can be valuable for debugging test failures, as the last  echoed
78              command  will be the one which has failed.  lit normally inserts
79              a no-op command (: in the case of bash) with argument  'RUN:  at
80              line  N'  before  each  command  pipeline,  and this option also
81              causes those no-op commands to be echoed to stdout to  help  you
82              locate  the  source  line  of  the  failed command.  This option
83              implies --verbose.
84
85       -a, --show-all
86              Show more information about all tests, for  example  the  entire
87              test commandline and output.
88
89       --no-progress-bar
90              Do not use curses based progress bar.
91
92       --show-unsupported
93              Show the names of unsupported tests.
94
95       --show-xfail
96              Show the names of tests that were expected to fail.
97

EXECUTION OPTIONS

99       --path=PATH
100              Specify an additional PATH to use when searching for executables
101              in tests.
102
103       --vg   Run individual tests under valgrind (using the  memcheck  tool).
104              The  --error-exitcode argument for valgrind is used so that val‐
105              grind failures will cause the program to exit  with  a  non-zero
106              status.
107
108              When this option is enabled, lit will also automatically provide
109              a "valgrind" feature that can be used to  conditionally  disable
110              (or expect failure in) certain tests.
111
112       --vg-arg=ARG
113              When  --vg  is  used,  specify an additional argument to pass to
114              valgrind itself.
115
116       --vg-leak
117              When --vg is used, enable memory leak checks.  When this  option
118              is enabled, lit will also automatically provide a "vg_leak" fea‐
119              ture that can be used to conditionally disable (or expect  fail‐
120              ure in) certain tests.
121
122       --time-tests
123              Track  the  wall  time  individual  tests  take  to  execute and
124              includes the results in the summary output.  This is useful  for
125              determining  which  tests  in a test suite take the most time to
126              execute.  Note that this option is most useful with -j 1.
127

SELECTION OPTIONS

129       --max-tests=N
130              Run at most N tests and then terminate.
131
132       --max-time=N
133              Spend at most N seconds (approximately) running tests  and  then
134              terminate.
135
136       --shuffle
137              Run the tests in a random order.
138
139       --num-shards=M
140              Divide  the  set of selected tests into M equal-sized subsets or
141              "shards", and run only one of  them.   Must  be  used  with  the
142              --run-shard=N  option, which selects the shard to run. The envi‐
143              ronment variable LIT_NUM_SHARDS can also be  used  in  place  of
144              this  option.  These  two options provide a coarse mechanism for
145              paritioning large testsuites, for parallel execution on separate
146              machines (say in a large testing farm).
147
148       --run-shard=N
149              Select  which  shard  to run, assuming the --num-shards=M option
150              was provided. The two options must be  used  together,  and  the
151              value  of  N must be in the range 1..M. The environment variable
152              LIT_RUN_SHARD can also be used in place of this option.
153
154       --filter=REGEXP
155              Run only those tests whose name matches the  regular  expression
156              specified  in REGEXP. The environment variable LIT_FILTER can be
157              also used in place of this option, which is especially useful in
158              environments where the call to lit is issued indirectly.
159

ADDITIONAL OPTIONS

161       --debug
162              Run  lit  in  debug mode, for debugging configuration issues and
163              lit itself.
164
165       --show-suites
166              List the discovered test suites and exit.
167
168       --show-tests
169              List all of the discovered tests and exit.
170

EXIT STATUS

172       lit will exit with an exit code of 1 if there are  any  FAIL  or  XPASS
173       results.   Otherwise, it will exit with the status 0.  Other exit codes
174       are used for non-test related failures (for example a user error or  an
175       internal program error).
176

TEST DISCOVERY

178       The  inputs  passed  to  lit  can be either individual tests, or entire
179       directories or hierarchies of tests to run.  When lit  starts  up,  the
180       first thing it does is convert the inputs into a complete list of tests
181       to run as part of test discovery.
182
183       In the lit model, every test must exist inside some  test  suite.   lit
184       resolves  the  inputs  specified  on the command line to test suites by
185       searching upwards from the input path  until  it  finds  a  lit.cfg  or
186       lit.site.cfg  file.   These files serve as both a marker of test suites
187       and as configuration files which lit loads in order to  understand  how
188       to find and run the tests inside the test suite.
189
190       Once  lit  has mapped the inputs into test suites it traverses the list
191       of inputs adding tests for individual files and  recursively  searching
192       for tests in directories.
193
194       This  behavior makes it easy to specify a subset of tests to run, while
195       still allowing the test suite  configuration  to  control  exactly  how
196       tests are interpreted.  In addition, lit always identifies tests by the
197       test suite they are in, and their relative path inside the test  suite.
198       For  appropriately configured projects, this allows lit to provide con‐
199       venient and flexible support for out-of-tree builds.
200

TEST STATUS RESULTS

202       Each test ultimately produces one of the following six results:
203
204       PASS
205          The test succeeded.
206
207       XFAIL
208          The test failed, but that is expected.  This is used for  test  for‐
209          mats which allow specifying that a test does not currently work, but
210          wish to leave it in the test suite.
211
212       XPASS
213          The test succeeded, but it was expected to fail.  This is  used  for
214          tests which were specified as expected to fail, but are now succeed‐
215          ing (generally because the feature they test was broken and has been
216          fixed).
217
218       FAIL
219          The test failed.
220
221       UNRESOLVED
222          The  test  result could not be determined.  For example, this occurs
223          when the test could not be run, the test itself is invalid,  or  the
224          test was interrupted.
225
226       UNSUPPORTED
227          The test is not supported in this environment.  This is used by test
228          formats which can report unsupported tests.
229
230       Depending on the test format tests may produce  additional  information
231       about  their  status  (generally  only  for  failures).  See the OUTPUT
232       OPTIONS section for more information.
233

LIT INFRASTRUCTURE

235       This section describes the lit testing architecture  for  users  inter‐
236       ested  in  creating  a  new lit testing implementation, or extending an
237       existing one.
238
239       lit proper is primarily an infrastructure for discovering  and  running
240       arbitrary  tests,  and to expose a single convenient interface to these
241       tests. lit itself doesn't know how to run tests, rather this  logic  is
242       defined by test suites.
243
244   TEST SUITES
245       As  described in TEST DISCOVERY, tests are always located inside a test
246       suite.  Test suites serve to define the format of the tests  they  con‐
247       tain, the logic for finding those tests, and any additional information
248       to run the tests.
249
250       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
251       lit.site.cfg  files  (see  also --config-prefix).  Test suites are ini‐
252       tially discovered by recursively searching up the  directory  hierarchy
253       for  all  the  input  files  passed  on  the command line.  You can use
254       --show-suites to display the discovered test suites at startup.
255
256       Once a test suite is discovered, its config  file  is  loaded.   Config
257       files  themselves  are Python modules which will be executed.  When the
258       config file is executed, two important global variables are predefined:
259
260       lit_config
261          The global lit configuration object (a  LitConfig  instance),  which
262          defines  the  builtin test formats, global configuration parameters,
263          and other helper routines for implementing test configurations.
264
265       config
266          This is the config object (a TestingConfig instance)  for  the  test
267          suite, which the config file is expected to populate.  The following
268          variables are also available on the config  object,  some  of  which
269          must be set by the config and others are optional or predefined:
270
271          name  [required]  The name of the test suite, for use in reports and
272          diagnostics.
273
274          test_format [required] The test format object which will be used  to
275          discover  and run tests in the test suite.  Generally this will be a
276          builtin test format available from the lit.formats module.
277
278          test_source_root The filesystem path to the test  suite  root.   For
279          out-of-dir  builds  this  is  the directory that will be scanned for
280          tests.
281
282          test_exec_root For out-of-dir builds, the path  to  the  test  suite
283          root  inside  the object directory.  This is where tests will be run
284          and temporary output files placed.
285
286          environment A dictionary representing the environment  to  use  when
287          executing tests in the suite.
288
289          suffixes For lit test formats which scan directories for tests, this
290          variable is a list of suffixes to identify  test  files.   Used  by:
291          ShTest.
292
293          substitutions For lit test formats which substitute variables into a
294          test script, the list of substitutions to perform.  Used by: ShTest.
295
296          unsupported Mark an unsupported directory, all tests within it  will
297          be reported as unsupported.  Used by: ShTest.
298
299          parent  The  parent configuration, this is the config object for the
300          directory containing the test suite, or None.
301
302          root The root configuration.  This is the top-most lit configuration
303          in the project.
304
305          pipefail Normally a test using a shell pipe fails if any of the com‐
306          mands on the pipe fail. If this is not desired, setting  this  vari‐
307          able  to  false  makes the test fail only if the last command in the
308          pipe fails.
309
310          available_features A set of features that  can  be  used  in  XFAIL,
311          REQUIRES, and UNSUPPORTED directives.
312
313   TEST DISCOVERY
314       Once  test  suites  are  located,  lit recursively traverses the source
315       directory (following test_source_root) looking  for  tests.   When  lit
316       enters  a  sub-directory, it first checks to see if a nested test suite
317       is defined in that directory.  If so, it loads that test  suite  recur‐
318       sively, otherwise it instantiates a local test config for the directory
319       (see LOCAL CONFIGURATION FILES).
320
321       Tests are identified by the test suite they are contained  within,  and
322       the  relative  path inside that suite.  Note that the relative path may
323       not refer to an actual  file  on  disk;  some  test  formats  (such  as
324       GoogleTest) define "virtual tests" which have a path that contains both
325       the path to the actual test file and a subpath to identify the  virtual
326       test.
327
328   LOCAL CONFIGURATION FILES
329       When  lit loads a subdirectory in a test suite, it instantiates a local
330       test configuration by cloning the configuration for the  parent  direc‐
331       tory  ---  the  root  of this configuration chain will always be a test
332       suite.  Once  the  test  configuration  is  cloned  lit  checks  for  a
333       lit.local.cfg  file in the subdirectory.  If present, this file will be
334       loaded and can be used to specialize the configuration for  each  indi‐
335       vidual  directory.   This facility can be used to define subdirectories
336       of optional tests, or to change other configuration parameters ---  for
337       example, to change the test format, or the suffixes which identify test
338       files.
339
340   PRE-DEFINED SUBSTITUTIONS
341       lit provides various patterns that can be used with  the  RUN  command.
342       These are defined in TestRunner.py. The base set of substitutions are:
343
344                       ┌───────────┬────────────────────────────┐
345                       │Macro      │ Substitution               │
346                       ├───────────┼────────────────────────────┤
347                       │%s         │ source  path  (path to the │
348                       │           │ file currently being run)  │
349                       ├───────────┼────────────────────────────┤
350                       │%S         │ source dir  (directory  of │
351                       │           │ the  file  currently being │
352                       │           │ run)                       │
353                       ├───────────┼────────────────────────────┤
354                       │%p         │ same as %S                 │
355                       ├───────────┼────────────────────────────┤
356                       │%{pathsep} │ path separator             │
357                       ├───────────┼────────────────────────────┤
358                       │%t         │ temporary file name unique │
359                       │           │ to the test                │
360                       ├───────────┼────────────────────────────┤
361                       │%T         │ parent   directory  of  %t │
362                       │           │ (not  unique,  deprecated, │
363                       │           │ do not use)                │
364                       ├───────────┼────────────────────────────┤
365                       │%%         │ %                          │
366                       └───────────┴────────────────────────────┘
367
368       Other  substitutions  are provided that are variations on this base set
369       and further substitution patterns can be defined by each  test  module.
370       See the modules LOCAL CONFIGURATION FILES.
371
372       More detailed information on substitutions can be found in the ../Test‐
373       ingGuide.
374
375   TEST RUN OUTPUT FORMAT
376       The lit output for a test run conforms to the following schema, in both
377       short  and  verbose modes (although in short mode no PASS lines will be
378       shown).  This schema has been chosen to be relatively easy to  reliably
379       parse  by  a  machine  (for  example in buildbot log scraping), and for
380       other tools to generate.
381
382       Each test result is expected to appear on a line that matches:
383
384          <result code>: <test name> (<progress info>)
385
386       where <result-code> is a standard  test  result  such  as  PASS,  FAIL,
387       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
388       of IMPROVED and REGRESSED are also allowed.
389
390       The <test name> field can consist of an arbitrary string containing  no
391       newline.
392
393       The  <progress  info>  field can be used to report progress information
394       such as (1/300) or can be empty, but even when  empty  the  parentheses
395       are required.
396
397       Each  test result may include additional (multiline) log information in
398       the following format:
399
400          <log delineator> TEST '(<test name>)' <trailing delineator>
401          ... log message ...
402          <log delineator>
403
404       where <test name> should be the name of a preceding reported test, <log
405       delineator> is a string of "*" characters at least four characters long
406       (the recommended length is 20), and <trailing delineator> is  an  arbi‐
407       trary (unparsed) string.
408
409       The following is an example of a test run output which consists of four
410       tests A, B, C, and D, and a log message for the failing test C:
411
412          PASS: A (1 of 4)
413          PASS: B (2 of 4)
414          FAIL: C (3 of 4)
415          ******************** TEST 'C' FAILED ********************
416          Test 'C' failed as a result of exit code 1.
417          ********************
418          PASS: D (4 of 4)
419
420   LIT EXAMPLE TESTS
421       The lit distribution contains several example implementations  of  test
422       suites in the ExampleTests directory.
423

SEE ALSO

425       valgrind(1)
426

AUTHOR

428       Maintained by the LLVM Team (https://llvm.org/).
429
431       2003-2020, LLVM Project
432
433
434
435
4369                                 2020-02-26                            LIT(1)
Impressum