1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed  to  be  a  lightweight testing tool with as simple a user
15       interface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be executed (potentially in parallel) and once
22       all  tests have been run lit will print summary information on the num‐
23       ber of tests which passed or failed (see TEST STATUS RESULTS).  The lit
24       program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit  also  includes  a  number of options for controlling how tests are
31       executed (specific features may depend on the particular test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       Users  interested  in  the  lit architecture or designing a lit testing
39       implementation should see LIT INFRASTRUCTURE.
40

GENERAL OPTIONS

42       -h, --help
43              Show the lit help message.
44
45       -j N, --threads=N
46              Run N tests in parallel.  By default, this is automatically cho‐
47              sen to match the number of detected available CPUs.
48
49       --config-prefix=NAME
50              Search  for  NAME.cfg  and NAME.site.cfg when searching for test
51              suites, instead of lit.cfg and lit.site.cfg.
52
53       -D NAME[=VALUE], --param NAME[=VALUE]
54              Add a user defined parameter NAME with the given VALUE  (or  the
55              empty string if not given).  The meaning and use of these param‐
56              eters is test suite dependent.
57

OUTPUT OPTIONS

59       -q, --quiet
60              Suppress any output except for test failures.
61
62       -s, --succinct
63              Show less output, for example don't show  information  on  tests
64              that pass.
65
66       -v, --verbose
67              Show  more  information on test failures, for example the entire
68              test output instead of just the test result.
69
70       -vv, --echo-all-commands
71              Echo all commands to stdout, as they are being  executed.   This
72              can  be valuable for debugging test failures, as the last echoed
73              command will be the one which has failed.  This  option  implies
74              --verbose.
75
76       -a, --show-all
77              Show  more  information  about all tests, for example the entire
78              test commandline and output.
79
80       --no-progress-bar
81              Do not use curses based progress bar.
82
83       --show-unsupported
84              Show the names of unsupported tests.
85
86       --show-xfail
87              Show the names of tests that were expected to fail.
88

EXECUTION OPTIONS

90       --path=PATH
91              Specify an additional PATH to use when searching for executables
92              in tests.
93
94       --vg   Run  individual  tests under valgrind (using the memcheck tool).
95              The --error-exitcode argument for valgrind is used so that  val‐
96              grind  failures  will  cause the program to exit with a non-zero
97              status.
98
99              When this option is enabled, lit will also automatically provide
100              a  "valgrind"  feature that can be used to conditionally disable
101              (or expect failure in) certain tests.
102
103       --vg-arg=ARG
104              When --vg is used, specify an additional  argument  to  pass  to
105              valgrind itself.
106
107       --vg-leak
108              When  --vg is used, enable memory leak checks.  When this option
109              is enabled, lit will also automatically provide a "vg_leak" fea‐
110              ture  that can be used to conditionally disable (or expect fail‐
111              ure in) certain tests.
112
113       --time-tests
114              Track the  wall  time  individual  tests  take  to  execute  and
115              includes  the results in the summary output.  This is useful for
116              determining which tests in a test suite take the  most  time  to
117              execute.  Note that this option is most useful with -j 1.
118

SELECTION OPTIONS

120       --max-tests=N
121              Run at most N tests and then terminate.
122
123       --max-time=N
124              Spend  at  most N seconds (approximately) running tests and then
125              terminate.
126
127       --shuffle
128              Run the tests in a random order.
129
130       --num-shards=M
131              Divide the set of selected tests into M equal-sized  subsets  or
132              "shards",  and  run  only  one  of  them.  Must be used with the
133              --run-shard=N option, which selects the shard to run. The  envi‐
134              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
135              this option. These two options provide a  coarse  mechanism  for
136              paritioning large testsuites, for parallel execution on separate
137              machines (say in a large testing farm).
138
139       --run-shard=N
140              Select which shard to run, assuming  the  --num-shards=M  option
141              was  provided.  The  two  options must be used together, and the
142              value of N must be in the range 1..M. The  environment  variable
143              LIT_RUN_SHARD can also be used in place of this option.
144
145       --filter=REGEXP
146              Run  only  those tests whose name matches the regular expression
147              specified in REGEXP. The environment variable LIT_FILTER can  be
148              also used in place of this option, which is especially useful in
149              environments where the call to lit is issued indirectly.
150

ADDITIONAL OPTIONS

152       --debug
153              Run lit in debug mode, for debugging  configuration  issues  and
154              lit itself.
155
156       --show-suites
157              List the discovered test suites and exit.
158
159       --show-tests
160              List all of the discovered tests and exit.
161

EXIT STATUS

163       lit  will  exit  with  an exit code of 1 if there are any FAIL or XPASS
164       results.  Otherwise, it will exit with the status 0.  Other exit  codes
165       are  used for non-test related failures (for example a user error or an
166       internal program error).
167

TEST DISCOVERY

169       The inputs passed to lit can be  either  individual  tests,  or  entire
170       directories  or  hierarchies  of tests to run.  When lit starts up, the
171       first thing it does is convert the inputs into a complete list of tests
172       to run as part of test discovery.
173
174       In  the  lit  model, every test must exist inside some test suite.  lit
175       resolves the inputs specified on the command line  to  test  suites  by
176       searching  upwards  from  the  input  path  until it finds a lit.cfg or
177       lit.site.cfg file.  These files serve as both a marker of  test  suites
178       and  as  configuration files which lit loads in order to understand how
179       to find and run the tests inside the test suite.
180
181       Once lit has mapped the inputs into test suites it traverses  the  list
182       of  inputs  adding tests for individual files and recursively searching
183       for tests in directories.
184
185       This behavior makes it easy to specify a subset of tests to run,  while
186       still  allowing  the  test  suite  configuration to control exactly how
187       tests are interpreted.  In addition, lit always identifies tests by the
188       test  suite they are in, and their relative path inside the test suite.
189       For appropriately configured projects, this allows lit to provide  con‐
190       venient and flexible support for out-of-tree builds.
191

TEST STATUS RESULTS

193       Each test ultimately produces one of the following six results:
194
195       PASS
196          The test succeeded.
197
198       XFAIL
199          The  test  failed, but that is expected.  This is used for test for‐
200          mats which allow specifying that a test does not currently work, but
201          wish to leave it in the test suite.
202
203       XPASS
204          The  test  succeeded, but it was expected to fail.  This is used for
205          tests which were specified as expected to fail, but are now succeed‐
206          ing (generally because the feature they test was broken and has been
207          fixed).
208
209       FAIL
210          The test failed.
211
212       UNRESOLVED
213          The test result could not be determined.  For example,  this  occurs
214          when  the  test could not be run, the test itself is invalid, or the
215          test was interrupted.
216
217       UNSUPPORTED
218          The test is not supported in this environment.  This is used by test
219          formats which can report unsupported tests.
220
221       Depending  on  the test format tests may produce additional information
222       about their status (generally  only  for  failures).   See  the  OUTPUT
223       OPTIONS section for more information.
224

LIT INFRASTRUCTURE

226       This  section  describes  the lit testing architecture for users inter‐
227       ested in creating a new lit testing  implementation,  or  extending  an
228       existing one.
229
230       lit  proper  is primarily an infrastructure for discovering and running
231       arbitrary tests, and to expose a single convenient interface  to  these
232       tests.  lit  itself doesn't know how to run tests, rather this logic is
233       defined by test suites.
234
235   TEST SUITES
236       As described in TEST DISCOVERY, tests are always located inside a  test
237       suite.   Test  suites serve to define the format of the tests they con‐
238       tain, the logic for finding those tests, and any additional information
239       to run the tests.
240
241       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
242       lit.site.cfg files (see also --config-prefix).  Test  suites  are  ini‐
243       tially  discovered  by recursively searching up the directory hierarchy
244       for all the input files passed  on  the  command  line.   You  can  use
245       --show-suites to display the discovered test suites at startup.
246
247       Once  a  test  suite  is discovered, its config file is loaded.  Config
248       files themselves are Python modules which will be executed.   When  the
249       config file is executed, two important global variables are predefined:
250
251       lit_config
252          The  global  lit  configuration object (a LitConfig instance), which
253          defines the builtin test formats, global  configuration  parameters,
254          and other helper routines for implementing test configurations.
255
256       config
257          This  is  the  config object (a TestingConfig instance) for the test
258          suite, which the config file is expected to populate.  The following
259          variables  are  also  available  on the config object, some of which
260          must be set by the config and others are optional or predefined:
261
262          name [required] The name of the test suite, for use in  reports  and
263          diagnostics.
264
265          test_format  [required] The test format object which will be used to
266          discover and run tests in the test suite.  Generally this will be  a
267          builtin test format available from the lit.formats module.
268
269          test_source_root  The  filesystem  path to the test suite root.  For
270          out-of-dir builds this is the directory that  will  be  scanned  for
271          tests.
272
273          test_exec_root  For  out-of-dir  builds,  the path to the test suite
274          root inside the object directory.  This is where tests will  be  run
275          and temporary output files placed.
276
277          environment  A  dictionary  representing the environment to use when
278          executing tests in the suite.
279
280          suffixes For lit test formats which scan directories for tests, this
281          variable  is  a  list  of suffixes to identify test files.  Used by:
282          ShTest.
283
284          substitutions For lit test formats which substitute variables into a
285          test script, the list of substitutions to perform.  Used by: ShTest.
286
287          unsupported  Mark an unsupported directory, all tests within it will
288          be reported as unsupported.  Used by: ShTest.
289
290          parent The parent configuration, this is the config object  for  the
291          directory containing the test suite, or None.
292
293          root The root configuration.  This is the top-most lit configuration
294          in the project.
295
296          pipefail Normally a test using a shell pipe fails if any of the com‐
297          mands  on  the pipe fail. If this is not desired, setting this vari‐
298          able to false makes the test fail only if the last  command  in  the
299          pipe fails.
300
301          available_features  A  set  of  features  that can be used in XFAIL,
302          REQUIRES, and UNSUPPORTED directives.
303
304   TEST DISCOVERY
305       Once test suites are located,  lit  recursively  traverses  the  source
306       directory  (following  test_source_root)  looking  for tests.  When lit
307       enters a sub-directory, it first checks to see if a nested  test  suite
308       is  defined  in that directory.  If so, it loads that test suite recur‐
309       sively, otherwise it instantiates a local test config for the directory
310       (see LOCAL CONFIGURATION FILES).
311
312       Tests  are  identified by the test suite they are contained within, and
313       the relative path inside that suite.  Note that the relative  path  may
314       not  refer  to  an  actual  file  on  disk;  some test formats (such as
315       GoogleTest) define "virtual tests" which have a path that contains both
316       the  path to the actual test file and a subpath to identify the virtual
317       test.
318
319   LOCAL CONFIGURATION FILES
320       When lit loads a subdirectory in a test suite, it instantiates a  local
321       test  configuration  by cloning the configuration for the parent direc‐
322       tory --- the root of this configuration chain will  always  be  a  test
323       suite.   Once  the  test  configuration  is  cloned  lit  checks  for a
324       lit.local.cfg file in the subdirectory.  If present, this file will  be
325       loaded  and  can be used to specialize the configuration for each indi‐
326       vidual directory.  This facility can be used to  define  subdirectories
327       of  optional tests, or to change other configuration parameters --- for
328       example, to change the test format, or the suffixes which identify test
329       files.
330
331   PRE-DEFINED SUBSTITUTIONS
332       lit  provides  various  patterns that can be used with the RUN command.
333       These are defined in TestRunner.py. The base set of substitutions are:
334
335                       ┌───────────┬────────────────────────────┐
336                       │Macro      │ Substitution               │
337                       └───────────┴────────────────────────────┘
338
339
340
341                       │%s         │ source path (path  to  the │
342                       │           │ file currently being run)  │
343                       ├───────────┼────────────────────────────┤
344                       │%S         │ source  dir  (directory of │
345                       │           │ the file  currently  being │
346                       │           │ run)                       │
347                       ├───────────┼────────────────────────────┤
348                       │%p         │ same as %S                 │
349                       ├───────────┼────────────────────────────┤
350                       │%{pathsep} │ path separator             │
351                       ├───────────┼────────────────────────────┤
352                       │%t         │ temporary file name unique │
353                       │           │ to the test                │
354                       ├───────────┼────────────────────────────┤
355                       │%T         │ temporary directory unique │
356                       │           │ to the test                │
357                       ├───────────┼────────────────────────────┤
358                       │%%         │ %                          │
359                       └───────────┴────────────────────────────┘
360
361       Other  substitutions  are provided that are variations on this base set
362       and further substitution patterns can be defined by each  test  module.
363       See the modules LOCAL CONFIGURATION FILES.
364
365       More detailed information on substitutions can be found in the ../Test‐
366       ingGuide.
367
368   TEST RUN OUTPUT FORMAT
369       The lit output for a test run conforms to the following schema, in both
370       short  and  verbose modes (although in short mode no PASS lines will be
371       shown).  This schema has been chosen to be relatively easy to  reliably
372       parse  by  a  machine  (for  example in buildbot log scraping), and for
373       other tools to generate.
374
375       Each test result is expected to appear on a line that matches:
376
377          <result code>: <test name> (<progress info>)
378
379       where <result-code> is a standard  test  result  such  as  PASS,  FAIL,
380       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
381       of IMPROVED and REGRESSED are also allowed.
382
383       The <test name> field can consist of an arbitrary string containing  no
384       newline.
385
386       The  <progress  info>  field can be used to report progress information
387       such as (1/300) or can be empty, but even when  empty  the  parentheses
388       are required.
389
390       Each  test result may include additional (multiline) log information in
391       the following format:
392
393          <log delineator> TEST '(<test name>)' <trailing delineator>
394          ... log message ...
395          <log delineator>
396
397       where <test name> should be the name of a preceding reported test, <log
398       delineator> is a string of "*" characters at least four characters long
399       (the recommended length is 20), and <trailing delineator> is  an  arbi‐
400       trary (unparsed) string.
401
402       The following is an example of a test run output which consists of four
403       tests A, B, C, and D, and a log message for the failing test C:
404
405          PASS: A (1 of 4)
406          PASS: B (2 of 4)
407          FAIL: C (3 of 4)
408          ******************** TEST 'C' FAILED ********************
409          Test 'C' failed as a result of exit code 1.
410          ********************
411          PASS: D (4 of 4)
412
413   LIT EXAMPLE TESTS
414       The lit distribution contains several example implementations  of  test
415       suites in the ExampleTests directory.
416

SEE ALSO

418       valgrind(1)
419

AUTHOR

421       Maintained by The LLVM Team (http://llvm.org/).
422
424       2003-2019, LLVM Project
425
426
427
428
4295                                 2019-02-01                            LIT(1)
Impressum