1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be executed (potentially in parallel) and once
22       all  tests have been run lit will print summary information on the num‐
23       ber of tests which passed or failed (see TEST STATUS RESULTS).  The lit
24       program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       Users interested in the lit architecture or designing a lit testing im‐
44       plementation should see LIT INFRASTRUCTURE.
45

GENERAL OPTIONS

47       -h, --help
48              Show the lit help message.
49
50       -j N, --workers=N
51              Run N tests in parallel.  By default, this is automatically cho‐
52              sen to match the number of detected available CPUs.
53
54       --config-prefix=NAME
55              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
56              suites, instead of lit.cfg and lit.site.cfg.
57
58       -D NAME[=VALUE], --param NAME[=VALUE]
59              Add  a  user defined parameter NAME with the given VALUE (or the
60              empty string if not given).  The meaning and use of these param‐
61              eters is test suite dependent.
62

OUTPUT OPTIONS

64       -q, --quiet
65              Suppress any output except for test failures.
66
67       -s, --succinct
68              Show  less  output,  for example don't show information on tests
69              that pass.  Also show a progress bar,  unless  --no-progress-bar
70              is specified.
71
72       -v, --verbose
73              Show  more  information on test failures, for example the entire
74              test output instead of just the test result.
75
76       -vv, --echo-all-commands
77              Echo all commands to stdout, as they are being  executed.   This
78              can  be valuable for debugging test failures, as the last echoed
79              command will be the one which has failed.  lit normally  inserts
80              a  no-op  command (: in the case of bash) with argument 'RUN: at
81              line N' before each  command  pipeline,  and  this  option  also
82              causes  those  no-op commands to be echoed to stdout to help you
83              locate the source line of the failed command.  This  option  im‐
84              plies --verbose.
85
86       -a, --show-all
87              Show  more  information  about all tests, for example the entire
88              test commandline and output.
89
90       --no-progress-bar
91              Do not use curses based progress bar.
92
93       --show-unsupported
94              Show the names of unsupported tests.
95
96       --show-xfail
97              Show the names of tests that were expected to fail.
98

EXECUTION OPTIONS

100       --path=PATH
101              Specify an additional PATH to use when searching for executables
102              in tests.
103
104       --vg   Run  individual  tests under valgrind (using the memcheck tool).
105              The --error-exitcode argument for valgrind is used so that  val‐
106              grind  failures  will  cause the program to exit with a non-zero
107              status.
108
109              When this option is enabled, lit will also automatically provide
110              a  "valgrind"  feature that can be used to conditionally disable
111              (or expect failure in) certain tests.
112
113       --vg-arg=ARG
114              When --vg is used, specify an additional  argument  to  pass  to
115              valgrind itself.
116
117       --vg-leak
118              When  --vg is used, enable memory leak checks.  When this option
119              is enabled, lit will also automatically provide a "vg_leak" fea‐
120              ture  that can be used to conditionally disable (or expect fail‐
121              ure in) certain tests.
122
123       --time-tests
124              Track the wall time individual tests take  to  execute  and  in‐
125              cludes  the  results  in the summary output.  This is useful for
126              determining which tests in a test suite take the  most  time  to
127              execute.  Note that this option is most useful with -j 1.
128

SELECTION OPTIONS

130       --max-failures N
131              Stop execution after the given number N of failures.  An integer
132              argument should be passed on the command line  prior  to  execu‐
133              tion.
134
135       --max-tests=N
136              Run at most N tests and then terminate.
137
138       --max-time=N
139              Spend  at  most N seconds (approximately) running tests and then
140              terminate.  Note that this is not an alias  for  --timeout;  the
141              two are different kinds of maximums.
142
143       --num-shards=M
144              Divide  the  set of selected tests into M equal-sized subsets or
145              "shards", and run only one of  them.   Must  be  used  with  the
146              --run-shard=N  option, which selects the shard to run. The envi‐
147              ronment variable LIT_NUM_SHARDS can also be  used  in  place  of
148              this  option.  These  two options provide a coarse mechanism for
149              partitioning large testsuites, for parallel execution  on  sepa‐
150              rate machines (say in a large testing farm).
151
152       --run-shard=N
153              Select  which  shard  to run, assuming the --num-shards=M option
154              was provided. The two options must be  used  together,  and  the
155              value  of  N must be in the range 1..M. The environment variable
156              LIT_RUN_SHARD can also be used in place of this option.
157
158       --shuffle
159              Run the tests in a random order.
160
161       --timeout=N
162              Spend at most N seconds (approximately) running each  individual
163              test.   0  means  no time limit, and 0 is the default. Note that
164              this is not an alias for --max-time; the two are different kinds
165              of maximums.
166
167       --filter=REGEXP
168              Run  only  those tests whose name matches the regular expression
169              specified in REGEXP. The environment variable LIT_FILTER can  be
170              also used in place of this option, which is especially useful in
171              environments where the call to lit is issued indirectly.
172

ADDITIONAL OPTIONS

174       --debug
175              Run lit in debug mode, for debugging  configuration  issues  and
176              lit itself.
177
178       --show-suites
179              List the discovered test suites and exit.
180
181       --show-tests
182              List all of the discovered tests and exit.
183

EXIT STATUS

185       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
186       sults.  Otherwise, it will exit with the status 0.   Other  exit  codes
187       are  used for non-test related failures (for example a user error or an
188       internal program error).
189

TEST DISCOVERY

191       The inputs passed to lit can be either individual tests, or entire  di‐
192       rectories  or  hierarchies  of  tests  to run.  When lit starts up, the
193       first thing it does is convert the inputs into a complete list of tests
194       to run as part of test discovery.
195
196       In  the  lit  model, every test must exist inside some test suite.  lit
197       resolves the inputs specified on the command line  to  test  suites  by
198       searching  upwards  from  the  input  path  until it finds a lit.cfg or
199       lit.site.cfg file.  These files serve as both a marker of  test  suites
200       and  as  configuration files which lit loads in order to understand how
201       to find and run the tests inside the test suite.
202
203       Once lit has mapped the inputs into test suites it traverses  the  list
204       of  inputs  adding tests for individual files and recursively searching
205       for tests in directories.
206
207       This behavior makes it easy to specify a subset of tests to run,  while
208       still  allowing  the  test  suite  configuration to control exactly how
209       tests are interpreted.  In addition, lit always identifies tests by the
210       test  suite they are in, and their relative path inside the test suite.
211       For appropriately configured projects, this allows lit to provide  con‐
212       venient and flexible support for out-of-tree builds.
213

TEST STATUS RESULTS

215       Each test ultimately produces one of the following eight results:
216
217       PASS
218          The test succeeded.
219
220       FLAKYPASS
221          The  test succeeded after being re-run more than once. This only ap‐
222          plies to tests containing an ALLOW_RETRIES: annotation.
223
224       XFAIL
225          The test failed, but that is expected.  This is used for  test  for‐
226          mats which allow specifying that a test does not currently work, but
227          wish to leave it in the test suite.
228
229       XPASS
230          The test succeeded, but it was expected to fail.  This is  used  for
231          tests which were specified as expected to fail, but are now succeed‐
232          ing (generally because the feature they test was broken and has been
233          fixed).
234
235       FAIL
236          The test failed.
237
238       UNRESOLVED
239          The  test  result could not be determined.  For example, this occurs
240          when the test could not be run, the test itself is invalid,  or  the
241          test was interrupted.
242
243       UNSUPPORTED
244          The test is not supported in this environment.  This is used by test
245          formats which can report unsupported tests.
246
247       TIMEOUT
248          The test was run, but it timed out before it was able  to  complete.
249          This is considered a failure.
250
251       Depending  on  the test format tests may produce additional information
252       about their status (generally only for failures).  See the  OUTPUT  OP‐
253       TIONS section for more information.
254

LIT INFRASTRUCTURE

256       This  section  describes  the lit testing architecture for users inter‐
257       ested in creating a new lit testing implementation, or extending an ex‐
258       isting one.
259
260       lit  proper  is primarily an infrastructure for discovering and running
261       arbitrary tests, and to expose a single convenient interface  to  these
262       tests.  lit  itself doesn't know how to run tests, rather this logic is
263       defined by test suites.
264
265   TEST SUITES
266       As described in TEST DISCOVERY, tests are always located inside a  test
267       suite.   Test  suites serve to define the format of the tests they con‐
268       tain, the logic for finding those tests, and any additional information
269       to run the tests.
270
271       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
272       lit.site.cfg files (see also --config-prefix).  Test  suites  are  ini‐
273       tially  discovered  by recursively searching up the directory hierarchy
274       for all the input files passed  on  the  command  line.   You  can  use
275       --show-suites to display the discovered test suites at startup.
276
277       Once  a  test  suite  is discovered, its config file is loaded.  Config
278       files themselves are Python modules which will be executed.   When  the
279       config file is executed, two important global variables are predefined:
280
281       lit_config
282          The  global  lit  configuration object (a LitConfig instance), which
283          defines the builtin test formats, global  configuration  parameters,
284          and other helper routines for implementing test configurations.
285
286       config
287          This  is  the  config object (a TestingConfig instance) for the test
288          suite, which the config file is expected to populate.  The following
289          variables  are  also  available  on the config object, some of which
290          must be set by the config and others are optional or predefined:
291
292          name [required] The name of the test suite, for use in  reports  and
293          diagnostics.
294
295          test_format  [required] The test format object which will be used to
296          discover and run tests in the test suite.  Generally this will be  a
297          builtin test format available from the lit.formats module.
298
299          test_source_root  The  filesystem  path to the test suite root.  For
300          out-of-dir builds this is the directory that  will  be  scanned  for
301          tests.
302
303          test_exec_root  For  out-of-dir  builds,  the path to the test suite
304          root inside the object directory.  This is where tests will  be  run
305          and temporary output files placed.
306
307          environment  A  dictionary  representing the environment to use when
308          executing tests in the suite.
309
310          suffixes For lit test formats which scan directories for tests, this
311          variable  is  a  list  of suffixes to identify test files.  Used by:
312          ShTest.
313
314          substitutions For lit test formats which substitute variables into a
315          test script, the list of substitutions to perform.  Used by: ShTest.
316
317          unsupported  Mark an unsupported directory, all tests within it will
318          be reported as unsupported.  Used by: ShTest.
319
320          parent The parent configuration, this is the config object  for  the
321          directory containing the test suite, or None.
322
323          root The root configuration.  This is the top-most lit configuration
324          in the project.
325
326          pipefail Normally a test using a shell pipe fails if any of the com‐
327          mands  on  the pipe fail. If this is not desired, setting this vari‐
328          able to false makes the test fail only if the last  command  in  the
329          pipe fails.
330
331          available_features  A set of features that can be used in XFAIL, RE‐
332          QUIRES, and UNSUPPORTED directives.
333
334   TEST DISCOVERY
335       Once test suites are located, lit recursively traverses the source  di‐
336       rectory  (following  test_source_root) looking for tests.  When lit en‐
337       ters a sub-directory, it first checks to see if a nested test suite  is
338       defined  in  that  directory.   If  so, it loads that test suite recur‐
339       sively, otherwise it instantiates a local test config for the directory
340       (see LOCAL CONFIGURATION FILES).
341
342       Tests  are  identified by the test suite they are contained within, and
343       the relative path inside that suite.  Note that the relative  path  may
344       not  refer  to  an  actual  file  on  disk;  some test formats (such as
345       GoogleTest) define "virtual tests" which have a path that contains both
346       the  path to the actual test file and a subpath to identify the virtual
347       test.
348
349   LOCAL CONFIGURATION FILES
350       When lit loads a subdirectory in a test suite, it instantiates a  local
351       test  configuration  by cloning the configuration for the parent direc‐
352       tory --- the root of this configuration chain will  always  be  a  test
353       suite.   Once the test configuration is cloned lit checks for a lit.lo‐
354       cal.cfg file in the subdirectory.  If present, this file will be loaded
355       and can be used to specialize the configuration for each individual di‐
356       rectory.  This facility can be used to  define  subdirectories  of  op‐
357       tional tests, or to change other configuration parameters --- for exam‐
358       ple, to change the test format, or the  suffixes  which  identify  test
359       files.
360
361   SUBSTITUTIONS
362       lit allows patterns to be substituted inside RUN commands. It also pro‐
363       vides the following base set of substitutions,  which  are  defined  in
364       TestRunner.py:
365
366                 ┌────────────────────────┬────────────────────────────┐
367                 │Macro                   │ Substitution               │
368                 ├────────────────────────┼────────────────────────────┤
369                 │%s                      │ source  path  (path to the │
370                 │                        │ file currently being run)  │
371                 ├────────────────────────┼────────────────────────────┤
372                 │%S                      │ source dir  (directory  of │
373                 │                        │ the  file  currently being │
374                 │                        │ run)                       │
375                 ├────────────────────────┼────────────────────────────┤
376                 │%p                      │ same as %S                 │
377                 ├────────────────────────┼────────────────────────────┤
378                 │%{pathsep}              │ path separator             │
379                 ├────────────────────────┼────────────────────────────┤
380                 │%t                      │ temporary file name unique │
381                 │                        │ to the test                │
382                 ├────────────────────────┼────────────────────────────┤
383                 │%basename_t             │ The last path component of │
384                 │                        │ %t but  without  the  .tmp 
385                 │                        │ extension                  │
386                 ├────────────────────────┼────────────────────────────┤
387                 │%T                      │ parent   directory  of  %t │
388                 │                        │ (not  unique,  deprecated, │
389                 │                        │ do not use)                │
390                 ├────────────────────────┼────────────────────────────┤
391                 │%%                      │ %                          │
392                 ├────────────────────────┼────────────────────────────┤
393                 │%/s                     │ %s but \ is replaced by /  
394                 └────────────────────────┴────────────────────────────┘
395
396
397                 │%/S                     │ %S but \ is replaced by /  
398                 ├────────────────────────┼────────────────────────────┤
399                 │%/p                     │ %p but \ is replaced by /  
400                 ├────────────────────────┼────────────────────────────┤
401                 │%/t                     │ %t but \ is replaced by /  
402                 ├────────────────────────┼────────────────────────────┤
403                 │%/T                     │ %T but \ is replaced by /  
404                 ├────────────────────────┼────────────────────────────┤
405                 │%{/s:regex_replacement} │ %/s but escaped for use in │
406                 │                        │ the replacement of a  s@@@ 
407                 │                        │ command in sed             │
408                 ├────────────────────────┼────────────────────────────┤
409                 │%{/S:regex_replacement} │ %/S but escaped for use in │
410                 │                        │ the replacement of a  s@@@ 
411                 │                        │ command in sed             │
412                 ├────────────────────────┼────────────────────────────┤
413                 │%{/p:regex_replacement} │ %/p but escaped for use in │
414                 │                        │ the replacement of a  s@@@ 
415                 │                        │ command in sed             │
416                 ├────────────────────────┼────────────────────────────┤
417                 │%{/t:regex_replacement} │ %/t but escaped for use in │
418                 │                        │ the replacement of a  s@@@ 
419                 │                        │ command in sed             │
420                 ├────────────────────────┼────────────────────────────┤
421                 │%{/T:regex_replacement} │ %/T but escaped for use in │
422                 │                        │ the replacement of a  s@@@ 
423                 │                        │ command in sed             │
424                 ├────────────────────────┼────────────────────────────┤
425                 │%:s                     │ On Windows, %/s but a : is │
426                 │                        │ removed if its the  second │
427                 │                        │ character.   Otherwise, %s │
428                 │                        │ but with a single  leading │
429                 │                        │ / removed.                 │
430                 ├────────────────────────┼────────────────────────────┤
431                 │%:S                     │ On Windows, %/S but a : is │
432                 │                        │ removed if its the  second │
433                 │                        │ character.   Otherwise, %S │
434                 │                        │ but with a single  leading │
435                 │                        │ / removed.                 │
436                 ├────────────────────────┼────────────────────────────┤
437                 │%:p                     │ On Windows, %/p but a : is │
438                 │                        │ removed if its the  second │
439                 │                        │ character.   Otherwise, %p │
440                 │                        │ but with a single  leading │
441                 │                        │ / removed.                 │
442                 ├────────────────────────┼────────────────────────────┤
443                 │%:t                     │ On Windows, %/t but a : is │
444                 │                        │ removed if its the  second │
445                 │                        │ character.   Otherwise, %t │
446                 │                        │ but with a single  leading │
447                 │                        │ / removed.                 │
448                 ├────────────────────────┼────────────────────────────┤
449                 │%:T                     │ On Windows, %/T but a : is │
450                 │                        │ removed if its the  second │
451                 │                        │ character.   Otherwise, %T │
452                 │                        │ but with a single  leading │
453                 │                        │ / removed.                 │
454                 └────────────────────────┴────────────────────────────┘
455
456       Other  substitutions  are provided that are variations on this base set
457       and further substitution patterns can be defined by each  test  module.
458       See the modules LOCAL CONFIGURATION FILES.
459
460       By  default, substitutions are expanded exactly once, so that if e.g. a
461       substitution %build is defined in top  of  another  substitution  %cxx,
462       %build  will  expand  to  %cxx  textually, not to what %cxx expands to.
463       However, if the recursiveExpansionLimit property of  the  TestingConfig
464       is set to a non-negative integer, substitutions will be expanded recur‐
465       sively until that limit is reached. It is an  error  if  the  limit  is
466       reached  and  expanding substitutions again would yield a different re‐
467       sult.
468
469       More detailed information on substitutions can be found in the ../Test‐
470       ingGuide.
471
472   TEST RUN OUTPUT FORMAT
473       The lit output for a test run conforms to the following schema, in both
474       short and verbose modes (although in short mode no PASS lines  will  be
475       shown).   This schema has been chosen to be relatively easy to reliably
476       parse by a machine (for example in  buildbot  log  scraping),  and  for
477       other tools to generate.
478
479       Each test result is expected to appear on a line that matches:
480
481          <result code>: <test name> (<progress info>)
482
483       where  <result-code>  is  a  standard  test  result such as PASS, FAIL,
484       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
485       of IMPROVED and REGRESSED are also allowed.
486
487       The  <test name> field can consist of an arbitrary string containing no
488       newline.
489
490       The <progress info> field can be used to  report  progress  information
491       such  as  (1/300)  or can be empty, but even when empty the parentheses
492       are required.
493
494       Each test result may include additional (multiline) log information  in
495       the following format:
496
497          <log delineator> TEST '(<test name>)' <trailing delineator>
498          ... log message ...
499          <log delineator>
500
501       where <test name> should be the name of a preceding reported test, <log
502       delineator> is a string of "*" characters at least four characters long
503       (the  recommended  length is 20), and <trailing delineator> is an arbi‐
504       trary (unparsed) string.
505
506       The following is an example of a test run output which consists of four
507       tests A, B, C, and D, and a log message for the failing test C:
508
509          PASS: A (1 of 4)
510          PASS: B (2 of 4)
511          FAIL: C (3 of 4)
512          ******************** TEST 'C' FAILED ********************
513          Test 'C' failed as a result of exit code 1.
514          ********************
515          PASS: D (4 of 4)
516
517   LIT EXAMPLE TESTS
518       The  lit  distribution contains several example implementations of test
519       suites in the ExampleTests directory.
520

SEE ALSO

522       valgrind(1)
523

AUTHOR

525       Maintained by the LLVM Team (https://llvm.org/).
526
528       2003-2021, LLVM Project
529
530
531
532
53312                                2021-05-17                            LIT(1)
Impressum