1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be executed (potentially in parallel) and once
22       all  tests have been run lit will print summary information on the num‐
23       ber of tests which passed or failed (see TEST STATUS RESULTS).  The lit
24       program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       Users interested in the lit architecture or designing a lit testing im‐
44       plementation should see LIT INFRASTRUCTURE.
45

GENERAL OPTIONS

47       -h, --help
48              Show the lit help message.
49
50       -j N, --workers=N
51              Run N tests in parallel.  By default, this is automatically cho‐
52              sen to match the number of detected available CPUs.
53
54       --config-prefix=NAME
55              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
56              suites, instead of lit.cfg and lit.site.cfg.
57
58       -D NAME[=VALUE], --param NAME[=VALUE]
59              Add  a  user defined parameter NAME with the given VALUE (or the
60              empty string if not given).  The meaning and use of these param‐
61              eters is test suite dependent.
62

OUTPUT OPTIONS

64       -q, --quiet
65              Suppress any output except for test failures.
66
67       -s, --succinct
68              Show  less  output,  for example don't show information on tests
69              that pass.
70
71       -v, --verbose
72              Show more information on test failures, for example  the  entire
73              test output instead of just the test result.
74
75       -vv, --echo-all-commands
76              Echo  all  commands to stdout, as they are being executed.  This
77              can be valuable for debugging test failures, as the last  echoed
78              command  will be the one which has failed.  lit normally inserts
79              a no-op command (: in the case of bash) with argument  'RUN:  at
80              line  N'  before  each  command  pipeline,  and this option also
81              causes those no-op commands to be echoed to stdout to  help  you
82              locate  the  source line of the failed command.  This option im‐
83              plies --verbose.
84
85       -a, --show-all
86              Show more information about all tests, for  example  the  entire
87              test commandline and output.
88
89       --no-progress-bar
90              Do not use curses based progress bar.
91
92       --show-unsupported
93              Show the names of unsupported tests.
94
95       --show-xfail
96              Show the names of tests that were expected to fail.
97

EXECUTION OPTIONS

99       --path=PATH
100              Specify an additional PATH to use when searching for executables
101              in tests.
102
103       --vg   Run individual tests under valgrind (using the  memcheck  tool).
104              The  --error-exitcode argument for valgrind is used so that val‐
105              grind failures will cause the program to exit  with  a  non-zero
106              status.
107
108              When this option is enabled, lit will also automatically provide
109              a "valgrind" feature that can be used to  conditionally  disable
110              (or expect failure in) certain tests.
111
112       --vg-arg=ARG
113              When  --vg  is  used,  specify an additional argument to pass to
114              valgrind itself.
115
116       --vg-leak
117              When --vg is used, enable memory leak checks.  When this  option
118              is enabled, lit will also automatically provide a "vg_leak" fea‐
119              ture that can be used to conditionally disable (or expect  fail‐
120              ure in) certain tests.
121
122       --time-tests
123              Track  the  wall  time  individual tests take to execute and in‐
124              cludes the results in the summary output.  This  is  useful  for
125              determining  which  tests  in a test suite take the most time to
126              execute.  Note that this option is most useful with -j 1.
127

SELECTION OPTIONS

129       --max-failures N
130              Stop execution after the given number N of failures.  An integer
131              argument  should  be  passed on the command line prior to execu‐
132              tion.
133
134       --max-tests=N
135              Run at most N tests and then terminate.
136
137       --max-time=N
138              Spend at most N seconds (approximately) running tests  and  then
139              terminate.   Note  that  this is not an alias for --timeout; the
140              two are different kinds of maximums.
141
142       --num-shards=M
143              Divide the set of selected tests into M equal-sized  subsets  or
144              "shards",  and  run  only  one  of  them.  Must be used with the
145              --run-shard=N option, which selects the shard to run. The  envi‐
146              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
147              this option. These two options provide a  coarse  mechanism  for
148              partitioning  large  testsuites, for parallel execution on sepa‐
149              rate machines (say in a large testing farm).
150
151       --run-shard=N
152              Select which shard to run, assuming  the  --num-shards=M  option
153              was  provided.  The  two  options must be used together, and the
154              value of N must be in the range 1..M. The  environment  variable
155              LIT_RUN_SHARD can also be used in place of this option.
156
157       --shuffle
158              Run the tests in a random order.
159
160       --timeout=N
161              Spend  at most N seconds (approximately) running each individual
162              test.  0 means no time limit, and 0 is the  default.  Note  that
163              this is not an alias for --max-time; the two are different kinds
164              of maximums.
165
166       --filter=REGEXP
167              Run only those tests whose name matches the  regular  expression
168              specified  in REGEXP. The environment variable LIT_FILTER can be
169              also used in place of this option, which is especially useful in
170              environments where the call to lit is issued indirectly.
171

ADDITIONAL OPTIONS

173       --debug
174              Run  lit  in  debug mode, for debugging configuration issues and
175              lit itself.
176
177       --show-suites
178              List the discovered test suites and exit.
179
180       --show-tests
181              List all of the discovered tests and exit.
182

EXIT STATUS

184       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
185       sults.   Otherwise,  it  will exit with the status 0.  Other exit codes
186       are used for non-test related failures (for example a user error or  an
187       internal program error).
188

TEST DISCOVERY

190       The  inputs passed to lit can be either individual tests, or entire di‐
191       rectories or hierarchies of tests to run.   When  lit  starts  up,  the
192       first thing it does is convert the inputs into a complete list of tests
193       to run as part of test discovery.
194
195       In the lit model, every test must exist inside some  test  suite.   lit
196       resolves  the  inputs  specified  on the command line to test suites by
197       searching upwards from the input path  until  it  finds  a  lit.cfg  or
198       lit.site.cfg  file.   These files serve as both a marker of test suites
199       and as configuration files which lit loads in order to  understand  how
200       to find and run the tests inside the test suite.
201
202       Once  lit  has mapped the inputs into test suites it traverses the list
203       of inputs adding tests for individual files and  recursively  searching
204       for tests in directories.
205
206       This  behavior makes it easy to specify a subset of tests to run, while
207       still allowing the test suite  configuration  to  control  exactly  how
208       tests are interpreted.  In addition, lit always identifies tests by the
209       test suite they are in, and their relative path inside the test  suite.
210       For  appropriately configured projects, this allows lit to provide con‐
211       venient and flexible support for out-of-tree builds.
212

TEST STATUS RESULTS

214       Each test ultimately produces one of the following eight results:
215
216       PASS
217          The test succeeded.
218
219       FLAKYPASS
220          The test succeeded after being re-run more than once. This only  ap‐
221          plies to tests containing an ALLOW_RETRIES: annotation.
222
223       XFAIL
224          The  test  failed, but that is expected.  This is used for test for‐
225          mats which allow specifying that a test does not currently work, but
226          wish to leave it in the test suite.
227
228       XPASS
229          The  test  succeeded, but it was expected to fail.  This is used for
230          tests which were specified as expected to fail, but are now succeed‐
231          ing (generally because the feature they test was broken and has been
232          fixed).
233
234       FAIL
235          The test failed.
236
237       UNRESOLVED
238          The test result could not be determined.  For example,  this  occurs
239          when  the  test could not be run, the test itself is invalid, or the
240          test was interrupted.
241
242       UNSUPPORTED
243          The test is not supported in this environment.  This is used by test
244          formats which can report unsupported tests.
245
246       TIMEOUT
247          The  test  was run, but it timed out before it was able to complete.
248          This is considered a failure.
249
250       Depending on the test format tests may produce  additional  information
251       about  their  status (generally only for failures).  See the OUTPUT OP‐
252       TIONS section for more information.
253

LIT INFRASTRUCTURE

255       This section describes the lit testing architecture  for  users  inter‐
256       ested in creating a new lit testing implementation, or extending an ex‐
257       isting one.
258
259       lit proper is primarily an infrastructure for discovering  and  running
260       arbitrary  tests,  and to expose a single convenient interface to these
261       tests. lit itself doesn't know how to run tests, rather this  logic  is
262       defined by test suites.
263
264   TEST SUITES
265       As  described in TEST DISCOVERY, tests are always located inside a test
266       suite.  Test suites serve to define the format of the tests  they  con‐
267       tain, the logic for finding those tests, and any additional information
268       to run the tests.
269
270       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
271       lit.site.cfg  files  (see  also --config-prefix).  Test suites are ini‐
272       tially discovered by recursively searching up the  directory  hierarchy
273       for  all  the  input  files  passed  on  the command line.  You can use
274       --show-suites to display the discovered test suites at startup.
275
276       Once a test suite is discovered, its config  file  is  loaded.   Config
277       files  themselves  are Python modules which will be executed.  When the
278       config file is executed, two important global variables are predefined:
279
280       lit_config
281          The global lit configuration object (a  LitConfig  instance),  which
282          defines  the  builtin test formats, global configuration parameters,
283          and other helper routines for implementing test configurations.
284
285       config
286          This is the config object (a TestingConfig instance)  for  the  test
287          suite, which the config file is expected to populate.  The following
288          variables are also available on the config  object,  some  of  which
289          must be set by the config and others are optional or predefined:
290
291          name  [required]  The name of the test suite, for use in reports and
292          diagnostics.
293
294          test_format [required] The test format object which will be used  to
295          discover  and run tests in the test suite.  Generally this will be a
296          builtin test format available from the lit.formats module.
297
298          test_source_root The filesystem path to the test  suite  root.   For
299          out-of-dir  builds  this  is  the directory that will be scanned for
300          tests.
301
302          test_exec_root For out-of-dir builds, the path  to  the  test  suite
303          root  inside  the object directory.  This is where tests will be run
304          and temporary output files placed.
305
306          environment A dictionary representing the environment  to  use  when
307          executing tests in the suite.
308
309          suffixes For lit test formats which scan directories for tests, this
310          variable is a list of suffixes to identify  test  files.   Used  by:
311          ShTest.
312
313          substitutions For lit test formats which substitute variables into a
314          test script, the list of substitutions to perform.  Used by: ShTest.
315
316          unsupported Mark an unsupported directory, all tests within it  will
317          be reported as unsupported.  Used by: ShTest.
318
319          parent  The  parent configuration, this is the config object for the
320          directory containing the test suite, or None.
321
322          root The root configuration.  This is the top-most lit configuration
323          in the project.
324
325          pipefail Normally a test using a shell pipe fails if any of the com‐
326          mands on the pipe fail. If this is not desired, setting  this  vari‐
327          able  to  false  makes the test fail only if the last command in the
328          pipe fails.
329
330          available_features A set of features that can be used in XFAIL,  RE‐
331          QUIRES, and UNSUPPORTED directives.
332
333   TEST DISCOVERY
334       Once  test suites are located, lit recursively traverses the source di‐
335       rectory (following test_source_root) looking for tests.  When  lit  en‐
336       ters  a sub-directory, it first checks to see if a nested test suite is
337       defined in that directory.  If so, it  loads  that  test  suite  recur‐
338       sively, otherwise it instantiates a local test config for the directory
339       (see LOCAL CONFIGURATION FILES).
340
341       Tests are identified by the test suite they are contained  within,  and
342       the  relative  path inside that suite.  Note that the relative path may
343       not refer to an actual  file  on  disk;  some  test  formats  (such  as
344       GoogleTest) define "virtual tests" which have a path that contains both
345       the path to the actual test file and a subpath to identify the  virtual
346       test.
347
348   LOCAL CONFIGURATION FILES
349       When  lit loads a subdirectory in a test suite, it instantiates a local
350       test configuration by cloning the configuration for the  parent  direc‐
351       tory  ---  the  root  of this configuration chain will always be a test
352       suite.  Once the test configuration is cloned lit checks for a  lit.lo‐
353       cal.cfg file in the subdirectory.  If present, this file will be loaded
354       and can be used to specialize the configuration for each individual di‐
355       rectory.   This  facility  can  be used to define subdirectories of op‐
356       tional tests, or to change other configuration parameters --- for exam‐
357       ple,  to  change  the  test format, or the suffixes which identify test
358       files.
359
360   SUBSTITUTIONS
361       lit allows patterns to be substituted inside RUN commands. It also pro‐
362       vides  the  following  base  set of substitutions, which are defined in
363       TestRunner.py:
364
365                 ┌────────────────────────┬────────────────────────────┐
366                 │Macro                   │ Substitution               │
367                 ├────────────────────────┼────────────────────────────┤
368                 │%s                      │ source path (path  to  the │
369                 │                        │ file currently being run)  │
370                 ├────────────────────────┼────────────────────────────┤
371                 │%S                      │ source  dir  (directory of │
372                 │                        │ the file  currently  being │
373                 │                        │ run)                       │
374                 ├────────────────────────┼────────────────────────────┤
375                 │%p                      │ same as %S                 │
376                 ├────────────────────────┼────────────────────────────┤
377                 │%{pathsep}              │ path separator             │
378                 ├────────────────────────┼────────────────────────────┤
379                 │%t                      │ temporary file name unique │
380                 │                        │ to the test                │
381                 ├────────────────────────┼────────────────────────────┤
382                 │%basename_t             │ The last path component of │
383                 │                        │ %t  but  without  the .tmp 
384                 │                        │ extension                  │
385                 ├────────────────────────┼────────────────────────────┤
386                 │%T                      │ parent  directory  of   %t │
387                 │                        │ (not  unique,  deprecated, │
388                 │                        │ do not use)                │
389                 ├────────────────────────┼────────────────────────────┤
390                 │%%                      │ %                          │
391                 ├────────────────────────┼────────────────────────────┤
392                 │%/s                     │ %s but \ is replaced by /  
393                 ├────────────────────────┼────────────────────────────┤
394                 │%/S                     │ %S but \ is replaced by /  
395                 ├────────────────────────┼────────────────────────────┤
396                 │%/p                     │ %p but \ is replaced by /  
397                 ├────────────────────────┼────────────────────────────┤
398                 │%/t                     │ %t but \ is replaced by /  
399                 ├────────────────────────┼────────────────────────────┤
400                 │%/T                     │ %T but \ is replaced by /  
401                 └────────────────────────┴────────────────────────────┘
402
403
404
405                 │%{/s:regex_replacement} │ %/s but escaped for use in │
406                 │                        │ the  replacement of a s@@@ 
407                 │                        │ command in sed             │
408                 ├────────────────────────┼────────────────────────────┤
409                 │%{/S:regex_replacement} │ %/S but escaped for use in │
410                 │                        │ the  replacement of a s@@@ 
411                 │                        │ command in sed             │
412                 ├────────────────────────┼────────────────────────────┤
413                 │%{/p:regex_replacement} │ %/p but escaped for use in │
414                 │                        │ the  replacement of a s@@@ 
415                 │                        │ command in sed             │
416                 ├────────────────────────┼────────────────────────────┤
417                 │%{/t:regex_replacement} │ %/t but escaped for use in │
418                 │                        │ the  replacement of a s@@@ 
419                 │                        │ command in sed             │
420                 ├────────────────────────┼────────────────────────────┤
421                 │%{/T:regex_replacement} │ %/T but escaped for use in │
422                 │                        │ the  replacement of a s@@@ 
423                 │                        │ command in sed             │
424                 ├────────────────────────┼────────────────────────────┤
425                 │%:s                     │ On Windows, %/s but a : is │
426                 │                        │ removed  if its the second │
427                 │                        │ character.  Otherwise,  %s │
428                 │                        │ but  with a single leading │
429                 │                        │ / removed.                 │
430                 ├────────────────────────┼────────────────────────────┤
431                 │%:S                     │ On Windows, %/S but a : is │
432                 │                        │ removed  if its the second │
433                 │                        │ character.  Otherwise,  %S │
434                 │                        │ but  with a single leading │
435                 │                        │ / removed.                 │
436                 ├────────────────────────┼────────────────────────────┤
437                 │%:p                     │ On Windows, %/p but a : is │
438                 │                        │ removed  if its the second │
439                 │                        │ character.  Otherwise,  %p │
440                 │                        │ but  with a single leading │
441                 │                        │ / removed.                 │
442                 ├────────────────────────┼────────────────────────────┤
443                 │%:t                     │ On Windows, %/t but a : is │
444                 │                        │ removed  if its the second │
445                 │                        │ character.  Otherwise,  %t │
446                 │                        │ but  with a single leading │
447                 │                        │ / removed.                 │
448                 ├────────────────────────┼────────────────────────────┤
449                 │%:T                     │ On Windows, %/T but a : is │
450                 │                        │ removed  if its the second │
451                 │                        │ character.  Otherwise,  %T │
452                 │                        │ but  with a single leading │
453                 │                        │ / removed.                 │
454                 └────────────────────────┴────────────────────────────┘
455
456       Other substitutions are provided that are variations on this  base  set
457       and  further  substitution patterns can be defined by each test module.
458       See the modules LOCAL CONFIGURATION FILES.
459
460       By default, substitutions are expanded exactly once, so that if e.g.  a
461       substitution  %build  is  defined  in top of another substitution %cxx,
462       %build will expand to %cxx textually, not  to  what  %cxx  expands  to.
463       However,  if  the recursiveExpansionLimit property of the TestingConfig
464       is set to a non-negative integer, substitutions will be expanded recur‐
465       sively  until  that  limit  is  reached. It is an error if the limit is
466       reached and expanding substitutions again would yield a  different  re‐
467       sult.
468
469       More detailed information on substitutions can be found in the ../Test‐
470       ingGuide.
471
472   TEST RUN OUTPUT FORMAT
473       The lit output for a test run conforms to the following schema, in both
474       short  and  verbose modes (although in short mode no PASS lines will be
475       shown).  This schema has been chosen to be relatively easy to  reliably
476       parse  by  a  machine  (for  example in buildbot log scraping), and for
477       other tools to generate.
478
479       Each test result is expected to appear on a line that matches:
480
481          <result code>: <test name> (<progress info>)
482
483       where <result-code> is a standard  test  result  such  as  PASS,  FAIL,
484       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
485       of IMPROVED and REGRESSED are also allowed.
486
487       The <test name> field can consist of an arbitrary string containing  no
488       newline.
489
490       The  <progress  info>  field can be used to report progress information
491       such as (1/300) or can be empty, but even when  empty  the  parentheses
492       are required.
493
494       Each  test result may include additional (multiline) log information in
495       the following format:
496
497          <log delineator> TEST '(<test name>)' <trailing delineator>
498          ... log message ...
499          <log delineator>
500
501       where <test name> should be the name of a preceding reported test, <log
502       delineator> is a string of "*" characters at least four characters long
503       (the recommended length is 20), and <trailing delineator> is  an  arbi‐
504       trary (unparsed) string.
505
506       The following is an example of a test run output which consists of four
507       tests A, B, C, and D, and a log message for the failing test C:
508
509          PASS: A (1 of 4)
510          PASS: B (2 of 4)
511          FAIL: C (3 of 4)
512          ******************** TEST 'C' FAILED ********************
513          Test 'C' failed as a result of exit code 1.
514          ********************
515          PASS: D (4 of 4)
516
517   LIT EXAMPLE TESTS
518       The lit distribution contains several example implementations  of  test
519       suites in the ExampleTests directory.
520

SEE ALSO

522       valgrind(1)
523

AUTHOR

525       Maintained by the LLVM Team (https://llvm.org/).
526
528       2003-2021, LLVM Project
529
530
531
532
53311                                2021-07-22                            LIT(1)
Impressum