1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be  executed  (potentially  concurrently)  and
22       once  all tests have been run lit will print summary information on the
23       number of tests which passed or failed (see TEST STATUS RESULTS).   The
24       lit program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       Users interested in the lit architecture or designing a lit testing im‐
44       plementation should see LIT INFRASTRUCTURE.
45

GENERAL OPTIONS

47       -h, --help
48              Show the lit help message.
49
50       -j N, --workers=N
51              Run N tests in parallel.  By default, this is automatically cho‐
52              sen to match the number of detected available CPUs.
53
54       --config-prefix=NAME
55              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
56              suites, instead of lit.cfg and lit.site.cfg.
57
58       -D NAME[=VALUE], --param NAME[=VALUE]
59              Add  a  user defined parameter NAME with the given VALUE (or the
60              empty string if not given).  The meaning and use of these param‐
61              eters is test suite dependent.
62

OUTPUT OPTIONS

64       -q, --quiet
65              Suppress any output except for test failures.
66
67       -s, --succinct
68              Show  less  output,  for example don't show information on tests
69              that pass.  Also show a progress bar,  unless  --no-progress-bar
70              is specified.
71
72       -v, --verbose
73              Show  more  information on test failures, for example the entire
74              test output instead of just the test result.
75
76       -vv, --echo-all-commands
77              Echo all commands to stdout, as they are being  executed.   This
78              can  be valuable for debugging test failures, as the last echoed
79              command will be the one which has failed.  lit normally  inserts
80              a  no-op  command (: in the case of bash) with argument 'RUN: at
81              line N' before each  command  pipeline,  and  this  option  also
82              causes  those  no-op commands to be echoed to stdout to help you
83              locate the source line of the failed command.  This  option  im‐
84              plies --verbose.
85
86       -a, --show-all
87              Show  more  information  about all tests, for example the entire
88              test commandline and output.
89
90       --no-progress-bar
91              Do not use curses based progress bar.
92
93       --show-unsupported
94              Show the names of unsupported tests.
95
96       --show-xfail
97              Show the names of tests that were expected to fail.
98

EXECUTION OPTIONS

100       --path=PATH
101              Specify an additional PATH to use when searching for executables
102              in tests.
103
104       --vg   Run  individual  tests under valgrind (using the memcheck tool).
105              The --error-exitcode argument for valgrind is used so that  val‐
106              grind  failures  will  cause the program to exit with a non-zero
107              status.
108
109              When this option is enabled, lit will also automatically provide
110              a  "valgrind"  feature that can be used to conditionally disable
111              (or expect failure in) certain tests.
112
113       --vg-arg=ARG
114              When --vg is used, specify an additional  argument  to  pass  to
115              valgrind itself.
116
117       --vg-leak
118              When  --vg is used, enable memory leak checks.  When this option
119              is enabled, lit will also automatically provide a "vg_leak" fea‐
120              ture  that can be used to conditionally disable (or expect fail‐
121              ure in) certain tests.
122
123       --time-tests
124              Track the wall time individual tests take  to  execute  and  in‐
125              cludes  the  results  in the summary output.  This is useful for
126              determining which tests in a test suite take the  most  time  to
127              execute.
128
129       --ignore-fail
130              Exit with status zero even if some tests fail.
131
132       --no-indirectly-run-check
133              Do  not  error if a test would not be run if the user had speci‐
134              fied the containing directory instead of  naming  the  test  di‐
135              rectly.
136

SELECTION OPTIONS

138       By  default,  lit  will  run failing tests first, then run tests in de‐
139       scending execution time order to optimize concurrency.   The  execution
140       order can be changed using the --order option.
141
142       The  timing  data  is  stored  in  the  test_exec_root  in a file named
143       .lit_test_times.txt. If this file does not exist, then lit  checks  the
144       test_source_root for the file to optionally accelerate clean builds.
145
146       --shuffle
147              Run the tests in a random order, not failing/slowest first. Dep‐
148              recated, use --order instead.
149
150       --max-failures N
151              Stop execution after the given number N of failures.  An integer
152              argument  should  be  passed on the command line prior to execu‐
153              tion.
154
155       --max-tests=N
156              Run at most N tests and then terminate.
157
158       --max-time=N
159              Spend at most N seconds (approximately) running tests  and  then
160              terminate.   Note  that  this is not an alias for --timeout; the
161              two are different kinds of maximums.
162
163       --num-shards=M
164              Divide the set of selected tests into M equal-sized  subsets  or
165              "shards",  and  run  only  one  of  them.  Must be used with the
166              --run-shard=N option, which selects the shard to run. The  envi‐
167              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
168              this option. These two options provide a  coarse  mechanism  for
169              partitioning  large  testsuites, for parallel execution on sepa‐
170              rate machines (say in a large testing farm).
171
172       --order={lexical,random,smart}
173              Define the order in which tests are run.  The  supported  values
174              are:
175
176              • lexical  - tests will be run in lexical order according to the
177                test file path. This option is useful  when  predictable  test
178                order is desired.
179
180              • random - tests will be run in random order.
181
182              • smart  -  tests that failed previously will be run first, then
183                the remaining tests, all in descending execution  time  order.
184                This is the default as it optimizes concurrency.
185
186       --run-shard=N
187              Select  which  shard  to run, assuming the --num-shards=M option
188              was provided. The two options must be  used  together,  and  the
189              value  of  N must be in the range 1..M. The environment variable
190              LIT_RUN_SHARD can also be used in place of this option.
191
192       --timeout=N
193              Spend at most N seconds (approximately) running each  individual
194              test.   0  means  no time limit, and 0 is the default. Note that
195              this is not an alias for --max-time; the two are different kinds
196              of maximums.
197
198       --filter=REGEXP
199              Run  only  those tests whose name matches the regular expression
200              specified in REGEXP. The environment variable LIT_FILTER can  be
201              also used in place of this option, which is especially useful in
202              environments where the call to lit is issued indirectly.
203
204       --filter-out=REGEXP
205              Filter out those tests whose name matches the regular expression
206              specified in REGEXP. The environment variable LIT_FILTER_OUT can
207              be also used in place of this option, which is especially useful
208              in environments where the call to lit is issued indirectly.
209
210       --xfail=LIST
211              Treat  those tests whose name is in the semicolon separated list
212              LIST as XFAIL. This can be helpful when one  does  not  want  to
213              modify the test suite. The environment variable LIT_XFAIL can be
214              also used in place of this option, which is especially useful in
215              environments where the call to lit is issued indirectly.
216
217              A  test  name  can specified as a file name relative to the test
218              suite directory.  For example:
219
220                 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
221
222              In this case, all of the following tests are treated as XFAIL:
223
224                 libomp :: affinity/kmp-hw-subset.c
225                 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
226                 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
227
228              Alternatively, a test name can be specified  as  the  full  test
229              name  reported  in  LIT  output.  For example, we can adjust the
230              previous example not to treat the nvptx64-nvidia-cuda version of
231              offloading/memory_manager.cpp as XFAIL:
232
233                 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
234
235       --xfail-not=LIST
236              Do  not  treat  the  specified  tests as XFAIL.  The environment
237              variable LIT_XFAIL_NOT can also be used in place of this option.
238              The   syntax   is   the  same  as  for  --xfail  and  LIT_XFAIL.
239              --xfail-not and LIT_XFAIL_NOT always override  all  other  XFAIL
240              specifications, including an --xfail appearing later on the com‐
241              mand line.  The primary purpose is to suppress an  XPASS  result
242              without modifying a test case that uses the XFAIL directive.
243

ADDITIONAL OPTIONS

245       --debug
246              Run  lit  in  debug mode, for debugging configuration issues and
247              lit itself.
248
249       --show-suites
250              List the discovered test suites and exit.
251
252       --show-tests
253              List all of the discovered tests and exit.
254

EXIT STATUS

256       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
257       sults.   Otherwise,  it  will exit with the status 0.  Other exit codes
258       are used for non-test related failures (for example a user error or  an
259       internal program error).
260

TEST DISCOVERY

262       The  inputs passed to lit can be either individual tests, or entire di‐
263       rectories or hierarchies of tests to run.   When  lit  starts  up,  the
264       first thing it does is convert the inputs into a complete list of tests
265       to run as part of test discovery.
266
267       In the lit model, every test must exist inside some  test  suite.   lit
268       resolves  the  inputs  specified  on the command line to test suites by
269       searching upwards from the input path  until  it  finds  a  lit.cfg  or
270       lit.site.cfg  file.   These files serve as both a marker of test suites
271       and as configuration files which lit loads in order to  understand  how
272       to find and run the tests inside the test suite.
273
274       Once  lit  has mapped the inputs into test suites it traverses the list
275       of inputs adding tests for individual files and  recursively  searching
276       for tests in directories.
277
278       This  behavior makes it easy to specify a subset of tests to run, while
279       still allowing the test suite  configuration  to  control  exactly  how
280       tests are interpreted.  In addition, lit always identifies tests by the
281       test suite they are in, and their relative path inside the test  suite.
282       For  appropriately configured projects, this allows lit to provide con‐
283       venient and flexible support for out-of-tree builds.
284

TEST STATUS RESULTS

286       Each test ultimately produces one of the following eight results:
287
288       PASS
289          The test succeeded.
290
291       FLAKYPASS
292          The test succeeded after being re-run more than once. This only  ap‐
293          plies to tests containing an ALLOW_RETRIES: annotation.
294
295       XFAIL
296          The  test  failed, but that is expected.  This is used for test for‐
297          mats which allow specifying that a test does not currently work, but
298          wish to leave it in the test suite.
299
300       XPASS
301          The  test  succeeded, but it was expected to fail.  This is used for
302          tests which were specified as expected to fail, but are now succeed‐
303          ing (generally because the feature they test was broken and has been
304          fixed).
305
306       FAIL
307          The test failed.
308
309       UNRESOLVED
310          The test result could not be determined.  For example,  this  occurs
311          when  the  test could not be run, the test itself is invalid, or the
312          test was interrupted.
313
314       UNSUPPORTED
315          The test is not supported in this environment.  This is used by test
316          formats which can report unsupported tests.
317
318       TIMEOUT
319          The  test  was run, but it timed out before it was able to complete.
320          This is considered a failure.
321
322       Depending on the test format tests may produce  additional  information
323       about  their  status (generally only for failures).  See the OUTPUT OP‐
324       TIONS section for more information.
325

LIT INFRASTRUCTURE

327       This section describes the lit testing architecture  for  users  inter‐
328       ested in creating a new lit testing implementation, or extending an ex‐
329       isting one.
330
331       lit proper is primarily an infrastructure for discovering  and  running
332       arbitrary  tests,  and to expose a single convenient interface to these
333       tests. lit itself doesn't know how to run tests, rather this  logic  is
334       defined by test suites.
335
336   TEST SUITES
337       As  described in TEST DISCOVERY, tests are always located inside a test
338       suite.  Test suites serve to define the format of the tests  they  con‐
339       tain, the logic for finding those tests, and any additional information
340       to run the tests.
341
342       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
343       lit.site.cfg  files  (see  also --config-prefix).  Test suites are ini‐
344       tially discovered by recursively searching up the  directory  hierarchy
345       for  all  the  input  files  passed  on  the command line.  You can use
346       --show-suites to display the discovered test suites at startup.
347
348       Once a test suite is discovered, its config  file  is  loaded.   Config
349       files  themselves  are Python modules which will be executed.  When the
350       config file is executed, two important global variables are predefined:
351
352       lit_config
353          The global lit configuration object (a  LitConfig  instance),  which
354          defines  the  builtin test formats, global configuration parameters,
355          and other helper routines for implementing test configurations.
356
357       config
358          This is the config object (a TestingConfig instance)  for  the  test
359          suite, which the config file is expected to populate.  The following
360          variables are also available on the config  object,  some  of  which
361          must be set by the config and others are optional or predefined:
362
363          name  [required]  The name of the test suite, for use in reports and
364          diagnostics.
365
366          test_format [required] The test format object which will be used  to
367          discover  and run tests in the test suite.  Generally this will be a
368          builtin test format available from the lit.formats module.
369
370          test_source_root The filesystem path to the test  suite  root.   For
371          out-of-dir  builds  this  is  the directory that will be scanned for
372          tests.
373
374          test_exec_root For out-of-dir builds, the path  to  the  test  suite
375          root  inside  the object directory.  This is where tests will be run
376          and temporary output files placed.
377
378          environment A dictionary representing the environment  to  use  when
379          executing tests in the suite.
380
381          standalone_tests  When true, mark a directory with tests expected to
382          be run standalone. Test discovery is disabled for that directory and
383          --no-indirectly-run-check  is  in  effect.  lit.suffixes and lit.ex‐
384          cludes must be empty when this variable is true.
385
386          suffixes For lit test formats which scan directories for tests, this
387          variable  is  a  list  of suffixes to identify test files.  Used by:
388          ShTest.
389
390          substitutions For lit test formats which substitute variables into a
391          test script, the list of substitutions to perform.  Used by: ShTest.
392
393          unsupported  Mark an unsupported directory, all tests within it will
394          be reported as unsupported.  Used by: ShTest.
395
396          parent The parent configuration, this is the config object  for  the
397          directory containing the test suite, or None.
398
399          root The root configuration.  This is the top-most lit configuration
400          in the project.
401
402          pipefail Normally a test using a shell pipe fails if any of the com‐
403          mands  on  the pipe fail. If this is not desired, setting this vari‐
404          able to false makes the test fail only if the last  command  in  the
405          pipe fails.
406
407          available_features  A set of features that can be used in XFAIL, RE‐
408          QUIRES, and UNSUPPORTED directives.
409
410   TEST DISCOVERY
411       Once test suites are located, lit recursively traverses the source  di‐
412       rectory  (following  test_source_root) looking for tests.  When lit en‐
413       ters a sub-directory, it first checks to see if a nested test suite  is
414       defined  in  that  directory.   If  so, it loads that test suite recur‐
415       sively, otherwise it instantiates a local test config for the directory
416       (see LOCAL CONFIGURATION FILES).
417
418       Tests  are  identified by the test suite they are contained within, and
419       the relative path inside that suite.  Note that the relative  path  may
420       not  refer  to  an  actual  file  on  disk;  some test formats (such as
421       GoogleTest) define "virtual tests" which have a path that contains both
422       the  path to the actual test file and a subpath to identify the virtual
423       test.
424
425   LOCAL CONFIGURATION FILES
426       When lit loads a subdirectory in a test suite, it instantiates a  local
427       test  configuration  by cloning the configuration for the parent direc‐
428       tory --- the root of this configuration chain will  always  be  a  test
429       suite.   Once the test configuration is cloned lit checks for a lit.lo‐
430       cal.cfg file in the subdirectory.  If present, this file will be loaded
431       and can be used to specialize the configuration for each individual di‐
432       rectory.  This facility can be used to  define  subdirectories  of  op‐
433       tional tests, or to change other configuration parameters --- for exam‐
434       ple, to change the test format, or the  suffixes  which  identify  test
435       files.
436
437   SUBSTITUTIONS
438       lit allows patterns to be substituted inside RUN commands. It also pro‐
439       vides the following base set of substitutions,  which  are  defined  in
440       TestRunner.py:
441
442                 ┌────────────────────────┬────────────────────────────┐
443                 │Macro                   │ Substitution               │
444                 ├────────────────────────┼────────────────────────────┤
445                 │%s                      │ source  path  (path to the │
446                 │                        │ file currently being run)  │
447                 ├────────────────────────┼────────────────────────────┤
448                 │%S                      │ source dir  (directory  of │
449                 │                        │ the  file  currently being │
450                 │                        │ run)                       │
451                 ├────────────────────────┼────────────────────────────┤
452                 │%p                      │ same as %S                 │
453                 ├────────────────────────┼────────────────────────────┤
454                 │%{pathsep}              │ path separator             │
455                 ├────────────────────────┼────────────────────────────┤
456                 │%t                      │ temporary file name unique │
457                 │                        │ to the test                │
458                 ├────────────────────────┼────────────────────────────┤
459                 │%basename_t             │ The last path component of │
460                 │                        │ %t but  without  the  .tmp 
461                 │                        │ extension                  │
462                 ├────────────────────────┼────────────────────────────┤
463                 │%T                      │ parent   directory  of  %t │
464                 │                        │ (not  unique,  deprecated, │
465                 │                        │ do not use)                │
466                 ├────────────────────────┼────────────────────────────┤
467                 │%%                      │ %                          │
468                 ├────────────────────────┼────────────────────────────┤
469                 │%/s                     │ %s but \ is replaced by /  
470                 ├────────────────────────┼────────────────────────────┤
471                 │%/S                     │ %S but \ is replaced by /  
472                 └────────────────────────┴────────────────────────────┘
473
474
475                 │%/p                     │ %p but \ is replaced by /  
476                 ├────────────────────────┼────────────────────────────┤
477                 │%/t                     │ %t but \ is replaced by /  
478                 ├────────────────────────┼────────────────────────────┤
479                 │%/T                     │ %T but \ is replaced by /  
480                 ├────────────────────────┼────────────────────────────┤
481                 │%{/s:regex_replacement} │ %/s but escaped for use in │
482                 │                        │ the replacement of a  s@@@ 
483                 │                        │ command in sed             │
484                 ├────────────────────────┼────────────────────────────┤
485                 │%{/S:regex_replacement} │ %/S but escaped for use in │
486                 │                        │ the replacement of a  s@@@ 
487                 │                        │ command in sed             │
488                 ├────────────────────────┼────────────────────────────┤
489                 │%{/p:regex_replacement} │ %/p but escaped for use in │
490                 │                        │ the replacement of a  s@@@ 
491                 │                        │ command in sed             │
492                 ├────────────────────────┼────────────────────────────┤
493                 │%{/t:regex_replacement} │ %/t but escaped for use in │
494                 │                        │ the replacement of a  s@@@ 
495                 │                        │ command in sed             │
496                 ├────────────────────────┼────────────────────────────┤
497                 │%{/T:regex_replacement} │ %/T but escaped for use in │
498                 │                        │ the replacement of a  s@@@ 
499                 │                        │ command in sed             │
500                 ├────────────────────────┼────────────────────────────┤
501                 │%:s                     │ On Windows, %/s but a : is │
502                 │                        │ removed if its the  second │
503                 │                        │ character.   Otherwise, %s │
504                 │                        │ but with a single  leading │
505                 │                        │ / removed.                 │
506                 ├────────────────────────┼────────────────────────────┤
507                 │%:S                     │ On Windows, %/S but a : is │
508                 │                        │ removed if its the  second │
509                 │                        │ character.   Otherwise, %S │
510                 │                        │ but with a single  leading │
511                 │                        │ / removed.                 │
512                 ├────────────────────────┼────────────────────────────┤
513                 │%:p                     │ On Windows, %/p but a : is │
514                 │                        │ removed if its the  second │
515                 │                        │ character.   Otherwise, %p │
516                 │                        │ but with a single  leading │
517                 │                        │ / removed.                 │
518                 ├────────────────────────┼────────────────────────────┤
519                 │%:t                     │ On Windows, %/t but a : is │
520                 │                        │ removed if its the  second │
521                 │                        │ character.   Otherwise, %t │
522                 │                        │ but with a single  leading │
523                 │                        │ / removed.                 │
524                 ├────────────────────────┼────────────────────────────┤
525                 │%:T                     │ On Windows, %/T but a : is │
526                 │                        │ removed if its the  second │
527                 │                        │ character.   Otherwise, %T │
528                 │                        │ but with a single  leading │
529                 │                        │ / removed.                 │
530                 └────────────────────────┴────────────────────────────┘
531
532       Other  substitutions  are provided that are variations on this base set
533       and further substitution patterns can be defined by each  test  module.
534       See the modules LOCAL CONFIGURATION FILES.
535
536       By  default, substitutions are expanded exactly once, so that if e.g. a
537       substitution %build is defined in top  of  another  substitution  %cxx,
538       %build  will  expand  to  %cxx  textually, not to what %cxx expands to.
539       However, if the recursiveExpansionLimit property of  the  TestingConfig
540       is set to a non-negative integer, substitutions will be expanded recur‐
541       sively until that limit is reached. It is an  error  if  the  limit  is
542       reached  and  expanding substitutions again would yield a different re‐
543       sult.
544
545       More detailed information on substitutions can be found in the ../Test‐
546       ingGuide.
547
548   TEST RUN OUTPUT FORMAT
549       The lit output for a test run conforms to the following schema, in both
550       short and verbose modes (although in short mode no PASS lines  will  be
551       shown).   This schema has been chosen to be relatively easy to reliably
552       parse by a machine (for example in  buildbot  log  scraping),  and  for
553       other tools to generate.
554
555       Each test result is expected to appear on a line that matches:
556
557          <result code>: <test name> (<progress info>)
558
559       where  <result-code>  is  a  standard  test  result such as PASS, FAIL,
560       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
561       of IMPROVED and REGRESSED are also allowed.
562
563       The  <test name> field can consist of an arbitrary string containing no
564       newline.
565
566       The <progress info> field can be used to  report  progress  information
567       such  as  (1/300)  or can be empty, but even when empty the parentheses
568       are required.
569
570       Each test result may include additional (multiline) log information  in
571       the following format:
572
573          <log delineator> TEST '(<test name>)' <trailing delineator>
574          ... log message ...
575          <log delineator>
576
577       where <test name> should be the name of a preceding reported test, <log
578       delineator> is a string of "*" characters at least four characters long
579       (the  recommended  length is 20), and <trailing delineator> is an arbi‐
580       trary (unparsed) string.
581
582       The following is an example of a test run output which consists of four
583       tests A, B, C, and D, and a log message for the failing test C:
584
585          PASS: A (1 of 4)
586          PASS: B (2 of 4)
587          FAIL: C (3 of 4)
588          ******************** TEST 'C' FAILED ********************
589          Test 'C' failed as a result of exit code 1.
590          ********************
591          PASS: D (4 of 4)
592
593   LIT EXAMPLE TESTS
594       The  lit  distribution contains several example implementations of test
595       suites in the ExampleTests directory.
596

SEE ALSO

598       valgrind(1)
599

AUTHOR

601       Maintained by the LLVM Team (https://llvm.org/).
602
604       2003-2022, LLVM Project
605
606
607
608
60914                                2022-03-23                            LIT(1)
Impressum