1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be  executed  (potentially  concurrently)  and
22       once  all tests have been run lit will print summary information on the
23       number of tests which passed or failed (see TEST STATUS RESULTS).   The
24       lit program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       lit can also read options from response files which  are  specified  as
44       inputs  using  the @path/to/file.rsp syntax. Arguments read from a file
45       must be one per line and are treated as if they were in the same  place
46       as  the  original  file referencing argument on the command line. A re‐
47       sponse file can reference other response files.
48
49       Users interested in the lit architecture or designing a lit testing im‐
50       plementation should see LIT INFRASTRUCTURE.
51

GENERAL OPTIONS

53       -h, --help
54              Show the lit help message.
55
56       -j N, --workers=N
57              Run N tests in parallel.  By default, this is automatically cho‐
58              sen to match the number of detected available CPUs.
59
60       --config-prefix=NAME
61              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
62              suites, instead of lit.cfg and lit.site.cfg.
63
64       -D NAME[=VALUE], --param NAME[=VALUE]
65              Add  a  user defined parameter NAME with the given VALUE (or the
66              empty string if not given).  The meaning and use of these param‐
67              eters is test suite dependent.
68

OUTPUT OPTIONS

70       -q, --quiet
71              Suppress any output except for test failures.
72
73       -s, --succinct
74              Show  less  output,  for example don't show information on tests
75              that pass.  Also show a progress bar,  unless  --no-progress-bar
76              is specified.
77
78       -v, --verbose
79              Show  more  information on test failures, for example the entire
80              test output instead of just the test result.
81
82       -vv, --echo-all-commands
83              Echo all commands to stdout, as they are being  executed.   This
84              can  be valuable for debugging test failures, as the last echoed
85              command will be the one which has failed.  lit normally  inserts
86              a  no-op  command (: in the case of bash) with argument 'RUN: at
87              line N' before each  command  pipeline,  and  this  option  also
88              causes  those  no-op commands to be echoed to stdout to help you
89              locate the source line of the failed command.  This  option  im‐
90              plies --verbose.
91
92       -a, --show-all
93              Show  more  information  about all tests, for example the entire
94              test commandline and output.
95
96       --no-progress-bar
97              Do not use curses based progress bar.
98
99       --show-unsupported
100              Show the names of unsupported tests.
101
102       --show-xfail
103              Show the names of tests that were expected to fail.
104

EXECUTION OPTIONS

106       --path=PATH
107              Specify an additional PATH to use when searching for executables
108              in tests.
109
110       --vg   Run  individual  tests under valgrind (using the memcheck tool).
111              The --error-exitcode argument for valgrind is used so that  val‐
112              grind  failures  will  cause the program to exit with a non-zero
113              status.
114
115              When this option is enabled, lit will also automatically provide
116              a  "valgrind"  feature that can be used to conditionally disable
117              (or expect failure in) certain tests.
118
119       --vg-arg=ARG
120              When --vg is used, specify an additional  argument  to  pass  to
121              valgrind itself.
122
123       --vg-leak
124              When  --vg is used, enable memory leak checks.  When this option
125              is enabled, lit will also automatically provide a "vg_leak" fea‐
126              ture  that can be used to conditionally disable (or expect fail‐
127              ure in) certain tests.
128
129       --time-tests
130              Track the wall time individual tests take  to  execute  and  in‐
131              cludes  the  results  in the summary output.  This is useful for
132              determining which tests in a test suite take the  most  time  to
133              execute.
134
135       --ignore-fail
136              Exit with status zero even if some tests fail.
137
138       --no-indirectly-run-check
139              Do  not  error if a test would not be run if the user had speci‐
140              fied the containing directory instead of  naming  the  test  di‐
141              rectly.
142

SELECTION OPTIONS

144       By  default,  lit  will  run failing tests first, then run tests in de‐
145       scending execution time order to optimize concurrency.   The  execution
146       order can be changed using the --order option.
147
148       The  timing  data  is  stored  in  the  test_exec_root  in a file named
149       .lit_test_times.txt. If this file does not exist, then lit  checks  the
150       test_source_root for the file to optionally accelerate clean builds.
151
152       --shuffle
153              Run the tests in a random order, not failing/slowest first. Dep‐
154              recated, use --order instead.
155
156       --max-failures N
157              Stop execution after the given number N of failures.  An integer
158              argument  should  be  passed on the command line prior to execu‐
159              tion.
160
161       --max-tests=N
162              Run at most N tests and then terminate.
163
164       --max-time=N
165              Spend at most N seconds (approximately) running tests  and  then
166              terminate.   Note  that  this is not an alias for --timeout; the
167              two are different kinds of maximums.
168
169       --num-shards=M
170              Divide the set of selected tests into M equal-sized  subsets  or
171              "shards",  and  run  only  one  of  them.  Must be used with the
172              --run-shard=N option, which selects the shard to run. The  envi‐
173              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
174              this option. These two options provide a  coarse  mechanism  for
175              partitioning  large  testsuites, for parallel execution on sepa‐
176              rate machines (say in a large testing farm).
177
178       --order={lexical,random,smart}
179              Define the order in which tests are run.  The  supported  values
180              are:
181
182              • lexical  - tests will be run in lexical order according to the
183                test file path. This option is useful  when  predictable  test
184                order is desired.
185
186              • random - tests will be run in random order.
187
188              • smart  -  tests that failed previously will be run first, then
189                the remaining tests, all in descending execution  time  order.
190                This is the default as it optimizes concurrency.
191
192       --run-shard=N
193              Select  which  shard  to run, assuming the --num-shards=M option
194              was provided. The two options must be  used  together,  and  the
195              value  of  N must be in the range 1..M. The environment variable
196              LIT_RUN_SHARD can also be used in place of this option.
197
198       --timeout=N
199              Spend at most N seconds (approximately) running each  individual
200              test.   0  means  no time limit, and 0 is the default. Note that
201              this is not an alias for --max-time; the two are different kinds
202              of maximums.
203
204       --filter=REGEXP
205              Run  only  those tests whose name matches the regular expression
206              specified in REGEXP. The environment variable LIT_FILTER can  be
207              also used in place of this option, which is especially useful in
208              environments where the call to lit is issued indirectly.
209
210       --filter-out=REGEXP
211              Filter out those tests whose name matches the regular expression
212              specified in REGEXP. The environment variable LIT_FILTER_OUT can
213              be also used in place of this option, which is especially useful
214              in environments where the call to lit is issued indirectly.
215
216       --xfail=LIST
217              Treat  those tests whose name is in the semicolon separated list
218              LIST as XFAIL. This can be helpful when one  does  not  want  to
219              modify the test suite. The environment variable LIT_XFAIL can be
220              also used in place of this option, which is especially useful in
221              environments where the call to lit is issued indirectly.
222
223              A  test  name  can specified as a file name relative to the test
224              suite directory.  For example:
225
226                 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
227
228              In this case, all of the following tests are treated as XFAIL:
229
230                 libomp :: affinity/kmp-hw-subset.c
231                 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
232                 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
233
234              Alternatively, a test name can be specified  as  the  full  test
235              name  reported  in  LIT  output.  For example, we can adjust the
236              previous example not to treat the nvptx64-nvidia-cuda version of
237              offloading/memory_manager.cpp as XFAIL:
238
239                 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
240
241       --xfail-not=LIST
242              Do  not  treat  the  specified  tests as XFAIL.  The environment
243              variable LIT_XFAIL_NOT can also be used in place of this option.
244              The   syntax   is   the  same  as  for  --xfail  and  LIT_XFAIL.
245              --xfail-not and LIT_XFAIL_NOT always override  all  other  XFAIL
246              specifications, including an --xfail appearing later on the com‐
247              mand line.  The primary purpose is to suppress an  XPASS  result
248              without modifying a test case that uses the XFAIL directive.
249

ADDITIONAL OPTIONS

251       --debug
252              Run  lit  in  debug mode, for debugging configuration issues and
253              lit itself.
254
255       --show-suites
256              List the discovered test suites and exit.
257
258       --show-tests
259              List all of the discovered tests and exit.
260

EXIT STATUS

262       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
263       sults.   Otherwise,  it  will exit with the status 0.  Other exit codes
264       are used for non-test related failures (for example a user error or  an
265       internal program error).
266

TEST DISCOVERY

268       The  inputs passed to lit can be either individual tests, or entire di‐
269       rectories or hierarchies of tests to run.   When  lit  starts  up,  the
270       first thing it does is convert the inputs into a complete list of tests
271       to run as part of test discovery.
272
273       In the lit model, every test must exist inside some  test  suite.   lit
274       resolves  the  inputs  specified  on the command line to test suites by
275       searching upwards from the input path  until  it  finds  a  lit.cfg  or
276       lit.site.cfg  file.   These files serve as both a marker of test suites
277       and as configuration files which lit loads in order to  understand  how
278       to find and run the tests inside the test suite.
279
280       Once  lit  has mapped the inputs into test suites it traverses the list
281       of inputs adding tests for individual files and  recursively  searching
282       for tests in directories.
283
284       This  behavior makes it easy to specify a subset of tests to run, while
285       still allowing the test suite  configuration  to  control  exactly  how
286       tests are interpreted.  In addition, lit always identifies tests by the
287       test suite they are in, and their relative path inside the test  suite.
288       For  appropriately configured projects, this allows lit to provide con‐
289       venient and flexible support for out-of-tree builds.
290

TEST STATUS RESULTS

292       Each test ultimately produces one of the following eight results:
293
294       PASS
295          The test succeeded.
296
297       FLAKYPASS
298          The test succeeded after being re-run more than once. This only  ap‐
299          plies to tests containing an ALLOW_RETRIES: annotation.
300
301       XFAIL
302          The  test  failed, but that is expected.  This is used for test for‐
303          mats which allow specifying that a test does not currently work, but
304          wish to leave it in the test suite.
305
306       XPASS
307          The  test  succeeded, but it was expected to fail.  This is used for
308          tests which were specified as expected to fail, but are now succeed‐
309          ing (generally because the feature they test was broken and has been
310          fixed).
311
312       FAIL
313          The test failed.
314
315       UNRESOLVED
316          The test result could not be determined.  For example,  this  occurs
317          when  the  test could not be run, the test itself is invalid, or the
318          test was interrupted.
319
320       UNSUPPORTED
321          The test is not supported in this environment.  This is used by test
322          formats which can report unsupported tests.
323
324       TIMEOUT
325          The  test  was run, but it timed out before it was able to complete.
326          This is considered a failure.
327
328       Depending on the test format tests may produce  additional  information
329       about  their  status (generally only for failures).  See the OUTPUT OP‐
330       TIONS section for more information.
331

LIT INFRASTRUCTURE

333       This section describes the lit testing architecture  for  users  inter‐
334       ested in creating a new lit testing implementation, or extending an ex‐
335       isting one.
336
337       lit proper is primarily an infrastructure for discovering  and  running
338       arbitrary  tests,  and to expose a single convenient interface to these
339       tests. lit itself doesn't know how to run tests, rather this  logic  is
340       defined by test suites.
341
342   TEST SUITES
343       As  described in TEST DISCOVERY, tests are always located inside a test
344       suite.  Test suites serve to define the format of the tests  they  con‐
345       tain, the logic for finding those tests, and any additional information
346       to run the tests.
347
348       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
349       lit.site.cfg  files  (see  also --config-prefix).  Test suites are ini‐
350       tially discovered by recursively searching up the  directory  hierarchy
351       for  all  the  input  files  passed  on  the command line.  You can use
352       --show-suites to display the discovered test suites at startup.
353
354       Once a test suite is discovered, its config  file  is  loaded.   Config
355       files  themselves  are Python modules which will be executed.  When the
356       config file is executed, two important global variables are predefined:
357
358       lit_config
359          The global lit configuration object (a  LitConfig  instance),  which
360          defines  the  builtin test formats, global configuration parameters,
361          and other helper routines for implementing test configurations.
362
363       config
364          This is the config object (a TestingConfig instance)  for  the  test
365          suite, which the config file is expected to populate.  The following
366          variables are also available on the config  object,  some  of  which
367          must be set by the config and others are optional or predefined:
368
369          name  [required]  The name of the test suite, for use in reports and
370          diagnostics.
371
372          test_format [required] The test format object which will be used  to
373          discover  and run tests in the test suite.  Generally this will be a
374          builtin test format available from the lit.formats module.
375
376          test_source_root The filesystem path to the test  suite  root.   For
377          out-of-dir  builds  this  is  the directory that will be scanned for
378          tests.
379
380          test_exec_root For out-of-dir builds, the path  to  the  test  suite
381          root  inside  the object directory.  This is where tests will be run
382          and temporary output files placed.
383
384          environment A dictionary representing the environment  to  use  when
385          executing tests in the suite.
386
387          standalone_tests  When true, mark a directory with tests expected to
388          be run standalone. Test discovery is disabled for that directory and
389          --no-indirectly-run-check  is  in  effect.  lit.suffixes and lit.ex‐
390          cludes must be empty when this variable is true.
391
392          suffixes For lit test formats which scan directories for tests, this
393          variable  is  a  list  of suffixes to identify test files.  Used by:
394          ShTest.
395
396          substitutions For lit test formats which substitute variables into a
397          test script, the list of substitutions to perform.  Used by: ShTest.
398
399          unsupported  Mark an unsupported directory, all tests within it will
400          be reported as unsupported.  Used by: ShTest.
401
402          parent The parent configuration, this is the config object  for  the
403          directory containing the test suite, or None.
404
405          root The root configuration.  This is the top-most lit configuration
406          in the project.
407
408          pipefail Normally a test using a shell pipe fails if any of the com‐
409          mands  on  the pipe fail. If this is not desired, setting this vari‐
410          able to false makes the test fail only if the last  command  in  the
411          pipe fails.
412
413          available_features  A set of features that can be used in XFAIL, RE‐
414          QUIRES, and UNSUPPORTED directives.
415
416   TEST DISCOVERY
417       Once test suites are located, lit recursively traverses the source  di‐
418       rectory  (following  test_source_root) looking for tests.  When lit en‐
419       ters a sub-directory, it first checks to see if a nested test suite  is
420       defined  in  that  directory.   If  so, it loads that test suite recur‐
421       sively, otherwise it instantiates a local test config for the directory
422       (see LOCAL CONFIGURATION FILES).
423
424       Tests  are  identified by the test suite they are contained within, and
425       the relative path inside that suite.  Note that the relative  path  may
426       not  refer  to  an  actual  file  on  disk;  some test formats (such as
427       GoogleTest) define "virtual tests" which have a path that contains both
428       the  path to the actual test file and a subpath to identify the virtual
429       test.
430
431   LOCAL CONFIGURATION FILES
432       When lit loads a subdirectory in a test suite, it instantiates a  local
433       test  configuration  by cloning the configuration for the parent direc‐
434       tory --- the root of this configuration chain will  always  be  a  test
435       suite.   Once the test configuration is cloned lit checks for a lit.lo‐
436       cal.cfg file in the subdirectory.  If present, this file will be loaded
437       and can be used to specialize the configuration for each individual di‐
438       rectory.  This facility can be used to  define  subdirectories  of  op‐
439       tional tests, or to change other configuration parameters --- for exam‐
440       ple, to change the test format, or the  suffixes  which  identify  test
441       files.
442
443   SUBSTITUTIONS
444       lit allows patterns to be substituted inside RUN commands. It also pro‐
445       vides the following base set of substitutions,  which  are  defined  in
446       TestRunner.py:
447
448                 ┌────────────────────────┬────────────────────────────┐
449                 │Macro                   │ Substitution               │
450                 ├────────────────────────┼────────────────────────────┤
451                 │%s                      │ source  path  (path to the │
452                 │                        │ file currently being run)  │
453                 ├────────────────────────┼────────────────────────────┤
454                 │%S                      │ source dir  (directory  of │
455                 │                        │ the  file  currently being │
456                 │                        │ run)                       │
457                 ├────────────────────────┼────────────────────────────┤
458                 │%p                      │ same as %S                 │
459                 ├────────────────────────┼────────────────────────────┤
460                 │%{pathsep}              │ path separator             │
461                 ├────────────────────────┼────────────────────────────┤
462                 │%{fs-src-root}          │ root  component  of   file │
463                 │                        │ system  paths  pointing to │
464                 │                        │ the LLVM checkout          │
465                 ├────────────────────────┼────────────────────────────┤
466                 │%{fs-tmp-root}          │ root  component  of   file │
467                 │                        │ system  paths  pointing to │
468                 │                        │ the test's  temporary  di‐ │
469                 │                        │ rectory                    │
470                 ├────────────────────────┼────────────────────────────┤
471                 │%{fs-sep}               │ file system path separator │
472                 ├────────────────────────┼────────────────────────────┤
473                 │%t                      │ temporary file name unique │
474                 │                        │ to the test                │
475                 └────────────────────────┴────────────────────────────┘
476
477                 │%basename_t             │ The last path component of │
478                 │                        │ %t  but  without  the .tmp 
479                 │                        │ extension                  │
480                 ├────────────────────────┼────────────────────────────┤
481                 │%T                      │ parent  directory  of   %t │
482                 │                        │ (not  unique,  deprecated, │
483                 │                        │ do not use)                │
484                 ├────────────────────────┼────────────────────────────┤
485                 │%%                      │ %                          │
486                 ├────────────────────────┼────────────────────────────┤
487                 │%/s                     │ %s but \ is replaced by /  
488                 ├────────────────────────┼────────────────────────────┤
489                 │%/S                     │ %S but \ is replaced by /  
490                 ├────────────────────────┼────────────────────────────┤
491                 │%/p                     │ %p but \ is replaced by /  
492                 ├────────────────────────┼────────────────────────────┤
493                 │%/t                     │ %t but \ is replaced by /  
494                 ├────────────────────────┼────────────────────────────┤
495                 │%/T                     │ %T but \ is replaced by /  
496                 ├────────────────────────┼────────────────────────────┤
497                 │%{/s:regex_replacement} │ %/s but escaped for use in │
498                 │                        │ the  replacement of a s@@@ 
499                 │                        │ command in sed             │
500                 ├────────────────────────┼────────────────────────────┤
501                 │%{/S:regex_replacement} │ %/S but escaped for use in │
502                 │                        │ the  replacement of a s@@@ 
503                 │                        │ command in sed             │
504                 ├────────────────────────┼────────────────────────────┤
505                 │%{/p:regex_replacement} │ %/p but escaped for use in │
506                 │                        │ the  replacement of a s@@@ 
507                 │                        │ command in sed             │
508                 ├────────────────────────┼────────────────────────────┤
509                 │%{/t:regex_replacement} │ %/t but escaped for use in │
510                 │                        │ the  replacement of a s@@@ 
511                 │                        │ command in sed             │
512                 ├────────────────────────┼────────────────────────────┤
513                 │%{/T:regex_replacement} │ %/T but escaped for use in │
514                 │                        │ the  replacement of a s@@@ 
515                 │                        │ command in sed             │
516                 ├────────────────────────┼────────────────────────────┤
517                 │%:s                     │ On Windows, %/s but a : is │
518                 │                        │ removed  if its the second │
519                 │                        │ character.  Otherwise,  %s │
520                 │                        │ but  with a single leading │
521                 │                        │ / removed.                 │
522                 ├────────────────────────┼────────────────────────────┤
523                 │%:S                     │ On Windows, %/S but a : is │
524                 │                        │ removed  if its the second │
525                 │                        │ character.  Otherwise,  %S │
526                 │                        │ but  with a single leading │
527                 │                        │ / removed.                 │
528                 ├────────────────────────┼────────────────────────────┤
529                 │%:p                     │ On Windows, %/p but a : is │
530                 │                        │ removed  if its the second │
531                 │                        │ character.  Otherwise,  %p │
532                 │                        │ but  with a single leading │
533                 │                        │ / removed.                 │
534                 ├────────────────────────┼────────────────────────────┤
535                 │%:t                     │ On Windows, %/t but a : is │
536                 │                        │ removed  if its the second │
537                 │                        │ character.  Otherwise,  %t │
538                 │                        │ but  with a single leading │
539                 │                        │ / removed.                 │
540                 └────────────────────────┴────────────────────────────┘
541
542
543
544
545
546                 │%:T                     │ On Windows, %/T but a : is │
547                 │                        │ removed  if its the second │
548                 │                        │ character.  Otherwise,  %T │
549                 │                        │ but  with a single leading │
550                 │                        │ / removed.                 │
551                 └────────────────────────┴────────────────────────────┘
552
553       Other substitutions are provided that are variations on this  base  set
554       and  further  substitution patterns can be defined by each test module.
555       See the modules LOCAL CONFIGURATION FILES.
556
557       More detailed information on substitutions can be  found  in  the  LLVM
558       Testing Infrastructure Guide.
559
560   TEST RUN OUTPUT FORMAT
561       The lit output for a test run conforms to the following schema, in both
562       short and verbose modes (although in short mode no PASS lines  will  be
563       shown).   This schema has been chosen to be relatively easy to reliably
564       parse by a machine (for example in  buildbot  log  scraping),  and  for
565       other tools to generate.
566
567       Each test result is expected to appear on a line that matches:
568
569          <result code>: <test name> (<progress info>)
570
571       where  <result-code>  is  a  standard  test  result such as PASS, FAIL,
572       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
573       of IMPROVED and REGRESSED are also allowed.
574
575       The  <test name> field can consist of an arbitrary string containing no
576       newline.
577
578       The <progress info> field can be used to  report  progress  information
579       such  as  (1/300)  or can be empty, but even when empty the parentheses
580       are required.
581
582       Each test result may include additional (multiline) log information  in
583       the following format:
584
585          <log delineator> TEST '(<test name>)' <trailing delineator>
586          ... log message ...
587          <log delineator>
588
589       where <test name> should be the name of a preceding reported test, <log
590       delineator> is a string of "*" characters at least four characters long
591       (the  recommended  length is 20), and <trailing delineator> is an arbi‐
592       trary (unparsed) string.
593
594       The following is an example of a test run output which consists of four
595       tests A, B, C, and D, and a log message for the failing test C:
596
597          PASS: A (1 of 4)
598          PASS: B (2 of 4)
599          FAIL: C (3 of 4)
600          ******************** TEST 'C' FAILED ********************
601          Test 'C' failed as a result of exit code 1.
602          ********************
603          PASS: D (4 of 4)
604
605   LIT EXAMPLE TESTS
606       The  lit  distribution contains several example implementations of test
607       suites in the ExampleTests directory.
608

SEE ALSO

610       valgrind(1)
611

AUTHOR

613       Maintained by the LLVM Team (https://llvm.org/).
614
616       2003-2023, LLVM Project
617
618
619
620
62116                                2023-07-20                            LIT(1)
Impressum