1LIT(1)                               LLVM                               LIT(1)
2
3
4

NAME

6       lit - LLVM Integrated Tester
7

SYNOPSIS

9       lit [options] [tests]
10

DESCRIPTION

12       lit  is a portable tool for executing LLVM and Clang style test suites,
13       summarizing their results, and providing indication of  failures.   lit
14       is  designed to be a lightweight testing tool with as simple a user in‐
15       terface as possible.
16
17       lit should be run with one or more tests to run specified on  the  com‐
18       mand line.  Tests can be either individual test files or directories to
19       search for tests (see TEST DISCOVERY).
20
21       Each specified test will be  executed  (potentially  concurrently)  and
22       once  all tests have been run lit will print summary information on the
23       number of tests which passed or failed (see TEST STATUS RESULTS).   The
24       lit program will execute with a non-zero exit code if any tests fail.
25
26       By default lit will use a succinct progress display and will only print
27       summary information for test failures.  See OUTPUT OPTIONS for  options
28       controlling the lit progress display and output.
29
30       lit also includes a number of options for controlling how tests are ex‐
31       ecuted (specific features may depend on the  particular  test  format).
32       See EXECUTION OPTIONS for more information.
33
34       Finally, lit also supports additional options for only running a subset
35       of the options specified on the command line, see SELECTION OPTIONS for
36       more information.
37
38       lit parses options from the environment variable LIT_OPTS after parsing
39       options from the command line.  LIT_OPTS is primarily useful  for  sup‐
40       plementing  or  overriding  the command-line options supplied to lit by
41       check targets defined by a project's build system.
42
43       Users interested in the lit architecture or designing a lit testing im‐
44       plementation should see LIT INFRASTRUCTURE.
45

GENERAL OPTIONS

47       -h, --help
48              Show the lit help message.
49
50       -j N, --workers=N
51              Run N tests in parallel.  By default, this is automatically cho‐
52              sen to match the number of detected available CPUs.
53
54       --config-prefix=NAME
55              Search for NAME.cfg and NAME.site.cfg when  searching  for  test
56              suites, instead of lit.cfg and lit.site.cfg.
57
58       -D NAME[=VALUE], --param NAME[=VALUE]
59              Add  a  user defined parameter NAME with the given VALUE (or the
60              empty string if not given).  The meaning and use of these param‐
61              eters is test suite dependent.
62

OUTPUT OPTIONS

64       -q, --quiet
65              Suppress any output except for test failures.
66
67       -s, --succinct
68              Show  less  output,  for example don't show information on tests
69              that pass.  Also show a progress bar,  unless  --no-progress-bar
70              is specified.
71
72       -v, --verbose
73              Show  more  information on test failures, for example the entire
74              test output instead of just the test result.
75
76       -vv, --echo-all-commands
77              Echo all commands to stdout, as they are being  executed.   This
78              can  be valuable for debugging test failures, as the last echoed
79              command will be the one which has failed.  lit normally  inserts
80              a  no-op  command (: in the case of bash) with argument 'RUN: at
81              line N' before each  command  pipeline,  and  this  option  also
82              causes  those  no-op commands to be echoed to stdout to help you
83              locate the source line of the failed command.  This  option  im‐
84              plies --verbose.
85
86       -a, --show-all
87              Show  more  information  about all tests, for example the entire
88              test commandline and output.
89
90       --no-progress-bar
91              Do not use curses based progress bar.
92
93       --show-unsupported
94              Show the names of unsupported tests.
95
96       --show-xfail
97              Show the names of tests that were expected to fail.
98

EXECUTION OPTIONS

100       --path=PATH
101              Specify an additional PATH to use when searching for executables
102              in tests.
103
104       --vg   Run  individual  tests under valgrind (using the memcheck tool).
105              The --error-exitcode argument for valgrind is used so that  val‐
106              grind  failures  will  cause the program to exit with a non-zero
107              status.
108
109              When this option is enabled, lit will also automatically provide
110              a  "valgrind"  feature that can be used to conditionally disable
111              (or expect failure in) certain tests.
112
113       --vg-arg=ARG
114              When --vg is used, specify an additional  argument  to  pass  to
115              valgrind itself.
116
117       --vg-leak
118              When  --vg is used, enable memory leak checks.  When this option
119              is enabled, lit will also automatically provide a "vg_leak" fea‐
120              ture  that can be used to conditionally disable (or expect fail‐
121              ure in) certain tests.
122
123       --time-tests
124              Track the wall time individual tests take  to  execute  and  in‐
125              cludes  the  results  in the summary output.  This is useful for
126              determining which tests in a test suite take the  most  time  to
127              execute.
128
129       --ignore-fail
130              Exit with status zero even if some tests fail.
131
132       --no-indirectly-run-check
133              Do  not  error if a test would not be run if the user had speci‐
134              fied the containing directory instead of  naming  the  test  di‐
135              rectly.
136

SELECTION OPTIONS

138       By  default,  lit  will  run failing tests first, then run tests in de‐
139       scending execution time order to optimize concurrency.
140
141       The timing data is  stored  in  the  test_exec_root  in  a  file  named
142       .lit_test_times.txt.  If  this file does not exist, then lit checks the
143       test_source_root for the file to optionally accelerate clean builds.
144
145       --shuffle
146              Run the tests in a random order, not failing/slowest first.
147
148       --max-failures N
149              Stop execution after the given number N of failures.  An integer
150              argument  should  be  passed on the command line prior to execu‐
151              tion.
152
153       --max-tests=N
154              Run at most N tests and then terminate.
155
156       --max-time=N
157              Spend at most N seconds (approximately) running tests  and  then
158              terminate.   Note  that  this is not an alias for --timeout; the
159              two are different kinds of maximums.
160
161       --num-shards=M
162              Divide the set of selected tests into M equal-sized  subsets  or
163              "shards",  and  run  only  one  of  them.  Must be used with the
164              --run-shard=N option, which selects the shard to run. The  envi‐
165              ronment  variable  LIT_NUM_SHARDS  can  also be used in place of
166              this option. These two options provide a  coarse  mechanism  for
167              partitioning  large  testsuites, for parallel execution on sepa‐
168              rate machines (say in a large testing farm).
169
170       --run-shard=N
171              Select which shard to run, assuming  the  --num-shards=M  option
172              was  provided.  The  two  options must be used together, and the
173              value of N must be in the range 1..M. The  environment  variable
174              LIT_RUN_SHARD can also be used in place of this option.
175
176       --timeout=N
177              Spend  at most N seconds (approximately) running each individual
178              test.  0 means no time limit, and 0 is the  default.  Note  that
179              this is not an alias for --max-time; the two are different kinds
180              of maximums.
181
182       --filter=REGEXP
183              Run only those tests whose name matches the  regular  expression
184              specified  in REGEXP. The environment variable LIT_FILTER can be
185              also used in place of this option, which is especially useful in
186              environments where the call to lit is issued indirectly.
187
188       --filter-out=REGEXP
189              Filter out those tests whose name matches the regular expression
190              specified in REGEXP. The environment variable LIT_FILTER_OUT can
191              be also used in place of this option, which is especially useful
192              in environments where the call to lit is issued indirectly.
193
194       --xfail=LIST
195              Treat those tests whose name is in the semicolon separated  list
196              LIST  as  XFAIL.  This  can be helpful when one does not want to
197              modify the test suite. The environment variable LIT_XFAIL can be
198              also used in place of this option, which is especially useful in
199              environments where the call to lit is issued indirectly.
200
201              A test name can specified as a file name relative  to  the  test
202              suite directory.  For example:
203
204                 LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"
205
206              In this case, all of the following tests are treated as XFAIL:
207
208                 libomp :: affinity/kmp-hw-subset.c
209                 libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
210                 libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp
211
212              Alternatively,  a  test  name  can be specified as the full test
213              name reported in LIT output.  For example,  we  can  adjust  the
214              previous example not to treat the nvptx64-nvidia-cuda version of
215              offloading/memory_manager.cpp as XFAIL:
216
217                 LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"
218
219       --xfail-not=LIST
220              Do not treat the specified  tests  as  XFAIL.   The  environment
221              variable LIT_XFAIL_NOT can also be used in place of this option.
222              The  syntax  is  the  same  as  for   --xfail   and   LIT_XFAIL.
223              --xfail-not  and  LIT_XFAIL_NOT  always override all other XFAIL
224              specifications, including an --xfail appearing later on the com‐
225              mand  line.   The primary purpose is to suppress an XPASS result
226              without modifying a test case that uses the XFAIL directive.
227

ADDITIONAL OPTIONS

229       --debug
230              Run lit in debug mode, for debugging  configuration  issues  and
231              lit itself.
232
233       --show-suites
234              List the discovered test suites and exit.
235
236       --show-tests
237              List all of the discovered tests and exit.
238

EXIT STATUS

240       lit will exit with an exit code of 1 if there are any FAIL or XPASS re‐
241       sults.  Otherwise, it will exit with the status 0.   Other  exit  codes
242       are  used for non-test related failures (for example a user error or an
243       internal program error).
244

TEST DISCOVERY

246       The inputs passed to lit can be either individual tests, or entire  di‐
247       rectories  or  hierarchies  of  tests  to run.  When lit starts up, the
248       first thing it does is convert the inputs into a complete list of tests
249       to run as part of test discovery.
250
251       In  the  lit  model, every test must exist inside some test suite.  lit
252       resolves the inputs specified on the command line  to  test  suites  by
253       searching  upwards  from  the  input  path  until it finds a lit.cfg or
254       lit.site.cfg file.  These files serve as both a marker of  test  suites
255       and  as  configuration files which lit loads in order to understand how
256       to find and run the tests inside the test suite.
257
258       Once lit has mapped the inputs into test suites it traverses  the  list
259       of  inputs  adding tests for individual files and recursively searching
260       for tests in directories.
261
262       This behavior makes it easy to specify a subset of tests to run,  while
263       still  allowing  the  test  suite  configuration to control exactly how
264       tests are interpreted.  In addition, lit always identifies tests by the
265       test  suite they are in, and their relative path inside the test suite.
266       For appropriately configured projects, this allows lit to provide  con‐
267       venient and flexible support for out-of-tree builds.
268

TEST STATUS RESULTS

270       Each test ultimately produces one of the following eight results:
271
272       PASS
273          The test succeeded.
274
275       FLAKYPASS
276          The  test succeeded after being re-run more than once. This only ap‐
277          plies to tests containing an ALLOW_RETRIES: annotation.
278
279       XFAIL
280          The test failed, but that is expected.  This is used for  test  for‐
281          mats which allow specifying that a test does not currently work, but
282          wish to leave it in the test suite.
283
284       XPASS
285          The test succeeded, but it was expected to fail.  This is  used  for
286          tests which were specified as expected to fail, but are now succeed‐
287          ing (generally because the feature they test was broken and has been
288          fixed).
289
290       FAIL
291          The test failed.
292
293       UNRESOLVED
294          The  test  result could not be determined.  For example, this occurs
295          when the test could not be run, the test itself is invalid,  or  the
296          test was interrupted.
297
298       UNSUPPORTED
299          The test is not supported in this environment.  This is used by test
300          formats which can report unsupported tests.
301
302       TIMEOUT
303          The test was run, but it timed out before it was able  to  complete.
304          This is considered a failure.
305
306       Depending  on  the test format tests may produce additional information
307       about their status (generally only for failures).  See the  OUTPUT  OP‐
308       TIONS section for more information.
309

LIT INFRASTRUCTURE

311       This  section  describes  the lit testing architecture for users inter‐
312       ested in creating a new lit testing implementation, or extending an ex‐
313       isting one.
314
315       lit  proper  is primarily an infrastructure for discovering and running
316       arbitrary tests, and to expose a single convenient interface  to  these
317       tests.  lit  itself doesn't know how to run tests, rather this logic is
318       defined by test suites.
319
320   TEST SUITES
321       As described in TEST DISCOVERY, tests are always located inside a  test
322       suite.   Test  suites serve to define the format of the tests they con‐
323       tain, the logic for finding those tests, and any additional information
324       to run the tests.
325
326       lit  identifies  test  suites  as  directories  containing  lit.cfg  or
327       lit.site.cfg files (see also --config-prefix).  Test  suites  are  ini‐
328       tially  discovered  by recursively searching up the directory hierarchy
329       for all the input files passed  on  the  command  line.   You  can  use
330       --show-suites to display the discovered test suites at startup.
331
332       Once  a  test  suite  is discovered, its config file is loaded.  Config
333       files themselves are Python modules which will be executed.   When  the
334       config file is executed, two important global variables are predefined:
335
336       lit_config
337          The  global  lit  configuration object (a LitConfig instance), which
338          defines the builtin test formats, global  configuration  parameters,
339          and other helper routines for implementing test configurations.
340
341       config
342          This  is  the  config object (a TestingConfig instance) for the test
343          suite, which the config file is expected to populate.  The following
344          variables  are  also  available  on the config object, some of which
345          must be set by the config and others are optional or predefined:
346
347          name [required] The name of the test suite, for use in  reports  and
348          diagnostics.
349
350          test_format  [required] The test format object which will be used to
351          discover and run tests in the test suite.  Generally this will be  a
352          builtin test format available from the lit.formats module.
353
354          test_source_root  The  filesystem  path to the test suite root.  For
355          out-of-dir builds this is the directory that  will  be  scanned  for
356          tests.
357
358          test_exec_root  For  out-of-dir  builds,  the path to the test suite
359          root inside the object directory.  This is where tests will  be  run
360          and temporary output files placed.
361
362          environment  A  dictionary  representing the environment to use when
363          executing tests in the suite.
364
365          standalone_tests When true, mark a directory with tests expected  to
366          be run standalone. Test discovery is disabled for that directory and
367          --no-indirectly-run-check is in  effect.  lit.suffixes  and  lit.ex‐
368          cludes must be empty when this variable is true.
369
370          suffixes For lit test formats which scan directories for tests, this
371          variable is a list of suffixes to identify  test  files.   Used  by:
372          ShTest.
373
374          substitutions For lit test formats which substitute variables into a
375          test script, the list of substitutions to perform.  Used by: ShTest.
376
377          unsupported Mark an unsupported directory, all tests within it  will
378          be reported as unsupported.  Used by: ShTest.
379
380          parent  The  parent configuration, this is the config object for the
381          directory containing the test suite, or None.
382
383          root The root configuration.  This is the top-most lit configuration
384          in the project.
385
386          pipefail Normally a test using a shell pipe fails if any of the com‐
387          mands on the pipe fail. If this is not desired, setting  this  vari‐
388          able  to  false  makes the test fail only if the last command in the
389          pipe fails.
390
391          available_features A set of features that can be used in XFAIL,  RE‐
392          QUIRES, and UNSUPPORTED directives.
393
394   TEST DISCOVERY
395       Once  test suites are located, lit recursively traverses the source di‐
396       rectory (following test_source_root) looking for tests.  When  lit  en‐
397       ters  a sub-directory, it first checks to see if a nested test suite is
398       defined in that directory.  If so, it  loads  that  test  suite  recur‐
399       sively, otherwise it instantiates a local test config for the directory
400       (see LOCAL CONFIGURATION FILES).
401
402       Tests are identified by the test suite they are contained  within,  and
403       the  relative  path inside that suite.  Note that the relative path may
404       not refer to an actual  file  on  disk;  some  test  formats  (such  as
405       GoogleTest) define "virtual tests" which have a path that contains both
406       the path to the actual test file and a subpath to identify the  virtual
407       test.
408
409   LOCAL CONFIGURATION FILES
410       When  lit loads a subdirectory in a test suite, it instantiates a local
411       test configuration by cloning the configuration for the  parent  direc‐
412       tory  ---  the  root  of this configuration chain will always be a test
413       suite.  Once the test configuration is cloned lit checks for a  lit.lo‐
414       cal.cfg file in the subdirectory.  If present, this file will be loaded
415       and can be used to specialize the configuration for each individual di‐
416       rectory.   This  facility  can  be used to define subdirectories of op‐
417       tional tests, or to change other configuration parameters --- for exam‐
418       ple,  to  change  the  test format, or the suffixes which identify test
419       files.
420
421   SUBSTITUTIONS
422       lit allows patterns to be substituted inside RUN commands. It also pro‐
423       vides  the  following  base  set of substitutions, which are defined in
424       TestRunner.py:
425
426                 ┌────────────────────────┬────────────────────────────┐
427                 │Macro                   │ Substitution               │
428                 ├────────────────────────┼────────────────────────────┤
429                 │%s                      │ source path (path  to  the │
430                 │                        │ file currently being run)  │
431                 ├────────────────────────┼────────────────────────────┤
432                 │%S                      │ source  dir  (directory of │
433                 │                        │ the file  currently  being │
434                 │                        │ run)                       │
435                 ├────────────────────────┼────────────────────────────┤
436                 │%p                      │ same as %S                 │
437                 ├────────────────────────┼────────────────────────────┤
438                 │%{pathsep}              │ path separator             │
439                 ├────────────────────────┼────────────────────────────┤
440                 │%t                      │ temporary file name unique │
441                 │                        │ to the test                │
442                 ├────────────────────────┼────────────────────────────┤
443                 │%basename_t             │ The last path component of │
444                 │                        │ %t  but  without  the .tmp 
445                 │                        │ extension                  │
446                 ├────────────────────────┼────────────────────────────┤
447                 │%T                      │ parent  directory  of   %t │
448                 │                        │ (not  unique,  deprecated, │
449                 │                        │ do not use)                │
450                 ├────────────────────────┼────────────────────────────┤
451                 │%%                      │ %                          │
452                 ├────────────────────────┼────────────────────────────┤
453                 │%/s                     │ %s but \ is replaced by /  
454                 ├────────────────────────┼────────────────────────────┤
455                 │%/S                     │ %S but \ is replaced by /  
456                 ├────────────────────────┼────────────────────────────┤
457                 │%/p                     │ %p but \ is replaced by /  
458                 ├────────────────────────┼────────────────────────────┤
459                 │%/t                     │ %t but \ is replaced by /  
460                 ├────────────────────────┼────────────────────────────┤
461                 │%/T                     │ %T but \ is replaced by /  
462                 ├────────────────────────┼────────────────────────────┤
463                 │%{/s:regex_replacement} │ %/s but escaped for use in │
464                 │                        │ the  replacement of a s@@@ 
465                 │                        │ command in sed             │
466                 ├────────────────────────┼────────────────────────────┤
467                 │%{/S:regex_replacement} │ %/S but escaped for use in │
468                 │                        │ the  replacement of a s@@@ 
469                 │                        │ command in sed             │
470                 ├────────────────────────┼────────────────────────────┤
471                 │%{/p:regex_replacement} │ %/p but escaped for use in │
472                 │                        │ the  replacement of a s@@@ 
473                 │                        │ command in sed             │
474                 ├────────────────────────┼────────────────────────────┤
475                 │%{/t:regex_replacement} │ %/t but escaped for use in │
476                 │                        │ the  replacement of a s@@@ 
477                 │                        │ command in sed             │
478                 └────────────────────────┴────────────────────────────┘
479
480
481
482
483                 │%{/T:regex_replacement} │ %/T but escaped for use in │
484                 │                        │ the  replacement of a s@@@ 
485                 │                        │ command in sed             │
486                 ├────────────────────────┼────────────────────────────┤
487                 │%:s                     │ On Windows, %/s but a : is │
488                 │                        │ removed  if its the second │
489                 │                        │ character.  Otherwise,  %s │
490                 │                        │ but  with a single leading │
491                 │                        │ / removed.                 │
492                 ├────────────────────────┼────────────────────────────┤
493                 │%:S                     │ On Windows, %/S but a : is │
494                 │                        │ removed  if its the second │
495                 │                        │ character.  Otherwise,  %S │
496                 │                        │ but  with a single leading │
497                 │                        │ / removed.                 │
498                 ├────────────────────────┼────────────────────────────┤
499                 │%:p                     │ On Windows, %/p but a : is │
500                 │                        │ removed  if its the second │
501                 │                        │ character.  Otherwise,  %p │
502                 │                        │ but  with a single leading │
503                 │                        │ / removed.                 │
504                 ├────────────────────────┼────────────────────────────┤
505                 │%:t                     │ On Windows, %/t but a : is │
506                 │                        │ removed  if its the second │
507                 │                        │ character.  Otherwise,  %t │
508                 │                        │ but  with a single leading │
509                 │                        │ / removed.                 │
510                 ├────────────────────────┼────────────────────────────┤
511                 │%:T                     │ On Windows, %/T but a : is │
512                 │                        │ removed  if its the second │
513                 │                        │ character.  Otherwise,  %T │
514                 │                        │ but  with a single leading │
515                 │                        │ / removed.                 │
516                 └────────────────────────┴────────────────────────────┘
517
518       Other substitutions are provided that are variations on this  base  set
519       and  further  substitution patterns can be defined by each test module.
520       See the modules LOCAL CONFIGURATION FILES.
521
522       By default, substitutions are expanded exactly once, so that if e.g.  a
523       substitution  %build  is  defined  in top of another substitution %cxx,
524       %build will expand to %cxx textually, not  to  what  %cxx  expands  to.
525       However,  if  the recursiveExpansionLimit property of the TestingConfig
526       is set to a non-negative integer, substitutions will be expanded recur‐
527       sively  until  that  limit  is  reached. It is an error if the limit is
528       reached and expanding substitutions again would yield a  different  re‐
529       sult.
530
531       More detailed information on substitutions can be found in the ../Test‐
532       ingGuide.
533
534   TEST RUN OUTPUT FORMAT
535       The lit output for a test run conforms to the following schema, in both
536       short  and  verbose modes (although in short mode no PASS lines will be
537       shown).  This schema has been chosen to be relatively easy to  reliably
538       parse  by  a  machine  (for  example in buildbot log scraping), and for
539       other tools to generate.
540
541       Each test result is expected to appear on a line that matches:
542
543          <result code>: <test name> (<progress info>)
544
545       where <result-code> is a standard  test  result  such  as  PASS,  FAIL,
546       XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED.  The performance result codes
547       of IMPROVED and REGRESSED are also allowed.
548
549       The <test name> field can consist of an arbitrary string containing  no
550       newline.
551
552       The  <progress  info>  field can be used to report progress information
553       such as (1/300) or can be empty, but even when  empty  the  parentheses
554       are required.
555
556       Each  test result may include additional (multiline) log information in
557       the following format:
558
559          <log delineator> TEST '(<test name>)' <trailing delineator>
560          ... log message ...
561          <log delineator>
562
563       where <test name> should be the name of a preceding reported test, <log
564       delineator> is a string of "*" characters at least four characters long
565       (the recommended length is 20), and <trailing delineator> is  an  arbi‐
566       trary (unparsed) string.
567
568       The following is an example of a test run output which consists of four
569       tests A, B, C, and D, and a log message for the failing test C:
570
571          PASS: A (1 of 4)
572          PASS: B (2 of 4)
573          FAIL: C (3 of 4)
574          ******************** TEST 'C' FAILED ********************
575          Test 'C' failed as a result of exit code 1.
576          ********************
577          PASS: D (4 of 4)
578
579   LIT EXAMPLE TESTS
580       The lit distribution contains several example implementations  of  test
581       suites in the ExampleTests directory.
582

SEE ALSO

584       valgrind(1)
585

AUTHOR

587       Maintained by the LLVM Team (https://llvm.org/).
588
590       2003-2022, LLVM Project
591
592
593
594
59513                                2022-02-02                            LIT(1)
Impressum