1CTEST(1) CMake CTEST(1)
2
6 ctest - CTest Command-Line Reference
7 Contents
9 • ctest(1)
10 • Synopsis
12 • Description
14 • Options
16 • Label Matching
18 • Label and Subproject Summary
20 • Build and Test Mode
22 • Dashboard Client
24 • Dashboard Client Steps
26 • Dashboard Client Modes
28 • Dashboard Client via CTest Command-Line
30 • Dashboard Client via CTest Script
32 • Dashboard Client Configuration
34 • CTest Start Step
36 • CTest Update Step
38 • CTest Configure Step
40 • CTest Build Step
42 • CTest Test Step
44 • CTest Coverage Step
46 • CTest MemCheck Step
48 • CTest Submit Step
50 • Show as JSON Object Model
52 • Resource Allocation
54 • Resource Specification File
56 • RESOURCE_GROUPS Property
58 • Environment Variables
60 • See Also
62
64 ctest [<options>]
65 ctest --build-and-test <path-to-source> <path-to-build>
66 --build-generator <generator> [<options>...]
67 [--build-options <opts>...] [--test-command <command> [<args>...]]
68 ctest {-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>}
69 [-- <dashboard-options>...]
70
72 The ctest executable is the CMake test driver program. CMake-generated
73 build trees created for projects that use the enable_testing() and
74 add_test() commands have testing support. This program will run the
75 tests and report results.
76
78 --preset <preset>, --preset=<preset>
79 Use a test preset to specify test options. The project binary
80 directory is inferred from the configurePreset key. The current
81 working directory must contain CMake preset files. See preset
82 for more details.
83 --list-presets
85 Lists the available test presets. The current working directory
86 must contain CMake preset files.
87 -C <cfg>, --build-config <cfg>
89 Choose configuration to test.
90 Some CMake-generated build trees can have multiple build config‐
92 urations in the same tree. This option can be used to specify
93 which one should be tested. Example configurations are Debug
94 and Release.
95 --progress
97 Enable short progress output from tests.
98 When the output of ctest is being sent directly to a terminal,
100 the progress through the set of tests is reported by updating
101 the same line rather than printing start and end messages for
102 each test on new lines. This can significantly reduce the ver‐
103 bosity of the test output. Test completion messages are still
104 output on their own line for failed tests and the final test
105 summary will also still be logged.
106 This option can also be enabled by setting the environment vari‐
108 able CTEST_PROGRESS_OUTPUT.
109 -V,--verbose
111 Enable verbose output from tests.
112 Test output is normally suppressed and only summary information
114 is displayed. This option will show all test output.
115 -VV,--extra-verbose
117 Enable more verbose output from tests.
118 Test output is normally suppressed and only summary information
120 is displayed. This option will show even more test output.
121 --debug
123 Displaying more verbose internals of CTest.
124 This feature will result in a large number of output that is
126 mostly useful for debugging dashboard problems.
127 --output-on-failure
129 Output anything outputted by the test program if the test should
130 fail. This option can also be enabled by setting the CTEST_OUT‐
131 PUT_ON_FAILURE environment variable
132 --stop-on-failure
134 Stop running the tests when the first failure happens.
135 -F Enable failover.
137 This option allows CTest to resume a test set execution that was
139 previously interrupted. If no interruption occurred, the -F op‐
140 tion will have no effect.
141 -j <jobs>, --parallel <jobs>
143 Run the tests in parallel using the given number of jobs.
144 This option tells CTest to run the tests in parallel using given
146 number of jobs. This option can also be set by setting the
147 CTEST_PARALLEL_LEVEL environment variable.
148 This option can be used with the PROCESSORS test property.
150 See Label and Subproject Summary.
152 --resource-spec-file <file>
154 Run CTest with resource allocation enabled, using the resource
155 specification file specified in <file>.
156 When ctest is run as a Dashboard Client this sets the Resource‐
158 SpecFile option of the CTest Test Step.
159 --test-load <level>
161 While running tests in parallel (e.g. with -j), try not to start
162 tests when they may cause the CPU load to pass above a given
163 threshold.
164 When ctest is run as a Dashboard Client this sets the TestLoad
166 option of the CTest Test Step.
167 -Q,--quiet
169 Make CTest quiet.
170 This option will suppress all the output. The output log file
172 will still be generated if the --output-log is specified. Op‐
173 tions such as --verbose, --extra-verbose, and --debug are ig‐
174 nored if --quiet is specified.
175 -O <file>, --output-log <file>
177 Output to log file.
178 This option tells CTest to write all its output to a <file> log
180 file.
181 --output-junit <file>
183 Write test results in JUnit format.
184 This option tells CTest to write test results to <file> in JUnit
186 XML format. If <file> already exists, it will be overwritten. If
187 using the -S option to run a dashboard script, use the OUT‐
188 PUT_JUNIT keyword with the ctest_test() command instead.
189 -N,--show-only[=<format>]
191 Disable actual execution of tests.
192 This option tells CTest to list the tests that would be run but
194 not actually run them. Useful in conjunction with the -R and -E
195 options.
196 <format> can be one of the following values.
198 human Human-friendly output. This is not guaranteed to be
200 stable. This is the default.
201 json-v1
203 Dump the test information in JSON format. See Show as
204 JSON Object Model.
205 -L <regex>, --label-regex <regex>
207 Run tests with labels matching regular expression as described
208 under string(REGEX).
209 This option tells CTest to run only the tests whose labels match
211 the given regular expression. When more than one -L option is
212 given, a test will only be run if each regular expression
213 matches at least one of the test's labels (i.e. the multiple -L
214 labels form an AND relationship). See Label Matching.
215 -R <regex>, --tests-regex <regex>
217 Run tests matching regular expression.
218 This option tells CTest to run only the tests whose names match
220 the given regular expression.
221 -E <regex>, --exclude-regex <regex>
223 Exclude tests matching regular expression.
224 This option tells CTest to NOT run the tests whose names match
226 the given regular expression.
227 -LE <regex>, --label-exclude <regex>
229 Exclude tests with labels matching regular expression.
230 This option tells CTest to NOT run the tests whose labels match
232 the given regular expression. When more than one -LE option is
233 given, a test will only be excluded if each regular expression
234 matches at least one of the test's labels (i.e. the multiple -LE
235 labels form an AND relationship). See Label Matching.
236 -FA <regex>, --fixture-exclude-any <regex>
238 Exclude fixtures matching <regex> from automatically adding any
239 tests to the test set.
240 If a test in the set of tests to be executed requires a particu‐
242 lar fixture, that fixture's setup and cleanup tests would nor‐
243 mally be added to the test set automatically. This option pre‐
244 vents adding setup or cleanup tests for fixtures matching the
245 <regex>. Note that all other fixture behavior is retained, in‐
246 cluding test dependencies and skipping tests that have fixture
247 setup tests that fail.
248 -FS <regex>, --fixture-exclude-setup <regex>
250 Same as -FA except only matching setup tests are excluded.
251 -FC <regex>, --fixture-exclude-cleanup <regex>
253 Same as -FA except only matching cleanup tests are excluded.
254 -D <dashboard>, --dashboard <dashboard>
256 Execute dashboard test.
257 This option tells CTest to act as a CDash client and perform a
259 dashboard test. All tests are <Mode><Test>, where <Mode> can be
260 Experimental, Nightly, and Continuous, and <Test> can be Start,
261 Update, Configure, Build, Test, Coverage, and Submit.
262 See Dashboard Client.
264 -D <var>:<type>=<value>
266 Define a variable for script mode.
267 Pass in variable values on the command line. Use in conjunction
269 with -S to pass variable values to a dashboard script. Parsing
270 -D arguments as variable values is only attempted if the value
271 following -D does not match any of the known dashboard types.
272 -M <model>, --test-model <model>
274 Sets the model for a dashboard.
275 This option tells CTest to act as a CDash client where the
277 <model> can be Experimental, Nightly, and Continuous. Combining
278 -M and -T is similar to -D.
279 See Dashboard Client.
281 -T <action>, --test-action <action>
283 Sets the dashboard action to perform.
284 This option tells CTest to act as a CDash client and perform
286 some action such as start, build, test etc. See Dashboard Client
287 Steps for the full list of actions. Combining -M and -T is sim‐
288 ilar to -D.
289 See Dashboard Client.
291 -S <script>, --script <script>
293 Execute a dashboard for a configuration.
294 This option tells CTest to load in a configuration script which
296 sets a number of parameters such as the binary and source direc‐
297 tories. Then CTest will do what is required to create and run a
298 dashboard. This option basically sets up a dashboard and then
299 runs ctest -D with the appropriate options.
300 See Dashboard Client.
302 -SP <script>, --script-new-process <script>
304 Execute a dashboard for a configuration.
305 This option does the same operations as -S but it will do them
307 in a separate process. This is primarily useful in cases where
308 the script may modify the environment and you do not want the
309 modified environment to impact other -S scripts.
310 See Dashboard Client.
312 -I [Start,End,Stride,test#,test#|Test file], --tests-information
314 Run a specific number of tests by number.
315 This option causes CTest to run tests starting at number Start,
317 ending at number End, and incrementing by Stride. Any addi‐
318 tional numbers after Stride are considered individual test num‐
319 bers. Start, End, or Stride can be empty. Optionally a file
320 can be given that contains the same syntax as the command line.
321 -U, --union
323 Take the Union of -I and -R.
324 When both -R and -I are specified by default the intersection of
326 tests are run. By specifying -U the union of tests is run in‐
327 stead.
328 --rerun-failed
330 Run only the tests that failed previously.
331 This option tells CTest to perform only the tests that failed
333 during its previous run. When this option is specified, CTest
334 ignores all other options intended to modify the list of tests
335 to run (-L, -R, -E, -LE, -I, etc). In the event that CTest runs
336 and no tests fail, subsequent calls to CTest with the --re‐
337 run-failed option will run the set of tests that most recently
338 failed (if any).
339 --repeat <mode>:<n>
341 Run tests repeatedly based on the given <mode> up to <n> times.
342 The modes are:
343 until-fail
345 Require each test to run <n> times without failing in or‐
346 der to pass. This is useful in finding sporadic failures
347 in test cases.
348 until-pass
350 Allow each test to run up to <n> times in order to pass.
351 Repeats tests if they fail for any reason. This is use‐
352 ful in tolerating sporadic failures in test cases.
353 after-timeout
355 Allow each test to run up to <n> times in order to pass.
356 Repeats tests only if they timeout. This is useful in
357 tolerating sporadic timeouts in test cases on busy ma‐
358 chines.
359 --repeat-until-fail <n>
361 Equivalent to --repeat until-fail:<n>.
362 --max-width <width>
364 Set the max width for a test name to output.
365 Set the maximum width for each test name to show in the output.
367 This allows the user to widen the output to avoid clipping the
368 test name which can be very annoying.
369 --interactive-debug-mode [0|1]
371 Set the interactive mode to 0 or 1.
372 This option causes CTest to run tests in either an interactive
374 mode or a non-interactive mode. In dashboard mode (Experimen‐
375 tal, Nightly, Continuous), the default is non-interactive. In
376 non-interactive mode, the environment variable DASH‐
377 BOARD_TEST_FROM_CTEST is set.
378 Prior to CMake 3.11, interactive mode on Windows allowed system
380 debug popup windows to appear. Now, due to CTest's use of libuv
381 to launch test processes, all system debug popup windows are al‐
382 ways blocked.
383 --no-label-summary
385 Disable timing summary information for labels.
386 This option tells CTest not to print summary information for
388 each label associated with the tests run. If there are no la‐
389 bels on the tests, nothing extra is printed.
390 See Label and Subproject Summary.
392 --no-subproject-summary
394 Disable timing summary information for subprojects.
395 This option tells CTest not to print summary information for
397 each subproject associated with the tests run. If there are no
398 subprojects on the tests, nothing extra is printed.
399 See Label and Subproject Summary.
401 --build-and-test See Build and Test Mode.
403 --test-dir <dir> Specify the directory in which to look for tests.
405 --test-output-size-passed <size>
407 Limit the output for passed tests to <size> bytes.
408 --test-output-size-failed <size>
410 Limit the output for failed tests to <size> bytes.
411 --overwrite
413 Overwrite CTest configuration option.
414 By default CTest uses configuration options from configuration
416 file. This option will overwrite the configuration option.
417 --force-new-ctest-process
419 Run child CTest instances as new processes.
420 By default CTest will run child CTest instances within the same
422 process. If this behavior is not desired, this argument will
423 enforce new processes for child CTest processes.
424 --schedule-random
426 Use a random order for scheduling tests.
427 This option will run the tests in a random order. It is com‐
429 monly used to detect implicit dependencies in a test suite.
430 --submit-index
432 Legacy option for old Dart2 dashboard server feature. Do not
433 use.
434 --timeout <seconds>
436 Set the default test timeout.
437 This option effectively sets a timeout on all tests that do not
439 already have a timeout set on them via the TIMEOUT property.
440 --stop-time <time>
442 Set a time at which all tests should stop running.
443 Set a real time of day at which all tests should timeout. Exam‐
445 ple: 7:00:00 -0400. Any time format understood by the curl date
446 parser is accepted. Local time is assumed if no timezone is
447 specified.
448 --print-labels
450 Print all available test labels.
451 This option will not run any tests, it will simply print the
453 list of all labels associated with the test set.
454 --no-tests=<[error|ignore]>
456 Regard no tests found either as error or ignore it.
457 If no tests were found, the default behavior of CTest is to al‐
459 ways log an error message but to return an error code in script
460 mode only. This option unifies the behavior of CTest by either
461 returning an error code if no tests were found or by ignoring
462 it.
463 --help,-help,-usage,-h,-H,/?
465 Print usage information and exit.
466 Usage describes the basic command line interface and its op‐
468 tions.
469 --version,-version,/V [<f>]
471 Show program name/version banner and exit.
472 If a file is specified, the version is written into it. The
474 help is printed to a named <f>ile if given.
475 --help-full [<f>]
477 Print all help manuals and exit.
478 All manuals are printed in a human-readable text format. The
480 help is printed to a named <f>ile if given.
481 --help-manual <man> [<f>]
483 Print one help manual and exit.
484 The specified manual is printed in a human-readable text format.
486 The help is printed to a named <f>ile if given.
487 --help-manual-list [<f>]
489 List help manuals available and exit.
490 The list contains all manuals for which help may be obtained by
492 using the --help-manual option followed by a manual name. The
493 help is printed to a named <f>ile if given.
494 --help-command <cmd> [<f>]
496 Print help for one command and exit.
497 The cmake-commands(7) manual entry for <cmd> is printed in a hu‐
499 man-readable text format. The help is printed to a named <f>ile
500 if given.
501 --help-command-list [<f>]
503 List commands with help available and exit.
504 The list contains all commands for which help may be obtained by
506 using the --help-command option followed by a command name. The
507 help is printed to a named <f>ile if given.
508 --help-commands [<f>]
510 Print cmake-commands manual and exit.
511 The cmake-commands(7) manual is printed in a human-readable text
513 format. The help is printed to a named <f>ile if given.
514 --help-module <mod> [<f>]
516 Print help for one module and exit.
517 The cmake-modules(7) manual entry for <mod> is printed in a hu‐
519 man-readable text format. The help is printed to a named <f>ile
520 if given.
521 --help-module-list [<f>]
523 List modules with help available and exit.
524 The list contains all modules for which help may be obtained by
526 using the --help-module option followed by a module name. The
527 help is printed to a named <f>ile if given.
528 --help-modules [<f>]
530 Print cmake-modules manual and exit.
531 The cmake-modules(7) manual is printed in a human-readable text
533 format. The help is printed to a named <f>ile if given.
534 --help-policy <cmp> [<f>]
536 Print help for one policy and exit.
537 The cmake-policies(7) manual entry for <cmp> is printed in a hu‐
539 man-readable text format. The help is printed to a named <f>ile
540 if given.
541 --help-policy-list [<f>]
543 List policies with help available and exit.
544 The list contains all policies for which help may be obtained by
546 using the --help-policy option followed by a policy name. The
547 help is printed to a named <f>ile if given.
548 --help-policies [<f>]
550 Print cmake-policies manual and exit.
551 The cmake-policies(7) manual is printed in a human-readable text
553 format. The help is printed to a named <f>ile if given.
554 --help-property <prop> [<f>]
556 Print help for one property and exit.
557 The cmake-properties(7) manual entries for <prop> are printed in
559 a human-readable text format. The help is printed to a named
560 <f>ile if given.
561 --help-property-list [<f>]
563 List properties with help available and exit.
564 The list contains all properties for which help may be obtained
566 by using the --help-property option followed by a property name.
567 The help is printed to a named <f>ile if given.
568 --help-properties [<f>]
570 Print cmake-properties manual and exit.
571 The cmake-properties(7) manual is printed in a human-readable
573 text format. The help is printed to a named <f>ile if given.
574 --help-variable <var> [<f>]
576 Print help for one variable and exit.
577 The cmake-variables(7) manual entry for <var> is printed in a
579 human-readable text format. The help is printed to a named
580 <f>ile if given.
581 --help-variable-list [<f>]
583 List variables with help available and exit.
584 The list contains all variables for which help may be obtained
586 by using the --help-variable option followed by a variable name.
587 The help is printed to a named <f>ile if given.
588 --help-variables [<f>]
590 Print cmake-variables manual and exit.
591 The cmake-variables(7) manual is printed in a human-readable
593 text format. The help is printed to a named <f>ile if given.
594
596 Tests may have labels attached to them. Tests may be included or ex‐
597 cluded from a test run by filtering on the labels. Each individual
598 filter is a regular expression applied to the labels attached to a
599 test.
600 When -L is used, in order for a test to be included in a test run, each
602 regular expression must match at least one label. Using more than one
603 -L option means "match all of these".
604 The -LE option works just like -L, but excludes tests rather than in‐
606 cluding them. A test is excluded if each regular expression matches at
607 least one label.
608 If a test has no labels attached to it, then -L will never include that
610 test, and -LE will never exclude that test. As an example of tests
611 with labels, consider five tests, with the following labels:
612 • test1 has labels tuesday and production
614 • test2 has labels tuesday and test
616 • test3 has labels wednesday and production
618 • test4 has label wednesday
620 • test5 has labels friday and test
622 Running ctest with -L tuesday -L test will select test2, which has both
624 labels. Running CTest with -L test will select test2 and test5, because
625 both of them have a label that matches that regular expression.
626 Because the matching works with regular expressions, take note that
628 running CTest with -L es will match all five tests. To select the
629 tuesday and wednesday tests together, use a single regular expression
630 that matches either of them, like -L "tue|wed".
631
633 CTest prints timing summary information for each LABEL and subproject
634 associated with the tests run. The label time summary will not include
635 labels that are mapped to subprojects.
636 New in version 3.22: Labels added dynamically during test execution are
638 also reported in the timing summary. See Additional Labels.
639 When the PROCESSORS test property is set, CTest will display a weighted
642 test timing result in label and subproject summaries. The time is re‐
643 ported with sec*proc instead of just sec.
644 The weighted time summary reported for each label or subproject j is
646 computed as:
647 Weighted Time Summary for Label/Subproject j =
649 sum(raw_test_time[j,i] * num_processors[j,i], i=1...num_tests[j])
650 for labels/subprojects j=1...total
652 where:
654 • raw_test_time[j,i]: Wall-clock time for the i test for the j label or
656 subproject
657 • num_processors[j,i]: Value of the CTest PROCESSORS property for the i
659 test for the j label or subproject
660 • num_tests[j]: Number of tests associated with the j label or subpro‐
662 ject
663 • total: Total number of labels or subprojects that have at least one
665 test run
666 Therefore, the weighted time summary for each label or subproject rep‐
668 resents the amount of time that CTest gave to run the tests for each
669 label or subproject and gives a good representation of the total ex‐
670 pense of the tests for each label or subproject when compared to other
671 labels or subprojects.
672 For example, if SubprojectA showed 100 sec*proc and SubprojectB showed
674 10 sec*proc, then CTest allocated approximately 10 times the CPU/core
675 time to run the tests for SubprojectA than for SubprojectB (e.g. so if
676 effort is going to be expended to reduce the cost of the test suite for
677 the whole project, then reducing the cost of the test suite for Subpro‐
678 jectA would likely have a larger impact than effort to reduce the cost
679 of the test suite for SubprojectB).
680
682 CTest provides a command-line signature to configure (i.e. run cmake
683 on), build, and/or execute a test:
684 ctest --build-and-test <path-to-source> <path-to-build>
686 --build-generator <generator>
687 [<options>...]
688 [--build-options <opts>...]
689 [--test-command <command> [<args>...]]
690 The configure and test steps are optional. The arguments to this com‐
692 mand line are the source and binary directories. The --build-generator
693 option must be provided to use --build-and-test. If --test-command is
694 specified then that will be run after the build is complete. Other op‐
695 tions that affect this mode include:
696 --build-target
698 Specify a specific target to build.
699 If left out the all target is built.
701 --build-nocmake
703 Run the build without running cmake first.
704 Skip the cmake step.
706 --build-run-dir
708 Specify directory to run programs from.
709 Directory where programs will be after it has been compiled.
711 --build-two-config
713 Run CMake twice.
714 --build-exe-dir
716 Specify the directory for the executable.
717 --build-generator
719 Specify the generator to use. See the cmake-generators(7) man‐
720 ual.
721 --build-generator-platform
723 Specify the generator-specific platform.
724 --build-generator-toolset
726 Specify the generator-specific toolset.
727 --build-project
729 Specify the name of the project to build.
730 --build-makeprogram
732 Specify the explicit make program to be used by CMake when con‐
733 figuring and building the project. Only applicable for Make and
734 Ninja based generators.
735 --build-noclean
737 Skip the make clean step.
738 --build-config-sample
740 A sample executable to use to determine the configuration that
741 should be used. e.g. Debug, Release etc.
742 --build-options
744 Additional options for configuring the build (i.e. for CMake,
745 not for the build tool). Note that if this is specified, the
746 --build-options keyword and its arguments must be the last op‐
747 tion given on the command line, with the possible exception of
748 --test-command.
749 --test-command
751 The command to run as the test step with the --build-and-test
752 option. All arguments following this keyword will be assumed to
753 be part of the test command line, so it must be the last option
754 given.
755 --test-timeout
757 The time limit in seconds
758
760 CTest can operate as a client for the CDash software quality dashboard
761 application. As a dashboard client, CTest performs a sequence of steps
762 to configure, build, and test software, and then submits the results to
763 a CDash server. The command-line signature used to submit to CDash is:
764 ctest (-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>)
766 [-- <dashboard-options>...]
767 Options for Dashboard Client include:
769 --group <group>
771 Specify what group you'd like to submit results to
772 Submit dashboard to specified group instead of default one. By
774 default, the dashboard is submitted to Nightly, Experimental, or
775 Continuous group, but by specifying this option, the group can
776 be arbitrary.
777 This replaces the deprecated option --track. Despite the name
779 change its behavior is unchanged.
780 -A <file>, --add-notes <file>
782 Add a notes file with submission.
783 This option tells CTest to include a notes file when submitting
785 dashboard.
786 --tomorrow-tag
788 Nightly or Experimental starts with next day tag.
789 This is useful if the build will not finish in one day.
791 --extra-submit <file>[;<file>]
793 Submit extra files to the dashboard.
794 This option will submit extra files to the dashboard.
796 --http1.0
798 Submit using HTTP 1.0.
799 This option will force CTest to use HTTP 1.0 to submit files to
801 the dashboard, instead of HTTP 1.1.
802 --no-compress-output
804 Do not compress test output when submitting.
805 This flag will turn off automatic compression of test output.
807 Use this to maintain compatibility with an older version of
808 CDash which doesn't support compressed test output.
809 Dashboard Client Steps
811 CTest defines an ordered list of testing steps of which some or all may
812 be run as a dashboard client:
813 Start Start a new dashboard submission to be composed of results
815 recorded by the following steps. See the CTest Start Step sec‐
816 tion below.
817 Update Update the source tree from its version control repository.
819 Record the old and new versions and the list of updated source
820 files. See the CTest Update Step section below.
821 Configure
823 Configure the software by running a command in the build tree.
824 Record the configuration output log. See the CTest Configure
825 Step section below.
826 Build Build the software by running a command in the build tree.
828 Record the build output log and detect warnings and errors. See
829 the CTest Build Step section below.
830 Test Test the software by loading a CTestTestfile.cmake from the
832 build tree and executing the defined tests. Record the output
833 and result of each test. See the CTest Test Step section below.
834 Coverage
836 Compute coverage of the source code by running a coverage analy‐
837 sis tool and recording its output. See the CTest Coverage Step
838 section below.
839 MemCheck
841 Run the software test suite through a memory check tool. Record
842 the test output, results, and issues reported by the tool. See
843 the CTest MemCheck Step section below.
844 Submit Submit results recorded from other testing steps to the software
846 quality dashboard server. See the CTest Submit Step section be‐
847 low.
848 Dashboard Client Modes
850 CTest defines three modes of operation as a dashboard client:
851 Nightly
853 This mode is intended to be invoked once per day, typically at
854 night. It enables the Start, Update, Configure, Build, Test,
855 Coverage, and Submit steps by default. Selected steps run even
856 if the Update step reports no changes to the source tree.
857 Continuous
859 This mode is intended to be invoked repeatedly throughout the
860 day. It enables the Start, Update, Configure, Build, Test, Cov‐
861 erage, and Submit steps by default, but exits after the Update
862 step if it reports no changes to the source tree.
863 Experimental
865 This mode is intended to be invoked by a developer to test local
866 changes. It enables the Start, Configure, Build, Test, Cover‐
867 age, and Submit steps by default.
868 Dashboard Client via CTest Command-Line
870 CTest can perform testing on an already-generated build tree. Run the
871 ctest command with the current working directory set to the build tree
872 and use one of these signatures:
873 ctest -D <mode>[<step>]
875 ctest -M <mode> [ -T <step> ]...
876 The <mode> must be one of the above Dashboard Client Modes, and each
878 <step> must be one of the above Dashboard Client Steps.
879 CTest reads the Dashboard Client Configuration settings from a file in
881 the build tree called either CTestConfiguration.ini or DartConfigura‐
882 tion.tcl (the names are historical). The format of the file is:
883 # Lines starting in '#' are comments.
885 # Other non-blank lines are key-value pairs.
886 <setting>: <value>
887 where <setting> is the setting name and <value> is the setting value.
889 In build trees generated by CMake, this configuration file is generated
891 by the CTest module if included by the project. The module uses vari‐
892 ables to obtain a value for each setting as documented with the set‐
893 tings below.
894 Dashboard Client via CTest Script
896 CTest can perform testing driven by a cmake-language(7) script that
897 creates and maintains the source and build tree as well as performing
898 the testing steps. Run the ctest command with the current working di‐
899 rectory set outside of any build tree and use one of these signatures:
900 ctest -S <script>
902 ctest -SP <script>
903 The <script> file must call CTest Commands commands to run testing
905 steps explicitly as documented below. The commands obtain Dashboard
906 Client Configuration settings from their arguments or from variables
907 set in the script.
908
910 The Dashboard Client Steps may be configured by named settings as docu‐
911 mented in the following sections.
912 CTest Start Step
914 Start a new dashboard submission to be composed of results recorded by
915 the following steps.
916 In a CTest Script, the ctest_start() command runs this step. Arguments
918 to the command may specify some of the step settings. The command
919 first runs the command-line specified by the CTEST_CHECKOUT_COMMAND
920 variable, if set, to initialize the source directory.
921 Configuration settings include:
923 BuildDirectory
925 The full path to the project build tree.
926 • CTest Script variable: CTEST_BINARY_DIRECTORY
928 • CTest module variable: PROJECT_BINARY_DIR
930 SourceDirectory
932 The full path to the project source tree.
933 • CTest Script variable: CTEST_SOURCE_DIRECTORY
935 • CTest module variable: PROJECT_SOURCE_DIR
937 CTest Update Step
939 In a CTest Script, the ctest_update() command runs this step. Argu‐
940 ments to the command may specify some of the step settings.
941 Configuration settings to specify the version control tool include:
943 BZRCommand
945 bzr command-line tool to use if source tree is managed by
946 Bazaar.
947 • CTest Script variable: CTEST_BZR_COMMAND
949 • CTest module variable: none
951 BZRUpdateOptions
953 Command-line options to the BZRCommand when updating the source.
954 • CTest Script variable: CTEST_BZR_UPDATE_OPTIONS
956 • CTest module variable: none
958 CVSCommand
960 cvs command-line tool to use if source tree is managed by CVS.
961 • CTest Script variable: CTEST_CVS_COMMAND
963 • CTest module variable: CVSCOMMAND
965 CVSUpdateOptions
967 Command-line options to the CVSCommand when updating the source.
968 • CTest Script variable: CTEST_CVS_UPDATE_OPTIONS
970 • CTest module variable: CVS_UPDATE_OPTIONS
972 GITCommand
974 git command-line tool to use if source tree is managed by Git.
975 • CTest Script variable: CTEST_GIT_COMMAND
977 • CTest module variable: GITCOMMAND
979 The source tree is updated by git fetch followed by git reset
981 --hard to the FETCH_HEAD. The result is the same as git pull
982 except that any local modifications are overwritten. Use GITUp‐
983 dateCustom to specify a different approach.
984 GITInitSubmodules
986 If set, CTest will update the repository's submodules before up‐
987 dating.
988 • CTest Script variable: CTEST_GIT_INIT_SUBMODULES
990 • CTest module variable: CTEST_GIT_INIT_SUBMODULES
992 GITUpdateCustom
994 Specify a custom command line (as a semicolon-separated list) to
995 run in the source tree (Git work tree) to update it instead of
996 running the GITCommand.
997 • CTest Script variable: CTEST_GIT_UPDATE_CUSTOM
999 • CTest module variable: CTEST_GIT_UPDATE_CUSTOM
1001 GITUpdateOptions
1003 Command-line options to the GITCommand when updating the source.
1004 • CTest Script variable: CTEST_GIT_UPDATE_OPTIONS
1006 • CTest module variable: GIT_UPDATE_OPTIONS
1008 HGCommand
1010 hg command-line tool to use if source tree is managed by Mercu‐
1011 rial.
1012 • CTest Script variable: CTEST_HG_COMMAND
1014 • CTest module variable: none
1016 HGUpdateOptions
1018 Command-line options to the HGCommand when updating the source.
1019 • CTest Script variable: CTEST_HG_UPDATE_OPTIONS
1021 • CTest module variable: none
1023 P4Client
1025 Value of the -c option to the P4Command.
1026 • CTest Script variable: CTEST_P4_CLIENT
1028 • CTest module variable: CTEST_P4_CLIENT
1030 P4Command
1032 p4 command-line tool to use if source tree is managed by Per‐
1033 force.
1034 • CTest Script variable: CTEST_P4_COMMAND
1036 • CTest module variable: P4COMMAND
1038 P4Options
1040 Command-line options to the P4Command for all invocations.
1041 • CTest Script variable: CTEST_P4_OPTIONS
1043 • CTest module variable: CTEST_P4_OPTIONS
1045 P4UpdateCustom
1047 Specify a custom command line (as a semicolon-separated list) to
1048 run in the source tree (Perforce tree) to update it instead of
1049 running the P4Command.
1050 • CTest Script variable: none
1052 • CTest module variable: CTEST_P4_UPDATE_CUSTOM
1054 P4UpdateOptions
1056 Command-line options to the P4Command when updating the source.
1057 • CTest Script variable: CTEST_P4_UPDATE_OPTIONS
1059 • CTest module variable: CTEST_P4_UPDATE_OPTIONS
1061 SVNCommand
1063 svn command-line tool to use if source tree is managed by Sub‐
1064 version.
1065 • CTest Script variable: CTEST_SVN_COMMAND
1067 • CTest module variable: SVNCOMMAND
1069 SVNOptions
1071 Command-line options to the SVNCommand for all invocations.
1072 • CTest Script variable: CTEST_SVN_OPTIONS
1074 • CTest module variable: CTEST_SVN_OPTIONS
1076 SVNUpdateOptions
1078 Command-line options to the SVNCommand when updating the source.
1079 • CTest Script variable: CTEST_SVN_UPDATE_OPTIONS
1081 • CTest module variable: SVN_UPDATE_OPTIONS
1083 UpdateCommand
1085 Specify the version-control command-line tool to use without de‐
1086 tecting the VCS that manages the source tree.
1087 • CTest Script variable: CTEST_UPDATE_COMMAND
1089 • CTest module variable: <VCS>COMMAND when UPDATE_TYPE is <vcs>,
1091 else UPDATE_COMMAND
1092 UpdateOptions
1094 Command-line options to the UpdateCommand.
1095 • CTest Script variable: CTEST_UPDATE_OPTIONS
1097 • CTest module variable: <VCS>_UPDATE_OPTIONS when UPDATE_TYPE
1099 is <vcs>, else UPDATE_OPTIONS
1100 UpdateType
1102 Specify the version-control system that manages the source tree
1103 if it cannot be detected automatically. The value may be bzr,
1104 cvs, git, hg, p4, or svn.
1105 • CTest Script variable: none, detected from source tree
1107 • CTest module variable: UPDATE_TYPE if set, else CTEST_UP‐
1109 DATE_TYPE
1110 UpdateVersionOnly
1112 Specify that you want the version control update command to only
1113 discover the current version that is checked out, and not to up‐
1114 date to a different version.
1115 • CTest Script variable: CTEST_UPDATE_VERSION_ONLY
1117 UpdateVersionOverride
1119 Specify the current version of your source tree.
1120 When this variable is set to a non-empty string, CTest will re‐
1122 port the value you specified rather than using the update com‐
1123 mand to discover the current version that is checked out. Use of
1124 this variable supersedes UpdateVersionOnly. Like UpdateVer‐
1125 sionOnly, using this variable tells CTest not to update the
1126 source tree to a different version.
1127 • CTest Script variable: CTEST_UPDATE_VERSION_OVERRIDE
1129 Additional configuration settings include:
1131 NightlyStartTime
1133 In the Nightly dashboard mode, specify the "nightly start time".
1134 With centralized version control systems (cvs and svn), the Up‐
1135 date step checks out the version of the software as of this time
1136 so that multiple clients choose a common version to test. This
1137 is not well-defined in distributed version-control systems so
1138 the setting is ignored.
1139 • CTest Script variable: CTEST_NIGHTLY_START_TIME
1141 • CTest module variable: NIGHTLY_START_TIME if set, else
1143 CTEST_NIGHTLY_START_TIME
1144 CTest Configure Step
1146 In a CTest Script, the ctest_configure() command runs this step. Argu‐
1147 ments to the command may specify some of the step settings.
1148 Configuration settings include:
1150 ConfigureCommand
1152 Command-line to launch the software configuration process. It
1153 will be executed in the location specified by the BuildDirectory
1154 setting.
1155 • CTest Script variable: CTEST_CONFIGURE_COMMAND
1157 • CTest module variable: CMAKE_COMMAND followed by
1159 PROJECT_SOURCE_DIR
1160 LabelsForSubprojects
1162 Specify a semicolon-separated list of labels that will be
1163 treated as subprojects. This mapping will be passed on to CDash
1164 when configure, test or build results are submitted.
1165 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1167 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1169 See Label and Subproject Summary.
1171 CTest Build Step
1173 In a CTest Script, the ctest_build() command runs this step. Arguments
1174 to the command may specify some of the step settings.
1175 Configuration settings include:
1177 DefaultCTestConfigurationType
1179 When the build system to be launched allows build-time selection
1180 of the configuration (e.g. Debug, Release), this specifies the
1181 default configuration to be built when no -C option is given to
1182 the ctest command. The value will be substituted into the value
1183 of MakeCommand to replace the literal string ${CTEST_CONFIGURA‐
1184 TION_TYPE} if it appears.
1185 • CTest Script variable: CTEST_CONFIGURATION_TYPE
1187 • CTest module variable: DEFAULT_CTEST_CONFIGURATION_TYPE, ini‐
1189 tialized by the CMAKE_CONFIG_TYPE environment variable
1190 LabelsForSubprojects
1192 Specify a semicolon-separated list of labels that will be
1193 treated as subprojects. This mapping will be passed on to CDash
1194 when configure, test or build results are submitted.
1195 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1197 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1199 See Label and Subproject Summary.
1201 MakeCommand
1203 Command-line to launch the software build process. It will be
1204 executed in the location specified by the BuildDirectory set‐
1205 ting.
1206 • CTest Script variable: CTEST_BUILD_COMMAND
1208 • CTest module variable: MAKECOMMAND, initialized by the
1210 build_command() command
1211 UseLaunchers
1213 For build trees generated by CMake using one of the Makefile
1214 Generators or the Ninja generator, specify whether the
1215 CTEST_USE_LAUNCHERS feature is enabled by the CTestUseLaunchers
1216 module (also included by the CTest module). When enabled, the
1217 generated build system wraps each invocation of the compiler,
1218 linker, or custom command line with a "launcher" that communi‐
1219 cates with CTest via environment variables and files to report
1220 granular build warning and error information. Otherwise, CTest
1221 must "scrape" the build output log for diagnostics.
1222 • CTest Script variable: CTEST_USE_LAUNCHERS
1224 • CTest module variable: CTEST_USE_LAUNCHERS
1226 CTest Test Step
1228 In a CTest Script, the ctest_test() command runs this step. Arguments
1229 to the command may specify some of the step settings.
1230 Configuration settings include:
1232 ResourceSpecFile
1234 Specify a resource specification file.
1235 • CTest Script variable: CTEST_RESOURCE_SPEC_FILE
1237 • CTest module variable: CTEST_RESOURCE_SPEC_FILE
1239 See Resource Allocation for more information.
1241 LabelsForSubprojects
1243 Specify a semicolon-separated list of labels that will be
1244 treated as subprojects. This mapping will be passed on to CDash
1245 when configure, test or build results are submitted.
1246 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1248 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1250 See Label and Subproject Summary.
1252 TestLoad
1254 While running tests in parallel (e.g. with -j), try not to start
1255 tests when they may cause the CPU load to pass above a given
1256 threshold.
1257 • CTest Script variable: CTEST_TEST_LOAD
1259 • CTest module variable: CTEST_TEST_LOAD
1261 TimeOut
1263 The default timeout for each test if not specified by the TIME‐
1264 OUT test property.
1265 • CTest Script variable: CTEST_TEST_TIMEOUT
1267 • CTest module variable: DART_TESTING_TIMEOUT
1269 To report extra test values to CDash, see Additional Test Measurements.
1271 CTest Coverage Step
1273 In a CTest Script, the ctest_coverage() command runs this step. Argu‐
1274 ments to the command may specify some of the step settings.
1275 Configuration settings include:
1277 CoverageCommand
1279 Command-line tool to perform software coverage analysis. It
1280 will be executed in the location specified by the BuildDirectory
1281 setting.
1282 • CTest Script variable: CTEST_COVERAGE_COMMAND
1284 • CTest module variable: COVERAGE_COMMAND
1286 CoverageExtraFlags
1288 Specify command-line options to the CoverageCommand tool.
1289 • CTest Script variable: CTEST_COVERAGE_EXTRA_FLAGS
1291 • CTest module variable: COVERAGE_EXTRA_FLAGS
1293 These options are the first arguments passed to CoverageCommand.
1295 CTest MemCheck Step
1297 In a CTest Script, the ctest_memcheck() command runs this step. Argu‐
1298 ments to the command may specify some of the step settings.
1299 Configuration settings include:
1301 MemoryCheckCommand
1303 Command-line tool to perform dynamic analysis. Test command
1304 lines will be launched through this tool.
1305 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND
1307 • CTest module variable: MEMORYCHECK_COMMAND
1309 MemoryCheckCommandOptions
1311 Specify command-line options to the MemoryCheckCommand tool.
1312 They will be placed prior to the test command line.
1313 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND_OPTIONS
1315 • CTest module variable: MEMORYCHECK_COMMAND_OPTIONS
1317 MemoryCheckType
1319 Specify the type of memory checking to perform.
1320 • CTest Script variable: CTEST_MEMORYCHECK_TYPE
1322 • CTest module variable: MEMORYCHECK_TYPE
1324 MemoryCheckSanitizerOptions
1326 Specify options to sanitizers when running with a sanitize-en‐
1327 abled build.
1328 • CTest Script variable: CTEST_MEMORYCHECK_SANITIZER_OPTIONS
1330 • CTest module variable: MEMORYCHECK_SANITIZER_OPTIONS
1332 MemoryCheckSuppressionFile
1334 Specify a file containing suppression rules for the MemoryCheck‐
1335 Command tool. It will be passed with options appropriate to the
1336 tool.
1337 • CTest Script variable: CTEST_MEMORYCHECK_SUPPRESSIONS_FILE
1339 • CTest module variable: MEMORYCHECK_SUPPRESSIONS_FILE
1341 Additional configuration settings include:
1343 BoundsCheckerCommand
1345 Specify a MemoryCheckCommand that is known to be command-line
1346 compatible with Bounds Checker.
1347 • CTest Script variable: none
1349 • CTest module variable: none
1351 PurifyCommand
1353 Specify a MemoryCheckCommand that is known to be command-line
1354 compatible with Purify.
1355 • CTest Script variable: none
1357 • CTest module variable: PURIFYCOMMAND
1359 ValgrindCommand
1361 Specify a MemoryCheckCommand that is known to be command-line
1362 compatible with Valgrind.
1363 • CTest Script variable: none
1365 • CTest module variable: VALGRIND_COMMAND
1367 ValgrindCommandOptions
1369 Specify command-line options to the ValgrindCommand tool. They
1370 will be placed prior to the test command line.
1371 • CTest Script variable: none
1373 • CTest module variable: VALGRIND_COMMAND_OPTIONS
1375 DrMemoryCommand
1377 Specify a MemoryCheckCommand that is known to be a command-line
1378 compatible with DrMemory.
1379 • CTest Script variable: none
1381 • CTest module variable: DRMEMORY_COMMAND
1383 DrMemoryCommandOptions
1385 Specify command-line options to the DrMemoryCommand tool. They
1386 will be placed prior to the test command line.
1387 • CTest Script variable: none
1389 • CTest module variable: DRMEMORY_COMMAND_OPTIONS
1391 CudaSanitizerCommand
1393 Specify a MemoryCheckCommand that is known to be a command-line
1394 compatible with cuda-memcheck or compute-sanitizer.
1395 • CTest Script variable: none
1397 • CTest module variable: CUDA_SANITIZER_COMMAND
1399 CudaSanitizerCommandOptions
1401 Specify command-line options to the CudaSanitizerCommand tool.
1402 They will be placed prior to the test command line.
1403 • CTest Script variable: none
1405 • CTest module variable: CUDA_SANITIZER_COMMAND_OPTIONS
1407 CTest Submit Step
1409 In a CTest Script, the ctest_submit() command runs this step. Argu‐
1410 ments to the command may specify some of the step settings.
1411 Configuration settings include:
1413 BuildName
1415 Describe the dashboard client platform with a short string.
1416 (Operating system, compiler, etc.)
1417 • CTest Script variable: CTEST_BUILD_NAME
1419 • CTest module variable: BUILDNAME
1421 CDashVersion
1423 Legacy option. Not used.
1424 • CTest Script variable: none, detected from server
1426 • CTest module variable: CTEST_CDASH_VERSION
1428 CTestSubmitRetryCount
1430 Specify a number of attempts to retry submission on network
1431 failure.
1432 • CTest Script variable: none, use the ctest_submit()
1434 RETRY_COUNT option.
1435 • CTest module variable: CTEST_SUBMIT_RETRY_COUNT
1437 CTestSubmitRetryDelay
1439 Specify a delay before retrying submission on network failure.
1440 • CTest Script variable: none, use the ctest_submit() RETRY_DE‐
1442 LAY option.
1443 • CTest module variable: CTEST_SUBMIT_RETRY_DELAY
1445 CurlOptions
1447 Specify a semicolon-separated list of options to control the
1448 Curl library that CTest uses internally to connect to the
1449 server. Possible options are CURLOPT_SSL_VERIFYPEER_OFF and
1450 CURLOPT_SSL_VERIFYHOST_OFF.
1451 • CTest Script variable: CTEST_CURL_OPTIONS
1453 • CTest module variable: CTEST_CURL_OPTIONS
1455 DropLocation
1457 Legacy option. When SubmitURL is not set, it is constructed
1458 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1459 DropLocation.
1460 • CTest Script variable: CTEST_DROP_LOCATION
1462 • CTest module variable: DROP_LOCATION if set, else
1464 CTEST_DROP_LOCATION
1465 DropMethod
1467 Legacy option. When SubmitURL is not set, it is constructed
1468 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1469 DropLocation.
1470 • CTest Script variable: CTEST_DROP_METHOD
1472 • CTest module variable: DROP_METHOD if set, else
1474 CTEST_DROP_METHOD
1475 DropSite
1477 Legacy option. When SubmitURL is not set, it is constructed
1478 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1479 DropLocation.
1480 • CTest Script variable: CTEST_DROP_SITE
1482 • CTest module variable: DROP_SITE if set, else CTEST_DROP_SITE
1484 DropSitePassword
1486 Legacy option. When SubmitURL is not set, it is constructed
1487 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1488 DropLocation.
1489 • CTest Script variable: CTEST_DROP_SITE_PASSWORD
1491 • CTest module variable: DROP_SITE_PASSWORD if set, else
1493 CTEST_DROP_SITE_PASWORD
1494 DropSiteUser
1496 Legacy option. When SubmitURL is not set, it is constructed
1497 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1498 DropLocation.
1499 • CTest Script variable: CTEST_DROP_SITE_USER
1501 • CTest module variable: DROP_SITE_USER if set, else
1503 CTEST_DROP_SITE_USER
1504 IsCDash
1506 Legacy option. Not used.
1507 • CTest Script variable: CTEST_DROP_SITE_CDASH
1509 • CTest module variable: CTEST_DROP_SITE_CDASH
1511 ScpCommand
1513 Legacy option. Not used.
1514 • CTest Script variable: CTEST_SCP_COMMAND
1516 • CTest module variable: SCPCOMMAND
1518 Site Describe the dashboard client host site with a short string.
1520 (Hostname, domain, etc.)
1521 • CTest Script variable: CTEST_SITE
1523 • CTest module variable: SITE, initialized by the site_name()
1525 command
1526 SubmitURL
1528 The http or https URL of the dashboard server to send the sub‐
1529 mission to.
1530 • CTest Script variable: CTEST_SUBMIT_URL
1532 • CTest module variable: SUBMIT_URL if set, else CTEST_SUB‐
1534 MIT_URL
1535 TriggerSite
1537 Legacy option. Not used.
1538 • CTest Script variable: CTEST_TRIGGER_SITE
1540 • CTest module variable: TRIGGER_SITE if set, else CTEST_TRIG‐
1542 GER_SITE
1543
1545 When the --show-only=json-v1 command line option is given, the test in‐
1546 formation is output in JSON format. Version 1.0 of the JSON object
1547 model is defined as follows:
1548 kind The string "ctestInfo".
1550 version
1552 A JSON object specifying the version components. Its members
1553 are
1554 major A non-negative integer specifying the major version com‐
1556 ponent.
1557 minor A non-negative integer specifying the minor version com‐
1559 ponent.
1560 backtraceGraph
1562 JSON object representing backtrace information with the follow‐
1563 ing members:
1564 commands
1566 List of command names.
1567 files List of file names.
1569 nodes List of node JSON objects with members:
1571 command
1573 Index into the commands member of the backtrace‐
1574 Graph.
1575 file Index into the files member of the backtraceGraph.
1577 line Line number in the file where the backtrace was
1579 added.
1580 parent Index into the nodes member of the backtraceGraph
1582 representing the parent in the graph.
1583 tests A JSON array listing information about each test. Each entry is
1585 a JSON object with members:
1586 name Test name.
1588 config Configuration that the test can run on. Empty string
1590 means any config.
1591 command
1593 List where the first element is the test command and the
1594 remaining elements are the command arguments.
1595 backtrace
1597 Index into the nodes member of the backtraceGraph.
1598 properties
1600 Test properties. Can contain keys for each of the sup‐
1601 ported test properties.
1602
1604 CTest provides a mechanism for tests to specify the resources that they
1605 need in a fine-grained way, and for users to specify the resources
1606 available on the running machine. This allows CTest to internally keep
1607 track of which resources are in use and which are free, scheduling
1608 tests in a way that prevents them from trying to claim resources that
1609 are not available.
1610 When the resource allocation feature is used, CTest will not oversub‐
1612 scribe resources. For example, if a resource has 8 slots, CTest will
1613 not run tests that collectively use more than 8 slots at a time. This
1614 has the effect of limiting how many tests can run at any given time,
1615 even if a high -j argument is used, if those tests all use some slots
1616 from the same resource. In addition, it means that a single test that
1617 uses more of a resource than is available on a machine will not run at
1618 all (and will be reported as Not Run).
1619 A common use case for this feature is for tests that require the use of
1621 a GPU. Multiple tests can simultaneously allocate memory from a GPU,
1622 but if too many tests try to do this at once, some of them will fail to
1623 allocate, resulting in a failed test, even though the test would have
1624 succeeded if it had the memory it needed. By using the resource alloca‐
1625 tion feature, each test can specify how much memory it requires from a
1626 GPU, allowing CTest to schedule tests in a way that running several of
1627 these tests at once does not exhaust the GPU's memory pool.
1628 Please note that CTest has no concept of what a GPU is or how much mem‐
1630 ory it has, nor does it have any way of communicating with a GPU to re‐
1631 trieve this information or perform any memory management. CTest simply
1632 keeps track of a list of abstract resource types, each of which has a
1633 certain number of slots available for tests to use. Each test specifies
1634 the number of slots that it requires from a certain resource, and CTest
1635 then schedules them in a way that prevents the total number of slots in
1636 use from exceeding the listed capacity. When a test is executed, and
1637 slots from a resource are allocated to that test, tests may assume that
1638 they have exclusive use of those slots for the duration of the test's
1639 process.
1640 The CTest resource allocation feature consists of two inputs:
1642 • The resource specification file, described below, which describes the
1644 resources available on the system.
1645 • The RESOURCE_GROUPS property of tests, which describes the resources
1647 required by the test.
1648 When CTest runs a test, the resources allocated to that test are passed
1650 in the form of a set of environment variables as described below. Using
1651 this information to decide which resource to connect to is left to the
1652 test writer.
1653 The RESOURCE_GROUPS property tells CTest what resources a test expects
1655 to use grouped in a way meaningful to the test. The test itself must
1656 read the environment variables to determine which resources have been
1657 allocated to each group. For example, each group may correspond to a
1658 process the test will spawn when executed.
1659 Note that even if a test specifies a RESOURCE_GROUPS property, it is
1661 still possible for that to test to run without any resource allocation
1662 (and without the corresponding environment variables) if the user does
1663 not pass a resource specification file. Passing this file, either
1664 through the --resource-spec-file command-line argument or the RE‐
1665 SOURCE_SPEC_FILE argument to ctest_test(), is what activates the re‐
1666 source allocation feature. Tests should check the CTEST_RE‐
1667 SOURCE_GROUP_COUNT environment variable to find out whether or not re‐
1668 source allocation is activated. This variable will always (and only) be
1669 defined if resource allocation is activated. If resource allocation is
1670 not activated, then the CTEST_RESOURCE_GROUP_COUNT variable will not
1671 exist, even if it exists for the parent ctest process. If a test abso‐
1672 lutely must have resource allocation, then it can return a failing exit
1673 code or use the SKIP_RETURN_CODE or SKIP_REGULAR_EXPRESSION properties
1674 to indicate a skipped test.
1675 Resource Specification File
1677 The resource specification file is a JSON file which is passed to
1678 CTest, either on the ctest(1) command line as --resource-spec-file, or
1679 as the RESOURCE_SPEC_FILE argument of ctest_test(). If a dashboard
1680 script is used and RESOURCE_SPEC_FILE is not specified, the value of
1681 CTEST_RESOURCE_SPEC_FILE in the dashboard script is used instead. If
1682 --resource-spec-file, RESOURCE_SPEC_FILE, and CTEST_RESOURCE_SPEC_FILE
1683 in the dashboard script are not specified, the value of CTEST_RE‐
1684 SOURCE_SPEC_FILE in the CMake build is used instead. If none of these
1685 are specified, no resource spec file is used.
1686 The resource specification file must be a JSON object. All examples in
1688 this document assume the following resource specification file:
1689 {
1691 "version": {
1692 "major": 1,
1693 "minor": 0
1694 },
1695 "local": [
1696 {
1697 "gpus": [
1698 {
1699 "id": "0",
1700 "slots": 2
1701 },
1702 {
1703 "id": "1",
1704 "slots": 4
1705 },
1706 {
1707 "id": "2",
1708 "slots": 2
1709 },
1710 {
1711 "id": "3"
1712 }
1713 ],
1714 "crypto_chips": [
1715 {
1716 "id": "card0",
1717 "slots": 4
1718 }
1719 ]
1720 }
1721 ]
1722 }
1723 The members are:
1725 version
1727 An object containing a major integer field and a minor integer
1728 field. Currently, the only supported version is major 1, minor
1729 0. Any other value is an error.
1730 local A JSON array of resource sets present on the system. Currently,
1732 this array is restricted to being of size 1.
1733 Each array element is a JSON object with members whose names are
1735 equal to the desired resource types, such as gpus. These names
1736 must start with a lowercase letter or an underscore, and subse‐
1737 quent characters can be a lowercase letter, a digit, or an un‐
1738 derscore. Uppercase letters are not allowed, because certain
1739 platforms have case-insensitive environment variables. See the
1740 Environment Variables section below for more information. It is
1741 recommended that the resource type name be the plural of a noun,
1742 such as gpus or crypto_chips (and not gpu or crypto_chip.)
1743 Please note that the names gpus and crypto_chips are just exam‐
1745 ples, and CTest does not interpret them in any way. You are free
1746 to make up any resource type you want to meet your own require‐
1747 ments.
1748 The value for each resource type is a JSON array consisting of
1750 JSON objects, each of which describe a specific instance of the
1751 specified resource. These objects have the following members:
1752 id A string consisting of an identifier for the resource.
1754 Each character in the identifier can be a lowercase let‐
1755 ter, a digit, or an underscore. Uppercase letters are
1756 not allowed.
1757 Identifiers must be unique within a resource type. How‐
1759 ever, they do not have to be unique across resource
1760 types. For example, it is valid to have a gpus resource
1761 named 0 and a crypto_chips resource named 0, but not two
1762 gpus resources both named 0.
1763 Please note that the IDs 0, 1, 2, 3, and card0 are just
1765 examples, and CTest does not interpret them in any way.
1766 You are free to make up any IDs you want to meet your own
1767 requirements.
1768 slots An optional unsigned number specifying the number of
1770 slots available on the resource. For example, this could
1771 be megabytes of RAM on a GPU, or cryptography units
1772 available on a cryptography chip. If slots is not speci‐
1773 fied, a default value of 1 is assumed.
1774 In the example file above, there are four GPUs with ID's 0 through 3.
1776 GPU 0 has 2 slots, GPU 1 has 4, GPU 2 has 2, and GPU 3 has a default of
1777 1 slot. There is also one cryptography chip with 4 slots.
1778 RESOURCE_GROUPS Property
1780 See RESOURCE_GROUPS for a description of this property.
1781 Environment Variables
1783 Once CTest has decided which resources to allocate to a test, it passes
1784 this information to the test executable as a series of environment
1785 variables. For each example below, we will assume that the test in
1786 question has a RESOURCE_GROUPS property of
1787 2,gpus:2;gpus:4,gpus:1,crypto_chips:2.
1788 The following variables are passed to the test process:
1790 CTEST_RESOURCE_GROUP_COUNT
1792 The total number of groups specified by the RESOURCE_GROUPS
1793 property. For example:
1794 • CTEST_RESOURCE_GROUP_COUNT=3
1796 This variable will only be defined if ctest(1) has been given a
1798 --resource-spec-file, or if ctest_test() has been given a RE‐
1799 SOURCE_SPEC_FILE. If no resource specification file has been
1800 given, this variable will not be defined.
1801 CTEST_RESOURCE_GROUP_<num>
1803 The list of resource types allocated to each group, with each
1804 item separated by a comma. <num> is a number from zero to
1805 CTEST_RESOURCE_GROUP_COUNT minus one. CTEST_RESOURCE_GROUP_<num>
1806 is defined for each <num> in this range. For example:
1807 • CTEST_RESOURCE_GROUP_0=gpus
1809 • CTEST_RESOURCE_GROUP_1=gpus
1811 • CTEST_RESOURCE_GROUP_2=crypto_chips,gpus
1813 CTEST_RESOURCE_GROUP_<num>_<resource-type>
1815 The list of resource IDs and number of slots from each ID allo‐
1816 cated to each group for a given resource type. This variable
1817 consists of a series of pairs, each pair separated by a semi‐
1818 colon, and with the two items in the pair separated by a comma.
1819 The first item in each pair is id: followed by the ID of a re‐
1820 source of type <resource-type>, and the second item is slots:
1821 followed by the number of slots from that resource allocated to
1822 the given group. For example:
1823 • CTEST_RESOURCE_GROUP_0_GPUS=id:0,slots:2
1825 • CTEST_RESOURCE_GROUP_1_GPUS=id:2,slots:2
1827 • CTEST_RESOURCE_GROUP_2_GPUS=id:1,slots:4;id:3,slots:1
1829 • CTEST_RESOURCE_GROUP_2_CRYPTO_CHIPS=id:card0,slots:2
1831 In this example, group 0 gets 2 slots from GPU 0, group 1 gets 2
1833 slots from GPU 2, and group 2 gets 4 slots from GPU 1, 1 slot
1834 from GPU 3, and 2 slots from cryptography chip card0.
1835 <num> is a number from zero to CTEST_RESOURCE_GROUP_COUNT minus
1837 one. <resource-type> is the name of a resource type, converted
1838 to uppercase. CTEST_RESOURCE_GROUP_<num>_<resource-type> is de‐
1839 fined for the product of each <num> in the range listed above
1840 and each resource type listed in CTEST_RESOURCE_GROUP_<num>.
1841 Because some platforms have case-insensitive names for environ‐
1843 ment variables, the names of resource types may not clash in a
1844 case-insensitive environment. Because of this, for the sake of
1845 simplicity, all resource types must be listed in all lowercase
1846 in the resource specification file and in the RESOURCE_GROUPS
1847 property, and they are converted to all uppercase in the
1848 CTEST_RESOURCE_GROUP_<num>_<resource-type> environment variable.
1849
1851 The following resources are available to get help using CMake:
1852 Home Page
1854 https://cmake.org
1855 The primary starting point for learning about CMake.
1857 Online Documentation and Community Resources
1859 https://cmake.org/documentation
1860 Links to available documentation and community resources may be
1862 found on this web page.
1863 Discourse Forum
1865 https://discourse.cmake.org
1866 The Discourse Forum hosts discussion and questions about CMake.
1868 : https://cdash.org
1870
1872 2000-2021 Kitware, Inc. and Contributors
18733.22.0 Dec 02, 2021 CTEST(1)