1CTEST(1) CMake CTEST(1)
2
6 ctest - CTest Command-Line Reference
7 Contents
9 • ctest(1)
10 • Synopsis
12 • Description
14 • Run Tests
16 • View Help
18 • Label Matching
20 • Label and Subproject Summary
22 • Build and Test Mode
24 • Dashboard Client
26 • Dashboard Client Steps
28 • Dashboard Client Modes
30 • Dashboard Client via CTest Command-Line
32 • Dashboard Client via CTest Script
34 • Dashboard Client Configuration
36 • CTest Start Step
38 • CTest Update Step
40 • CTest Configure Step
42 • CTest Build Step
44 • CTest Test Step
46 • CTest Coverage Step
48 • CTest MemCheck Step
50 • CTest Submit Step
52 • Show as JSON Object Model
54 • Resource Allocation
56 • Resource Specification File
58 • RESOURCE_GROUPS Property
60 • Environment Variables
62 • See Also
64
66 Run Tests
67 ctest [<options>]
68 Build and Test Mode
70 ctest --build-and-test <path-to-source> <path-to-build>
71 --build-generator <generator> [<options>...]
72 [--build-options <opts>...]
73 [--test-command <command> [<args>...]]
74 Dashboard Client
76 ctest -D <dashboard> [-- <dashboard-options>...]
77 ctest -M <model> -T <action> [-- <dashboard-options>...]
78 ctest -S <script> [-- <dashboard-options>...]
79 ctest -SP <script> [-- <dashboard-options>...]
80 View Help
82 ctest --help[-<topic>]
83
85 The ctest executable is the CMake test driver program. CMake-generated
86 build trees created for projects that use the enable_testing() and
87 add_test() commands have testing support. This program will run the
88 tests and report results.
89
91 --preset <preset>, --preset=<preset>
92 Use a test preset to specify test options. The project binary
93 directory is inferred from the configurePreset key. The current
94 working directory must contain CMake preset files. See preset
95 for more details.
96 --list-presets
98 Lists the available test presets. The current working directory
99 must contain CMake preset files.
100 -C <cfg>, --build-config <cfg>
102 Choose configuration to test.
103 Some CMake-generated build trees can have multiple build config‐
105 urations in the same tree. This option can be used to specify
106 which one should be tested. Example configurations are Debug
107 and Release.
108 --progress
110 Enable short progress output from tests.
111 When the output of ctest is being sent directly to a terminal,
113 the progress through the set of tests is reported by updating
114 the same line rather than printing start and end messages for
115 each test on new lines. This can significantly reduce the ver‐
116 bosity of the test output. Test completion messages are still
117 output on their own line for failed tests and the final test
118 summary will also still be logged.
119 This option can also be enabled by setting the environment vari‐
121 able CTEST_PROGRESS_OUTPUT.
122 -V, --verbose
124 Enable verbose output from tests.
125 Test output is normally suppressed and only summary information
127 is displayed. This option will show all test output.
128 -VV, --extra-verbose
130 Enable more verbose output from tests.
131 Test output is normally suppressed and only summary information
133 is displayed. This option will show even more test output.
134 --debug
136 Displaying more verbose internals of CTest.
137 This feature will result in a large number of output that is
139 mostly useful for debugging dashboard problems.
140 --output-on-failure
142 Output anything outputted by the test program if the test should
143 fail. This option can also be enabled by setting the
144 CTEST_OUTPUT_ON_FAILURE environment variable
145 --stop-on-failure
147 Stop running the tests when the first failure happens.
148 -F Enable failover.
150 This option allows CTest to resume a test set execution that was
152 previously interrupted. If no interruption occurred, the -F op‐
153 tion will have no effect.
154 -j <jobs>, --parallel <jobs>
156 Run the tests in parallel using the given number of jobs.
157 This option tells CTest to run the tests in parallel using given
159 number of jobs. This option can also be set by setting the
160 CTEST_PARALLEL_LEVEL environment variable.
161 This option can be used with the PROCESSORS test property.
163 See Label and Subproject Summary.
165 --resource-spec-file <file>
167 Run CTest with resource allocation enabled, using the resource
168 specification file specified in <file>.
169 When ctest is run as a Dashboard Client this sets the Resource‐
171 SpecFile option of the CTest Test Step.
172 --test-load <level>
174 While running tests in parallel (e.g. with -j), try not to start
175 tests when they may cause the CPU load to pass above a given
176 threshold.
177 When ctest is run as a Dashboard Client this sets the TestLoad
179 option of the CTest Test Step.
180 -Q, --quiet
182 Make CTest quiet.
183 This option will suppress all the output. The output log file
185 will still be generated if the --output-log is specified. Op‐
186 tions such as --verbose, --extra-verbose, and --debug are ig‐
187 nored if --quiet is specified.
188 -O <file>, --output-log <file>
190 Output to log file.
191 This option tells CTest to write all its output to a <file> log
193 file.
194 --output-junit <file>
196 New in version 3.21.
197 Write test results in JUnit format.
200 This option tells CTest to write test results to <file> in JUnit
202 XML format. If <file> already exists, it will be overwritten. If
203 using the -S option to run a dashboard script, use the OUT‐
204 PUT_JUNIT keyword with the ctest_test() command instead.
205 -N, --show-only[=<format>]
207 Disable actual execution of tests.
208 This option tells CTest to list the tests that would be run but
210 not actually run them. Useful in conjunction with the -R and -E
211 options.
212 New in version 3.14: The --show-only option accepts a <format>
214 value.
215 <format> can be one of the following values.
218 human Human-friendly output. This is not guaranteed to be
220 stable. This is the default.
221 json-v1
223 Dump the test information in JSON format. See Show as
224 JSON Object Model.
225 -L <regex>, --label-regex <regex>
227 Run tests with labels matching regular expression as described
228 under string(REGEX).
229 This option tells CTest to run only the tests whose labels match
231 the given regular expression. When more than one -L option is
232 given, a test will only be run if each regular expression
233 matches at least one of the test's labels (i.e. the multiple -L
234 labels form an AND relationship). See Label Matching.
235 -R <regex>, --tests-regex <regex>
237 Run tests matching regular expression.
238 This option tells CTest to run only the tests whose names match
240 the given regular expression.
241 -E <regex>, --exclude-regex <regex>
243 Exclude tests matching regular expression.
244 This option tells CTest to NOT run the tests whose names match
246 the given regular expression.
247 -LE <regex>, --label-exclude <regex>
249 Exclude tests with labels matching regular expression.
250 This option tells CTest to NOT run the tests whose labels match
252 the given regular expression. When more than one -LE option is
253 given, a test will only be excluded if each regular expression
254 matches at least one of the test's labels (i.e. the multiple -LE
255 labels form an AND relationship). See Label Matching.
256 -FA <regex>, --fixture-exclude-any <regex>
258 Exclude fixtures matching <regex> from automatically adding any
259 tests to the test set.
260 If a test in the set of tests to be executed requires a particu‐
262 lar fixture, that fixture's setup and cleanup tests would nor‐
263 mally be added to the test set automatically. This option pre‐
264 vents adding setup or cleanup tests for fixtures matching the
265 <regex>. Note that all other fixture behavior is retained, in‐
266 cluding test dependencies and skipping tests that have fixture
267 setup tests that fail.
268 -FS <regex>, --fixture-exclude-setup <regex>
270 Same as -FA except only matching setup tests are excluded.
271 -FC <regex>, --fixture-exclude-cleanup <regex>
273 Same as -FA except only matching cleanup tests are excluded.
274 -I [Start,End,Stride,test#,test#|Test file], --tests-information
276 Run a specific number of tests by number.
277 This option causes CTest to run tests starting at number Start,
279 ending at number End, and incrementing by Stride. Any addi‐
280 tional numbers after Stride are considered individual test num‐
281 bers. Start, End, or Stride can be empty. Optionally a file
282 can be given that contains the same syntax as the command line.
283 -U, --union
285 Take the Union of -I and -R.
286 When both -R and -I are specified by default the intersection of
288 tests are run. By specifying -U the union of tests is run in‐
289 stead.
290 --rerun-failed
292 Run only the tests that failed previously.
293 This option tells CTest to perform only the tests that failed
295 during its previous run. When this option is specified, CTest
296 ignores all other options intended to modify the list of tests
297 to run ( -L, -R, -E, -LE, -I, etc). In the event that CTest
298 runs and no tests fail, subsequent calls to CTest with the --re‐
299 run-failed option will run the set of tests that most recently
300 failed (if any).
301 --repeat <mode>:<n>
303 Run tests repeatedly based on the given <mode> up to <n> times.
304 The modes are:
305 until-fail
307 Require each test to run <n> times without failing in or‐
308 der to pass. This is useful in finding sporadic failures
309 in test cases.
310 until-pass
312 Allow each test to run up to <n> times in order to pass.
313 Repeats tests if they fail for any reason. This is use‐
314 ful in tolerating sporadic failures in test cases.
315 after-timeout
317 Allow each test to run up to <n> times in order to pass.
318 Repeats tests only if they timeout. This is useful in
319 tolerating sporadic timeouts in test cases on busy ma‐
320 chines.
321 --repeat-until-fail <n>
323 Equivalent to --repeat until-fail:<n>.
324 --max-width <width>
326 Set the max width for a test name to output.
327 Set the maximum width for each test name to show in the output.
329 This allows the user to widen the output to avoid clipping the
330 test name which can be very annoying.
331 --interactive-debug-mode [0|1]
333 Set the interactive mode to 0 or 1.
334 This option causes CTest to run tests in either an interactive
336 mode or a non-interactive mode. In dashboard mode (Experimen‐
337 tal, Nightly, Continuous), the default is non-interactive. In
338 non-interactive mode, the environment variable
339 DASHBOARD_TEST_FROM_CTEST is set.
340 Prior to CMake 3.11, interactive mode on Windows allowed system
342 debug popup windows to appear. Now, due to CTest's use of libuv
343 to launch test processes, all system debug popup windows are al‐
344 ways blocked.
345 --no-label-summary
347 Disable timing summary information for labels.
348 This option tells CTest not to print summary information for
350 each label associated with the tests run. If there are no la‐
351 bels on the tests, nothing extra is printed.
352 See Label and Subproject Summary.
354 --no-subproject-summary
356 Disable timing summary information for subprojects.
357 This option tells CTest not to print summary information for
359 each subproject associated with the tests run. If there are no
360 subprojects on the tests, nothing extra is printed.
361 See Label and Subproject Summary.
363 --test-dir <dir>
365 Specify the directory in which to look for tests.
366 --test-output-size-passed <size>
368 New in version 3.4.
369 Limit the output for passed tests to <size> bytes.
372 --test-output-size-failed <size>
374 New in version 3.4.
375 Limit the output for failed tests to <size> bytes.
378 --test-output-truncation <mode>
380 New in version 3.24.
381 Truncate tail (default), middle or head of test output once max‐
384 imum output size is reached.
385 --overwrite
387 Overwrite CTest configuration option.
388 By default CTest uses configuration options from configuration
390 file. This option will overwrite the configuration option.
391 --force-new-ctest-process
393 Run child CTest instances as new processes.
394 By default CTest will run child CTest instances within the same
396 process. If this behavior is not desired, this argument will
397 enforce new processes for child CTest processes.
398 --schedule-random
400 Use a random order for scheduling tests.
401 This option will run the tests in a random order. It is com‐
403 monly used to detect implicit dependencies in a test suite.
404 --submit-index
406 Legacy option for old Dart2 dashboard server feature. Do not
407 use.
408 --timeout <seconds>
410 Set the default test timeout.
411 This option effectively sets a timeout on all tests that do not
413 already have a timeout set on them via the TIMEOUT property.
414 --stop-time <time>
416 Set a time at which all tests should stop running.
417 Set a real time of day at which all tests should timeout. Exam‐
419 ple: 7:00:00 -0400. Any time format understood by the curl date
420 parser is accepted. Local time is assumed if no timezone is
421 specified.
422 --print-labels
424 Print all available test labels.
425 This option will not run any tests, it will simply print the
427 list of all labels associated with the test set.
428 --no-tests=<action>
430 Regard no tests found either as error (when <action> is set to
431 error) or ignore it (when <action> is set to ignore).
432 If no tests were found, the default behavior of CTest is to al‐
434 ways log an error message but to return an error code in script
435 mode only. This option unifies the behavior of CTest by either
436 returning an error code if no tests were found or by ignoring
437 it.
438
440 To print version details or selected pages from the CMake documenta‐
441 tion, use one of the following options:
442 -version [<file>], --version [<file>], /V [<file>]
444 Show program name/version banner and exit. The output is
445 printed to a named <file> if given.
446 -h, -H, --help, -help, -usage, /?
448 Print usage information and exit.
449 Usage describes the basic command line interface and its op‐
451 tions.
452 --help-full [<file>]
454 Print all help manuals and exit.
455 All manuals are printed in a human-readable text format. The
457 output is printed to a named <file> if given.
458 --help-manual <man> [<file>]
460 Print one help manual and exit.
461 The specified manual is printed in a human-readable text format.
463 The output is printed to a named <file> if given.
464 --help-manual-list [<file>]
466 List help manuals available and exit.
467 The list contains all manuals for which help may be obtained by
469 using the --help-manual option followed by a manual name. The
470 output is printed to a named <file> if given.
471 --help-command <cmd> [<file>]
473 Print help for one command and exit.
474 The cmake-commands(7) manual entry for <cmd> is printed in a hu‐
476 man-readable text format. The output is printed to a named
477 <file> if given.
478 --help-command-list [<file>]
480 List commands with help available and exit.
481 The list contains all commands for which help may be obtained by
483 using the --help-command option followed by a command name. The
484 output is printed to a named <file> if given.
485 --help-commands [<file>]
487 Print cmake-commands manual and exit.
488 The cmake-commands(7) manual is printed in a human-readable text
490 format. The output is printed to a named <file> if given.
491 --help-module <mod> [<file>]
493 Print help for one module and exit.
494 The cmake-modules(7) manual entry for <mod> is printed in a hu‐
496 man-readable text format. The output is printed to a named
497 <file> if given.
498 --help-module-list [<file>]
500 List modules with help available and exit.
501 The list contains all modules for which help may be obtained by
503 using the --help-module option followed by a module name. The
504 output is printed to a named <file> if given.
505 --help-modules [<file>]
507 Print cmake-modules manual and exit.
508 The cmake-modules(7) manual is printed in a human-readable text
510 format. The output is printed to a named <file> if given.
511 --help-policy <cmp> [<file>]
513 Print help for one policy and exit.
514 The cmake-policies(7) manual entry for <cmp> is printed in a hu‐
516 man-readable text format. The output is printed to a named
517 <file> if given.
518 --help-policy-list [<file>]
520 List policies with help available and exit.
521 The list contains all policies for which help may be obtained by
523 using the --help-policy option followed by a policy name. The
524 output is printed to a named <file> if given.
525 --help-policies [<file>]
527 Print cmake-policies manual and exit.
528 The cmake-policies(7) manual is printed in a human-readable text
530 format. The output is printed to a named <file> if given.
531 --help-property <prop> [<file>]
533 Print help for one property and exit.
534 The cmake-properties(7) manual entries for <prop> are printed in
536 a human-readable text format. The output is printed to a named
537 <file> if given.
538 --help-property-list [<file>]
540 List properties with help available and exit.
541 The list contains all properties for which help may be obtained
543 by using the --help-property option followed by a property name.
544 The output is printed to a named <file> if given.
545 --help-properties [<file>]
547 Print cmake-properties manual and exit.
548 The cmake-properties(7) manual is printed in a human-readable
550 text format. The output is printed to a named <file> if given.
551 --help-variable <var> [<file>]
553 Print help for one variable and exit.
554 The cmake-variables(7) manual entry for <var> is printed in a
556 human-readable text format. The output is printed to a named
557 <file> if given.
558 --help-variable-list [<file>]
560 List variables with help available and exit.
561 The list contains all variables for which help may be obtained
563 by using the --help-variable option followed by a variable name.
564 The output is printed to a named <file> if given.
565 --help-variables [<file>]
567 Print cmake-variables manual and exit.
568 The cmake-variables(7) manual is printed in a human-readable
570 text format. The output is printed to a named <file> if given.
571
573 Tests may have labels attached to them. Tests may be included or ex‐
574 cluded from a test run by filtering on the labels. Each individual
575 filter is a regular expression applied to the labels attached to a
576 test.
577 When -L is used, in order for a test to be included in a test run, each
579 regular expression must match at least one label. Using more than one
580 -L option means "match all of these".
581 The -LE option works just like -L, but excludes tests rather than in‐
583 cluding them. A test is excluded if each regular expression matches at
584 least one label.
585 If a test has no labels attached to it, then -L will never include that
587 test, and -LE will never exclude that test. As an example of tests
588 with labels, consider five tests, with the following labels:
589 • test1 has labels tuesday and production
591 • test2 has labels tuesday and test
593 • test3 has labels wednesday and production
595 • test4 has label wednesday
597 • test5 has labels friday and test
599 Running ctest with -L tuesday -L test will select test2, which has both
601 labels. Running CTest with -L test will select test2 and test5, because
602 both of them have a label that matches that regular expression.
603 Because the matching works with regular expressions, take note that
605 running CTest with -L es will match all five tests. To select the
606 tuesday and wednesday tests together, use a single regular expression
607 that matches either of them, like -L "tue|wed".
608
610 CTest prints timing summary information for each LABEL and subproject
611 associated with the tests run. The label time summary will not include
612 labels that are mapped to subprojects.
613 New in version 3.22: Labels added dynamically during test execution are
615 also reported in the timing summary. See Additional Labels.
616 When the PROCESSORS test property is set, CTest will display a weighted
619 test timing result in label and subproject summaries. The time is re‐
620 ported with sec*proc instead of just sec.
621 The weighted time summary reported for each label or subproject j is
623 computed as:
624 Weighted Time Summary for Label/Subproject j =
626 sum(raw_test_time[j,i] * num_processors[j,i], i=1...num_tests[j])
627 for labels/subprojects j=1...total
629 where:
631 • raw_test_time[j,i]: Wall-clock time for the i test for the j label or
633 subproject
634 • num_processors[j,i]: Value of the CTest PROCESSORS property for the i
636 test for the j label or subproject
637 • num_tests[j]: Number of tests associated with the j label or subpro‐
639 ject
640 • total: Total number of labels or subprojects that have at least one
642 test run
643 Therefore, the weighted time summary for each label or subproject rep‐
645 resents the amount of time that CTest gave to run the tests for each
646 label or subproject and gives a good representation of the total ex‐
647 pense of the tests for each label or subproject when compared to other
648 labels or subprojects.
649 For example, if SubprojectA showed 100 sec*proc and SubprojectB showed
651 10 sec*proc, then CTest allocated approximately 10 times the CPU/core
652 time to run the tests for SubprojectA than for SubprojectB (e.g. so if
653 effort is going to be expended to reduce the cost of the test suite for
654 the whole project, then reducing the cost of the test suite for Subpro‐
655 jectA would likely have a larger impact than effort to reduce the cost
656 of the test suite for SubprojectB).
657
659 CTest provides a command-line signature to configure (i.e. run cmake
660 on), build, and/or execute a test:
661 ctest --build-and-test <path-to-source> <path-to-build>
663 --build-generator <generator>
664 [<options>...]
665 [--build-options <opts>...]
666 [--test-command <command> [<args>...]]
667 The configure and test steps are optional. The arguments to this com‐
669 mand line are the source and binary directories. The --build-generator
670 option must be provided to use --build-and-test. If --test-command is
671 specified then that will be run after the build is complete. Other op‐
672 tions that affect this mode include:
673 --build-and-test
675 Switch into the build and test mode.
676 --build-target
678 Specify a specific target to build. The option can be given
679 multiple times with different targets, in which case each target
680 is built in turn. A clean will be done before building each
681 target unless the --build-noclean option is given.
682 If no --build-target is specified, the all target is built.
684 --build-nocmake
686 Run the build without running cmake first.
687 Skip the cmake step.
689 --build-run-dir
691 Specify directory to run programs from.
692 Directory where programs will be after it has been compiled.
694 --build-two-config
696 Run CMake twice.
697 --build-exe-dir
699 Specify the directory for the executable.
700 --build-generator
702 Specify the generator to use. See the cmake-generators(7) man‐
703 ual.
704 --build-generator-platform
706 Specify the generator-specific platform.
707 --build-generator-toolset
709 Specify the generator-specific toolset.
710 --build-project
712 Specify the name of the project to build.
713 --build-makeprogram
715 Specify the explicit make program to be used by CMake when con‐
716 figuring and building the project. Only applicable for Make and
717 Ninja based generators.
718 --build-noclean
720 Skip the make clean step.
721 --build-config-sample
723 A sample executable to use to determine the configuration that
724 should be used. e.g. Debug, Release etc.
725 --build-options
727 Additional options for configuring the build (i.e. for CMake,
728 not for the build tool). Note that if this is specified, the
729 --build-options keyword and its arguments must be the last op‐
730 tion given on the command line, with the possible exception of
731 --test-command.
732 --test-command
734 The command to run as the test step with the --build-and-test
735 option. All arguments following this keyword will be assumed to
736 be part of the test command line, so it must be the last option
737 given.
738 --test-timeout
740 The time limit in seconds
741
743 CTest can operate as a client for the CDash software quality dashboard
744 application. As a dashboard client, CTest performs a sequence of steps
745 to configure, build, and test software, and then submits the results to
746 a CDash server. The command-line signature used to submit to CDash is:
747 ctest -D <dashboard> [-- <dashboard-options>...]
749 ctest -M <model> -T <action> [-- <dashboard-options>...]
750 ctest -S <script> [-- <dashboard-options>...]
751 ctest -SP <script> [-- <dashboard-options>...]
752 Options for Dashboard Client include:
754 -D <dashboard>, --dashboard <dashboard>
756 Execute dashboard test.
757 This option tells CTest to act as a CDash client and perform a
759 dashboard test. All tests are <Mode><Test>, where <Mode> can be
760 Experimental, Nightly, and Continuous, and <Test> can be Start,
761 Update, Configure, Build, Test, Coverage, and Submit.
762 If <dashboard> is not one of the recognized <Mode><Test> values,
764 this will be treated as a variable definition instead (see the
765 dashboard-options further below).
766 -M <model>, --test-model <model>
768 Sets the model for a dashboard.
769 This option tells CTest to act as a CDash client where the
771 <model> can be Experimental, Nightly, and Continuous. Combining
772 -M and -T is similar to -D.
773 -T <action>, --test-action <action>
775 Sets the dashboard action to perform.
776 This option tells CTest to act as a CDash client and perform
778 some action such as start, build, test etc. See Dashboard Client
779 Steps for the full list of actions. Combining -M and -T is sim‐
780 ilar to -D.
781 -S <script>, --script <script>
783 Execute a dashboard for a configuration.
784 This option tells CTest to load in a configuration script which
786 sets a number of parameters such as the binary and source direc‐
787 tories. Then CTest will do what is required to create and run a
788 dashboard. This option basically sets up a dashboard and then
789 runs ctest -D with the appropriate options.
790 -SP <script>, --script-new-process <script>
792 Execute a dashboard for a configuration.
793 This option does the same operations as -S but it will do them
795 in a separate process. This is primarily useful in cases where
796 the script may modify the environment and you do not want the
797 modified environment to impact other -S scripts.
798 The available <dashboard-options> are the following:
800 -D <var>:<type>=<value>
802 Define a variable for script mode.
803 Pass in variable values on the command line. Use in conjunction
805 with -S to pass variable values to a dashboard script. Parsing
806 -D arguments as variable values is only attempted if the value
807 following -D does not match any of the known dashboard types.
808 --group <group>
810 Specify what group you'd like to submit results to
811 Submit dashboard to specified group instead of default one. By
813 default, the dashboard is submitted to Nightly, Experimental, or
814 Continuous group, but by specifying this option, the group can
815 be arbitrary.
816 This replaces the deprecated option --track. Despite the name
818 change its behavior is unchanged.
819 -A <file>, --add-notes <file>
821 Add a notes file with submission.
822 This option tells CTest to include a notes file when submitting
824 dashboard.
825 --tomorrow-tag
827 Nightly or Experimental starts with next day tag.
828 This is useful if the build will not finish in one day.
830 --extra-submit <file>[;<file>]
832 Submit extra files to the dashboard.
833 This option will submit extra files to the dashboard.
835 --http1.0
837 Submit using HTTP 1.0.
838 This option will force CTest to use HTTP 1.0 to submit files to
840 the dashboard, instead of HTTP 1.1.
841 --no-compress-output
843 Do not compress test output when submitting.
844 This flag will turn off automatic compression of test output.
846 Use this to maintain compatibility with an older version of
847 CDash which doesn't support compressed test output.
848 Dashboard Client Steps
850 CTest defines an ordered list of testing steps of which some or all may
851 be run as a dashboard client:
852 Start Start a new dashboard submission to be composed of results
854 recorded by the following steps. See the CTest Start Step sec‐
855 tion below.
856 Update Update the source tree from its version control repository.
858 Record the old and new versions and the list of updated source
859 files. See the CTest Update Step section below.
860 Configure
862 Configure the software by running a command in the build tree.
863 Record the configuration output log. See the CTest Configure
864 Step section below.
865 Build Build the software by running a command in the build tree.
867 Record the build output log and detect warnings and errors. See
868 the CTest Build Step section below.
869 Test Test the software by loading a CTestTestfile.cmake from the
871 build tree and executing the defined tests. Record the output
872 and result of each test. See the CTest Test Step section below.
873 Coverage
875 Compute coverage of the source code by running a coverage analy‐
876 sis tool and recording its output. See the CTest Coverage Step
877 section below.
878 MemCheck
880 Run the software test suite through a memory check tool. Record
881 the test output, results, and issues reported by the tool. See
882 the CTest MemCheck Step section below.
883 Submit Submit results recorded from other testing steps to the software
885 quality dashboard server. See the CTest Submit Step section be‐
886 low.
887 Dashboard Client Modes
889 CTest defines three modes of operation as a dashboard client:
890 Nightly
892 This mode is intended to be invoked once per day, typically at
893 night. It enables the Start, Update, Configure, Build, Test,
894 Coverage, and Submit steps by default. Selected steps run even
895 if the Update step reports no changes to the source tree.
896 Continuous
898 This mode is intended to be invoked repeatedly throughout the
899 day. It enables the Start, Update, Configure, Build, Test, Cov‐
900 erage, and Submit steps by default, but exits after the Update
901 step if it reports no changes to the source tree.
902 Experimental
904 This mode is intended to be invoked by a developer to test local
905 changes. It enables the Start, Configure, Build, Test, Cover‐
906 age, and Submit steps by default.
907 Dashboard Client via CTest Command-Line
909 CTest can perform testing on an already-generated build tree. Run the
910 ctest command with the current working directory set to the build tree
911 and use one of these signatures:
912 ctest -D <mode>[<step>]
914 ctest -M <mode> [-T <step>]...
915 The <mode> must be one of the above Dashboard Client Modes, and each
917 <step> must be one of the above Dashboard Client Steps.
918 CTest reads the Dashboard Client Configuration settings from a file in
920 the build tree called either CTestConfiguration.ini or DartConfigura‐
921 tion.tcl (the names are historical). The format of the file is:
922 # Lines starting in '#' are comments.
924 # Other non-blank lines are key-value pairs.
925 <setting>: <value>
926 where <setting> is the setting name and <value> is the setting value.
928 In build trees generated by CMake, this configuration file is generated
930 by the CTest module if included by the project. The module uses vari‐
931 ables to obtain a value for each setting as documented with the set‐
932 tings below.
933 Dashboard Client via CTest Script
935 CTest can perform testing driven by a cmake-language(7) script that
936 creates and maintains the source and build tree as well as performing
937 the testing steps. Run the ctest command with the current working di‐
938 rectory set outside of any build tree and use one of these signatures:
939 ctest -S <script>
941 ctest -SP <script>
942 The <script> file must call CTest Commands commands to run testing
944 steps explicitly as documented below. The commands obtain Dashboard
945 Client Configuration settings from their arguments or from variables
946 set in the script.
947
949 The Dashboard Client Steps may be configured by named settings as docu‐
950 mented in the following sections.
951 CTest Start Step
953 Start a new dashboard submission to be composed of results recorded by
954 the following steps.
955 In a CTest Script, the ctest_start() command runs this step. Arguments
957 to the command may specify some of the step settings. The command
958 first runs the command-line specified by the CTEST_CHECKOUT_COMMAND
959 variable, if set, to initialize the source directory.
960 Configuration settings include:
962 BuildDirectory
964 The full path to the project build tree.
965 • CTest Script variable: CTEST_BINARY_DIRECTORY
967 • CTest module variable: PROJECT_BINARY_DIR
969 SourceDirectory
971 The full path to the project source tree.
972 • CTest Script variable: CTEST_SOURCE_DIRECTORY
974 • CTest module variable: PROJECT_SOURCE_DIR
976 CTest Update Step
978 In a CTest Script, the ctest_update() command runs this step. Argu‐
979 ments to the command may specify some of the step settings.
980 Configuration settings to specify the version control tool include:
982 BZRCommand
984 bzr command-line tool to use if source tree is managed by
985 Bazaar.
986 • CTest Script variable: CTEST_BZR_COMMAND
988 • CTest module variable: none
990 BZRUpdateOptions
992 Command-line options to the BZRCommand when updating the source.
993 • CTest Script variable: CTEST_BZR_UPDATE_OPTIONS
995 • CTest module variable: none
997 CVSCommand
999 cvs command-line tool to use if source tree is managed by CVS.
1000 • CTest Script variable: CTEST_CVS_COMMAND
1002 • CTest module variable: CVSCOMMAND
1004 CVSUpdateOptions
1006 Command-line options to the CVSCommand when updating the source.
1007 • CTest Script variable: CTEST_CVS_UPDATE_OPTIONS
1009 • CTest module variable: CVS_UPDATE_OPTIONS
1011 GITCommand
1013 git command-line tool to use if source tree is managed by Git.
1014 • CTest Script variable: CTEST_GIT_COMMAND
1016 • CTest module variable: GITCOMMAND
1018 The source tree is updated by git fetch followed by git reset
1020 --hard to the FETCH_HEAD. The result is the same as git pull
1021 except that any local modifications are overwritten. Use GITUp‐
1022 dateCustom to specify a different approach.
1023 GITInitSubmodules
1025 If set, CTest will update the repository's submodules before up‐
1026 dating.
1027 • CTest Script variable: CTEST_GIT_INIT_SUBMODULES
1029 • CTest module variable: CTEST_GIT_INIT_SUBMODULES
1031 GITUpdateCustom
1033 Specify a custom command line (as a semicolon-separated list) to
1034 run in the source tree (Git work tree) to update it instead of
1035 running the GITCommand.
1036 • CTest Script variable: CTEST_GIT_UPDATE_CUSTOM
1038 • CTest module variable: CTEST_GIT_UPDATE_CUSTOM
1040 GITUpdateOptions
1042 Command-line options to the GITCommand when updating the source.
1043 • CTest Script variable: CTEST_GIT_UPDATE_OPTIONS
1045 • CTest module variable: GIT_UPDATE_OPTIONS
1047 HGCommand
1049 hg command-line tool to use if source tree is managed by Mercu‐
1050 rial.
1051 • CTest Script variable: CTEST_HG_COMMAND
1053 • CTest module variable: none
1055 HGUpdateOptions
1057 Command-line options to the HGCommand when updating the source.
1058 • CTest Script variable: CTEST_HG_UPDATE_OPTIONS
1060 • CTest module variable: none
1062 P4Client
1064 Value of the -c option to the P4Command.
1065 • CTest Script variable: CTEST_P4_CLIENT
1067 • CTest module variable: CTEST_P4_CLIENT
1069 P4Command
1071 p4 command-line tool to use if source tree is managed by Per‐
1072 force.
1073 • CTest Script variable: CTEST_P4_COMMAND
1075 • CTest module variable: P4COMMAND
1077 P4Options
1079 Command-line options to the P4Command for all invocations.
1080 • CTest Script variable: CTEST_P4_OPTIONS
1082 • CTest module variable: CTEST_P4_OPTIONS
1084 P4UpdateCustom
1086 Specify a custom command line (as a semicolon-separated list) to
1087 run in the source tree (Perforce tree) to update it instead of
1088 running the P4Command.
1089 • CTest Script variable: none
1091 • CTest module variable: CTEST_P4_UPDATE_CUSTOM
1093 P4UpdateOptions
1095 Command-line options to the P4Command when updating the source.
1096 • CTest Script variable: CTEST_P4_UPDATE_OPTIONS
1098 • CTest module variable: CTEST_P4_UPDATE_OPTIONS
1100 SVNCommand
1102 svn command-line tool to use if source tree is managed by Sub‐
1103 version.
1104 • CTest Script variable: CTEST_SVN_COMMAND
1106 • CTest module variable: SVNCOMMAND
1108 SVNOptions
1110 Command-line options to the SVNCommand for all invocations.
1111 • CTest Script variable: CTEST_SVN_OPTIONS
1113 • CTest module variable: CTEST_SVN_OPTIONS
1115 SVNUpdateOptions
1117 Command-line options to the SVNCommand when updating the source.
1118 • CTest Script variable: CTEST_SVN_UPDATE_OPTIONS
1120 • CTest module variable: SVN_UPDATE_OPTIONS
1122 UpdateCommand
1124 Specify the version-control command-line tool to use without de‐
1125 tecting the VCS that manages the source tree.
1126 • CTest Script variable: CTEST_UPDATE_COMMAND
1128 • CTest module variable: <VCS>COMMAND when UPDATE_TYPE is <vcs>,
1130 else UPDATE_COMMAND
1131 UpdateOptions
1133 Command-line options to the UpdateCommand.
1134 • CTest Script variable: CTEST_UPDATE_OPTIONS
1136 • CTest module variable: <VCS>_UPDATE_OPTIONS when UPDATE_TYPE
1138 is <vcs>, else UPDATE_OPTIONS
1139 UpdateType
1141 Specify the version-control system that manages the source tree
1142 if it cannot be detected automatically. The value may be bzr,
1143 cvs, git, hg, p4, or svn.
1144 • CTest Script variable: none, detected from source tree
1146 • CTest module variable: UPDATE_TYPE if set, else CTEST_UP‐
1148 DATE_TYPE
1149 UpdateVersionOnly
1151 Specify that you want the version control update command to only
1152 discover the current version that is checked out, and not to up‐
1153 date to a different version.
1154 • CTest Script variable: CTEST_UPDATE_VERSION_ONLY
1156 UpdateVersionOverride
1158 Specify the current version of your source tree.
1159 When this variable is set to a non-empty string, CTest will re‐
1161 port the value you specified rather than using the update com‐
1162 mand to discover the current version that is checked out. Use of
1163 this variable supersedes UpdateVersionOnly. Like UpdateVer‐
1164 sionOnly, using this variable tells CTest not to update the
1165 source tree to a different version.
1166 • CTest Script variable: CTEST_UPDATE_VERSION_OVERRIDE
1168 Additional configuration settings include:
1170 NightlyStartTime
1172 In the Nightly dashboard mode, specify the "nightly start time".
1173 With centralized version control systems (cvs and svn), the Up‐
1174 date step checks out the version of the software as of this time
1175 so that multiple clients choose a common version to test. This
1176 is not well-defined in distributed version-control systems so
1177 the setting is ignored.
1178 • CTest Script variable: CTEST_NIGHTLY_START_TIME
1180 • CTest module variable: NIGHTLY_START_TIME if set, else
1182 CTEST_NIGHTLY_START_TIME
1183 CTest Configure Step
1185 In a CTest Script, the ctest_configure() command runs this step. Argu‐
1186 ments to the command may specify some of the step settings.
1187 Configuration settings include:
1189 ConfigureCommand
1191 Command-line to launch the software configuration process. It
1192 will be executed in the location specified by the BuildDirectory
1193 setting.
1194 • CTest Script variable: CTEST_CONFIGURE_COMMAND
1196 • CTest module variable: CMAKE_COMMAND followed by
1198 PROJECT_SOURCE_DIR
1199 LabelsForSubprojects
1201 Specify a semicolon-separated list of labels that will be
1202 treated as subprojects. This mapping will be passed on to CDash
1203 when configure, test or build results are submitted.
1204 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1206 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1208 See Label and Subproject Summary.
1210 CTest Build Step
1212 In a CTest Script, the ctest_build() command runs this step. Arguments
1213 to the command may specify some of the step settings.
1214 Configuration settings include:
1216 DefaultCTestConfigurationType
1218 When the build system to be launched allows build-time selection
1219 of the configuration (e.g. Debug, Release), this specifies the
1220 default configuration to be built when no -C option is given to
1221 the ctest command. The value will be substituted into the value
1222 of MakeCommand to replace the literal string ${CTEST_CONFIGURA‐
1223 TION_TYPE} if it appears.
1224 • CTest Script variable: CTEST_CONFIGURATION_TYPE
1226 • CTest module variable: DEFAULT_CTEST_CONFIGURATION_TYPE, ini‐
1228 tialized by the CMAKE_CONFIG_TYPE environment variable
1229 LabelsForSubprojects
1231 Specify a semicolon-separated list of labels that will be
1232 treated as subprojects. This mapping will be passed on to CDash
1233 when configure, test or build results are submitted.
1234 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1236 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1238 See Label and Subproject Summary.
1240 MakeCommand
1242 Command-line to launch the software build process. It will be
1243 executed in the location specified by the BuildDirectory set‐
1244 ting.
1245 • CTest Script variable: CTEST_BUILD_COMMAND
1247 • CTest module variable: MAKECOMMAND, initialized by the
1249 build_command() command
1250 UseLaunchers
1252 For build trees generated by CMake using one of the Makefile
1253 Generators or the Ninja generator, specify whether the
1254 CTEST_USE_LAUNCHERS feature is enabled by the CTestUseLaunchers
1255 module (also included by the CTest module). When enabled, the
1256 generated build system wraps each invocation of the compiler,
1257 linker, or custom command line with a "launcher" that communi‐
1258 cates with CTest via environment variables and files to report
1259 granular build warning and error information. Otherwise, CTest
1260 must "scrape" the build output log for diagnostics.
1261 • CTest Script variable: CTEST_USE_LAUNCHERS
1263 • CTest module variable: CTEST_USE_LAUNCHERS
1265 CTest Test Step
1267 In a CTest Script, the ctest_test() command runs this step. Arguments
1268 to the command may specify some of the step settings.
1269 Configuration settings include:
1271 ResourceSpecFile
1273 Specify a resource specification file.
1274 • CTest Script variable: CTEST_RESOURCE_SPEC_FILE
1276 • CTest module variable: CTEST_RESOURCE_SPEC_FILE
1278 See Resource Allocation for more information.
1280 LabelsForSubprojects
1282 Specify a semicolon-separated list of labels that will be
1283 treated as subprojects. This mapping will be passed on to CDash
1284 when configure, test or build results are submitted.
1285 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1287 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1289 See Label and Subproject Summary.
1291 TestLoad
1293 While running tests in parallel (e.g. with -j), try not to start
1294 tests when they may cause the CPU load to pass above a given
1295 threshold.
1296 • CTest Script variable: CTEST_TEST_LOAD
1298 • CTest module variable: CTEST_TEST_LOAD
1300 TimeOut
1302 The default timeout for each test if not specified by the
1303 TIMEOUT test property.
1304 • CTest Script variable: CTEST_TEST_TIMEOUT
1306 • CTest module variable: DART_TESTING_TIMEOUT
1308 To report extra test values to CDash, see Additional Test Measurements.
1310 CTest Coverage Step
1312 In a CTest Script, the ctest_coverage() command runs this step. Argu‐
1313 ments to the command may specify some of the step settings.
1314 Configuration settings include:
1316 CoverageCommand
1318 Command-line tool to perform software coverage analysis. It
1319 will be executed in the location specified by the BuildDirectory
1320 setting.
1321 • CTest Script variable: CTEST_COVERAGE_COMMAND
1323 • CTest module variable: COVERAGE_COMMAND
1325 CoverageExtraFlags
1327 Specify command-line options to the CoverageCommand tool.
1328 • CTest Script variable: CTEST_COVERAGE_EXTRA_FLAGS
1330 • CTest module variable: COVERAGE_EXTRA_FLAGS
1332 These options are the first arguments passed to CoverageCommand.
1334 CTest MemCheck Step
1336 In a CTest Script, the ctest_memcheck() command runs this step. Argu‐
1337 ments to the command may specify some of the step settings.
1338 Configuration settings include:
1340 MemoryCheckCommand
1342 Command-line tool to perform dynamic analysis. Test command
1343 lines will be launched through this tool.
1344 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND
1346 • CTest module variable: MEMORYCHECK_COMMAND
1348 MemoryCheckCommandOptions
1350 Specify command-line options to the MemoryCheckCommand tool.
1351 They will be placed prior to the test command line.
1352 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND_OPTIONS
1354 • CTest module variable: MEMORYCHECK_COMMAND_OPTIONS
1356 MemoryCheckType
1358 Specify the type of memory checking to perform.
1359 • CTest Script variable: CTEST_MEMORYCHECK_TYPE
1361 • CTest module variable: MEMORYCHECK_TYPE
1363 MemoryCheckSanitizerOptions
1365 Specify options to sanitizers when running with a sanitize-en‐
1366 abled build.
1367 • CTest Script variable: CTEST_MEMORYCHECK_SANITIZER_OPTIONS
1369 • CTest module variable: MEMORYCHECK_SANITIZER_OPTIONS
1371 MemoryCheckSuppressionFile
1373 Specify a file containing suppression rules for the MemoryCheck‐
1374 Command tool. It will be passed with options appropriate to the
1375 tool.
1376 • CTest Script variable: CTEST_MEMORYCHECK_SUPPRESSIONS_FILE
1378 • CTest module variable: MEMORYCHECK_SUPPRESSIONS_FILE
1380 Additional configuration settings include:
1382 BoundsCheckerCommand
1384 Specify a MemoryCheckCommand that is known to be command-line
1385 compatible with Bounds Checker.
1386 • CTest Script variable: none
1388 • CTest module variable: none
1390 PurifyCommand
1392 Specify a MemoryCheckCommand that is known to be command-line
1393 compatible with Purify.
1394 • CTest Script variable: none
1396 • CTest module variable: PURIFYCOMMAND
1398 ValgrindCommand
1400 Specify a MemoryCheckCommand that is known to be command-line
1401 compatible with Valgrind.
1402 • CTest Script variable: none
1404 • CTest module variable: VALGRIND_COMMAND
1406 ValgrindCommandOptions
1408 Specify command-line options to the ValgrindCommand tool. They
1409 will be placed prior to the test command line.
1410 • CTest Script variable: none
1412 • CTest module variable: VALGRIND_COMMAND_OPTIONS
1414 DrMemoryCommand
1416 Specify a MemoryCheckCommand that is known to be a command-line
1417 compatible with DrMemory.
1418 • CTest Script variable: none
1420 • CTest module variable: DRMEMORY_COMMAND
1422 DrMemoryCommandOptions
1424 Specify command-line options to the DrMemoryCommand tool. They
1425 will be placed prior to the test command line.
1426 • CTest Script variable: none
1428 • CTest module variable: DRMEMORY_COMMAND_OPTIONS
1430 CudaSanitizerCommand
1432 Specify a MemoryCheckCommand that is known to be a command-line
1433 compatible with cuda-memcheck or compute-sanitizer.
1434 • CTest Script variable: none
1436 • CTest module variable: CUDA_SANITIZER_COMMAND
1438 CudaSanitizerCommandOptions
1440 Specify command-line options to the CudaSanitizerCommand tool.
1441 They will be placed prior to the test command line.
1442 • CTest Script variable: none
1444 • CTest module variable: CUDA_SANITIZER_COMMAND_OPTIONS
1446 CTest Submit Step
1448 In a CTest Script, the ctest_submit() command runs this step. Argu‐
1449 ments to the command may specify some of the step settings.
1450 Configuration settings include:
1452 BuildName
1454 Describe the dashboard client platform with a short string.
1455 (Operating system, compiler, etc.)
1456 • CTest Script variable: CTEST_BUILD_NAME
1458 • CTest module variable: BUILDNAME
1460 CDashVersion
1462 Legacy option. Not used.
1463 • CTest Script variable: none, detected from server
1465 • CTest module variable: CTEST_CDASH_VERSION
1467 CTestSubmitRetryCount
1469 Specify a number of attempts to retry submission on network
1470 failure.
1471 • CTest Script variable: none, use the ctest_submit()
1473 RETRY_COUNT option.
1474 • CTest module variable: CTEST_SUBMIT_RETRY_COUNT
1476 CTestSubmitRetryDelay
1478 Specify a delay before retrying submission on network failure.
1479 • CTest Script variable: none, use the ctest_submit() RETRY_DE‐
1481 LAY option.
1482 • CTest module variable: CTEST_SUBMIT_RETRY_DELAY
1484 CurlOptions
1486 Specify a semicolon-separated list of options to control the
1487 Curl library that CTest uses internally to connect to the
1488 server. Possible options are CURLOPT_SSL_VERIFYPEER_OFF and
1489 CURLOPT_SSL_VERIFYHOST_OFF.
1490 • CTest Script variable: CTEST_CURL_OPTIONS
1492 • CTest module variable: CTEST_CURL_OPTIONS
1494 DropLocation
1496 Legacy option. When SubmitURL is not set, it is constructed
1497 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1498 DropLocation.
1499 • CTest Script variable: CTEST_DROP_LOCATION
1501 • CTest module variable: DROP_LOCATION if set, else
1503 CTEST_DROP_LOCATION
1504 DropMethod
1506 Legacy option. When SubmitURL is not set, it is constructed
1507 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1508 DropLocation.
1509 • CTest Script variable: CTEST_DROP_METHOD
1511 • CTest module variable: DROP_METHOD if set, else
1513 CTEST_DROP_METHOD
1514 DropSite
1516 Legacy option. When SubmitURL is not set, it is constructed
1517 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1518 DropLocation.
1519 • CTest Script variable: CTEST_DROP_SITE
1521 • CTest module variable: DROP_SITE if set, else CTEST_DROP_SITE
1523 DropSitePassword
1525 Legacy option. When SubmitURL is not set, it is constructed
1526 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1527 DropLocation.
1528 • CTest Script variable: CTEST_DROP_SITE_PASSWORD
1530 • CTest module variable: DROP_SITE_PASSWORD if set, else
1532 CTEST_DROP_SITE_PASWORD
1533 DropSiteUser
1535 Legacy option. When SubmitURL is not set, it is constructed
1536 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1537 DropLocation.
1538 • CTest Script variable: CTEST_DROP_SITE_USER
1540 • CTest module variable: DROP_SITE_USER if set, else
1542 CTEST_DROP_SITE_USER
1543 IsCDash
1545 Legacy option. Not used.
1546 • CTest Script variable: CTEST_DROP_SITE_CDASH
1548 • CTest module variable: CTEST_DROP_SITE_CDASH
1550 ScpCommand
1552 Legacy option. Not used.
1553 • CTest Script variable: CTEST_SCP_COMMAND
1555 • CTest module variable: SCPCOMMAND
1557 Site Describe the dashboard client host site with a short string.
1559 (Hostname, domain, etc.)
1560 • CTest Script variable: CTEST_SITE
1562 • CTest module variable: SITE, initialized by the site_name()
1564 command
1565 SubmitURL
1567 The http or https URL of the dashboard server to send the sub‐
1568 mission to.
1569 • CTest Script variable: CTEST_SUBMIT_URL
1571 • CTest module variable: SUBMIT_URL if set, else CTEST_SUB‐
1573 MIT_URL
1574 SubmitInactivityTimeout
1576 The time to wait for the submission after which it is canceled
1577 if not completed. Specify a zero value to disable timeout.
1578 • CTest Script variable: CTEST_SUBMIT_INACTIVITY_TIMEOUT
1580 • CTest module variable: CTEST_SUBMIT_INACTIVITY_TIMEOUT
1582 TriggerSite
1584 Legacy option. Not used.
1585 • CTest Script variable: CTEST_TRIGGER_SITE
1587 • CTest module variable: TRIGGER_SITE if set, else CTEST_TRIG‐
1589 GER_SITE
1590
1592 New in version 3.14.
1593 When the --show-only=json-v1 command line option is given, the test in‐
1596 formation is output in JSON format. Version 1.0 of the JSON object
1597 model is defined as follows:
1598 kind The string "ctestInfo".
1600 version
1602 A JSON object specifying the version components. Its members
1603 are
1604 major A non-negative integer specifying the major version com‐
1606 ponent.
1607 minor A non-negative integer specifying the minor version com‐
1609 ponent.
1610 backtraceGraph
1612 JSON object representing backtrace information with the follow‐
1613 ing members:
1614 commands
1616 List of command names.
1617 files List of file names.
1619 nodes List of node JSON objects with members:
1621 command
1623 Index into the commands member of the backtrace‐
1624 Graph.
1625 file Index into the files member of the backtraceGraph.
1627 line Line number in the file where the backtrace was
1629 added.
1630 parent Index into the nodes member of the backtraceGraph
1632 representing the parent in the graph.
1633 tests A JSON array listing information about each test. Each entry is
1635 a JSON object with members:
1636 name Test name.
1638 config Configuration that the test can run on. Empty string
1640 means any config.
1641 command
1643 List where the first element is the test command and the
1644 remaining elements are the command arguments.
1645 backtrace
1647 Index into the nodes member of the backtraceGraph.
1648 properties
1650 Test properties. Can contain keys for each of the sup‐
1651 ported test properties.
1652
1654 CTest provides a mechanism for tests to specify the resources that they
1655 need in a fine-grained way, and for users to specify the resources
1656 available on the running machine. This allows CTest to internally keep
1657 track of which resources are in use and which are free, scheduling
1658 tests in a way that prevents them from trying to claim resources that
1659 are not available.
1660 When the resource allocation feature is used, CTest will not oversub‐
1662 scribe resources. For example, if a resource has 8 slots, CTest will
1663 not run tests that collectively use more than 8 slots at a time. This
1664 has the effect of limiting how many tests can run at any given time,
1665 even if a high -j argument is used, if those tests all use some slots
1666 from the same resource. In addition, it means that a single test that
1667 uses more of a resource than is available on a machine will not run at
1668 all (and will be reported as Not Run).
1669 A common use case for this feature is for tests that require the use of
1671 a GPU. Multiple tests can simultaneously allocate memory from a GPU,
1672 but if too many tests try to do this at once, some of them will fail to
1673 allocate, resulting in a failed test, even though the test would have
1674 succeeded if it had the memory it needed. By using the resource alloca‐
1675 tion feature, each test can specify how much memory it requires from a
1676 GPU, allowing CTest to schedule tests in a way that running several of
1677 these tests at once does not exhaust the GPU's memory pool.
1678 Please note that CTest has no concept of what a GPU is or how much mem‐
1680 ory it has, nor does it have any way of communicating with a GPU to re‐
1681 trieve this information or perform any memory management. CTest simply
1682 keeps track of a list of abstract resource types, each of which has a
1683 certain number of slots available for tests to use. Each test specifies
1684 the number of slots that it requires from a certain resource, and CTest
1685 then schedules them in a way that prevents the total number of slots in
1686 use from exceeding the listed capacity. When a test is executed, and
1687 slots from a resource are allocated to that test, tests may assume that
1688 they have exclusive use of those slots for the duration of the test's
1689 process.
1690 The CTest resource allocation feature consists of two inputs:
1692 • The resource specification file, described below, which describes the
1694 resources available on the system.
1695 • The RESOURCE_GROUPS property of tests, which describes the resources
1697 required by the test.
1698 When CTest runs a test, the resources allocated to that test are passed
1700 in the form of a set of environment variables as described below. Using
1701 this information to decide which resource to connect to is left to the
1702 test writer.
1703 The RESOURCE_GROUPS property tells CTest what resources a test expects
1705 to use grouped in a way meaningful to the test. The test itself must
1706 read the environment variables to determine which resources have been
1707 allocated to each group. For example, each group may correspond to a
1708 process the test will spawn when executed.
1709 Note that even if a test specifies a RESOURCE_GROUPS property, it is
1711 still possible for that to test to run without any resource allocation
1712 (and without the corresponding environment variables) if the user does
1713 not pass a resource specification file. Passing this file, either
1714 through the --resource-spec-file command-line argument or the RE‐
1715 SOURCE_SPEC_FILE argument to ctest_test(), is what activates the re‐
1716 source allocation feature. Tests should check the CTEST_RE‐
1717 SOURCE_GROUP_COUNT environment variable to find out whether or not re‐
1718 source allocation is activated. This variable will always (and only) be
1719 defined if resource allocation is activated. If resource allocation is
1720 not activated, then the CTEST_RESOURCE_GROUP_COUNT variable will not
1721 exist, even if it exists for the parent ctest process. If a test abso‐
1722 lutely must have resource allocation, then it can return a failing exit
1723 code or use the SKIP_RETURN_CODE or SKIP_REGULAR_EXPRESSION properties
1724 to indicate a skipped test.
1725 Resource Specification File
1727 The resource specification file is a JSON file which is passed to
1728 CTest, either on the command line as ctest --resource-spec-file, or as
1729 the RESOURCE_SPEC_FILE argument of ctest_test(). If a dashboard script
1730 is used and RESOURCE_SPEC_FILE is not specified, the value of
1731 CTEST_RESOURCE_SPEC_FILE in the dashboard script is used instead. If
1732 --resource-spec-file, RESOURCE_SPEC_FILE, and CTEST_RESOURCE_SPEC_FILE
1733 in the dashboard script are not specified, the value of
1734 CTEST_RESOURCE_SPEC_FILE in the CMake build is used instead. If none of
1735 these are specified, no resource spec file is used.
1736 The resource specification file must be a JSON object. All examples in
1738 this document assume the following resource specification file:
1739 {
1741 "version": {
1742 "major": 1,
1743 "minor": 0
1744 },
1745 "local": [
1746 {
1747 "gpus": [
1748 {
1749 "id": "0",
1750 "slots": 2
1751 },
1752 {
1753 "id": "1",
1754 "slots": 4
1755 },
1756 {
1757 "id": "2",
1758 "slots": 2
1759 },
1760 {
1761 "id": "3"
1762 }
1763 ],
1764 "crypto_chips": [
1765 {
1766 "id": "card0",
1767 "slots": 4
1768 }
1769 ]
1770 }
1771 ]
1772 }
1773 The members are:
1775 version
1777 An object containing a major integer field and a minor integer
1778 field. Currently, the only supported version is major 1, minor
1779 0. Any other value is an error.
1780 local A JSON array of resource sets present on the system. Currently,
1782 this array is restricted to being of size 1.
1783 Each array element is a JSON object with members whose names are
1785 equal to the desired resource types, such as gpus. These names
1786 must start with a lowercase letter or an underscore, and subse‐
1787 quent characters can be a lowercase letter, a digit, or an un‐
1788 derscore. Uppercase letters are not allowed, because certain
1789 platforms have case-insensitive environment variables. See the
1790 Environment Variables section below for more information. It is
1791 recommended that the resource type name be the plural of a noun,
1792 such as gpus or crypto_chips (and not gpu or crypto_chip.)
1793 Please note that the names gpus and crypto_chips are just exam‐
1795 ples, and CTest does not interpret them in any way. You are free
1796 to make up any resource type you want to meet your own require‐
1797 ments.
1798 The value for each resource type is a JSON array consisting of
1800 JSON objects, each of which describe a specific instance of the
1801 specified resource. These objects have the following members:
1802 id A string consisting of an identifier for the resource.
1804 Each character in the identifier can be a lowercase let‐
1805 ter, a digit, or an underscore. Uppercase letters are
1806 not allowed.
1807 Identifiers must be unique within a resource type. How‐
1809 ever, they do not have to be unique across resource
1810 types. For example, it is valid to have a gpus resource
1811 named 0 and a crypto_chips resource named 0, but not two
1812 gpus resources both named 0.
1813 Please note that the IDs 0, 1, 2, 3, and card0 are just
1815 examples, and CTest does not interpret them in any way.
1816 You are free to make up any IDs you want to meet your own
1817 requirements.
1818 slots An optional unsigned number specifying the number of
1820 slots available on the resource. For example, this could
1821 be megabytes of RAM on a GPU, or cryptography units
1822 available on a cryptography chip. If slots is not speci‐
1823 fied, a default value of 1 is assumed.
1824 In the example file above, there are four GPUs with ID's 0 through 3.
1826 GPU 0 has 2 slots, GPU 1 has 4, GPU 2 has 2, and GPU 3 has a default of
1827 1 slot. There is also one cryptography chip with 4 slots.
1828 RESOURCE_GROUPS Property
1830 See RESOURCE_GROUPS for a description of this property.
1831 Environment Variables
1833 Once CTest has decided which resources to allocate to a test, it passes
1834 this information to the test executable as a series of environment
1835 variables. For each example below, we will assume that the test in
1836 question has a RESOURCE_GROUPS property of
1837 2,gpus:2;gpus:4,gpus:1,crypto_chips:2.
1838 The following variables are passed to the test process:
1840 CTEST_RESOURCE_GROUP_COUNT
1842 The total number of groups specified by the RESOURCE_GROUPS
1843 property. For example:
1844 • CTEST_RESOURCE_GROUP_COUNT=3
1846 This variable will only be defined if ctest(1) has been given a
1848 --resource-spec-file, or if ctest_test() has been given a RE‐
1849 SOURCE_SPEC_FILE. If no resource specification file has been
1850 given, this variable will not be defined.
1851 CTEST_RESOURCE_GROUP_<num>
1853 The list of resource types allocated to each group, with each
1854 item separated by a comma. <num> is a number from zero to
1855 CTEST_RESOURCE_GROUP_COUNT minus one. CTEST_RESOURCE_GROUP_<num>
1856 is defined for each <num> in this range. For example:
1857 • CTEST_RESOURCE_GROUP_0=gpus
1859 • CTEST_RESOURCE_GROUP_1=gpus
1861 • CTEST_RESOURCE_GROUP_2=crypto_chips,gpus
1863 CTEST_RESOURCE_GROUP_<num>_<resource-type>
1865 The list of resource IDs and number of slots from each ID allo‐
1866 cated to each group for a given resource type. This variable
1867 consists of a series of pairs, each pair separated by a semi‐
1868 colon, and with the two items in the pair separated by a comma.
1869 The first item in each pair is id: followed by the ID of a re‐
1870 source of type <resource-type>, and the second item is slots:
1871 followed by the number of slots from that resource allocated to
1872 the given group. For example:
1873 • CTEST_RESOURCE_GROUP_0_GPUS=id:0,slots:2
1875 • CTEST_RESOURCE_GROUP_1_GPUS=id:2,slots:2
1877 • CTEST_RESOURCE_GROUP_2_GPUS=id:1,slots:4;id:3,slots:1
1879 • CTEST_RESOURCE_GROUP_2_CRYPTO_CHIPS=id:card0,slots:2
1881 In this example, group 0 gets 2 slots from GPU 0, group 1 gets 2
1883 slots from GPU 2, and group 2 gets 4 slots from GPU 1, 1 slot
1884 from GPU 3, and 2 slots from cryptography chip card0.
1885 <num> is a number from zero to CTEST_RESOURCE_GROUP_COUNT minus
1887 one. <resource-type> is the name of a resource type, converted
1888 to uppercase. CTEST_RESOURCE_GROUP_<num>_<resource-type> is de‐
1889 fined for the product of each <num> in the range listed above
1890 and each resource type listed in CTEST_RESOURCE_GROUP_<num>.
1891 Because some platforms have case-insensitive names for environ‐
1893 ment variables, the names of resource types may not clash in a
1894 case-insensitive environment. Because of this, for the sake of
1895 simplicity, all resource types must be listed in all lowercase
1896 in the resource specification file and in the RESOURCE_GROUPS
1897 property, and they are converted to all uppercase in the
1898 CTEST_RESOURCE_GROUP_<num>_<resource-type> environment variable.
1899
1901 The following resources are available to get help using CMake:
1902 Home Page
1904 https://cmake.org
1905 The primary starting point for learning about CMake.
1907 Online Documentation and Community Resources
1909 https://cmake.org/documentation
1910 Links to available documentation and community resources may be
1912 found on this web page.
1913 Discourse Forum
1915 https://discourse.cmake.org
1916 The Discourse Forum hosts discussion and questions about CMake.
1918 : https://cdash.org
1920
1922 2000-2023 Kitware, Inc. and Contributors
19233.25.2 Jan 19, 2023 CTEST(1)