1CTEST(1) CMake CTEST(1)
2
6 ctest - CTest Command-Line Reference
7 Contents
9 • ctest(1)
10 • Synopsis
12 • Description
14 • Run Tests
16 • View Help
18 • Label Matching
20 • Label and Subproject Summary
22 • Build and Test Mode
24 • Dashboard Client
26 • Dashboard Client Steps
28 • Dashboard Client Modes
30 • Dashboard Client via CTest Command-Line
32 • Dashboard Client via CTest Script
34 • Dashboard Client Configuration
36 • CTest Start Step
38 • CTest Update Step
40 • CTest Configure Step
42 • CTest Build Step
44 • CTest Test Step
46 • CTest Coverage Step
48 • CTest MemCheck Step
50 • CTest Submit Step
52 • Show as JSON Object Model
54 • Resource Allocation
56 • Resource Specification File
58 • RESOURCE_GROUPS Property
60 • Environment Variables
62 • See Also
64
66 Run Tests
67 ctest [<options>] [--test-dir <path-to-build>]
68 Build and Test Mode
70 ctest --build-and-test <path-to-source> <path-to-build>
71 --build-generator <generator> [<options>...]
72 [--build-options <opts>...]
73 [--test-command <command> [<args>...]]
74 Dashboard Client
76 ctest -D <dashboard> [-- <dashboard-options>...]
77 ctest -M <model> -T <action> [-- <dashboard-options>...]
78 ctest -S <script> [-- <dashboard-options>...]
79 ctest -SP <script> [-- <dashboard-options>...]
80 View Help
82 ctest --help[-<topic>]
83
85 The ctest executable is the CMake test driver program. CMake-generated
86 build trees created for projects that use the enable_testing() and
87 add_test() commands have testing support. This program will run the
88 tests and report results.
89
91 --preset <preset>, --preset=<preset>
92 Use a test preset to specify test options. The project binary
93 directory is inferred from the configurePreset key. The current
94 working directory must contain CMake preset files. See preset
95 for more details.
96 --list-presets
98 Lists the available test presets. The current working directory
99 must contain CMake preset files.
100 -C <cfg>, --build-config <cfg>
102 Choose configuration to test.
103 Some CMake-generated build trees can have multiple build config‐
105 urations in the same tree. This option can be used to specify
106 which one should be tested. Example configurations are Debug
107 and Release.
108 --progress
110 Enable short progress output from tests.
111 When the output of ctest is being sent directly to a terminal,
113 the progress through the set of tests is reported by updating
114 the same line rather than printing start and end messages for
115 each test on new lines. This can significantly reduce the ver‐
116 bosity of the test output. Test completion messages are still
117 output on their own line for failed tests and the final test
118 summary will also still be logged.
119 This option can also be enabled by setting the environment vari‐
121 able CTEST_PROGRESS_OUTPUT.
122 -V, --verbose
124 Enable verbose output from tests.
125 Test output is normally suppressed and only summary information
127 is displayed. This option will show all test output.
128 -VV, --extra-verbose
130 Enable more verbose output from tests.
131 Test output is normally suppressed and only summary information
133 is displayed. This option will show even more test output.
134 --debug
136 Displaying more verbose internals of CTest.
137 This feature will result in a large number of output that is
139 mostly useful for debugging dashboard problems.
140 --output-on-failure
142 Output anything outputted by the test program if the test should
143 fail. This option can also be enabled by setting the
144 CTEST_OUTPUT_ON_FAILURE environment variable
145 --stop-on-failure
147 Stop running the tests when the first failure happens.
148 -F Enable failover.
150 This option allows CTest to resume a test set execution that was
152 previously interrupted. If no interruption occurred, the -F op‐
153 tion will have no effect.
154 -j <jobs>, --parallel <jobs>
156 Run the tests in parallel using the given number of jobs.
157 This option tells CTest to run the tests in parallel using given
159 number of jobs. This option can also be set by setting the
160 CTEST_PARALLEL_LEVEL environment variable.
161 This option can be used with the PROCESSORS test property.
163 See Label and Subproject Summary.
165 --resource-spec-file <file>
167 Run CTest with resource allocation enabled, using the resource
168 specification file specified in <file>.
169 When ctest is run as a Dashboard Client this sets the Resource‐
171 SpecFile option of the CTest Test Step.
172 --test-load <level>
174 While running tests in parallel (e.g. with -j), try not to start
175 tests when they may cause the CPU load to pass above a given
176 threshold.
177 When ctest is run as a Dashboard Client this sets the TestLoad
179 option of the CTest Test Step.
180 -Q, --quiet
182 Make CTest quiet.
183 This option will suppress all the output. The output log file
185 will still be generated if the --output-log is specified. Op‐
186 tions such as --verbose, --extra-verbose, and --debug are ig‐
187 nored if --quiet is specified.
188 -O <file>, --output-log <file>
190 Output to log file.
191 This option tells CTest to write all its output to a <file> log
193 file.
194 --output-junit <file>
196 New in version 3.21.
197 Write test results in JUnit format.
200 This option tells CTest to write test results to <file> in JUnit
202 XML format. If <file> already exists, it will be overwritten. If
203 using the -S option to run a dashboard script, use the OUT‐
204 PUT_JUNIT keyword with the ctest_test() command instead.
205 -N, --show-only[=<format>]
207 Disable actual execution of tests.
208 This option tells CTest to list the tests that would be run but
210 not actually run them. Useful in conjunction with the -R and -E
211 options.
212 New in version 3.14: The --show-only option accepts a <format>
214 value.
215 <format> can be one of the following values.
218 human Human-friendly output. This is not guaranteed to be
220 stable. This is the default.
221 json-v1
223 Dump the test information in JSON format. See Show as
224 JSON Object Model.
225 -L <regex>, --label-regex <regex>
227 Run tests with labels matching regular expression as described
228 under string(REGEX).
229 This option tells CTest to run only the tests whose labels match
231 the given regular expression. When more than one -L option is
232 given, a test will only be run if each regular expression
233 matches at least one of the test's labels (i.e. the multiple -L
234 labels form an AND relationship). See Label Matching.
235 -R <regex>, --tests-regex <regex>
237 Run tests matching regular expression.
238 This option tells CTest to run only the tests whose names match
240 the given regular expression.
241 -E <regex>, --exclude-regex <regex>
243 Exclude tests matching regular expression.
244 This option tells CTest to NOT run the tests whose names match
246 the given regular expression.
247 -LE <regex>, --label-exclude <regex>
249 Exclude tests with labels matching regular expression.
250 This option tells CTest to NOT run the tests whose labels match
252 the given regular expression. When more than one -LE option is
253 given, a test will only be excluded if each regular expression
254 matches at least one of the test's labels (i.e. the multiple -LE
255 labels form an AND relationship). See Label Matching.
256 -FA <regex>, --fixture-exclude-any <regex>
258 Exclude fixtures matching <regex> from automatically adding any
259 tests to the test set.
260 If a test in the set of tests to be executed requires a particu‐
262 lar fixture, that fixture's setup and cleanup tests would nor‐
263 mally be added to the test set automatically. This option pre‐
264 vents adding setup or cleanup tests for fixtures matching the
265 <regex>. Note that all other fixture behavior is retained, in‐
266 cluding test dependencies and skipping tests that have fixture
267 setup tests that fail.
268 -FS <regex>, --fixture-exclude-setup <regex>
270 Same as -FA except only matching setup tests are excluded.
271 -FC <regex>, --fixture-exclude-cleanup <regex>
273 Same as -FA except only matching cleanup tests are excluded.
274 -I [Start,End,Stride,test#,test#|Test file], --tests-information
276 Run a specific number of tests by number.
277 This option causes CTest to run tests starting at number Start,
279 ending at number End, and incrementing by Stride. Any addi‐
280 tional numbers after Stride are considered individual test num‐
281 bers. Start, End, or Stride can be empty. Optionally a file
282 can be given that contains the same syntax as the command line.
283 -U, --union
285 Take the Union of -I and -R.
286 When both -R and -I are specified by default the intersection of
288 tests are run. By specifying -U the union of tests is run in‐
289 stead.
290 --rerun-failed
292 Run only the tests that failed previously.
293 This option tells CTest to perform only the tests that failed
295 during its previous run. When this option is specified, CTest
296 ignores all other options intended to modify the list of tests
297 to run ( -L, -R, -E, -LE, -I, etc). In the event that CTest
298 runs and no tests fail, subsequent calls to CTest with the --re‐
299 run-failed option will run the set of tests that most recently
300 failed (if any).
301 --repeat <mode>:<n>
303 Run tests repeatedly based on the given <mode> up to <n> times.
304 The modes are:
305 until-fail
307 Require each test to run <n> times without failing in or‐
308 der to pass. This is useful in finding sporadic failures
309 in test cases.
310 until-pass
312 Allow each test to run up to <n> times in order to pass.
313 Repeats tests if they fail for any reason. This is use‐
314 ful in tolerating sporadic failures in test cases.
315 after-timeout
317 Allow each test to run up to <n> times in order to pass.
318 Repeats tests only if they timeout. This is useful in
319 tolerating sporadic timeouts in test cases on busy ma‐
320 chines.
321 --repeat-until-fail <n>
323 Equivalent to --repeat until-fail:<n>.
324 --max-width <width>
326 Set the max width for a test name to output.
327 Set the maximum width for each test name to show in the output.
329 This allows the user to widen the output to avoid clipping the
330 test name which can be very annoying.
331 --interactive-debug-mode [0|1]
333 Set the interactive mode to 0 or 1.
334 This option causes CTest to run tests in either an interactive
336 mode or a non-interactive mode. In dashboard mode (Experimen‐
337 tal, Nightly, Continuous), the default is non-interactive. In
338 non-interactive mode, the environment variable
339 DASHBOARD_TEST_FROM_CTEST is set.
340 Prior to CMake 3.11, interactive mode on Windows allowed system
342 debug popup windows to appear. Now, due to CTest's use of libuv
343 to launch test processes, all system debug popup windows are al‐
344 ways blocked.
345 --no-label-summary
347 Disable timing summary information for labels.
348 This option tells CTest not to print summary information for
350 each label associated with the tests run. If there are no la‐
351 bels on the tests, nothing extra is printed.
352 See Label and Subproject Summary.
354 --no-subproject-summary
356 Disable timing summary information for subprojects.
357 This option tells CTest not to print summary information for
359 each subproject associated with the tests run. If there are no
360 subprojects on the tests, nothing extra is printed.
361 See Label and Subproject Summary.
363 --test-dir <dir>
365 Specify the directory in which to look for tests, typically a
366 CMake project build directory. If not specified, the current di‐
367 rectory is used.
368 --test-output-size-passed <size>
370 New in version 3.4.
371 Limit the output for passed tests to <size> bytes.
374 --test-output-size-failed <size>
376 New in version 3.4.
377 Limit the output for failed tests to <size> bytes.
380 --test-output-truncation <mode>
382 New in version 3.24.
383 Truncate tail (default), middle or head of test output once max‐
386 imum output size is reached.
387 --overwrite
389 Overwrite CTest configuration option.
390 By default CTest uses configuration options from configuration
392 file. This option will overwrite the configuration option.
393 --force-new-ctest-process
395 Run child CTest instances as new processes.
396 By default CTest will run child CTest instances within the same
398 process. If this behavior is not desired, this argument will
399 enforce new processes for child CTest processes.
400 --schedule-random
402 Use a random order for scheduling tests.
403 This option will run the tests in a random order. It is com‐
405 monly used to detect implicit dependencies in a test suite.
406 --submit-index
408 Legacy option for old Dart2 dashboard server feature. Do not
409 use.
410 --timeout <seconds>
412 Set the default test timeout.
413 This option effectively sets a timeout on all tests that do not
415 already have a timeout set on them via the TIMEOUT property.
416 --stop-time <time>
418 Set a time at which all tests should stop running.
419 Set a real time of day at which all tests should timeout. Exam‐
421 ple: 7:00:00 -0400. Any time format understood by the curl date
422 parser is accepted. Local time is assumed if no timezone is
423 specified.
424 --print-labels
426 Print all available test labels.
427 This option will not run any tests, it will simply print the
429 list of all labels associated with the test set.
430 --no-tests=<action>
432 Regard no tests found either as error (when <action> is set to
433 error) or ignore it (when <action> is set to ignore).
434 If no tests were found, the default behavior of CTest is to al‐
436 ways log an error message but to return an error code in script
437 mode only. This option unifies the behavior of CTest by either
438 returning an error code if no tests were found or by ignoring
439 it.
440 New in version 3.26.
442 This option can also be set by setting the CTEST_NO_TESTS_ACTION
445 environment variable.
446
448 To print version details or selected pages from the CMake documenta‐
449 tion, use one of the following options:
450 -version [<file>], --version [<file>], /V [<file>]
452 Show program name/version banner and exit. The output is
453 printed to a named <file> if given.
454 -h, -H, --help, -help, -usage, /?
456 Print usage information and exit.
457 Usage describes the basic command line interface and its op‐
459 tions.
460 --help-full [<file>]
462 Print all help manuals and exit.
463 All manuals are printed in a human-readable text format. The
465 output is printed to a named <file> if given.
466 --help-manual <man> [<file>]
468 Print one help manual and exit.
469 The specified manual is printed in a human-readable text format.
471 The output is printed to a named <file> if given.
472 --help-manual-list [<file>]
474 List help manuals available and exit.
475 The list contains all manuals for which help may be obtained by
477 using the --help-manual option followed by a manual name. The
478 output is printed to a named <file> if given.
479 --help-command <cmd> [<file>]
481 Print help for one command and exit.
482 The cmake-commands(7) manual entry for <cmd> is printed in a hu‐
484 man-readable text format. The output is printed to a named
485 <file> if given.
486 --help-command-list [<file>]
488 List commands with help available and exit.
489 The list contains all commands for which help may be obtained by
491 using the --help-command option followed by a command name. The
492 output is printed to a named <file> if given.
493 --help-commands [<file>]
495 Print cmake-commands manual and exit.
496 The cmake-commands(7) manual is printed in a human-readable text
498 format. The output is printed to a named <file> if given.
499 --help-module <mod> [<file>]
501 Print help for one module and exit.
502 The cmake-modules(7) manual entry for <mod> is printed in a hu‐
504 man-readable text format. The output is printed to a named
505 <file> if given.
506 --help-module-list [<file>]
508 List modules with help available and exit.
509 The list contains all modules for which help may be obtained by
511 using the --help-module option followed by a module name. The
512 output is printed to a named <file> if given.
513 --help-modules [<file>]
515 Print cmake-modules manual and exit.
516 The cmake-modules(7) manual is printed in a human-readable text
518 format. The output is printed to a named <file> if given.
519 --help-policy <cmp> [<file>]
521 Print help for one policy and exit.
522 The cmake-policies(7) manual entry for <cmp> is printed in a hu‐
524 man-readable text format. The output is printed to a named
525 <file> if given.
526 --help-policy-list [<file>]
528 List policies with help available and exit.
529 The list contains all policies for which help may be obtained by
531 using the --help-policy option followed by a policy name. The
532 output is printed to a named <file> if given.
533 --help-policies [<file>]
535 Print cmake-policies manual and exit.
536 The cmake-policies(7) manual is printed in a human-readable text
538 format. The output is printed to a named <file> if given.
539 --help-property <prop> [<file>]
541 Print help for one property and exit.
542 The cmake-properties(7) manual entries for <prop> are printed in
544 a human-readable text format. The output is printed to a named
545 <file> if given.
546 --help-property-list [<file>]
548 List properties with help available and exit.
549 The list contains all properties for which help may be obtained
551 by using the --help-property option followed by a property name.
552 The output is printed to a named <file> if given.
553 --help-properties [<file>]
555 Print cmake-properties manual and exit.
556 The cmake-properties(7) manual is printed in a human-readable
558 text format. The output is printed to a named <file> if given.
559 --help-variable <var> [<file>]
561 Print help for one variable and exit.
562 The cmake-variables(7) manual entry for <var> is printed in a
564 human-readable text format. The output is printed to a named
565 <file> if given.
566 --help-variable-list [<file>]
568 List variables with help available and exit.
569 The list contains all variables for which help may be obtained
571 by using the --help-variable option followed by a variable name.
572 The output is printed to a named <file> if given.
573 --help-variables [<file>]
575 Print cmake-variables manual and exit.
576 The cmake-variables(7) manual is printed in a human-readable
578 text format. The output is printed to a named <file> if given.
579
581 Tests may have labels attached to them. Tests may be included or ex‐
582 cluded from a test run by filtering on the labels. Each individual
583 filter is a regular expression applied to the labels attached to a
584 test.
585 When -L is used, in order for a test to be included in a test run, each
587 regular expression must match at least one label. Using more than one
588 -L option means "match all of these".
589 The -LE option works just like -L, but excludes tests rather than in‐
591 cluding them. A test is excluded if each regular expression matches at
592 least one label.
593 If a test has no labels attached to it, then -L will never include that
595 test, and -LE will never exclude that test. As an example of tests
596 with labels, consider five tests, with the following labels:
597 • test1 has labels tuesday and production
599 • test2 has labels tuesday and test
601 • test3 has labels wednesday and production
603 • test4 has label wednesday
605 • test5 has labels friday and test
607 Running ctest with -L tuesday -L test will select test2, which has both
609 labels. Running CTest with -L test will select test2 and test5, because
610 both of them have a label that matches that regular expression.
611 Because the matching works with regular expressions, take note that
613 running CTest with -L es will match all five tests. To select the
614 tuesday and wednesday tests together, use a single regular expression
615 that matches either of them, like -L "tue|wed".
616
618 CTest prints timing summary information for each LABEL and subproject
619 associated with the tests run. The label time summary will not include
620 labels that are mapped to subprojects.
621 New in version 3.22: Labels added dynamically during test execution are
623 also reported in the timing summary. See Additional Labels.
624 When the PROCESSORS test property is set, CTest will display a weighted
627 test timing result in label and subproject summaries. The time is re‐
628 ported with sec*proc instead of just sec.
629 The weighted time summary reported for each label or subproject j is
631 computed as:
632 Weighted Time Summary for Label/Subproject j =
634 sum(raw_test_time[j,i] * num_processors[j,i], i=1...num_tests[j])
635 for labels/subprojects j=1...total
637 where:
639 • raw_test_time[j,i]: Wall-clock time for the i test for the j label or
641 subproject
642 • num_processors[j,i]: Value of the CTest PROCESSORS property for the i
644 test for the j label or subproject
645 • num_tests[j]: Number of tests associated with the j label or subpro‐
647 ject
648 • total: Total number of labels or subprojects that have at least one
650 test run
651 Therefore, the weighted time summary for each label or subproject rep‐
653 resents the amount of time that CTest gave to run the tests for each
654 label or subproject and gives a good representation of the total ex‐
655 pense of the tests for each label or subproject when compared to other
656 labels or subprojects.
657 For example, if SubprojectA showed 100 sec*proc and SubprojectB showed
659 10 sec*proc, then CTest allocated approximately 10 times the CPU/core
660 time to run the tests for SubprojectA than for SubprojectB (e.g. so if
661 effort is going to be expended to reduce the cost of the test suite for
662 the whole project, then reducing the cost of the test suite for Subpro‐
663 jectA would likely have a larger impact than effort to reduce the cost
664 of the test suite for SubprojectB).
665
667 CTest provides a command-line signature to configure (i.e. run cmake
668 on), build, and/or execute a test:
669 ctest --build-and-test <path-to-source> <path-to-build>
671 --build-generator <generator>
672 [<options>...]
673 [--build-options <opts>...]
674 [--test-command <command> [<args>...]]
675 The configure and test steps are optional. The arguments to this com‐
677 mand line are the source and binary directories. The --build-generator
678 option must be provided to use --build-and-test. If --test-command is
679 specified then that will be run after the build is complete. Other op‐
680 tions that affect this mode include:
681 --build-and-test
683 Switch into the build and test mode.
684 --build-target
686 Specify a specific target to build. The option can be given
687 multiple times with different targets, in which case each target
688 is built in turn. A clean will be done before building each
689 target unless the --build-noclean option is given.
690 If no --build-target is specified, the all target is built.
692 --build-nocmake
694 Run the build without running cmake first.
695 Skip the cmake step.
697 --build-run-dir
699 Specify directory to run programs from.
700 Directory where programs will be after it has been compiled.
702 --build-two-config
704 Run CMake twice.
705 --build-exe-dir
707 Specify the directory for the executable.
708 --build-generator
710 Specify the generator to use. See the cmake-generators(7) man‐
711 ual.
712 --build-generator-platform
714 Specify the generator-specific platform.
715 --build-generator-toolset
717 Specify the generator-specific toolset.
718 --build-project
720 Specify the name of the project to build.
721 --build-makeprogram
723 Specify the explicit make program to be used by CMake when con‐
724 figuring and building the project. Only applicable for Make and
725 Ninja based generators.
726 --build-noclean
728 Skip the make clean step.
729 --build-config-sample
731 A sample executable to use to determine the configuration that
732 should be used. e.g. Debug, Release etc.
733 --build-options
735 Additional options for configuring the build (i.e. for CMake,
736 not for the build tool). Note that if this is specified, the
737 --build-options keyword and its arguments must be the last op‐
738 tion given on the command line, with the possible exception of
739 --test-command.
740 --test-command
742 The command to run as the test step with the --build-and-test
743 option. All arguments following this keyword will be assumed to
744 be part of the test command line, so it must be the last option
745 given.
746 --test-timeout
748 The time limit in seconds
749
751 CTest can operate as a client for the CDash software quality dashboard
752 application. As a dashboard client, CTest performs a sequence of steps
753 to configure, build, and test software, and then submits the results to
754 a CDash server. The command-line signature used to submit to CDash is:
755 ctest -D <dashboard> [-- <dashboard-options>...]
757 ctest -M <model> -T <action> [-- <dashboard-options>...]
758 ctest -S <script> [-- <dashboard-options>...]
759 ctest -SP <script> [-- <dashboard-options>...]
760 Options for Dashboard Client include:
762 -D <dashboard>, --dashboard <dashboard>
764 Execute dashboard test.
765 This option tells CTest to act as a CDash client and perform a
767 dashboard test. All tests are <Mode><Test>, where <Mode> can be
768 Experimental, Nightly, and Continuous, and <Test> can be Start,
769 Update, Configure, Build, Test, Coverage, and Submit.
770 If <dashboard> is not one of the recognized <Mode><Test> values,
772 this will be treated as a variable definition instead (see the
773 dashboard-options further below).
774 -M <model>, --test-model <model>
776 Sets the model for a dashboard.
777 This option tells CTest to act as a CDash client where the
779 <model> can be Experimental, Nightly, and Continuous. Combining
780 -M and -T is similar to -D.
781 -T <action>, --test-action <action>
783 Sets the dashboard action to perform.
784 This option tells CTest to act as a CDash client and perform
786 some action such as start, build, test etc. See Dashboard Client
787 Steps for the full list of actions. Combining -M and -T is sim‐
788 ilar to -D.
789 -S <script>, --script <script>
791 Execute a dashboard for a configuration.
792 This option tells CTest to load in a configuration script which
794 sets a number of parameters such as the binary and source direc‐
795 tories. Then CTest will do what is required to create and run a
796 dashboard. This option basically sets up a dashboard and then
797 runs ctest -D with the appropriate options.
798 -SP <script>, --script-new-process <script>
800 Execute a dashboard for a configuration.
801 This option does the same operations as -S but it will do them
803 in a separate process. This is primarily useful in cases where
804 the script may modify the environment and you do not want the
805 modified environment to impact other -S scripts.
806 The available <dashboard-options> are the following:
808 -D <var>:<type>=<value>
810 Define a variable for script mode.
811 Pass in variable values on the command line. Use in conjunction
813 with -S to pass variable values to a dashboard script. Parsing
814 -D arguments as variable values is only attempted if the value
815 following -D does not match any of the known dashboard types.
816 --group <group>
818 Specify what group you'd like to submit results to
819 Submit dashboard to specified group instead of default one. By
821 default, the dashboard is submitted to Nightly, Experimental, or
822 Continuous group, but by specifying this option, the group can
823 be arbitrary.
824 This replaces the deprecated option --track. Despite the name
826 change its behavior is unchanged.
827 -A <file>, --add-notes <file>
829 Add a notes file with submission.
830 This option tells CTest to include a notes file when submitting
832 dashboard.
833 --tomorrow-tag
835 Nightly or Experimental starts with next day tag.
836 This is useful if the build will not finish in one day.
838 --extra-submit <file>[;<file>]
840 Submit extra files to the dashboard.
841 This option will submit extra files to the dashboard.
843 --http1.0
845 Submit using HTTP 1.0.
846 This option will force CTest to use HTTP 1.0 to submit files to
848 the dashboard, instead of HTTP 1.1.
849 --no-compress-output
851 Do not compress test output when submitting.
852 This flag will turn off automatic compression of test output.
854 Use this to maintain compatibility with an older version of
855 CDash which doesn't support compressed test output.
856 Dashboard Client Steps
858 CTest defines an ordered list of testing steps of which some or all may
859 be run as a dashboard client:
860 Start Start a new dashboard submission to be composed of results
862 recorded by the following steps. See the CTest Start Step sec‐
863 tion below.
864 Update Update the source tree from its version control repository.
866 Record the old and new versions and the list of updated source
867 files. See the CTest Update Step section below.
868 Configure
870 Configure the software by running a command in the build tree.
871 Record the configuration output log. See the CTest Configure
872 Step section below.
873 Build Build the software by running a command in the build tree.
875 Record the build output log and detect warnings and errors. See
876 the CTest Build Step section below.
877 Test Test the software by loading a CTestTestfile.cmake from the
879 build tree and executing the defined tests. Record the output
880 and result of each test. See the CTest Test Step section below.
881 Coverage
883 Compute coverage of the source code by running a coverage analy‐
884 sis tool and recording its output. See the CTest Coverage Step
885 section below.
886 MemCheck
888 Run the software test suite through a memory check tool. Record
889 the test output, results, and issues reported by the tool. See
890 the CTest MemCheck Step section below.
891 Submit Submit results recorded from other testing steps to the software
893 quality dashboard server. See the CTest Submit Step section be‐
894 low.
895 Dashboard Client Modes
897 CTest defines three modes of operation as a dashboard client:
898 Nightly
900 This mode is intended to be invoked once per day, typically at
901 night. It enables the Start, Update, Configure, Build, Test,
902 Coverage, and Submit steps by default. Selected steps run even
903 if the Update step reports no changes to the source tree.
904 Continuous
906 This mode is intended to be invoked repeatedly throughout the
907 day. It enables the Start, Update, Configure, Build, Test, Cov‐
908 erage, and Submit steps by default, but exits after the Update
909 step if it reports no changes to the source tree.
910 Experimental
912 This mode is intended to be invoked by a developer to test local
913 changes. It enables the Start, Configure, Build, Test, Cover‐
914 age, and Submit steps by default.
915 Dashboard Client via CTest Command-Line
917 CTest can perform testing on an already-generated build tree. Run the
918 ctest command with the current working directory set to the build tree
919 and use one of these signatures:
920 ctest -D <mode>[<step>]
922 ctest -M <mode> [-T <step>]...
923 The <mode> must be one of the above Dashboard Client Modes, and each
925 <step> must be one of the above Dashboard Client Steps.
926 CTest reads the Dashboard Client Configuration settings from a file in
928 the build tree called either CTestConfiguration.ini or DartConfigura‐
929 tion.tcl (the names are historical). The format of the file is:
930 # Lines starting in '#' are comments.
932 # Other non-blank lines are key-value pairs.
933 <setting>: <value>
934 where <setting> is the setting name and <value> is the setting value.
936 In build trees generated by CMake, this configuration file is generated
938 by the CTest module if included by the project. The module uses vari‐
939 ables to obtain a value for each setting as documented with the set‐
940 tings below.
941 Dashboard Client via CTest Script
943 CTest can perform testing driven by a cmake-language(7) script that
944 creates and maintains the source and build tree as well as performing
945 the testing steps. Run the ctest command with the current working di‐
946 rectory set outside of any build tree and use one of these signatures:
947 ctest -S <script>
949 ctest -SP <script>
950 The <script> file must call CTest Commands commands to run testing
952 steps explicitly as documented below. The commands obtain Dashboard
953 Client Configuration settings from their arguments or from variables
954 set in the script.
955
957 The Dashboard Client Steps may be configured by named settings as docu‐
958 mented in the following sections.
959 CTest Start Step
961 Start a new dashboard submission to be composed of results recorded by
962 the following steps.
963 In a CTest Script, the ctest_start() command runs this step. Arguments
965 to the command may specify some of the step settings. The command
966 first runs the command-line specified by the CTEST_CHECKOUT_COMMAND
967 variable, if set, to initialize the source directory.
968 Configuration settings include:
970 BuildDirectory
972 The full path to the project build tree.
973 • CTest Script variable: CTEST_BINARY_DIRECTORY
975 • CTest module variable: PROJECT_BINARY_DIR
977 SourceDirectory
979 The full path to the project source tree.
980 • CTest Script variable: CTEST_SOURCE_DIRECTORY
982 • CTest module variable: PROJECT_SOURCE_DIR
984 CTest Update Step
986 In a CTest Script, the ctest_update() command runs this step. Argu‐
987 ments to the command may specify some of the step settings.
988 Configuration settings to specify the version control tool include:
990 BZRCommand
992 bzr command-line tool to use if source tree is managed by
993 Bazaar.
994 • CTest Script variable: CTEST_BZR_COMMAND
996 • CTest module variable: none
998 BZRUpdateOptions
1000 Command-line options to the BZRCommand when updating the source.
1001 • CTest Script variable: CTEST_BZR_UPDATE_OPTIONS
1003 • CTest module variable: none
1005 CVSCommand
1007 cvs command-line tool to use if source tree is managed by CVS.
1008 • CTest Script variable: CTEST_CVS_COMMAND
1010 • CTest module variable: CVSCOMMAND
1012 CVSUpdateOptions
1014 Command-line options to the CVSCommand when updating the source.
1015 • CTest Script variable: CTEST_CVS_UPDATE_OPTIONS
1017 • CTest module variable: CVS_UPDATE_OPTIONS
1019 GITCommand
1021 git command-line tool to use if source tree is managed by Git.
1022 • CTest Script variable: CTEST_GIT_COMMAND
1024 • CTest module variable: GITCOMMAND
1026 The source tree is updated by git fetch followed by git reset
1028 --hard to the FETCH_HEAD. The result is the same as git pull
1029 except that any local modifications are overwritten. Use GITUp‐
1030 dateCustom to specify a different approach.
1031 GITInitSubmodules
1033 If set, CTest will update the repository's submodules before up‐
1034 dating.
1035 • CTest Script variable: CTEST_GIT_INIT_SUBMODULES
1037 • CTest module variable: CTEST_GIT_INIT_SUBMODULES
1039 GITUpdateCustom
1041 Specify a custom command line (as a semicolon-separated list) to
1042 run in the source tree (Git work tree) to update it instead of
1043 running the GITCommand.
1044 • CTest Script variable: CTEST_GIT_UPDATE_CUSTOM
1046 • CTest module variable: CTEST_GIT_UPDATE_CUSTOM
1048 GITUpdateOptions
1050 Command-line options to the GITCommand when updating the source.
1051 • CTest Script variable: CTEST_GIT_UPDATE_OPTIONS
1053 • CTest module variable: GIT_UPDATE_OPTIONS
1055 HGCommand
1057 hg command-line tool to use if source tree is managed by Mercu‐
1058 rial.
1059 • CTest Script variable: CTEST_HG_COMMAND
1061 • CTest module variable: none
1063 HGUpdateOptions
1065 Command-line options to the HGCommand when updating the source.
1066 • CTest Script variable: CTEST_HG_UPDATE_OPTIONS
1068 • CTest module variable: none
1070 P4Client
1072 Value of the -c option to the P4Command.
1073 • CTest Script variable: CTEST_P4_CLIENT
1075 • CTest module variable: CTEST_P4_CLIENT
1077 P4Command
1079 p4 command-line tool to use if source tree is managed by Per‐
1080 force.
1081 • CTest Script variable: CTEST_P4_COMMAND
1083 • CTest module variable: P4COMMAND
1085 P4Options
1087 Command-line options to the P4Command for all invocations.
1088 • CTest Script variable: CTEST_P4_OPTIONS
1090 • CTest module variable: CTEST_P4_OPTIONS
1092 P4UpdateCustom
1094 Specify a custom command line (as a semicolon-separated list) to
1095 run in the source tree (Perforce tree) to update it instead of
1096 running the P4Command.
1097 • CTest Script variable: none
1099 • CTest module variable: CTEST_P4_UPDATE_CUSTOM
1101 P4UpdateOptions
1103 Command-line options to the P4Command when updating the source.
1104 • CTest Script variable: CTEST_P4_UPDATE_OPTIONS
1106 • CTest module variable: CTEST_P4_UPDATE_OPTIONS
1108 SVNCommand
1110 svn command-line tool to use if source tree is managed by Sub‐
1111 version.
1112 • CTest Script variable: CTEST_SVN_COMMAND
1114 • CTest module variable: SVNCOMMAND
1116 SVNOptions
1118 Command-line options to the SVNCommand for all invocations.
1119 • CTest Script variable: CTEST_SVN_OPTIONS
1121 • CTest module variable: CTEST_SVN_OPTIONS
1123 SVNUpdateOptions
1125 Command-line options to the SVNCommand when updating the source.
1126 • CTest Script variable: CTEST_SVN_UPDATE_OPTIONS
1128 • CTest module variable: SVN_UPDATE_OPTIONS
1130 UpdateCommand
1132 Specify the version-control command-line tool to use without de‐
1133 tecting the VCS that manages the source tree.
1134 • CTest Script variable: CTEST_UPDATE_COMMAND
1136 • CTest module variable: <VCS>COMMAND when UPDATE_TYPE is <vcs>,
1138 else UPDATE_COMMAND
1139 UpdateOptions
1141 Command-line options to the UpdateCommand.
1142 • CTest Script variable: CTEST_UPDATE_OPTIONS
1144 • CTest module variable: <VCS>_UPDATE_OPTIONS when UPDATE_TYPE
1146 is <vcs>, else UPDATE_OPTIONS
1147 UpdateType
1149 Specify the version-control system that manages the source tree
1150 if it cannot be detected automatically. The value may be bzr,
1151 cvs, git, hg, p4, or svn.
1152 • CTest Script variable: none, detected from source tree
1154 • CTest module variable: UPDATE_TYPE if set, else CTEST_UP‐
1156 DATE_TYPE
1157 UpdateVersionOnly
1159 Specify that you want the version control update command to only
1160 discover the current version that is checked out, and not to up‐
1161 date to a different version.
1162 • CTest Script variable: CTEST_UPDATE_VERSION_ONLY
1164 UpdateVersionOverride
1166 Specify the current version of your source tree.
1167 When this variable is set to a non-empty string, CTest will re‐
1169 port the value you specified rather than using the update com‐
1170 mand to discover the current version that is checked out. Use of
1171 this variable supersedes UpdateVersionOnly. Like UpdateVer‐
1172 sionOnly, using this variable tells CTest not to update the
1173 source tree to a different version.
1174 • CTest Script variable: CTEST_UPDATE_VERSION_OVERRIDE
1176 Additional configuration settings include:
1178 NightlyStartTime
1180 In the Nightly dashboard mode, specify the "nightly start time".
1181 With centralized version control systems (cvs and svn), the Up‐
1182 date step checks out the version of the software as of this time
1183 so that multiple clients choose a common version to test. This
1184 is not well-defined in distributed version-control systems so
1185 the setting is ignored.
1186 • CTest Script variable: CTEST_NIGHTLY_START_TIME
1188 • CTest module variable: NIGHTLY_START_TIME if set, else
1190 CTEST_NIGHTLY_START_TIME
1191 CTest Configure Step
1193 In a CTest Script, the ctest_configure() command runs this step. Argu‐
1194 ments to the command may specify some of the step settings.
1195 Configuration settings include:
1197 ConfigureCommand
1199 Command-line to launch the software configuration process. It
1200 will be executed in the location specified by the BuildDirectory
1201 setting.
1202 • CTest Script variable: CTEST_CONFIGURE_COMMAND
1204 • CTest module variable: CMAKE_COMMAND followed by
1206 PROJECT_SOURCE_DIR
1207 LabelsForSubprojects
1209 Specify a semicolon-separated list of labels that will be
1210 treated as subprojects. This mapping will be passed on to CDash
1211 when configure, test or build results are submitted.
1212 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1214 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1216 See Label and Subproject Summary.
1218 CTest Build Step
1220 In a CTest Script, the ctest_build() command runs this step. Arguments
1221 to the command may specify some of the step settings.
1222 Configuration settings include:
1224 DefaultCTestConfigurationType
1226 When the build system to be launched allows build-time selection
1227 of the configuration (e.g. Debug, Release), this specifies the
1228 default configuration to be built when no -C option is given to
1229 the ctest command. The value will be substituted into the value
1230 of MakeCommand to replace the literal string ${CTEST_CONFIGURA‐
1231 TION_TYPE} if it appears.
1232 • CTest Script variable: CTEST_CONFIGURATION_TYPE
1234 • CTest module variable: DEFAULT_CTEST_CONFIGURATION_TYPE, ini‐
1236 tialized by the CMAKE_CONFIG_TYPE environment variable
1237 LabelsForSubprojects
1239 Specify a semicolon-separated list of labels that will be
1240 treated as subprojects. This mapping will be passed on to CDash
1241 when configure, test or build results are submitted.
1242 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1244 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1246 See Label and Subproject Summary.
1248 MakeCommand
1250 Command-line to launch the software build process. It will be
1251 executed in the location specified by the BuildDirectory set‐
1252 ting.
1253 • CTest Script variable: CTEST_BUILD_COMMAND
1255 • CTest module variable: MAKECOMMAND, initialized by the
1257 build_command() command
1258 UseLaunchers
1260 For build trees generated by CMake using one of the Makefile
1261 Generators or the Ninja generator, specify whether the
1262 CTEST_USE_LAUNCHERS feature is enabled by the CTestUseLaunchers
1263 module (also included by the CTest module). When enabled, the
1264 generated build system wraps each invocation of the compiler,
1265 linker, or custom command line with a "launcher" that communi‐
1266 cates with CTest via environment variables and files to report
1267 granular build warning and error information. Otherwise, CTest
1268 must "scrape" the build output log for diagnostics.
1269 • CTest Script variable: CTEST_USE_LAUNCHERS
1271 • CTest module variable: CTEST_USE_LAUNCHERS
1273 CTest Test Step
1275 In a CTest Script, the ctest_test() command runs this step. Arguments
1276 to the command may specify some of the step settings.
1277 Configuration settings include:
1279 ResourceSpecFile
1281 Specify a resource specification file.
1282 • CTest Script variable: CTEST_RESOURCE_SPEC_FILE
1284 • CTest module variable: CTEST_RESOURCE_SPEC_FILE
1286 See Resource Allocation for more information.
1288 LabelsForSubprojects
1290 Specify a semicolon-separated list of labels that will be
1291 treated as subprojects. This mapping will be passed on to CDash
1292 when configure, test or build results are submitted.
1293 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1295 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1297 See Label and Subproject Summary.
1299 TestLoad
1301 While running tests in parallel (e.g. with -j), try not to start
1302 tests when they may cause the CPU load to pass above a given
1303 threshold.
1304 • CTest Script variable: CTEST_TEST_LOAD
1306 • CTest module variable: CTEST_TEST_LOAD
1308 TimeOut
1310 The default timeout for each test if not specified by the
1311 TIMEOUT test property or the --timeout flag.
1312 • CTest Script variable: CTEST_TEST_TIMEOUT
1314 • CTest module variable: DART_TESTING_TIMEOUT
1316 To report extra test values to CDash, see Additional Test Measurements.
1318 CTest Coverage Step
1320 In a CTest Script, the ctest_coverage() command runs this step. Argu‐
1321 ments to the command may specify some of the step settings.
1322 Configuration settings include:
1324 CoverageCommand
1326 Command-line tool to perform software coverage analysis. It
1327 will be executed in the location specified by the BuildDirectory
1328 setting.
1329 • CTest Script variable: CTEST_COVERAGE_COMMAND
1331 • CTest module variable: COVERAGE_COMMAND
1333 CoverageExtraFlags
1335 Specify command-line options to the CoverageCommand tool.
1336 • CTest Script variable: CTEST_COVERAGE_EXTRA_FLAGS
1338 • CTest module variable: COVERAGE_EXTRA_FLAGS
1340 These options are the first arguments passed to CoverageCommand.
1342 CTest MemCheck Step
1344 In a CTest Script, the ctest_memcheck() command runs this step. Argu‐
1345 ments to the command may specify some of the step settings.
1346 Configuration settings include:
1348 MemoryCheckCommand
1350 Command-line tool to perform dynamic analysis. Test command
1351 lines will be launched through this tool.
1352 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND
1354 • CTest module variable: MEMORYCHECK_COMMAND
1356 MemoryCheckCommandOptions
1358 Specify command-line options to the MemoryCheckCommand tool.
1359 They will be placed prior to the test command line.
1360 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND_OPTIONS
1362 • CTest module variable: MEMORYCHECK_COMMAND_OPTIONS
1364 MemoryCheckType
1366 Specify the type of memory checking to perform.
1367 • CTest Script variable: CTEST_MEMORYCHECK_TYPE
1369 • CTest module variable: MEMORYCHECK_TYPE
1371 MemoryCheckSanitizerOptions
1373 Specify options to sanitizers when running with a sanitize-en‐
1374 abled build.
1375 • CTest Script variable: CTEST_MEMORYCHECK_SANITIZER_OPTIONS
1377 • CTest module variable: MEMORYCHECK_SANITIZER_OPTIONS
1379 MemoryCheckSuppressionFile
1381 Specify a file containing suppression rules for the MemoryCheck‐
1382 Command tool. It will be passed with options appropriate to the
1383 tool.
1384 • CTest Script variable: CTEST_MEMORYCHECK_SUPPRESSIONS_FILE
1386 • CTest module variable: MEMORYCHECK_SUPPRESSIONS_FILE
1388 Additional configuration settings include:
1390 BoundsCheckerCommand
1392 Specify a MemoryCheckCommand that is known to be command-line
1393 compatible with Bounds Checker.
1394 • CTest Script variable: none
1396 • CTest module variable: none
1398 PurifyCommand
1400 Specify a MemoryCheckCommand that is known to be command-line
1401 compatible with Purify.
1402 • CTest Script variable: none
1404 • CTest module variable: PURIFYCOMMAND
1406 ValgrindCommand
1408 Specify a MemoryCheckCommand that is known to be command-line
1409 compatible with Valgrind.
1410 • CTest Script variable: none
1412 • CTest module variable: VALGRIND_COMMAND
1414 ValgrindCommandOptions
1416 Specify command-line options to the ValgrindCommand tool. They
1417 will be placed prior to the test command line.
1418 • CTest Script variable: none
1420 • CTest module variable: VALGRIND_COMMAND_OPTIONS
1422 DrMemoryCommand
1424 Specify a MemoryCheckCommand that is known to be a command-line
1425 compatible with DrMemory.
1426 • CTest Script variable: none
1428 • CTest module variable: DRMEMORY_COMMAND
1430 DrMemoryCommandOptions
1432 Specify command-line options to the DrMemoryCommand tool. They
1433 will be placed prior to the test command line.
1434 • CTest Script variable: none
1436 • CTest module variable: DRMEMORY_COMMAND_OPTIONS
1438 CudaSanitizerCommand
1440 Specify a MemoryCheckCommand that is known to be a command-line
1441 compatible with cuda-memcheck or compute-sanitizer.
1442 • CTest Script variable: none
1444 • CTest module variable: CUDA_SANITIZER_COMMAND
1446 CudaSanitizerCommandOptions
1448 Specify command-line options to the CudaSanitizerCommand tool.
1449 They will be placed prior to the test command line.
1450 • CTest Script variable: none
1452 • CTest module variable: CUDA_SANITIZER_COMMAND_OPTIONS
1454 CTest Submit Step
1456 In a CTest Script, the ctest_submit() command runs this step. Argu‐
1457 ments to the command may specify some of the step settings.
1458 Configuration settings include:
1460 BuildName
1462 Describe the dashboard client platform with a short string.
1463 (Operating system, compiler, etc.)
1464 • CTest Script variable: CTEST_BUILD_NAME
1466 • CTest module variable: BUILDNAME
1468 CDashVersion
1470 Legacy option. Not used.
1471 • CTest Script variable: none, detected from server
1473 • CTest module variable: CTEST_CDASH_VERSION
1475 CTestSubmitRetryCount
1477 Specify a number of attempts to retry submission on network
1478 failure.
1479 • CTest Script variable: none, use the ctest_submit()
1481 RETRY_COUNT option.
1482 • CTest module variable: CTEST_SUBMIT_RETRY_COUNT
1484 CTestSubmitRetryDelay
1486 Specify a delay before retrying submission on network failure.
1487 • CTest Script variable: none, use the ctest_submit() RETRY_DE‐
1489 LAY option.
1490 • CTest module variable: CTEST_SUBMIT_RETRY_DELAY
1492 CurlOptions
1494 Specify a semicolon-separated list of options to control the
1495 Curl library that CTest uses internally to connect to the
1496 server. Possible options are CURLOPT_SSL_VERIFYPEER_OFF and
1497 CURLOPT_SSL_VERIFYHOST_OFF.
1498 • CTest Script variable: CTEST_CURL_OPTIONS
1500 • CTest module variable: CTEST_CURL_OPTIONS
1502 DropLocation
1504 Legacy option. When SubmitURL is not set, it is constructed
1505 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1506 DropLocation.
1507 • CTest Script variable: CTEST_DROP_LOCATION
1509 • CTest module variable: DROP_LOCATION if set, else
1511 CTEST_DROP_LOCATION
1512 DropMethod
1514 Legacy option. When SubmitURL is not set, it is constructed
1515 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1516 DropLocation.
1517 • CTest Script variable: CTEST_DROP_METHOD
1519 • CTest module variable: DROP_METHOD if set, else
1521 CTEST_DROP_METHOD
1522 DropSite
1524 Legacy option. When SubmitURL is not set, it is constructed
1525 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1526 DropLocation.
1527 • CTest Script variable: CTEST_DROP_SITE
1529 • CTest module variable: DROP_SITE if set, else CTEST_DROP_SITE
1531 DropSitePassword
1533 Legacy option. When SubmitURL is not set, it is constructed
1534 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1535 DropLocation.
1536 • CTest Script variable: CTEST_DROP_SITE_PASSWORD
1538 • CTest module variable: DROP_SITE_PASSWORD if set, else
1540 CTEST_DROP_SITE_PASWORD
1541 DropSiteUser
1543 Legacy option. When SubmitURL is not set, it is constructed
1544 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1545 DropLocation.
1546 • CTest Script variable: CTEST_DROP_SITE_USER
1548 • CTest module variable: DROP_SITE_USER if set, else
1550 CTEST_DROP_SITE_USER
1551 IsCDash
1553 Legacy option. Not used.
1554 • CTest Script variable: CTEST_DROP_SITE_CDASH
1556 • CTest module variable: CTEST_DROP_SITE_CDASH
1558 ScpCommand
1560 Legacy option. Not used.
1561 • CTest Script variable: CTEST_SCP_COMMAND
1563 • CTest module variable: SCPCOMMAND
1565 Site Describe the dashboard client host site with a short string.
1567 (Hostname, domain, etc.)
1568 • CTest Script variable: CTEST_SITE
1570 • CTest module variable: SITE, initialized by the site_name()
1572 command
1573 SubmitURL
1575 The http or https URL of the dashboard server to send the sub‐
1576 mission to.
1577 • CTest Script variable: CTEST_SUBMIT_URL
1579 • CTest module variable: SUBMIT_URL if set, else CTEST_SUB‐
1581 MIT_URL
1582 SubmitInactivityTimeout
1584 The time to wait for the submission after which it is canceled
1585 if not completed. Specify a zero value to disable timeout.
1586 • CTest Script variable: CTEST_SUBMIT_INACTIVITY_TIMEOUT
1588 • CTest module variable: CTEST_SUBMIT_INACTIVITY_TIMEOUT
1590 TriggerSite
1592 Legacy option. Not used.
1593 • CTest Script variable: CTEST_TRIGGER_SITE
1595 • CTest module variable: TRIGGER_SITE if set, else CTEST_TRIG‐
1597 GER_SITE
1598
1600 New in version 3.14.
1601 When the --show-only=json-v1 command line option is given, the test in‐
1604 formation is output in JSON format. Version 1.0 of the JSON object
1605 model is defined as follows:
1606 kind The string "ctestInfo".
1608 version
1610 A JSON object specifying the version components. Its members
1611 are
1612 major A non-negative integer specifying the major version com‐
1614 ponent.
1615 minor A non-negative integer specifying the minor version com‐
1617 ponent.
1618 backtraceGraph
1620 JSON object representing backtrace information with the follow‐
1621 ing members:
1622 commands
1624 List of command names.
1625 files List of file names.
1627 nodes List of node JSON objects with members:
1629 command
1631 Index into the commands member of the backtrace‐
1632 Graph.
1633 file Index into the files member of the backtraceGraph.
1635 line Line number in the file where the backtrace was
1637 added.
1638 parent Index into the nodes member of the backtraceGraph
1640 representing the parent in the graph.
1641 tests A JSON array listing information about each test. Each entry is
1643 a JSON object with members:
1644 name Test name.
1646 config Configuration that the test can run on. Empty string
1648 means any config.
1649 command
1651 List where the first element is the test command and the
1652 remaining elements are the command arguments.
1653 backtrace
1655 Index into the nodes member of the backtraceGraph.
1656 properties
1658 Test properties. Can contain keys for each of the sup‐
1659 ported test properties.
1660
1662 CTest provides a mechanism for tests to specify the resources that they
1663 need in a fine-grained way, and for users to specify the resources
1664 available on the running machine. This allows CTest to internally keep
1665 track of which resources are in use and which are free, scheduling
1666 tests in a way that prevents them from trying to claim resources that
1667 are not available.
1668 When the resource allocation feature is used, CTest will not oversub‐
1670 scribe resources. For example, if a resource has 8 slots, CTest will
1671 not run tests that collectively use more than 8 slots at a time. This
1672 has the effect of limiting how many tests can run at any given time,
1673 even if a high -j argument is used, if those tests all use some slots
1674 from the same resource. In addition, it means that a single test that
1675 uses more of a resource than is available on a machine will not run at
1676 all (and will be reported as Not Run).
1677 A common use case for this feature is for tests that require the use of
1679 a GPU. Multiple tests can simultaneously allocate memory from a GPU,
1680 but if too many tests try to do this at once, some of them will fail to
1681 allocate, resulting in a failed test, even though the test would have
1682 succeeded if it had the memory it needed. By using the resource alloca‐
1683 tion feature, each test can specify how much memory it requires from a
1684 GPU, allowing CTest to schedule tests in a way that running several of
1685 these tests at once does not exhaust the GPU's memory pool.
1686 Please note that CTest has no concept of what a GPU is or how much mem‐
1688 ory it has, nor does it have any way of communicating with a GPU to re‐
1689 trieve this information or perform any memory management. CTest simply
1690 keeps track of a list of abstract resource types, each of which has a
1691 certain number of slots available for tests to use. Each test specifies
1692 the number of slots that it requires from a certain resource, and CTest
1693 then schedules them in a way that prevents the total number of slots in
1694 use from exceeding the listed capacity. When a test is executed, and
1695 slots from a resource are allocated to that test, tests may assume that
1696 they have exclusive use of those slots for the duration of the test's
1697 process.
1698 The CTest resource allocation feature consists of two inputs:
1700 • The resource specification file, described below, which describes the
1702 resources available on the system.
1703 • The RESOURCE_GROUPS property of tests, which describes the resources
1705 required by the test.
1706 When CTest runs a test, the resources allocated to that test are passed
1708 in the form of a set of environment variables as described below. Using
1709 this information to decide which resource to connect to is left to the
1710 test writer.
1711 The RESOURCE_GROUPS property tells CTest what resources a test expects
1713 to use grouped in a way meaningful to the test. The test itself must
1714 read the environment variables to determine which resources have been
1715 allocated to each group. For example, each group may correspond to a
1716 process the test will spawn when executed.
1717 Note that even if a test specifies a RESOURCE_GROUPS property, it is
1719 still possible for that to test to run without any resource allocation
1720 (and without the corresponding environment variables) if the user does
1721 not pass a resource specification file. Passing this file, either
1722 through the --resource-spec-file command-line argument or the RE‐
1723 SOURCE_SPEC_FILE argument to ctest_test(), is what activates the re‐
1724 source allocation feature. Tests should check the CTEST_RE‐
1725 SOURCE_GROUP_COUNT environment variable to find out whether or not re‐
1726 source allocation is activated. This variable will always (and only) be
1727 defined if resource allocation is activated. If resource allocation is
1728 not activated, then the CTEST_RESOURCE_GROUP_COUNT variable will not
1729 exist, even if it exists for the parent ctest process. If a test abso‐
1730 lutely must have resource allocation, then it can return a failing exit
1731 code or use the SKIP_RETURN_CODE or SKIP_REGULAR_EXPRESSION properties
1732 to indicate a skipped test.
1733 Resource Specification File
1735 The resource specification file is a JSON file which is passed to
1736 CTest, either on the command line as ctest --resource-spec-file, or as
1737 the RESOURCE_SPEC_FILE argument of ctest_test(). If a dashboard script
1738 is used and RESOURCE_SPEC_FILE is not specified, the value of
1739 CTEST_RESOURCE_SPEC_FILE in the dashboard script is used instead. If
1740 --resource-spec-file, RESOURCE_SPEC_FILE, and CTEST_RESOURCE_SPEC_FILE
1741 in the dashboard script are not specified, the value of
1742 CTEST_RESOURCE_SPEC_FILE in the CMake build is used instead. If none of
1743 these are specified, no resource spec file is used.
1744 The resource specification file must be a JSON object. All examples in
1746 this document assume the following resource specification file:
1747 {
1749 "version": {
1750 "major": 1,
1751 "minor": 0
1752 },
1753 "local": [
1754 {
1755 "gpus": [
1756 {
1757 "id": "0",
1758 "slots": 2
1759 },
1760 {
1761 "id": "1",
1762 "slots": 4
1763 },
1764 {
1765 "id": "2",
1766 "slots": 2
1767 },
1768 {
1769 "id": "3"
1770 }
1771 ],
1772 "crypto_chips": [
1773 {
1774 "id": "card0",
1775 "slots": 4
1776 }
1777 ]
1778 }
1779 ]
1780 }
1781 The members are:
1783 version
1785 An object containing a major integer field and a minor integer
1786 field. Currently, the only supported version is major 1, minor
1787 0. Any other value is an error.
1788 local A JSON array of resource sets present on the system. Currently,
1790 this array is restricted to being of size 1.
1791 Each array element is a JSON object with members whose names are
1793 equal to the desired resource types, such as gpus. These names
1794 must start with a lowercase letter or an underscore, and subse‐
1795 quent characters can be a lowercase letter, a digit, or an un‐
1796 derscore. Uppercase letters are not allowed, because certain
1797 platforms have case-insensitive environment variables. See the
1798 Environment Variables section below for more information. It is
1799 recommended that the resource type name be the plural of a noun,
1800 such as gpus or crypto_chips (and not gpu or crypto_chip.)
1801 Please note that the names gpus and crypto_chips are just exam‐
1803 ples, and CTest does not interpret them in any way. You are free
1804 to make up any resource type you want to meet your own require‐
1805 ments.
1806 The value for each resource type is a JSON array consisting of
1808 JSON objects, each of which describe a specific instance of the
1809 specified resource. These objects have the following members:
1810 id A string consisting of an identifier for the resource.
1812 Each character in the identifier can be a lowercase let‐
1813 ter, a digit, or an underscore. Uppercase letters are
1814 not allowed.
1815 Identifiers must be unique within a resource type. How‐
1817 ever, they do not have to be unique across resource
1818 types. For example, it is valid to have a gpus resource
1819 named 0 and a crypto_chips resource named 0, but not two
1820 gpus resources both named 0.
1821 Please note that the IDs 0, 1, 2, 3, and card0 are just
1823 examples, and CTest does not interpret them in any way.
1824 You are free to make up any IDs you want to meet your own
1825 requirements.
1826 slots An optional unsigned number specifying the number of
1828 slots available on the resource. For example, this could
1829 be megabytes of RAM on a GPU, or cryptography units
1830 available on a cryptography chip. If slots is not speci‐
1831 fied, a default value of 1 is assumed.
1832 In the example file above, there are four GPUs with ID's 0 through 3.
1834 GPU 0 has 2 slots, GPU 1 has 4, GPU 2 has 2, and GPU 3 has a default of
1835 1 slot. There is also one cryptography chip with 4 slots.
1836 RESOURCE_GROUPS Property
1838 See RESOURCE_GROUPS for a description of this property.
1839 Environment Variables
1841 Once CTest has decided which resources to allocate to a test, it passes
1842 this information to the test executable as a series of environment
1843 variables. For each example below, we will assume that the test in
1844 question has a RESOURCE_GROUPS property of
1845 2,gpus:2;gpus:4,gpus:1,crypto_chips:2.
1846 The following variables are passed to the test process:
1848 CTEST_RESOURCE_GROUP_COUNT
1850 The total number of groups specified by the RESOURCE_GROUPS
1851 property. For example:
1852 • CTEST_RESOURCE_GROUP_COUNT=3
1854 This variable will only be defined if ctest(1) has been given a
1856 --resource-spec-file, or if ctest_test() has been given a RE‐
1857 SOURCE_SPEC_FILE. If no resource specification file has been
1858 given, this variable will not be defined.
1859 CTEST_RESOURCE_GROUP_<num>
1861 The list of resource types allocated to each group, with each
1862 item separated by a comma. <num> is a number from zero to
1863 CTEST_RESOURCE_GROUP_COUNT minus one. CTEST_RESOURCE_GROUP_<num>
1864 is defined for each <num> in this range. For example:
1865 • CTEST_RESOURCE_GROUP_0=gpus
1867 • CTEST_RESOURCE_GROUP_1=gpus
1869 • CTEST_RESOURCE_GROUP_2=crypto_chips,gpus
1871 CTEST_RESOURCE_GROUP_<num>_<resource-type>
1873 The list of resource IDs and number of slots from each ID allo‐
1874 cated to each group for a given resource type. This variable
1875 consists of a series of pairs, each pair separated by a semi‐
1876 colon, and with the two items in the pair separated by a comma.
1877 The first item in each pair is id: followed by the ID of a re‐
1878 source of type <resource-type>, and the second item is slots:
1879 followed by the number of slots from that resource allocated to
1880 the given group. For example:
1881 • CTEST_RESOURCE_GROUP_0_GPUS=id:0,slots:2
1883 • CTEST_RESOURCE_GROUP_1_GPUS=id:2,slots:2
1885 • CTEST_RESOURCE_GROUP_2_GPUS=id:1,slots:4;id:3,slots:1
1887 • CTEST_RESOURCE_GROUP_2_CRYPTO_CHIPS=id:card0,slots:2
1889 In this example, group 0 gets 2 slots from GPU 0, group 1 gets 2
1891 slots from GPU 2, and group 2 gets 4 slots from GPU 1, 1 slot
1892 from GPU 3, and 2 slots from cryptography chip card0.
1893 <num> is a number from zero to CTEST_RESOURCE_GROUP_COUNT minus
1895 one. <resource-type> is the name of a resource type, converted
1896 to uppercase. CTEST_RESOURCE_GROUP_<num>_<resource-type> is de‐
1897 fined for the product of each <num> in the range listed above
1898 and each resource type listed in CTEST_RESOURCE_GROUP_<num>.
1899 Because some platforms have case-insensitive names for environ‐
1901 ment variables, the names of resource types may not clash in a
1902 case-insensitive environment. Because of this, for the sake of
1903 simplicity, all resource types must be listed in all lowercase
1904 in the resource specification file and in the RESOURCE_GROUPS
1905 property, and they are converted to all uppercase in the
1906 CTEST_RESOURCE_GROUP_<num>_<resource-type> environment variable.
1907
1909 The following resources are available to get help using CMake:
1910 Home Page
1912 https://cmake.org
1913 The primary starting point for learning about CMake.
1915 Online Documentation and Community Resources
1917 https://cmake.org/documentation
1918 Links to available documentation and community resources may be
1920 found on this web page.
1921 Discourse Forum
1923 https://discourse.cmake.org
1924 The Discourse Forum hosts discussion and questions about CMake.
1926 : https://cdash.org
1928
1930 2000-2023 Kitware, Inc. and Contributors
19313.27.7 Oct 07, 2023 CTEST(1)