1CTEST(1) CMake CTEST(1)
2
6 ctest - CTest Command-Line Reference
7 Contents
9 • ctest(1)
10 • Synopsis
12 • Description
14 • Options
16 • Label and Subproject Summary
18 • Build and Test Mode
20 • Dashboard Client
22 • Dashboard Client Steps
24 • Dashboard Client Modes
26 • Dashboard Client via CTest Command-Line
28 • Dashboard Client via CTest Script
30 • Dashboard Client Configuration
32 • CTest Start Step
34 • CTest Update Step
36 • CTest Configure Step
38 • CTest Build Step
40 • CTest Test Step
42 • CTest Coverage Step
44 • CTest MemCheck Step
46 • CTest Submit Step
48 • Show as JSON Object Model
50 • Resource Allocation
52 • Resource Specification File
54 • RESOURCE_GROUPS Property
56 • Environment Variables
58 • See Also
60
62 ctest [<options>]
63 ctest --build-and-test <path-to-source> <path-to-build>
64 --build-generator <generator> [<options>...]
65 [--build-options <opts>...] [--test-command <command> [<args>...]]
66 ctest {-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>}
67 [-- <dashboard-options>...]
68
70 The ctest executable is the CMake test driver program. CMake-generated
71 build trees created for projects that use the enable_testing() and
72 add_test() commands have testing support. This program will run the
73 tests and report results.
74
76 --preset <preset>, --preset=<preset>
77 Use a test preset to specify test options. The project binary
78 directory is inferred from the configurePreset key. The current
79 working directory must contain CMake preset files. See preset
80 for more details.
81 --list-presets
83 Lists the available test presets. The current working directory
84 must contain CMake preset files.
85 -C <cfg>, --build-config <cfg>
87 Choose configuration to test.
88 Some CMake-generated build trees can have multiple build config‐
90 urations in the same tree. This option can be used to specify
91 which one should be tested. Example configurations are Debug
92 and Release.
93 --progress
95 Enable short progress output from tests.
96 When the output of ctest is being sent directly to a terminal,
98 the progress through the set of tests is reported by updating
99 the same line rather than printing start and end messages for
100 each test on new lines. This can significantly reduce the ver‐
101 bosity of the test output. Test completion messages are still
102 output on their own line for failed tests and the final test
103 summary will also still be logged.
104 This option can also be enabled by setting the environment vari‐
106 able CTEST_PROGRESS_OUTPUT.
107 -V,--verbose
109 Enable verbose output from tests.
110 Test output is normally suppressed and only summary information
112 is displayed. This option will show all test output.
113 -VV,--extra-verbose
115 Enable more verbose output from tests.
116 Test output is normally suppressed and only summary information
118 is displayed. This option will show even more test output.
119 --debug
121 Displaying more verbose internals of CTest.
122 This feature will result in a large number of output that is
124 mostly useful for debugging dashboard problems.
125 --output-on-failure
127 Output anything outputted by the test program if the test should
128 fail. This option can also be enabled by setting the CTEST_OUT‐
129 PUT_ON_FAILURE environment variable
130 --stop-on-failure
132 Stop running the tests when the first failure happens.
133 -F Enable failover.
135 This option allows CTest to resume a test set execution that was
137 previously interrupted. If no interruption occurred, the -F op‐
138 tion will have no effect.
139 -j <jobs>, --parallel <jobs>
141 Run the tests in parallel using the given number of jobs.
142 This option tells CTest to run the tests in parallel using given
144 number of jobs. This option can also be set by setting the
145 CTEST_PARALLEL_LEVEL environment variable.
146 This option can be used with the PROCESSORS test property.
148 See Label and Subproject Summary.
150 --resource-spec-file <file>
152 Run CTest with resource allocation enabled, using the resource
153 specification file specified in <file>.
154 When ctest is run as a Dashboard Client this sets the Resource‐
156 SpecFile option of the CTest Test Step.
157 --test-load <level>
159 While running tests in parallel (e.g. with -j), try not to start
160 tests when they may cause the CPU load to pass above a given
161 threshold.
162 When ctest is run as a Dashboard Client this sets the TestLoad
164 option of the CTest Test Step.
165 -Q,--quiet
167 Make CTest quiet.
168 This option will suppress all the output. The output log file
170 will still be generated if the --output-log is specified. Op‐
171 tions such as --verbose, --extra-verbose, and --debug are ig‐
172 nored if --quiet is specified.
173 -O <file>, --output-log <file>
175 Output to log file.
176 This option tells CTest to write all its output to a <file> log
178 file.
179 -N,--show-only[=<format>]
181 Disable actual execution of tests.
182 This option tells CTest to list the tests that would be run but
184 not actually run them. Useful in conjunction with the -R and -E
185 options.
186 <format> can be one of the following values.
188 human Human-friendly output. This is not guaranteed to be
190 stable. This is the default.
191 json-v1
193 Dump the test information in JSON format. See Show as
194 JSON Object Model.
195 -L <regex>, --label-regex <regex>
197 Run tests with labels matching regular expression.
198 This option tells CTest to run only the tests whose labels match
200 the given regular expression.
201 -R <regex>, --tests-regex <regex>
203 Run tests matching regular expression.
204 This option tells CTest to run only the tests whose names match
206 the given regular expression.
207 -E <regex>, --exclude-regex <regex>
209 Exclude tests matching regular expression.
210 This option tells CTest to NOT run the tests whose names match
212 the given regular expression.
213 -LE <regex>, --label-exclude <regex>
215 Exclude tests with labels matching regular expression.
216 This option tells CTest to NOT run the tests whose labels match
218 the given regular expression.
219 -FA <regex>, --fixture-exclude-any <regex>
221 Exclude fixtures matching <regex> from automatically adding any
222 tests to the test set.
223 If a test in the set of tests to be executed requires a particu‐
225 lar fixture, that fixture's setup and cleanup tests would nor‐
226 mally be added to the test set automatically. This option pre‐
227 vents adding setup or cleanup tests for fixtures matching the
228 <regex>. Note that all other fixture behavior is retained, in‐
229 cluding test dependencies and skipping tests that have fixture
230 setup tests that fail.
231 -FS <regex>, --fixture-exclude-setup <regex>
233 Same as -FA except only matching setup tests are excluded.
234 -FC <regex>, --fixture-exclude-cleanup <regex>
236 Same as -FA except only matching cleanup tests are excluded.
237 -D <dashboard>, --dashboard <dashboard>
239 Execute dashboard test.
240 This option tells CTest to act as a CDash client and perform a
242 dashboard test. All tests are <Mode><Test>, where <Mode> can be
243 Experimental, Nightly, and Continuous, and <Test> can be Start,
244 Update, Configure, Build, Test, Coverage, and Submit.
245 See Dashboard Client.
247 -D <var>:<type>=<value>
249 Define a variable for script mode.
250 Pass in variable values on the command line. Use in conjunction
252 with -S to pass variable values to a dashboard script. Parsing
253 -D arguments as variable values is only attempted if the value
254 following -D does not match any of the known dashboard types.
255 -M <model>, --test-model <model>
257 Sets the model for a dashboard.
258 This option tells CTest to act as a CDash client where the
260 <model> can be Experimental, Nightly, and Continuous. Combining
261 -M and -T is similar to -D.
262 See Dashboard Client.
264 -T <action>, --test-action <action>
266 Sets the dashboard action to perform.
267 This option tells CTest to act as a CDash client and perform
269 some action such as start, build, test etc. See Dashboard Client
270 Steps for the full list of actions. Combining -M and -T is sim‐
271 ilar to -D.
272 See Dashboard Client.
274 -S <script>, --script <script>
276 Execute a dashboard for a configuration.
277 This option tells CTest to load in a configuration script which
279 sets a number of parameters such as the binary and source direc‐
280 tories. Then CTest will do what is required to create and run a
281 dashboard. This option basically sets up a dashboard and then
282 runs ctest -D with the appropriate options.
283 See Dashboard Client.
285 -SP <script>, --script-new-process <script>
287 Execute a dashboard for a configuration.
288 This option does the same operations as -S but it will do them
290 in a separate process. This is primarily useful in cases where
291 the script may modify the environment and you do not want the
292 modified environment to impact other -S scripts.
293 See Dashboard Client.
295 -I [Start,End,Stride,test#,test#|Test file], --tests-information
297 Run a specific number of tests by number.
298 This option causes CTest to run tests starting at number Start,
300 ending at number End, and incrementing by Stride. Any addi‐
301 tional numbers after Stride are considered individual test num‐
302 bers. Start, End, or Stride can be empty. Optionally a file
303 can be given that contains the same syntax as the command line.
304 -U, --union
306 Take the Union of -I and -R.
307 When both -R and -I are specified by default the intersection of
309 tests are run. By specifying -U the union of tests is run in‐
310 stead.
311 --rerun-failed
313 Run only the tests that failed previously.
314 This option tells CTest to perform only the tests that failed
316 during its previous run. When this option is specified, CTest
317 ignores all other options intended to modify the list of tests
318 to run (-L, -R, -E, -LE, -I, etc). In the event that CTest runs
319 and no tests fail, subsequent calls to CTest with the --re‐
320 run-failed option will run the set of tests that most recently
321 failed (if any).
322 --repeat <mode>:<n>
324 Run tests repeatedly based on the given <mode> up to <n> times.
325 The modes are:
326 until-fail
328 Require each test to run <n> times without failing in or‐
329 der to pass. This is useful in finding sporadic failures
330 in test cases.
331 until-pass
333 Allow each test to run up to <n> times in order to pass.
334 Repeats tests if they fail for any reason. This is use‐
335 ful in tolerating sporadic failures in test cases.
336 after-timeout
338 Allow each test to run up to <n> times in order to pass.
339 Repeats tests only if they timeout. This is useful in
340 tolerating sporadic timeouts in test cases on busy ma‐
341 chines.
342 --repeat-until-fail <n>
344 Equivalent to --repeat until-fail:<n>.
345 --max-width <width>
347 Set the max width for a test name to output.
348 Set the maximum width for each test name to show in the output.
350 This allows the user to widen the output to avoid clipping the
351 test name which can be very annoying.
352 --interactive-debug-mode [0|1]
354 Set the interactive mode to 0 or 1.
355 This option causes CTest to run tests in either an interactive
357 mode or a non-interactive mode. On Windows this means that in
358 non-interactive mode, all system debug pop up windows are
359 blocked. In dashboard mode (Experimental, Nightly, Continuous),
360 the default is non-interactive. When just running tests not for
361 a dashboard the default is to allow popups and interactive de‐
362 bugging.
363 --no-label-summary
365 Disable timing summary information for labels.
366 This option tells CTest not to print summary information for
368 each label associated with the tests run. If there are no la‐
369 bels on the tests, nothing extra is printed.
370 See Label and Subproject Summary.
372 --no-subproject-summary
374 Disable timing summary information for subprojects.
375 This option tells CTest not to print summary information for
377 each subproject associated with the tests run. If there are no
378 subprojects on the tests, nothing extra is printed.
379 See Label and Subproject Summary.
381 --build-and-test See Build and Test Mode.
383 --test-dir <dir> Specify the directory in which to look for tests.
385 --test-output-size-passed <size>
387 Limit the output for passed tests to <size> bytes.
388 --test-output-size-failed <size>
390 Limit the output for failed tests to <size> bytes.
391 --overwrite
393 Overwrite CTest configuration option.
394 By default CTest uses configuration options from configuration
396 file. This option will overwrite the configuration option.
397 --force-new-ctest-process
399 Run child CTest instances as new processes.
400 By default CTest will run child CTest instances within the same
402 process. If this behavior is not desired, this argument will
403 enforce new processes for child CTest processes.
404 --schedule-random
406 Use a random order for scheduling tests.
407 This option will run the tests in a random order. It is com‐
409 monly used to detect implicit dependencies in a test suite.
410 --submit-index
412 Legacy option for old Dart2 dashboard server feature. Do not
413 use.
414 --timeout <seconds>
416 Set the default test timeout.
417 This option effectively sets a timeout on all tests that do not
419 already have a timeout set on them via the TIMEOUT property.
420 --stop-time <time>
422 Set a time at which all tests should stop running.
423 Set a real time of day at which all tests should timeout. Exam‐
425 ple: 7:00:00 -0400. Any time format understood by the curl date
426 parser is accepted. Local time is assumed if no timezone is
427 specified.
428 --print-labels
430 Print all available test labels.
431 This option will not run any tests, it will simply print the
433 list of all labels associated with the test set.
434 --no-tests=<[error|ignore]>
436 Regard no tests found either as error or ignore it.
437 If no tests were found, the default behavior of CTest is to al‐
439 ways log an error message but to return an error code in script
440 mode only. This option unifies the behavior of CTest by either
441 returning an error code if no tests were found or by ignoring
442 it.
443 --help,-help,-usage,-h,-H,/?
445 Print usage information and exit.
446 Usage describes the basic command line interface and its op‐
448 tions.
449 --version,-version,/V [<f>]
451 Show program name/version banner and exit.
452 If a file is specified, the version is written into it. The
454 help is printed to a named <f>ile if given.
455 --help-full [<f>]
457 Print all help manuals and exit.
458 All manuals are printed in a human-readable text format. The
460 help is printed to a named <f>ile if given.
461 --help-manual <man> [<f>]
463 Print one help manual and exit.
464 The specified manual is printed in a human-readable text format.
466 The help is printed to a named <f>ile if given.
467 --help-manual-list [<f>]
469 List help manuals available and exit.
470 The list contains all manuals for which help may be obtained by
472 using the --help-manual option followed by a manual name. The
473 help is printed to a named <f>ile if given.
474 --help-command <cmd> [<f>]
476 Print help for one command and exit.
477 The cmake-commands(7) manual entry for <cmd> is printed in a hu‐
479 man-readable text format. The help is printed to a named <f>ile
480 if given.
481 --help-command-list [<f>]
483 List commands with help available and exit.
484 The list contains all commands for which help may be obtained by
486 using the --help-command option followed by a command name. The
487 help is printed to a named <f>ile if given.
488 --help-commands [<f>]
490 Print cmake-commands manual and exit.
491 The cmake-commands(7) manual is printed in a human-readable text
493 format. The help is printed to a named <f>ile if given.
494 --help-module <mod> [<f>]
496 Print help for one module and exit.
497 The cmake-modules(7) manual entry for <mod> is printed in a hu‐
499 man-readable text format. The help is printed to a named <f>ile
500 if given.
501 --help-module-list [<f>]
503 List modules with help available and exit.
504 The list contains all modules for which help may be obtained by
506 using the --help-module option followed by a module name. The
507 help is printed to a named <f>ile if given.
508 --help-modules [<f>]
510 Print cmake-modules manual and exit.
511 The cmake-modules(7) manual is printed in a human-readable text
513 format. The help is printed to a named <f>ile if given.
514 --help-policy <cmp> [<f>]
516 Print help for one policy and exit.
517 The cmake-policies(7) manual entry for <cmp> is printed in a hu‐
519 man-readable text format. The help is printed to a named <f>ile
520 if given.
521 --help-policy-list [<f>]
523 List policies with help available and exit.
524 The list contains all policies for which help may be obtained by
526 using the --help-policy option followed by a policy name. The
527 help is printed to a named <f>ile if given.
528 --help-policies [<f>]
530 Print cmake-policies manual and exit.
531 The cmake-policies(7) manual is printed in a human-readable text
533 format. The help is printed to a named <f>ile if given.
534 --help-property <prop> [<f>]
536 Print help for one property and exit.
537 The cmake-properties(7) manual entries for <prop> are printed in
539 a human-readable text format. The help is printed to a named
540 <f>ile if given.
541 --help-property-list [<f>]
543 List properties with help available and exit.
544 The list contains all properties for which help may be obtained
546 by using the --help-property option followed by a property name.
547 The help is printed to a named <f>ile if given.
548 --help-properties [<f>]
550 Print cmake-properties manual and exit.
551 The cmake-properties(7) manual is printed in a human-readable
553 text format. The help is printed to a named <f>ile if given.
554 --help-variable <var> [<f>]
556 Print help for one variable and exit.
557 The cmake-variables(7) manual entry for <var> is printed in a
559 human-readable text format. The help is printed to a named
560 <f>ile if given.
561 --help-variable-list [<f>]
563 List variables with help available and exit.
564 The list contains all variables for which help may be obtained
566 by using the --help-variable option followed by a variable name.
567 The help is printed to a named <f>ile if given.
568 --help-variables [<f>]
570 Print cmake-variables manual and exit.
571 The cmake-variables(7) manual is printed in a human-readable
573 text format. The help is printed to a named <f>ile if given.
574
576 CTest prints timing summary information for each LABEL and subproject
577 associated with the tests run. The label time summary will not include
578 labels that are mapped to subprojects.
579 When the PROCESSORS test property is set, CTest will display a weighted
581 test timing result in label and subproject summaries. The time is re‐
582 ported with sec*proc instead of just sec.
583 The weighted time summary reported for each label or subproject j is
585 computed as:
586 Weighted Time Summary for Label/Subproject j =
588 sum(raw_test_time[j,i] * num_processors[j,i], i=1...num_tests[j])
589 for labels/subprojects j=1...total
591 where:
593 • raw_test_time[j,i]: Wall-clock time for the i test for the j label or
595 subproject
596 • num_processors[j,i]: Value of the CTest PROCESSORS property for the i
598 test for the j label or subproject
599 • num_tests[j]: Number of tests associated with the j label or subpro‐
601 ject
602 • total: Total number of labels or subprojects that have at least one
604 test run
605 Therefore, the weighted time summary for each label or subproject rep‐
607 resents the amount of time that CTest gave to run the tests for each
608 label or subproject and gives a good representation of the total ex‐
609 pense of the tests for each label or subproject when compared to other
610 labels or subprojects.
611 For example, if SubprojectA showed 100 sec*proc and SubprojectB showed
613 10 sec*proc, then CTest allocated approximately 10 times the CPU/core
614 time to run the tests for SubprojectA than for SubprojectB (e.g. so if
615 effort is going to be expended to reduce the cost of the test suite for
616 the whole project, then reducing the cost of the test suite for Subpro‐
617 jectA would likely have a larger impact than effort to reduce the cost
618 of the test suite for SubprojectB).
619
621 CTest provides a command-line signature to configure (i.e. run cmake
622 on), build, and/or execute a test:
623 ctest --build-and-test <path-to-source> <path-to-build>
625 --build-generator <generator>
626 [<options>...]
627 [--build-options <opts>...]
628 [--test-command <command> [<args>...]]
629 The configure and test steps are optional. The arguments to this com‐
631 mand line are the source and binary directories. The --build-generator
632 option must be provided to use --build-and-test. If --test-command is
633 specified then that will be run after the build is complete. Other op‐
634 tions that affect this mode include:
635 --build-target
637 Specify a specific target to build.
638 If left out the all target is built.
640 --build-nocmake
642 Run the build without running cmake first.
643 Skip the cmake step.
645 --build-run-dir
647 Specify directory to run programs from.
648 Directory where programs will be after it has been compiled.
650 --build-two-config
652 Run CMake twice.
653 --build-exe-dir
655 Specify the directory for the executable.
656 --build-generator
658 Specify the generator to use. See the cmake-generators(7) man‐
659 ual.
660 --build-generator-platform
662 Specify the generator-specific platform.
663 --build-generator-toolset
665 Specify the generator-specific toolset.
666 --build-project
668 Specify the name of the project to build.
669 --build-makeprogram
671 Specify the explicit make program to be used by CMake when con‐
672 figuring and building the project. Only applicable for Make and
673 Ninja based generators.
674 --build-noclean
676 Skip the make clean step.
677 --build-config-sample
679 A sample executable to use to determine the configuration that
680 should be used. e.g. Debug, Release etc.
681 --build-options
683 Additional options for configuring the build (i.e. for CMake,
684 not for the build tool). Note that if this is specified, the
685 --build-options keyword and its arguments must be the last op‐
686 tion given on the command line, with the possible exception of
687 --test-command.
688 --test-command
690 The command to run as the test step with the --build-and-test
691 option. All arguments following this keyword will be assumed to
692 be part of the test command line, so it must be the last option
693 given.
694 --test-timeout
696 The time limit in seconds
697
699 CTest can operate as a client for the CDash software quality dashboard
700 application. As a dashboard client, CTest performs a sequence of steps
701 to configure, build, and test software, and then submits the results to
702 a CDash server. The command-line signature used to submit to CDash is:
703 ctest (-D <dashboard> | -M <model> -T <action> | -S <script> | -SP <script>)
705 [-- <dashboard-options>...]
706 Options for Dashboard Client include:
708 --group <group>
710 Specify what group you'd like to submit results to
711 Submit dashboard to specified group instead of default one. By
713 default, the dashboard is submitted to Nightly, Experimental, or
714 Continuous group, but by specifying this option, the group can
715 be arbitrary.
716 This replaces the deprecated option --track. Despite the name
718 change its behavior is unchanged.
719 -A <file>, --add-notes <file>
721 Add a notes file with submission.
722 This option tells CTest to include a notes file when submitting
724 dashboard.
725 --tomorrow-tag
727 Nightly or Experimental starts with next day tag.
728 This is useful if the build will not finish in one day.
730 --extra-submit <file>[;<file>]
732 Submit extra files to the dashboard.
733 This option will submit extra files to the dashboard.
735 --http1.0
737 Submit using HTTP 1.0.
738 This option will force CTest to use HTTP 1.0 to submit files to
740 the dashboard, instead of HTTP 1.1.
741 --no-compress-output
743 Do not compress test output when submitting.
744 This flag will turn off automatic compression of test output.
746 Use this to maintain compatibility with an older version of
747 CDash which doesn't support compressed test output.
748 Dashboard Client Steps
750 CTest defines an ordered list of testing steps of which some or all may
751 be run as a dashboard client:
752 Start Start a new dashboard submission to be composed of results
754 recorded by the following steps. See the CTest Start Step sec‐
755 tion below.
756 Update Update the source tree from its version control repository.
758 Record the old and new versions and the list of updated source
759 files. See the CTest Update Step section below.
760 Configure
762 Configure the software by running a command in the build tree.
763 Record the configuration output log. See the CTest Configure
764 Step section below.
765 Build Build the software by running a command in the build tree.
767 Record the build output log and detect warnings and errors. See
768 the CTest Build Step section below.
769 Test Test the software by loading a CTestTestfile.cmake from the
771 build tree and executing the defined tests. Record the output
772 and result of each test. See the CTest Test Step section below.
773 Coverage
775 Compute coverage of the source code by running a coverage analy‐
776 sis tool and recording its output. See the CTest Coverage Step
777 section below.
778 MemCheck
780 Run the software test suite through a memory check tool. Record
781 the test output, results, and issues reported by the tool. See
782 the CTest MemCheck Step section below.
783 Submit Submit results recorded from other testing steps to the software
785 quality dashboard server. See the CTest Submit Step section be‐
786 low.
787 Dashboard Client Modes
789 CTest defines three modes of operation as a dashboard client:
790 Nightly
792 This mode is intended to be invoked once per day, typically at
793 night. It enables the Start, Update, Configure, Build, Test,
794 Coverage, and Submit steps by default. Selected steps run even
795 if the Update step reports no changes to the source tree.
796 Continuous
798 This mode is intended to be invoked repeatedly throughout the
799 day. It enables the Start, Update, Configure, Build, Test, Cov‐
800 erage, and Submit steps by default, but exits after the Update
801 step if it reports no changes to the source tree.
802 Experimental
804 This mode is intended to be invoked by a developer to test local
805 changes. It enables the Start, Configure, Build, Test, Cover‐
806 age, and Submit steps by default.
807 Dashboard Client via CTest Command-Line
809 CTest can perform testing on an already-generated build tree. Run the
810 ctest command with the current working directory set to the build tree
811 and use one of these signatures:
812 ctest -D <mode>[<step>]
814 ctest -M <mode> [ -T <step> ]...
815 The <mode> must be one of the above Dashboard Client Modes, and each
817 <step> must be one of the above Dashboard Client Steps.
818 CTest reads the Dashboard Client Configuration settings from a file in
820 the build tree called either CTestConfiguration.ini or DartConfigura‐
821 tion.tcl (the names are historical). The format of the file is:
822 # Lines starting in '#' are comments.
824 # Other non-blank lines are key-value pairs.
825 <setting>: <value>
826 where <setting> is the setting name and <value> is the setting value.
828 In build trees generated by CMake, this configuration file is generated
830 by the CTest module if included by the project. The module uses vari‐
831 ables to obtain a value for each setting as documented with the set‐
832 tings below.
833 Dashboard Client via CTest Script
835 CTest can perform testing driven by a cmake-language(7) script that
836 creates and maintains the source and build tree as well as performing
837 the testing steps. Run the ctest command with the current working di‐
838 rectory set outside of any build tree and use one of these signatures:
839 ctest -S <script>
841 ctest -SP <script>
842 The <script> file must call CTest Commands commands to run testing
844 steps explicitly as documented below. The commands obtain Dashboard
845 Client Configuration settings from their arguments or from variables
846 set in the script.
847
849 The Dashboard Client Steps may be configured by named settings as docu‐
850 mented in the following sections.
851 CTest Start Step
853 Start a new dashboard submission to be composed of results recorded by
854 the following steps.
855 In a CTest Script, the ctest_start() command runs this step. Arguments
857 to the command may specify some of the step settings. The command
858 first runs the command-line specified by the CTEST_CHECKOUT_COMMAND
859 variable, if set, to initialize the source directory.
860 Configuration settings include:
862 BuildDirectory
864 The full path to the project build tree.
865 • CTest Script variable: CTEST_BINARY_DIRECTORY
867 • CTest module variable: PROJECT_BINARY_DIR
869 SourceDirectory
871 The full path to the project source tree.
872 • CTest Script variable: CTEST_SOURCE_DIRECTORY
874 • CTest module variable: PROJECT_SOURCE_DIR
876 CTest Update Step
878 In a CTest Script, the ctest_update() command runs this step. Argu‐
879 ments to the command may specify some of the step settings.
880 Configuration settings to specify the version control tool include:
882 BZRCommand
884 bzr command-line tool to use if source tree is managed by
885 Bazaar.
886 • CTest Script variable: CTEST_BZR_COMMAND
888 • CTest module variable: none
890 BZRUpdateOptions
892 Command-line options to the BZRCommand when updating the source.
893 • CTest Script variable: CTEST_BZR_UPDATE_OPTIONS
895 • CTest module variable: none
897 CVSCommand
899 cvs command-line tool to use if source tree is managed by CVS.
900 • CTest Script variable: CTEST_CVS_COMMAND
902 • CTest module variable: CVSCOMMAND
904 CVSUpdateOptions
906 Command-line options to the CVSCommand when updating the source.
907 • CTest Script variable: CTEST_CVS_UPDATE_OPTIONS
909 • CTest module variable: CVS_UPDATE_OPTIONS
911 GITCommand
913 git command-line tool to use if source tree is managed by Git.
914 • CTest Script variable: CTEST_GIT_COMMAND
916 • CTest module variable: GITCOMMAND
918 The source tree is updated by git fetch followed by git reset
920 --hard to the FETCH_HEAD. The result is the same as git pull
921 except that any local modifications are overwritten. Use GITUp‐
922 dateCustom to specify a different approach.
923 GITInitSubmodules
925 If set, CTest will update the repository's submodules before up‐
926 dating.
927 • CTest Script variable: CTEST_GIT_INIT_SUBMODULES
929 • CTest module variable: CTEST_GIT_INIT_SUBMODULES
931 GITUpdateCustom
933 Specify a custom command line (as a semicolon-separated list) to
934 run in the source tree (Git work tree) to update it instead of
935 running the GITCommand.
936 • CTest Script variable: CTEST_GIT_UPDATE_CUSTOM
938 • CTest module variable: CTEST_GIT_UPDATE_CUSTOM
940 GITUpdateOptions
942 Command-line options to the GITCommand when updating the source.
943 • CTest Script variable: CTEST_GIT_UPDATE_OPTIONS
945 • CTest module variable: GIT_UPDATE_OPTIONS
947 HGCommand
949 hg command-line tool to use if source tree is managed by Mercu‐
950 rial.
951 • CTest Script variable: CTEST_HG_COMMAND
953 • CTest module variable: none
955 HGUpdateOptions
957 Command-line options to the HGCommand when updating the source.
958 • CTest Script variable: CTEST_HG_UPDATE_OPTIONS
960 • CTest module variable: none
962 P4Client
964 Value of the -c option to the P4Command.
965 • CTest Script variable: CTEST_P4_CLIENT
967 • CTest module variable: CTEST_P4_CLIENT
969 P4Command
971 p4 command-line tool to use if source tree is managed by Per‐
972 force.
973 • CTest Script variable: CTEST_P4_COMMAND
975 • CTest module variable: P4COMMAND
977 P4Options
979 Command-line options to the P4Command for all invocations.
980 • CTest Script variable: CTEST_P4_OPTIONS
982 • CTest module variable: CTEST_P4_OPTIONS
984 P4UpdateCustom
986 Specify a custom command line (as a semicolon-separated list) to
987 run in the source tree (Perforce tree) to update it instead of
988 running the P4Command.
989 • CTest Script variable: none
991 • CTest module variable: CTEST_P4_UPDATE_CUSTOM
993 P4UpdateOptions
995 Command-line options to the P4Command when updating the source.
996 • CTest Script variable: CTEST_P4_UPDATE_OPTIONS
998 • CTest module variable: CTEST_P4_UPDATE_OPTIONS
1000 SVNCommand
1002 svn command-line tool to use if source tree is managed by Sub‐
1003 version.
1004 • CTest Script variable: CTEST_SVN_COMMAND
1006 • CTest module variable: SVNCOMMAND
1008 SVNOptions
1010 Command-line options to the SVNCommand for all invocations.
1011 • CTest Script variable: CTEST_SVN_OPTIONS
1013 • CTest module variable: CTEST_SVN_OPTIONS
1015 SVNUpdateOptions
1017 Command-line options to the SVNCommand when updating the source.
1018 • CTest Script variable: CTEST_SVN_UPDATE_OPTIONS
1020 • CTest module variable: SVN_UPDATE_OPTIONS
1022 UpdateCommand
1024 Specify the version-control command-line tool to use without de‐
1025 tecting the VCS that manages the source tree.
1026 • CTest Script variable: CTEST_UPDATE_COMMAND
1028 • CTest module variable: <VCS>COMMAND when UPDATE_TYPE is <vcs>,
1030 else UPDATE_COMMAND
1031 UpdateOptions
1033 Command-line options to the UpdateCommand.
1034 • CTest Script variable: CTEST_UPDATE_OPTIONS
1036 • CTest module variable: <VCS>_UPDATE_OPTIONS when UPDATE_TYPE
1038 is <vcs>, else UPDATE_OPTIONS
1039 UpdateType
1041 Specify the version-control system that manages the source tree
1042 if it cannot be detected automatically. The value may be bzr,
1043 cvs, git, hg, p4, or svn.
1044 • CTest Script variable: none, detected from source tree
1046 • CTest module variable: UPDATE_TYPE if set, else CTEST_UP‐
1048 DATE_TYPE
1049 UpdateVersionOnly
1051 Specify that you want the version control update command to only
1052 discover the current version that is checked out, and not to up‐
1053 date to a different version.
1054 • CTest Script variable: CTEST_UPDATE_VERSION_ONLY
1056 UpdateVersionOverride
1058 Specify the current version of your source tree.
1059 When this variable is set to a non-empty string, CTest will re‐
1061 port the value you specified rather than using the update com‐
1062 mand to discover the current version that is checked out. Use of
1063 this variable supersedes UpdateVersionOnly. Like UpdateVer‐
1064 sionOnly, using this variable tells CTest not to update the
1065 source tree to a different version.
1066 • CTest Script variable: CTEST_UPDATE_VERSION_OVERRIDE
1068 Additional configuration settings include:
1070 NightlyStartTime
1072 In the Nightly dashboard mode, specify the "nightly start time".
1073 With centralized version control systems (cvs and svn), the Up‐
1074 date step checks out the version of the software as of this time
1075 so that multiple clients choose a common version to test. This
1076 is not well-defined in distributed version-control systems so
1077 the setting is ignored.
1078 • CTest Script variable: CTEST_NIGHTLY_START_TIME
1080 • CTest module variable: NIGHTLY_START_TIME if set, else
1082 CTEST_NIGHTLY_START_TIME
1083 CTest Configure Step
1085 In a CTest Script, the ctest_configure() command runs this step. Argu‐
1086 ments to the command may specify some of the step settings.
1087 Configuration settings include:
1089 ConfigureCommand
1091 Command-line to launch the software configuration process. It
1092 will be executed in the location specified by the BuildDirectory
1093 setting.
1094 • CTest Script variable: CTEST_CONFIGURE_COMMAND
1096 • CTest module variable: CMAKE_COMMAND followed by
1098 PROJECT_SOURCE_DIR
1099 LabelsForSubprojects
1101 Specify a semicolon-separated list of labels that will be
1102 treated as subprojects. This mapping will be passed on to CDash
1103 when configure, test or build results are submitted.
1104 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1106 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1108 See Label and Subproject Summary.
1110 CTest Build Step
1112 In a CTest Script, the ctest_build() command runs this step. Arguments
1113 to the command may specify some of the step settings.
1114 Configuration settings include:
1116 DefaultCTestConfigurationType
1118 When the build system to be launched allows build-time selection
1119 of the configuration (e.g. Debug, Release), this specifies the
1120 default configuration to be built when no -C option is given to
1121 the ctest command. The value will be substituted into the value
1122 of MakeCommand to replace the literal string ${CTEST_CONFIGURA‐
1123 TION_TYPE} if it appears.
1124 • CTest Script variable: CTEST_CONFIGURATION_TYPE
1126 • CTest module variable: DEFAULT_CTEST_CONFIGURATION_TYPE, ini‐
1128 tialized by the CMAKE_CONFIG_TYPE environment variable
1129 LabelsForSubprojects
1131 Specify a semicolon-separated list of labels that will be
1132 treated as subprojects. This mapping will be passed on to CDash
1133 when configure, test or build results are submitted.
1134 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1136 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1138 See Label and Subproject Summary.
1140 MakeCommand
1142 Command-line to launch the software build process. It will be
1143 executed in the location specified by the BuildDirectory set‐
1144 ting.
1145 • CTest Script variable: CTEST_BUILD_COMMAND
1147 • CTest module variable: MAKECOMMAND, initialized by the
1149 build_command() command
1150 UseLaunchers
1152 For build trees generated by CMake using one of the Makefile
1153 Generators or the Ninja generator, specify whether the
1154 CTEST_USE_LAUNCHERS feature is enabled by the CTestUseLaunchers
1155 module (also included by the CTest module). When enabled, the
1156 generated build system wraps each invocation of the compiler,
1157 linker, or custom command line with a "launcher" that communi‐
1158 cates with CTest via environment variables and files to report
1159 granular build warning and error information. Otherwise, CTest
1160 must "scrape" the build output log for diagnostics.
1161 • CTest Script variable: CTEST_USE_LAUNCHERS
1163 • CTest module variable: CTEST_USE_LAUNCHERS
1165 CTest Test Step
1167 In a CTest Script, the ctest_test() command runs this step. Arguments
1168 to the command may specify some of the step settings.
1169 Configuration settings include:
1171 ResourceSpecFile
1173 Specify a resource specification file.
1174 • CTest Script variable: CTEST_RESOURCE_SPEC_FILE
1176 • CTest module variable: CTEST_RESOURCE_SPEC_FILE
1178 See Resource Allocation for more information.
1180 LabelsForSubprojects
1182 Specify a semicolon-separated list of labels that will be
1183 treated as subprojects. This mapping will be passed on to CDash
1184 when configure, test or build results are submitted.
1185 • CTest Script variable: CTEST_LABELS_FOR_SUBPROJECTS
1187 • CTest module variable: CTEST_LABELS_FOR_SUBPROJECTS
1189 See Label and Subproject Summary.
1191 TestLoad
1193 While running tests in parallel (e.g. with -j), try not to start
1194 tests when they may cause the CPU load to pass above a given
1195 threshold.
1196 • CTest Script variable: CTEST_TEST_LOAD
1198 • CTest module variable: CTEST_TEST_LOAD
1200 TimeOut
1202 The default timeout for each test if not specified by the TIME‐
1203 OUT test property.
1204 • CTest Script variable: CTEST_TEST_TIMEOUT
1206 • CTest module variable: DART_TESTING_TIMEOUT
1208 CTest Coverage Step
1210 In a CTest Script, the ctest_coverage() command runs this step. Argu‐
1211 ments to the command may specify some of the step settings.
1212 Configuration settings include:
1214 CoverageCommand
1216 Command-line tool to perform software coverage analysis. It
1217 will be executed in the location specified by the BuildDirectory
1218 setting.
1219 • CTest Script variable: CTEST_COVERAGE_COMMAND
1221 • CTest module variable: COVERAGE_COMMAND
1223 CoverageExtraFlags
1225 Specify command-line options to the CoverageCommand tool.
1226 • CTest Script variable: CTEST_COVERAGE_EXTRA_FLAGS
1228 • CTest module variable: COVERAGE_EXTRA_FLAGS
1230 These options are the first arguments passed to CoverageCommand.
1232 CTest MemCheck Step
1234 In a CTest Script, the ctest_memcheck() command runs this step. Argu‐
1235 ments to the command may specify some of the step settings.
1236 Configuration settings include:
1238 MemoryCheckCommand
1240 Command-line tool to perform dynamic analysis. Test command
1241 lines will be launched through this tool.
1242 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND
1244 • CTest module variable: MEMORYCHECK_COMMAND
1246 MemoryCheckCommandOptions
1248 Specify command-line options to the MemoryCheckCommand tool.
1249 They will be placed prior to the test command line.
1250 • CTest Script variable: CTEST_MEMORYCHECK_COMMAND_OPTIONS
1252 • CTest module variable: MEMORYCHECK_COMMAND_OPTIONS
1254 MemoryCheckType
1256 Specify the type of memory checking to perform.
1257 • CTest Script variable: CTEST_MEMORYCHECK_TYPE
1259 • CTest module variable: MEMORYCHECK_TYPE
1261 MemoryCheckSanitizerOptions
1263 Specify options to sanitizers when running with a sanitize-en‐
1264 abled build.
1265 • CTest Script variable: CTEST_MEMORYCHECK_SANITIZER_OPTIONS
1267 • CTest module variable: MEMORYCHECK_SANITIZER_OPTIONS
1269 MemoryCheckSuppressionFile
1271 Specify a file containing suppression rules for the MemoryCheck‐
1272 Command tool. It will be passed with options appropriate to the
1273 tool.
1274 • CTest Script variable: CTEST_MEMORYCHECK_SUPPRESSIONS_FILE
1276 • CTest module variable: MEMORYCHECK_SUPPRESSIONS_FILE
1278 Additional configuration settings include:
1280 BoundsCheckerCommand
1282 Specify a MemoryCheckCommand that is known to be command-line
1283 compatible with Bounds Checker.
1284 • CTest Script variable: none
1286 • CTest module variable: none
1288 PurifyCommand
1290 Specify a MemoryCheckCommand that is known to be command-line
1291 compatible with Purify.
1292 • CTest Script variable: none
1294 • CTest module variable: PURIFYCOMMAND
1296 ValgrindCommand
1298 Specify a MemoryCheckCommand that is known to be command-line
1299 compatible with Valgrind.
1300 • CTest Script variable: none
1302 • CTest module variable: VALGRIND_COMMAND
1304 ValgrindCommandOptions
1306 Specify command-line options to the ValgrindCommand tool. They
1307 will be placed prior to the test command line.
1308 • CTest Script variable: none
1310 • CTest module variable: VALGRIND_COMMAND_OPTIONS
1312 DrMemoryCommand
1314 Specify a MemoryCheckCommand that is known to be a command-line
1315 compatible with DrMemory.
1316 • CTest Script variable: none
1318 • CTest module variable: DRMEMORY_COMMAND
1320 DrMemoryCommandOptions
1322 Specify command-line options to the DrMemoryCommand tool. They
1323 will be placed prior to the test command line.
1324 • CTest Script variable: none
1326 • CTest module variable: DRMEMORY_COMMAND_OPTIONS
1328 CudaSanitizerCommand
1330 Specify a MemoryCheckCommand that is known to be a command-line
1331 compatible with cuda-memcheck or compute-sanitizer.
1332 • CTest Script variable: none
1334 • CTest module variable: CUDA_SANITIZER_COMMAND
1336 CudaSanitizerCommandOptions
1338 Specify command-line options to the CudaSanitizerCommand tool.
1339 They will be placed prior to the test command line.
1340 • CTest Script variable: none
1342 • CTest module variable: CUDA_SANITIZER_COMMAND_OPTIONS
1344 CTest Submit Step
1346 In a CTest Script, the ctest_submit() command runs this step. Argu‐
1347 ments to the command may specify some of the step settings.
1348 Configuration settings include:
1350 BuildName
1352 Describe the dashboard client platform with a short string.
1353 (Operating system, compiler, etc.)
1354 • CTest Script variable: CTEST_BUILD_NAME
1356 • CTest module variable: BUILDNAME
1358 CDashVersion
1360 Legacy option. Not used.
1361 • CTest Script variable: none, detected from server
1363 • CTest module variable: CTEST_CDASH_VERSION
1365 CTestSubmitRetryCount
1367 Specify a number of attempts to retry submission on network
1368 failure.
1369 • CTest Script variable: none, use the ctest_submit()
1371 RETRY_COUNT option.
1372 • CTest module variable: CTEST_SUBMIT_RETRY_COUNT
1374 CTestSubmitRetryDelay
1376 Specify a delay before retrying submission on network failure.
1377 • CTest Script variable: none, use the ctest_submit() RETRY_DE‐
1379 LAY option.
1380 • CTest module variable: CTEST_SUBMIT_RETRY_DELAY
1382 CurlOptions
1384 Specify a semicolon-separated list of options to control the
1385 Curl library that CTest uses internally to connect to the
1386 server. Possible options are CURLOPT_SSL_VERIFYPEER_OFF and
1387 CURLOPT_SSL_VERIFYHOST_OFF.
1388 • CTest Script variable: CTEST_CURL_OPTIONS
1390 • CTest module variable: CTEST_CURL_OPTIONS
1392 DropLocation
1394 Legacy option. When SubmitURL is not set, it is constructed
1395 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1396 DropLocation.
1397 • CTest Script variable: CTEST_DROP_LOCATION
1399 • CTest module variable: DROP_LOCATION if set, else
1401 CTEST_DROP_LOCATION
1402 DropMethod
1404 Legacy option. When SubmitURL is not set, it is constructed
1405 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1406 DropLocation.
1407 • CTest Script variable: CTEST_DROP_METHOD
1409 • CTest module variable: DROP_METHOD if set, else
1411 CTEST_DROP_METHOD
1412 DropSite
1414 Legacy option. When SubmitURL is not set, it is constructed
1415 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1416 DropLocation.
1417 • CTest Script variable: CTEST_DROP_SITE
1419 • CTest module variable: DROP_SITE if set, else CTEST_DROP_SITE
1421 DropSitePassword
1423 Legacy option. When SubmitURL is not set, it is constructed
1424 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1425 DropLocation.
1426 • CTest Script variable: CTEST_DROP_SITE_PASSWORD
1428 • CTest module variable: DROP_SITE_PASSWORD if set, else
1430 CTEST_DROP_SITE_PASWORD
1431 DropSiteUser
1433 Legacy option. When SubmitURL is not set, it is constructed
1434 from DropMethod, DropSiteUser, DropSitePassword, DropSite, and
1435 DropLocation.
1436 • CTest Script variable: CTEST_DROP_SITE_USER
1438 • CTest module variable: DROP_SITE_USER if set, else
1440 CTEST_DROP_SITE_USER
1441 IsCDash
1443 Legacy option. Not used.
1444 • CTest Script variable: CTEST_DROP_SITE_CDASH
1446 • CTest module variable: CTEST_DROP_SITE_CDASH
1448 ScpCommand
1450 Legacy option. Not used.
1451 • CTest Script variable: CTEST_SCP_COMMAND
1453 • CTest module variable: SCPCOMMAND
1455 Site Describe the dashboard client host site with a short string.
1457 (Hostname, domain, etc.)
1458 • CTest Script variable: CTEST_SITE
1460 • CTest module variable: SITE, initialized by the site_name()
1462 command
1463 SubmitURL
1465 The http or https URL of the dashboard server to send the sub‐
1466 mission to.
1467 • CTest Script variable: CTEST_SUBMIT_URL
1469 • CTest module variable: SUBMIT_URL if set, else CTEST_SUB‐
1471 MIT_URL
1472 TriggerSite
1474 Legacy option. Not used.
1475 • CTest Script variable: CTEST_TRIGGER_SITE
1477 • CTest module variable: TRIGGER_SITE if set, else CTEST_TRIG‐
1479 GER_SITE
1480
1482 When the --show-only=json-v1 command line option is given, the test in‐
1483 formation is output in JSON format. Version 1.0 of the JSON object
1484 model is defined as follows:
1485 kind The string "ctestInfo".
1487 version
1489 A JSON object specifying the version components. Its members
1490 are
1491 major A non-negative integer specifying the major version com‐
1493 ponent.
1494 minor A non-negative integer specifying the minor version com‐
1496 ponent.
1497 backtraceGraph
1499 JSON object representing backtrace information with the follow‐
1500 ing members:
1501 commands
1503 List of command names.
1504 files List of file names.
1506 nodes List of node JSON objects with members:
1508 command
1510 Index into the commands member of the backtrace‐
1511 Graph.
1512 file Index into the files member of the backtraceGraph.
1514 line Line number in the file where the backtrace was
1516 added.
1517 parent Index into the nodes member of the backtraceGraph
1519 representing the parent in the graph.
1520 tests A JSON array listing information about each test. Each entry is
1522 a JSON object with members:
1523 name Test name.
1525 config Configuration that the test can run on. Empty string
1527 means any config.
1528 command
1530 List where the first element is the test command and the
1531 remaining elements are the command arguments.
1532 backtrace
1534 Index into the nodes member of the backtraceGraph.
1535 properties
1537 Test properties. Can contain keys for each of the sup‐
1538 ported test properties.
1539
1541 CTest provides a mechanism for tests to specify the resources that they
1542 need in a fine-grained way, and for users to specify the resources
1543 available on the running machine. This allows CTest to internally keep
1544 track of which resources are in use and which are free, scheduling
1545 tests in a way that prevents them from trying to claim resources that
1546 are not available.
1547 When the resource allocation feature is used, CTest will not oversub‐
1549 scribe resources. For example, if a resource has 8 slots, CTest will
1550 not run tests that collectively use more than 8 slots at a time. This
1551 has the effect of limiting how many tests can run at any given time,
1552 even if a high -j argument is used, if those tests all use some slots
1553 from the same resource. In addition, it means that a single test that
1554 uses more of a resource than is available on a machine will not run at
1555 all (and will be reported as Not Run).
1556 A common use case for this feature is for tests that require the use of
1558 a GPU. Multiple tests can simultaneously allocate memory from a GPU,
1559 but if too many tests try to do this at once, some of them will fail to
1560 allocate, resulting in a failed test, even though the test would have
1561 succeeded if it had the memory it needed. By using the resource alloca‐
1562 tion feature, each test can specify how much memory it requires from a
1563 GPU, allowing CTest to schedule tests in a way that running several of
1564 these tests at once does not exhaust the GPU's memory pool.
1565 Please note that CTest has no concept of what a GPU is or how much mem‐
1567 ory it has, nor does it have any way of communicating with a GPU to re‐
1568 trieve this information or perform any memory management. CTest simply
1569 keeps track of a list of abstract resource types, each of which has a
1570 certain number of slots available for tests to use. Each test specifies
1571 the number of slots that it requires from a certain resource, and CTest
1572 then schedules them in a way that prevents the total number of slots in
1573 use from exceeding the listed capacity. When a test is executed, and
1574 slots from a resource are allocated to that test, tests may assume that
1575 they have exclusive use of those slots for the duration of the test's
1576 process.
1577 The CTest resource allocation feature consists of two inputs:
1579 • The resource specification file, described below, which describes the
1581 resources available on the system.
1582 • The RESOURCE_GROUPS property of tests, which describes the resources
1584 required by the test.
1585 When CTest runs a test, the resources allocated to that test are passed
1587 in the form of a set of environment variables as described below. Using
1588 this information to decide which resource to connect to is left to the
1589 test writer.
1590 The RESOURCE_GROUPS property tells CTest what resources a test expects
1592 to use grouped in a way meaningful to the test. The test itself must
1593 read the environment variables to determine which resources have been
1594 allocated to each group. For example, each group may correspond to a
1595 process the test will spawn when executed.
1596 Note that even if a test specifies a RESOURCE_GROUPS property, it is
1598 still possible for that to test to run without any resource allocation
1599 (and without the corresponding environment variables) if the user does
1600 not pass a resource specification file. Passing this file, either
1601 through the --resource-spec-file command-line argument or the RE‐
1602 SOURCE_SPEC_FILE argument to ctest_test(), is what activates the re‐
1603 source allocation feature. Tests should check the CTEST_RE‐
1604 SOURCE_GROUP_COUNT environment variable to find out whether or not re‐
1605 source allocation is activated. This variable will always (and only) be
1606 defined if resource allocation is activated. If resource allocation is
1607 not activated, then the CTEST_RESOURCE_GROUP_COUNT variable will not
1608 exist, even if it exists for the parent ctest process. If a test abso‐
1609 lutely must have resource allocation, then it can return a failing exit
1610 code or use the SKIP_RETURN_CODE or SKIP_REGULAR_EXPRESSION properties
1611 to indicate a skipped test.
1612 Resource Specification File
1614 The resource specification file is a JSON file which is passed to
1615 CTest, either on the ctest(1) command line as --resource-spec-file, or
1616 as the RESOURCE_SPEC_FILE argument of ctest_test(). If a dashboard
1617 script is used and RESOURCE_SPEC_FILE is not specified, the value of
1618 CTEST_RESOURCE_SPEC_FILE in the dashboard script is used instead. If
1619 --resource-spec-file, RESOURCE_SPEC_FILE, and CTEST_RESOURCE_SPEC_FILE
1620 in the dashboard script are not specified, the value of CTEST_RE‐
1621 SOURCE_SPEC_FILE in the CMake build is used instead. If none of these
1622 are specified, no resource spec file is used.
1623 The resource specification file must be a JSON object. All examples in
1625 this document assume the following resource specification file:
1626 {
1628 "version": {
1629 "major": 1,
1630 "minor": 0
1631 },
1632 "local": [
1633 {
1634 "gpus": [
1635 {
1636 "id": "0",
1637 "slots": 2
1638 },
1639 {
1640 "id": "1",
1641 "slots": 4
1642 },
1643 {
1644 "id": "2",
1645 "slots": 2
1646 },
1647 {
1648 "id": "3"
1649 }
1650 ],
1651 "crypto_chips": [
1652 {
1653 "id": "card0",
1654 "slots": 4
1655 }
1656 ]
1657 }
1658 ]
1659 }
1660 The members are:
1662 version
1664 An object containing a major integer field and a minor integer
1665 field. Currently, the only supported version is major 1, minor
1666 0. Any other value is an error.
1667 local A JSON array of resource sets present on the system. Currently,
1669 this array is restricted to being of size 1.
1670 Each array element is a JSON object with members whose names are
1672 equal to the desired resource types, such as gpus. These names
1673 must start with a lowercase letter or an underscore, and subse‐
1674 quent characters can be a lowercase letter, a digit, or an un‐
1675 derscore. Uppercase letters are not allowed, because certain
1676 platforms have case-insensitive environment variables. See the
1677 Environment Variables section below for more information. It is
1678 recommended that the resource type name be the plural of a noun,
1679 such as gpus or crypto_chips (and not gpu or crypto_chip.)
1680 Please note that the names gpus and crypto_chips are just exam‐
1682 ples, and CTest does not interpret them in any way. You are free
1683 to make up any resource type you want to meet your own require‐
1684 ments.
1685 The value for each resource type is a JSON array consisting of
1687 JSON objects, each of which describe a specific instance of the
1688 specified resource. These objects have the following members:
1689 id A string consisting of an identifier for the resource.
1691 Each character in the identifier can be a lowercase let‐
1692 ter, a digit, or an underscore. Uppercase letters are
1693 not allowed.
1694 Identifiers must be unique within a resource type. How‐
1696 ever, they do not have to be unique across resource
1697 types. For example, it is valid to have a gpus resource
1698 named 0 and a crypto_chips resource named 0, but not two
1699 gpus resources both named 0.
1700 Please note that the IDs 0, 1, 2, 3, and card0 are just
1702 examples, and CTest does not interpret them in any way.
1703 You are free to make up any IDs you want to meet your own
1704 requirements.
1705 slots An optional unsigned number specifying the number of
1707 slots available on the resource. For example, this could
1708 be megabytes of RAM on a GPU, or cryptography units
1709 available on a cryptography chip. If slots is not speci‐
1710 fied, a default value of 1 is assumed.
1711 In the example file above, there are four GPUs with ID's 0 through 3.
1713 GPU 0 has 2 slots, GPU 1 has 4, GPU 2 has 2, and GPU 3 has a default of
1714 1 slot. There is also one cryptography chip with 4 slots.
1715 RESOURCE_GROUPS Property
1717 See RESOURCE_GROUPS for a description of this property.
1718 Environment Variables
1720 Once CTest has decided which resources to allocate to a test, it passes
1721 this information to the test executable as a series of environment
1722 variables. For each example below, we will assume that the test in
1723 question has a RESOURCE_GROUPS property of
1724 2,gpus:2;gpus:4,gpus:1,crypto_chips:2.
1725 The following variables are passed to the test process:
1727 CTEST_RESOURCE_GROUP_COUNT
1729 The total number of groups specified by the RESOURCE_GROUPS
1730 property. For example:
1731 • CTEST_RESOURCE_GROUP_COUNT=3
1733 This variable will only be defined if ctest(1) has been given a
1735 --resource-spec-file, or if ctest_test() has been given a RE‐
1736 SOURCE_SPEC_FILE. If no resource specification file has been
1737 given, this variable will not be defined.
1738 CTEST_RESOURCE_GROUP_<num>
1740 The list of resource types allocated to each group, with each
1741 item separated by a comma. <num> is a number from zero to
1742 CTEST_RESOURCE_GROUP_COUNT minus one. CTEST_RESOURCE_GROUP_<num>
1743 is defined for each <num> in this range. For example:
1744 • CTEST_RESOURCE_GROUP_0=gpus
1746 • CTEST_RESOURCE_GROUP_1=gpus
1748 • CTEST_RESOURCE_GROUP_2=crypto_chips,gpus
1750 CTEST_RESOURCE_GROUP_<num>_<resource-type>
1752 The list of resource IDs and number of slots from each ID allo‐
1753 cated to each group for a given resource type. This variable
1754 consists of a series of pairs, each pair separated by a semi‐
1755 colon, and with the two items in the pair separated by a comma.
1756 The first item in each pair is id: followed by the ID of a re‐
1757 source of type <resource-type>, and the second item is slots:
1758 followed by the number of slots from that resource allocated to
1759 the given group. For example:
1760 • CTEST_RESOURCE_GROUP_0_GPUS=id:0,slots:2
1762 • CTEST_RESOURCE_GROUP_1_GPUS=id:2,slots:2
1764 • CTEST_RESOURCE_GROUP_2_GPUS=id:1,slots:4;id:3,slots:1
1766 • CTEST_RESOURCE_GROUP_2_CRYPTO_CHIPS=id:card0,slots:2
1768 In this example, group 0 gets 2 slots from GPU 0, group 1 gets 2
1770 slots from GPU 2, and group 2 gets 4 slots from GPU 1, 1 slot
1771 from GPU 3, and 2 slots from cryptography chip card0.
1772 <num> is a number from zero to CTEST_RESOURCE_GROUP_COUNT minus
1774 one. <resource-type> is the name of a resource type, converted
1775 to uppercase. CTEST_RESOURCE_GROUP_<num>_<resource-type> is de‐
1776 fined for the product of each <num> in the range listed above
1777 and each resource type listed in CTEST_RESOURCE_GROUP_<num>.
1778 Because some platforms have case-insensitive names for environ‐
1780 ment variables, the names of resource types may not clash in a
1781 case-insensitive environment. Because of this, for the sake of
1782 simplicity, all resource types must be listed in all lowercase
1783 in the resource specification file and in the RESOURCE_GROUPS
1784 property, and they are converted to all uppercase in the
1785 CTEST_RESOURCE_GROUP_<num>_<resource-type> environment variable.
1786
1788 The following resources are available to get help using CMake:
1789 Home Page
1791 https://cmake.org
1792 The primary starting point for learning about CMake.
1794 Online Documentation and Community Resources
1796 https://cmake.org/documentation
1797 Links to available documentation and community resources may be
1799 found on this web page.
1800 Discourse Forum
1802 https://discourse.cmake.org
1803 The Discourse Forum hosts discussion and questions about CMake.
1805
1807 2000-2021 Kitware, Inc. and Contributors
18083.20.3 May 30, 2021 CTEST(1)