1tcltest(n) Tcl Bundled Packages tcltest(n)
2______________________________________________________________________________
6
8 tcltest - Test harness support code and utilities
9
11 package require tcltest ?2.2.5?
12 tcltest::test name description ?option value ...?
14 tcltest::test name description ?constraints? body result
15 tcltest::loadTestedCommands
17 tcltest::makeDirectory name ?directory?
18 tcltest::removeDirectory name ?directory?
19 tcltest::makeFile contents name ?directory?
20 tcltest::removeFile name ?directory?
21 tcltest::viewFile name ?directory?
22 tcltest::cleanupTests ?runningMultipleTests?
23 tcltest::runAllTests
24 tcltest::configure
26 tcltest::configure option
27 tcltest::configure option value ?option value ...?
28 tcltest::customMatch mode command
29 tcltest::testConstraint constraint ?value?
30 tcltest::outputChannel ?channelID?
31 tcltest::errorChannel ?channelID?
32 tcltest::interpreter ?interp?
33 tcltest::debug ?level?
35 tcltest::errorFile ?filename?
36 tcltest::limitConstraints ?boolean?
37 tcltest::loadFile ?filename?
38 tcltest::loadScript ?script?
39 tcltest::match ?patternList?
40 tcltest::matchDirectories ?patternList?
41 tcltest::matchFiles ?patternList?
42 tcltest::outputFile ?filename?
43 tcltest::preserveCore ?level?
44 tcltest::singleProcess ?boolean?
45 tcltest::skip ?patternList?
46 tcltest::skipDirectories ?patternList?
47 tcltest::skipFiles ?patternList?
48 tcltest::temporaryDirectory ?directory?
49 tcltest::testsDirectory ?directory?
50 tcltest::verbose ?level?
51 tcltest::test name description optionList
53 tcltest::bytestring string
54 tcltest::normalizeMsg msg
55 tcltest::normalizePath pathVar
56 tcltest::workingDirectory ?dir?
57_________________________________________________________________
58
60 The tcltest package provides several utility commands useful in the
61 construction of test suites for code instrumented to be run by evalua‐
62 tion of Tcl commands. Notably the built-in commands of the Tcl library
63 itself are tested by a test suite using the tcltest package.
64 All the commands provided by the tcltest package are defined in and
66 exported from the ::tcltest namespace, as indicated in the SYNOPSIS
67 above. In the following sections, all commands will be described by
68 their simple names, in the interest of brevity.
69 The central command of tcltest is [test] that defines and runs a test.
71 Testing with [test] involves evaluation of a Tcl script and comparing
72 the result to an expected result, as configured and controlled by a
73 number of options. Several other commands provided by tcltest govern
74 the configuration of [test] and the collection of many [test] commands
75 into test suites.
76 See CREATING TEST SUITES WITH TCLTEST below for an extended example of
78 how to use the commands of tcltest to produce test suites for your Tcl-
79 enabled code.
80
82 test name description ?option value ...?
83 Defines and possibly runs a test with the name name and descrip‐
84 tion description. The name and description of a test are used
85 in messages reported by [test] during the test, as configured by
86 the options of tcltest. The remaining option value arguments to
87 [test] define the test, including the scripts to run, the condi‐
88 tions under which to run them, the expected result, and the
89 means by which the expected and actual results should be com‐
90 pared. See TESTS below for a complete description of the valid
91 options and how they define a test. The [test] command returns
92 an empty string.
93 test name description ?constraints? body result
95 This form of [test] is provided to support test suites written
96 for version 1 of the tcltest package, and also a simpler inter‐
97 face for a common usage. It is the same as [test name descrip‐
98 tion -constraints constraints -body body -result result]. All
99 other options to [test] take their default values. When con‐
100 straints is omitted, this form of [test] can be distinguished
101 from the first because all options begin with ``-''.
102 loadTestedCommands
104 Evaluates in the caller's context the script specified by [con‐
105 figure -load] or [configure -loadfile]. Returns the result of
106 that script evaluation, including any error raised by the
107 script. Use this command and the related configuration options
108 to provide the commands to be tested to the interpreter running
109 the test suite.
110 makeFile contents name ?directory?
112 Creates a file named name relative to directory directory and
113 write contents to that file using the encoding [encoding sys‐
114 tem]. If contents does not end with a newline, a newline will
115 be appended so that the file named name does end with a newline.
116 Because the system encoding is used, this command is only suit‐
117 able for making text files. The file will be removed by the
118 next evaluation of [cleanupTests], unless it is removed by
119 [removeFile] first. The default value of directory is the
120 directory [configure -tmpdir]. Returns the full path of the
121 file created. Use this command to create any text file required
122 by a test with contents as needed.
123 removeFile name ?directory?
125 Forces the file referenced by name to be removed. This file
126 name should be relative to directory. The default value of
127 directory is the directory [configure -tmpdir]. Returns an
128 empty string. Use this command to delete files created by
129 [makeFile].
130 makeDirectory name ?directory?
132 Creates a directory named name relative to directory directory.
133 The directory will be removed by the next evaluation of
134 [cleanupTests], unless it is removed by [removeDirectory] first.
135 The default value of directory is the directory [configure
136 -tmpdir]. Returns the full path of the directory created. Use
137 this command to create any directories that are required to
138 exist by a test.
139 removeDirectory name ?directory?
141 Forces the directory referenced by name to be removed. This
142 directory should be relative to directory. The default value of
143 directory is the directory [configure -tmpdir]. Returns an
144 empty string. Use this command to delete any directories cre‐
145 ated by [makeDirectory].
146 viewFile file ?directory?
148 Returns the contents of file, except for any final newline, just
149 as [read -nonewline] would return. This file name should be
150 relative to directory. The default value of directory is the
151 directory [configure -tmpdir]. Use this command as a convenient
152 way to turn the contents of a file generated by a test into the
153 result of that test for matching against an expected result.
154 The contents of the file are read using the system encoding, so
155 its usefulness is limited to text files.
156 cleanupTests
158 Intended to clean up and summarize after several tests have been
159 run. Typically called once per test file, at the end of the
160 file after all tests have been completed. For best effective‐
161 ness, be sure that the [cleanupTests] is evaluated even if an
162 error occurs earlier in the test file evaluation.
163 Prints statistics about the tests run and removes files that
165 were created by [makeDirectory] and [makeFile] since the last
166 [cleanupTests]. Names of files and directories in the directory
167 [configure -tmpdir] created since the last [cleanupTests], but
168 not created by [makeFile] or [makeDirectory] are printed to
169 [outputChannel]. This command also restores the original shell
170 environment, as described by the ::env array. Returns an empty
171 string.
172 runAllTests
174 This is a master command meant to run an entire suite of tests,
175 spanning multiple files and/or directories, as governed by the
176 configurable options of tcltest. See RUNNING ALL TESTS below
177 for a complete description of the many variations possible with
178 [runAllTests].
179
181 configure
182 Returns the list of configurable options supported by tcltest.
183 See CONFIGURABLE OPTIONS below for the full list of options,
184 their valid values, and their effect on tcltest operations.
185 configure option
187 Returns the current value of the supported configurable option
188 option. Raises an error if option is not a supported config‐
189 urable option.
190 configure option value ?option value ...?
192 Sets the value of each configurable option option to the corre‐
193 sponding value value, in order. Raises an error if an option is
194 not a supported configurable option, or if value is not a valid
195 value for the corresponding option, or if a value is not pro‐
196 vided. When an error is raised, the operation of [configure] is
197 halted, and subsequent option value arguments are not processed.
198 If the environment variable ::env(TCLTEST_OPTIONS) exists when
200 the tcltest package is loaded (by [package require tcltest])
201 then its value is taken as a list of arguments to pass to [con‐
202 figure]. This allows the default values of the configuration
203 options to be set by the environment.
204 customMatch mode script
206 Registers mode as a new legal value of the -match option to
207 [test]. When the -match mode option is passed to [test], the
208 script script will be evaluated to compare the actual result of
209 evaluating the body of the test to the expected result. To per‐
210 form the match, the script is completed with two additional
211 words, the expected result, and the actual result, and the com‐
212 pleted script is evaluated in the global namespace. The com‐
213 pleted script is expected to return a boolean value indicating
214 whether or not the results match. The built-in matching modes
215 of [test] are exact, glob, and regexp.
216 testConstraint constraint ?boolean?
218 Sets or returns the boolean value associated with the named con‐
219 straint. See TEST CONSTRAINTS below for more information.
220 interpreter ?executableName?
222 Sets or returns the name of the executable to be [exec]ed by
223 [runAllTests] to run each test file when [configure -singleproc]
224 is false. The default value for [interpreter] is the name of
225 the currently running program as returned by [info nameofexe‐
226 cutable].
227 outputChannel ?channelID?
229 Sets or returns the output channel ID. This defaults to stdout.
230 Any test that prints test related output should send that output
231 to [outputChannel] rather than letting that output default to
232 stdout.
233 errorChannel ?channelID?
235 Sets or returns the error channel ID. This defaults to stderr.
236 Any test that prints error messages should send that output to
237 [errorChannel] rather than printing directly to stderr.
238
240 debug ?level?
241 Same as [configure -debug ?level?].
242 errorFile ?filename?
244 Same as [configure -errfile ?filename?].
245 limitConstraints ?boolean?
247 Same as [configure -limitconstraints ?boolean?].
248 loadFile ?filename?
250 Same as [configure -loadfile ?filename?].
251 loadScript ?script?
253 Same as [configure -load ?script?].
254 match ?patternList?
256 Same as [configure -match ?patternList?].
257 matchDirectories ?patternList?
259 Same as [configure -relateddir ?patternList?].
260 matchFiles ?patternList?
262 Same as [configure -file ?patternList?].
263 outputFile ?filename?
265 Same as [configure -outfile ?filename?].
266 preserveCore ?level?
268 Same as [configure -preservecore ?level?].
269 singleProcess ?boolean?
271 Same as [configure -singleproc ?boolean?].
272 skip ?patternList?
274 Same as [configure -skip ?patternList?].
275 skipDirectories ?patternList?
277 Same as [configure -asidefromdir ?patternList?].
278 skipFiles ?patternList?
280 Same as [configure -notfile ?patternList?].
281 temporaryDirectory ?directory?
283 Same as [configure -tmpdir ?directory?].
284 testsDirectory ?directory?
286 Same as [configure -testdir ?directory?].
287 verbose ?level?
289 Same as [configure -verbose ?level?].
290
292 The remaining commands provided by tcltest have better alternatives
293 provided by tcltest or Tcl itself. They are retained to support exist‐
294 ing test suites, but should be avoided in new code.
295 test name description optionList
297 This form of [test] was provided to enable passing many options
298 spanning several lines to [test] as a single argument quoted by
299 braces, rather than needing to backslash quote the newlines
300 between arguments to [test]. The optionList argument is
301 expected to be a list with an even number of elements represent‐
302 ing option and value arguments to pass to [test]. However,
303 these values are not passed directly, as in the alternate forms
304 of [switch]. Instead, this form makes an unfortunate attempt to
305 overthrow Tcl's substitution rules by performing substitutions
306 on some of the list elements as an attempt to implement a ``do
307 what I mean'' interpretation of a brace-enclosed ``block''. The
308 result is nearly impossible to document clearly, and for that
309 reason this form is not recommended. See the examples in CREAT‐
310 ING TEST SUITES WITH TCLTEST below to see that this form is
311 really not necessary to avoid backslash-quoted newlines. If you
312 insist on using this form, examine the source code of tcltest if
313 you want to know the substitution details, or just enclose the
314 third through last argument to [test] in braces and hope for the
315 best.
316 workingDirectory ?directoryName?
318 Sets or returns the current working directory when the test
319 suite is running. The default value for workingDirectory is the
320 directory in which the test suite was launched. The Tcl com‐
321 mands [cd] and [pwd] are sufficient replacements.
322 normalizeMsg msg
324 Returns the result of removing the ``extra'' newlines from msg,
325 where ``extra'' is rather imprecise. Tcl offers plenty of
326 string processing commands to modify strings as you wish, and
327 [customMatch] allows flexible matching of actual and expected
328 results.
329 normalizePath pathVar
331 Resolves symlinks in a path, thus creating a path without inter‐
332 nal redirection. It is assumed that pathVar is absolute. path‐
333 Var is modified in place. The Tcl command [file normalize] is a
334 sufficient replacement.
335 bytestring string
337 Construct a string that consists of the requested sequence of
338 bytes, as opposed to a string of properly formed UTF-8 charac‐
339 ters using the value supplied in string. This allows the tester
340 to create denormalized or improperly formed strings to pass to C
341 procedures that are supposed to accept strings with embedded
342 NULL types and confirm that a string result has a certain pat‐
343 tern of bytes. This is exactly equivalent to the Tcl command
344 [encoding convertfrom identity].
345
347 The [test] command is the heart of the tcltest package. Its essential
348 function is to evaluate a Tcl script and compare the result with an
349 expected result. The options of [test] define the test script, the
350 environment in which to evaluate it, the expected result, and how the
351 compare the actual result to the expected result. Some configuration
352 options of tcltest also influence how [test] operates.
353 The valid options for [test] are summarized:
355 test name description
356 ?-constraints keywordList|expression?
357 ?-setup setupScript?
358 ?-body testScript?
359 ?-cleanup cleanupScript?
360 ?-result expectedAnswer?
361 ?-output expectedOutput?
362 ?-errorOutput expectedError?
363 ?-returnCodes codeList?
364 ?-match mode?
365 The name may be any string. It is conventional to choose a name
366 according to the pattern:
367 target-majorNum.minorNum
368 For white-box (regression) tests, the target should be the name of the
369 C function or Tcl procedure being tested. For black-box tests, the
370 target should be the name of the feature being tested. Some conven‐
371 tions call for the names of black-box tests to have the suffix _bb.
372 Related tests should share a major number. As a test suite evolves, it
373 is best to have the same test name continue to correspond to the same
374 test, so that it remains meaningful to say things like ``Test foo-1.3
375 passed in all releases up to 3.4, but began failing in release 3.5.''
376 During evaluation of [test], the name will be compared to the lists of
378 string matching patterns returned by [configure -match], and [configure
379 -skip]. The test will be run only if name matches any of the patterns
380 from [configure -match] and matches none of the patterns from [config‐
381 ure -skip].
382 The description should be a short textual description of the test. The
384 description is included in output produced by the test, typically test
385 failure messages. Good description values should briefly explain the
386 purpose of the test to users of a test suite. The name of a Tcl or C
387 function being tested should be included in the description for regres‐
388 sion tests. If the test case exists to reproduce a bug, include the
389 bug ID in the description.
390 Valid attributes and associated values are:
392 -constraints keywordList|expression
394 The optional -constraints attribute can be list of one or more
395 keywords or an expression. If the -constraints value is a list
396 of keywords, each of these keywords should be the name of a con‐
397 straint defined by a call to [testConstraint]. If any of the
398 listed constraints is false or does not exist, the test is
399 skipped. If the -constraints value is an expression, that
400 expression is evaluated. If the expression evaluates to true,
401 then the test is run. Note that the expression form of -con‐
402 straints may interfere with the operation of [configure -con‐
403 straints] and [configure -limitconstraints], and is not recom‐
404 mended. Appropriate constraints should be added to any tests
405 that should not always be run. That is, conditional evaluation
406 of a test should be accomplished by the -constraints option, not
407 by conditional evaluation of [test]. In that way, the same num‐
408 ber of tests are always reported by the test suite, though the
409 number skipped may change based on the testing environment. The
410 default value is an empty list. See TEST CONSTRAINTS below for
411 a list of built-in constraints and information on how to add
412 your own constraints.
413 -setup script
415 The optional -setup attribute indicates a script that will be
416 run before the script indicated by the -body attribute. If
417 evaluation of script raises an error, the test will fail. The
418 default value is an empty script.
419 -body script
421 The -body attribute indicates the script to run to carry out the
422 test. It must return a result that can be checked for correct‐
423 ness. If evaluation of script raises an error, the test will
424 fail. The default value is an empty script.
425 -cleanup script
427 The optional -cleanup attribute indicates a script that will be
428 run after the script indicated by the -body attribute. If eval‐
429 uation of script raises an error, the test will fail. The
430 default value is an empty script.
431 -match mode
433 The -match attribute determines how expected answers supplied by
434 -result, -output, and -errorOutput are compared. Valid values
435 for mode are regexp, glob, exact, and any value registered by a
436 prior call to [customMatch]. The default value is exact.
437 -result expectedValue
439 The -result attribute supplies the expectedValue against which
440 the return value from script will be compared. The default value
441 is an empty string.
442 -output expectedValue
444 The -output attribute supplies the expectedValue against which
445 any output sent to stdout or [outputChannel] during evaluation
446 of the script(s) will be compared. Note that only output
447 printed using [::puts] is used for comparison. If -output is
448 not specified, output sent to stdout and [outputChannel] is not
449 processed for comparison.
450 -errorOutput expectedValue
452 The -errorOutput attribute supplies the expectedValue against
453 which any output sent to stderr or [errorChannel] during evalua‐
454 tion of the script(s) will be compared. Note that only output
455 printed using [::puts] is used for comparison. If -errorOutput
456 is not specified, output sent to stderr and [errorChannel] is
457 not processed for comparison.
458 -returnCodes expectedCodeList
460 The optional -returnCodes attribute supplies expectedCodeList, a
461 list of return codes that may be accepted from evaluation of the
462 -body script. If evaluation of the -body script returns a code
463 not in the expectedCodeList, the test fails. All return codes
464 known to [return], in both numeric and symbolic form, including
465 extended return codes, are acceptable elements in the expected‐
466 CodeList. Default value is {ok return}.
467 To pass, a test must successfully evaluate its -setup, -body, and
469 -cleanup scripts. The return code of the -body script and its result
470 must match expected values, and if specified, output and error data
471 from the test must match expected -output and -errorOutput values. If
472 any of these conditions are not met, then the test fails. Note that
473 all scripts are evaluated in the context of the caller of [test].
474 As long as [test] is called with valid syntax and legal values for all
476 attributes, it will not raise an error. Test failures are instead
477 reported as output written to [outputChannel]. In default operation, a
478 successful test produces no output. The output messages produced by
479 [test] are controlled by the [configure -verbose] option as described
480 in CONFIGURABLE OPTIONS below. Any output produced by the test scripts
481 themselves should be produced using [::puts] to [outputChannel] or
482 [errorChannel], so that users of the test suite may easily capture out‐
483 put with the [configure -outfile] and [configure -errfile] options, and
484 so that the -output and -errorOutput attributes work properly.
485
487 Constraints are used to determine whether or not a test should be
488 skipped. Each constraint has a name, which may be any string, and a
489 boolean value. Each [test] has a -constraints value which is a list of
490 constraint names. There are two modes of constraint control. Most
491 frequently, the default mode is used, indicated by a setting of [con‐
492 figure -limitconstraints] to false. The test will run only if all con‐
493 straints in the list are true-valued. Thus, the -constraints option of
494 [test] is a convenient, symbolic way to define any conditions required
495 for the test to be possible or meaningful. For example, a [test] with
496 -constraints unix will only be run if the constraint unix is true,
497 which indicates the test suite is being run on a Unix platform.
498 Each [test] should include whatever -constraints are required to con‐
500 strain it to run only where appropriate. Several constraints are pre-
501 defined in the tcltest package, listed below. The registration of
502 user-defined constraints is performed by the [testConstraint] command.
503 User-defined constraints may appear within a test file, or within the
504 script specified by the [configure -load] or [configure -loadfile]
505 options.
506 The following is a list of constraints pre-defined by the tcltest pack‐
508 age itself:
509 singleTestInterp
511 test can only be run if all test files are sourced into a single
512 interpreter
513 unix test can only be run on any Unix platform
515 win test can only be run on any Windows platform
517 nt test can only be run on any Windows NT platform
519 95 test can only be run on any Windows 95 platform
521 98 test can only be run on any Windows 98 platform
523 mac test can only be run on any Mac platform
525 unixOrWin
527 test can only be run on a Unix or Windows platform
528 macOrWin
530 test can only be run on a Mac or Windows platform
531 macOrUnix
533 test can only be run on a Mac or Unix platform
534 tempNotWin
536 test can not be run on Windows. This flag is used to temporar‐
537 ily disable a test.
538 tempNotMac
540 test can not be run on a Mac. This flag is used to temporarily
541 disable a test.
542 unixCrash
544 test crashes if it's run on Unix. This flag is used to tempo‐
545 rarily disable a test.
546 winCrash
548 test crashes if it's run on Windows. This flag is used to tem‐
549 porarily disable a test.
550 macCrash
552 test crashes if it's run on a Mac. This flag is used to tempo‐
553 rarily disable a test.
554 emptyTest
556 test is empty, and so not worth running, but it remains as a
557 place-holder for a test to be written in the future. This con‐
558 straint has value false to cause tests to be skipped unless the
559 user specifies otherwise.
560 knownBug
562 test is known to fail and the bug is not yet fixed. This con‐
563 straint has value false to cause tests to be skipped unless the
564 user specifies otherwise.
565 nonPortable
567 test can only be run in some known development environment.
568 Some tests are inherently non-portable because they depend on
569 things like word length, file system configuration, window man‐
570 ager, etc. This constraint has value false to cause tests to be
571 skipped unless the user specifies otherwise.
572 userInteraction
574 test requires interaction from the user. This constraint has
575 value false to causes tests to be skipped unless the user speci‐
576 fies otherwise.
577 interactive
579 test can only be run in if the interpreter is in interactive
580 mode (when the global tcl_interactive variable is set to 1).
581 nonBlockFiles
583 test can only be run if platform supports setting files into
584 nonblocking mode
585 asyncPipeClose
587 test can only be run if platform supports async flush and async
588 close on a pipe
589 unixExecs
591 test can only be run if this machine has Unix-style commands
592 cat, echo, sh, wc, rm, sleep, fgrep, ps, chmod, and mkdir avail‐
593 able
594 hasIsoLocale
596 test can only be run if can switch to an ISO locale
597 root test can only run if Unix user is root
599 notRoot
601 test can only run if Unix user is not root
602 eformat
604 test can only run if app has a working version of sprintf with
605 respect to the "e" format of floating-point numbers.
606 stdio test can only be run if [interpreter] can be [open]ed as a pipe.
608 The alternative mode of constraint control is enabled by setting [con‐
610 figure -limitconstraints] to true. With that configuration setting,
611 all existing constraints other than those in the constraint list
612 returned by [configure -constraints] are set to false. When the value
613 of [configure -constraints] is set, all those constraints are set to
614 true. The effect is that when both options [configure -constraints]
615 and [configure -limitconstraints] are in use, only those tests includ‐
616 ing only constraints from the [configure -constraints] list are run;
617 all others are skipped. For example, one might set up a configuration
618 with
619 configure -constraints knownBug \
620 -limitconstraints true \
621 -verbose pass
622 to run exactly those tests that exercise known bugs, and discover
623 whether any of them pass, indicating the bug had been fixed.
624
626 The single command [runAllTests] is evaluated to run an entire test
627 suite, spanning many files and directories. The configuration options
628 of tcltest control the precise operations. The [runAllTests] command
629 begins by printing a summary of its configuration to [outputChannel].
630 Test files to be evaluated are sought in the directory [configure
632 -testdir]. The list of files in that directory that match any of the
633 patterns in [configure -file] and match none of the patterns in [con‐
634 figure -notfile] is generated and sorted. Then each file will be eval‐
635 uated in turn. If [configure -singleproc] is true, then each file will
636 be [source]d in the caller's context. If it is false, then a copy of
637 [interpreter] will be [exec]d to evaluate each file. The multi-process
638 operation is useful when testing can cause errors so severe that a
639 process terminates. Although such an error may terminate a child
640 process evaluating one file, the master process can continue with the
641 rest of the test suite. In multi-process operation, the configuration
642 of tcltest in the master process is passed to the child processes as
643 command line arguments, with the exception of [configure -outfile].
644 The [runAllTests] command in the master process collects all output
645 from the child processes and collates their results into one master
646 report. Any reports of individual test failures, or messages requested
647 by a [configure -verbose] setting are passed directly on to [out‐
648 putChannel] by the master process.
649 After evaluating all selected test files, a summary of the results is
651 printed to [outputChannel]. The summary includes the total number of
652 [test]s evaluated, broken down into those skipped, those passed, and
653 those failed. The summary also notes the number of files evaluated,
654 and the names of any files with failing tests or errors. A list of the
655 constraints that caused tests to be skipped, and the number of tests
656 skipped for each is also printed. Also, messages are printed if it
657 appears that evaluation of a test file has caused any temporary files
658 to be left behind in [configure -tmpdir].
659 Having completed and summarized all selected test files, [runAllTests]
661 then recursively acts on subdirectories of [configure -testdir]. All
662 subdirectories that match any of the patterns in [configure -related‐
663 dir] and do not match any of the patterns in [configure -asidefromdir]
664 are examined. If a file named all.tcl is found in such a directory, it
665 will be [source]d in the caller's context. Whether or not an examined
666 directory contains an all.tcl file, its subdirectories are also scanned
667 against the [configure -relateddir] and [configure -asidefromdir] pat‐
668 terns. In this way, many directories in a directory tree can have all
669 their test files evaluated by a single [runAllTests] command.
670
672 The [configure] command is used to set and query the configurable
673 options of tcltest. The valid options are:
674 -singleproc boolean
676 Controls whether or not [runAllTests] spawns a child process for
677 each test file. No spawning when boolean is true. Default
678 value is false.
679 -debug level
681 Sets the debug level to level, an integer value indicating how
682 much debugging information should be printed to stdout. Note
683 that debug messages always go to stdout, independent of the
684 value of [configure -outfile]. Default value is 0. Levels are
685 defined as:
686 0 Do not display any debug information.
688 1 Display information regarding whether a test is skipped
690 because it doesn't match any of the tests that were spec‐
691 ified using by [configure -match] (userSpecifiedNonMatch)
692 or matches any of the tests specified by [configure
693 -skip] (userSpecifiedSkip). Also print warnings about
694 possible lack of cleanup or balance in test files. Also
695 print warnings about any re-use of test names.
696 2 Display the flag array parsed by the command line proces‐
698 sor, the contents of the ::env array, and all user-
699 defined variables that exist in the current namespace as
700 they are used.
701 3 Display information regarding what individual procs in
703 the test harness are doing.
704 -verbose level
706 Sets the type of output verbosity desired to level, a list of
707 zero or more of the elements body, pass, skip, start, and error.
708 Default value is {body error}. Levels are defined as:
709 body (b)
711 Display the body of failed tests
712 pass (p)
714 Print output when a test passes
715 skip (s)
717 Print output when a test is skipped
718 start (t)
720 Print output whenever a test starts
721 error (e)
723 Print errorInfo and errorCode, if they exist, when a test
724 return code does not match its expected return code
725 The single letter abbreviations noted above are also recognized so that
726 [configure -verbose pt] is the same as [configure -verbose {pass
727 start}].
728 -preservecore level
730 Sets the core preservation level to level. This level deter‐
731 mines how stringent checks for core files are. Default value is
732 0. Levels are defined as:
733 0 No checking - do not check for core files at the end of
735 each test command, but do check for them in [runAllTests]
736 after all test files have been evaluated.
737 1 Also check for core files at the end of each [test] com‐
739 mand.
740 2 Check for core files at all times described above, and
742 save a copy of each core file produced in [configure
743 -tmpdir].
744 -limitconstraints boolean
746 Sets the mode by which [test] honors constraints as described in
747 TESTS above. Default value is false.
748 -constraints list
750 Sets all the constraints in list to true. Also used in combina‐
751 tion with [configure -limitconstraints true] to control an
752 alternative constraint mode as described in TESTS above.
753 Default value is an empty list.
754 -tmpdir directory
756 Sets the temporary directory to be used by [makeFile], [makeDi‐
757 rectory], [viewFile], [removeFile], and [removeDirectory] as the
758 default directory where temporary files and directories created
759 by test files should be created. Default value is [workingDi‐
760 rectory].
761 -testdir directory
763 Sets the directory searched by [runAllTests] for test files and
764 subdirectories. Default value is [workingDirectory].
765 -file patternList
767 Sets the list of patterns used by [runAllTests] to determine
768 what test files to evaluate. Default value is *.test.
769 -notfile patternList
771 Sets the list of patterns used by [runAllTests] to determine
772 what test files to skip. Default value is l.*.test, so that any
773 SCCS lock files are skipped.
774 -relateddir patternList
776 Sets the list of patterns used by [runAllTests] to determine
777 what subdirectories to search for an all.tcl file. Default
778 value is *.
779 -asidefromdir patternList
781 Sets the list of patterns used by [runAllTests] to determine
782 what subdirectories to skip when searching for an all.tcl file.
783 Default value is an empty list.
784 -match patternList
786 Set the list of patterns used by [test] to determine whether a
787 test should be run. Default value is *.
788 -skip patternList
790 Set the list of patterns used by [test] to determine whether a
791 test should be skipped. Default value is an empty list.
792 -load script
794 Sets a script to be evaluated by [loadTestedCommands]. Default
795 value is an empty script.
796 -loadfile filename
798 Sets the filename from which to read a script to be evaluated by
799 [loadTestedCommands]. This is an alternative to -load. They
800 cannot be used together.
801 -outfile filename
803 Sets the file to which all output produced by tcltest should be
804 written. A file named filename will be [open]ed for writing,
805 and the resulting channel will be set as the value of [out‐
806 putChannel].
807 -errfile filename
809 Sets the file to which all error output produced by tcltest
810 should be written. A file named filename will be [open]ed for
811 writing, and the resulting channel will be set as the value of
812 [errorChannel].
813
815 The fundamental element of a test suite is the individual [test] com‐
816 mand. We begin with several examples.
817 [1] Test of a script that returns normally.
819 test example-1.0 {normal return} {
820 format %s value
821 } value
822 [2] Test of a script that requires context setup and cleanup. Note
824 the bracing and indenting style that avoids any need for line
825 continuation.
826 test example-1.1 {test file existence} -setup {
827 set file [makeFile {} test]
828 } -body {
829 file exists $file
830 } -cleanup {
831 removeFile test
832 } -result 1
833 [3] Test of a script that raises an error.
835 test example-1.2 {error return} -body {
836 error message
837 } -returnCodes error -result message
838 [4] Test with a constraint.
840 test example-1.3 {user owns created files} -constraints {
841 unix
842 } -setup {
843 set file [makeFile {} test]
844 } -body {
845 file attributes $file -owner
846 } -cleanup {
847 removeFile test
848 } -result $::tcl_platform(user)
849 At the next higher layer of organization, several [test] commands are
851 gathered together into a single test file. Test files should have
852 names with the .test extension, because that is the default pattern
853 used by [runAllTests] to find test files. It is a good rule of thumb
854 to have one test file for each source code file of your project. It is
855 good practice to edit the test file and the source code file together,
856 keeping tests synchronized with code changes.
857 Most of the code in the test file should be the [test] commands. Use
859 constraints to skip tests, rather than conditional evaluation of
860 [test]. That is, do this:
861 [5]
863 testConstraint X [expr $myRequirement]
864 test goodConditionalTest {} X {
865 # body
866 } result
867 and do not do this:
868 [6]
870 if $myRequirement {
871 test badConditionalTest {} {
872 #body
873 } result
874 }
875 Use the -setup and -cleanup options to establish and release all con‐
877 text requirements of the test body. Do not make tests depend on prior
878 tests in the file. Those prior tests might be skipped. If several
879 consecutive tests require the same context, the appropriate setup and
880 cleanup scripts may be stored in variable for passing to each tests
881 -setup and -cleanup options. This is a better solution than performing
882 setup outside of [test] commands, because the setup will only be done
883 if necessary, and any errors during setup will be reported, and not
884 cause the test file to abort.
885 A test file should be able to be combined with other test files and not
887 interfere with them, even when [configure -singleproc 1] causes all
888 files to be evaluated in a common interpreter. A simple way to achieve
889 this is to have your tests define all their commands and variables in a
890 namespace that is deleted when the test file evaluation is complete. A
891 good namespace to use is a child namespace test of the namespace of the
892 module you are testing.
893 A test file should also be able to be evaluated directly as a script,
895 not depending on being called by a master [runAllTests]. This means
896 that each test file should process command line arguments to give the
897 tester all the configuration control that tcltest provides.
898 After all [test]s in a test file, the command [cleanupTests] should be
900 called.
901 [7] Here is a sketch of a sample test file illustrating those
903 points:
904 package require tcltest 2.2
905 eval ::tcltest::configure $argv
906 package require example
907 namespace eval ::example::test {
908 namespace import ::tcltest::*
909 testConstraint X [expr {...}]
910 variable SETUP {#common setup code}
911 variable CLEANUP {#common cleanup code}
912 test example-1 {} -setup $SETUP -body {
913 # First test
914 } -cleanup $CLEANUP -result {...}
915 test example-2 {} -constraints X -setup $SETUP -body {
916 # Second test; constrained
917 } -cleanup $CLEANUP -result {...}
918 test example-3 {} {
919 # Third test; no context required
920 } {...}
921 cleanupTests
922 }
923 namespace delete ::example::test
924 The next level of organization is a full test suite, made up of several
926 test files. One script is used to control the entire suite. The basic
927 function of this script is to call [runAllTests] after doing any neces‐
928 sary setup. This script is usually named all.tcl because that's the
929 default name used by [runAllTests] when combining multiple test suites
930 into one testing run.
931 [8] Here is a sketch of a sample test suite master script:
933 package require Tcl 8.4
934 package require tcltest 2.2
935 package require example
936 ::tcltest::configure -testdir [file dirname [file normalize [info script]]]
937 eval ::tcltest::configure $argv
938 ::tcltest::runAllTests
939
941 A number of commands and variables in the ::tcltest namespace provided
942 by earlier releases of tcltest have not been documented here. They are
943 no longer part of the supported public interface of tcltest and should
944 not be used in new test suites. However, to continue to support exist‐
945 ing test suites written to the older interface specifications, many of
946 those deprecated commands and variables still work as before. For
947 example, in many circumstances, [configure] will be automatically
948 called shortly after [package require tcltest 2.1] succeeds with argu‐
949 ments from the variable ::argv. This is to support test suites that
950 depend on the old behavior that tcltest was automatically configured
951 from command line arguments. New test files should not depend on this,
952 but should explicitly include
953 eval ::tcltest::configure $::argv
954 to establish a configuration from command line arguments.
955
957 There are two known issues related to nested evaluations of [test].
958 The first issue relates to the stack level in which test scripts are
959 executed. Tests nested within other tests may be executed at the same
960 stack level as the outermost test. For example, in the following code:
961 test level-1.1 {level 1} {
962 -body {
963 test level-2.1 {level 2} {
964 }
965 }
966 }
967 any script executed in level-2.1 may be executed at the same stack
968 level as the script defined for level-1.1.
969 In addition, while two [test]s have been run, results will only be
971 reported by [cleanupTests] for tests at the same level as test
972 level-1.1. However, test results for all tests run prior to level-1.1
973 will be available when test level-2.1 runs. What this means is that if
974 you try to access the test results for test level-2.1, it will may say
975 that 'm' tests have run, 'n' tests have been skipped, 'o' tests have
976 passed and 'p' tests have failed, where 'm', 'n', 'o', and 'p' refer to
977 tests that were run at the same test level as test level-1.1.
978 Implementation of output and error comparison in the test command
980 depends on usage of ::puts in your application code. Output is inter‐
981 cepted by redefining the ::puts command while the defined test script
982 is being run. Errors thrown by C procedures or printed directly from C
983 applications will not be caught by the test command. Therefore, usage
984 of the -output and -errorOuput options to [test] is useful only for
985 pure Tcl applications that use [::puts] to produce output.
986
989 test, test harness, test suite
990tcltest 2.2 tcltest(n)