1CMDTEST(1) General Commands Manual CMDTEST(1)
2
6 cmdtest - blackbox testing of Unix command line tools
7
9 cmdtest [-cCOMMAND] [--command=COMMAND] [--config=FILE] [--dump-config]
10 [--dump-setting-names] [--generate-manpage=TEMPLATE] [-h] [--help]
11 [--help-all] [--list-config-files] [--version] [--no-default-configs]
12 [--dump-memory-profile=METHOD] [-k] [--keep] [--no-keep] [--log=FILE]
13 [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE]
14 [--memory-dump-interval=SECONDS] [--output=FILE] [-tTEST] [--test=TEST]
15 [--timings] [--no-timings] [FILE]...
16
18 cmdtest black box tests Unix command line tools. Given some test
19 scripts, their inputs, and expected outputs, it verifies that the com‐
20 mand line produces the expected output. If not, it reports problems,
21 and shows the differences.
22 Each test case foo consists of the following files:
24 foo.script
26 a script to run the test (this is required)
27 foo.stdin
29 the file fed to standard input
30 foo.stdout
32 the expected output to the standard output
33 foo.stderr
35 the expected output to the standard error
36 foo.exit
38 the expected exit code
39 foo.setup
41 a shell script to run before the test
42 foo.teardown
44 a shell script to run after test
45 Usually, a single test is not enough. All tests are put into the same
47 directory, and they may share some setup and teardown code:
48 setup-once
50 a shell script to run once, before any tests
51 setup a shell script to run before each test
53 teardown
55 a shell script to run after each test
56 teardown-once
58 a shell script to run once, after all tests
59 cmdtest is given the name of the directory with all the tests, or sev‐
61 eral such directories, and it does the following:
62 · execute setup-once
64 · for each test case (unique prefix foo):
66 — execute setup
68 — execute foo.setup
70 — execute the command, by running foo.script, and redirecting
72 standard input to come from foo.stdin, and capturing standard
73 output and error and exit codes
74 — execute foo.teardown
76 — execute teardown
78 — report result of test: does exit code match foo.exit, standard
80 output match foo.stdout, and standard error match foo.stderr?
81 · execute teardown-once
83 Except for foo.script, all of these files are optional. If a setup or
85 teardown script is missing, it is simply not executed. If one of the
86 standard input, output, or error files is missing, it is treated as if
87 it were empty. If the exit code file is missing, it is treated as if
88 it specified an exit code of zero.
89 The shell scripts may use the following environment variables:
91 DATADIR
93 a temporary directory where files may be created by the test
94 TESTNAME
96 name of the current test (will be empty for setup-once and tear‐
97 down-once)
98 SRCDIR directory from which cmdtest was launched
100
102 -c, --command=COMMAND
103 ignored for backwards compatibility
104 --generate-manpage=TEMPLATE
106 fill in manual page TEMPLATE
107 -h, --help
109 show this help message and exit
110 -k, --keep
112 keep temporary data on failure
113 --no-keep
115 opposite of --keep
116 --output=FILE
118 write output to FILE, instead of standard output
119 -t, --test=TEST
121 run only TEST (can be given many times)
122 --timings
124 report how long each test takes
125 --no-timings
127 opposite of --timings
128 --version
130 show program's version number and exit
131 Configuration files and settings
133 --config=FILE
134 add FILE to config files
135 --dump-config
137 write out the entire current configuration
138 --dump-setting-names
140 write out all names of settings and quit
141 --help-all
143 show all options
144 --list-config-files
146 list all possible config files
147 --no-default-configs
149 clear list of configuration files to read
150 Logging
152 --log=FILE
153 write log entries to FILE (default is to not write log files at
154 all); use "syslog" to log to system log, "stderr" to log to the
155 standard error output, or "none" to disable logging
156 --log-keep=N
158 keep last N logs (10)
159 --log-level=LEVEL
161 log at LEVEL, one of debug, info, warning, error, critical, fa‐
162 tal (default: debug)
163 --log-max=SIZE
165 rotate logs larger than SIZE, zero for never (default: 0)
166 --log-mode=MODE
168 set permissions of new log files to MODE (octal; default 0600)
169 Peformance
171 --dump-memory-profile=METHOD
172 make memory profiling dumps using METHOD, which is one of: none,
173 simple, or meliae (default: simple)
174 --memory-dump-interval=SECONDS
176 make memory profiling dumps at least SECONDS apart
177
179 To test that the echo(1) command outputs the expected string, create a
180 file called echo-tests/hello.script containing the following content:
181 #!/bin/sh
183 echo hello, world
184 Also create the file echo-tests/hello.stdout containing:
186 hello, world
188 Then you can run the tests:
190 $ cmdtest echo-tests
192 test 1/1
193 1/1 tests OK, 0 failures
194 If you change the stdout file to be something else, cmdtest will report
196 the differences:
197 $ cmdtest echo-tests
199 FAIL: hello: stdout diff:
200 --- echo-tests/hello.stdout 2011-09-11 19:14:47 +0100
201 +++ echo-tests/hello.stdout-actual 2011-09-11 19:14:49 +0100
202 @@ -1 +1 @@
203 -something else
204 +hello, world
205 test 1/1
207 0/1 tests OK, 1 failures
208 Furthermore, the echo-tests directory will contain the actual output
210 files, and diffs from the expected files. If one of the actual output
211 files is actually correct, you can actualy rename it to be the expected
212 file. Actually, that's a very convenient way of creating the expected
213 output files: you run the test, fixing things, until you've manually
214 checked the actual output is correct, then you rename the file.
215
217 cliapp(5).
218 CMDTEST(1)