1CMDTEST(1)                  General Commands Manual                 CMDTEST(1)
2
3
4

NAME

6       cmdtest - blackbox testing of Unix command line tools
7

SYNOPSIS

9       cmdtest [-cCOMMAND] [--command=COMMAND] [--config=FILE] [--dump-config]
10       [--dump-setting-names]  [--generate-manpage=TEMPLATE]   [-h]   [--help]
11       [--help-all]  [--list-config-files]  [--version] [--no-default-configs]
12       [--dump-memory-profile=METHOD] [-k] [--keep]  [--no-keep]  [--log=FILE]
13       [--log-keep=N]  [--log-level=LEVEL]  [--log-max=SIZE] [--log-mode=MODE]
14       [--memory-dump-interval=SECONDS] [--output=FILE] [-tTEST] [--test=TEST]
15       [--timings] [--no-timings] [FILE]...
16

DESCRIPTION

18       cmdtest  black  box  tests  Unix  command  line tools.  Given some test
19       scripts, their inputs, and expected outputs, it verifies that the  com‐
20       mand  line  produces the expected output.  If not, it reports problems,
21       and shows the differences.
22
23       Each test case foo consists of the following files:
24
25       foo.script
26              a script to run the test (this is required)
27
28       foo.stdin
29              the file fed to standard input
30
31       foo.stdout
32              the expected output to the standard output
33
34       foo.stderr
35              the expected output to the standard error
36
37       foo.exit
38              the expected exit code
39
40       foo.setup
41              a shell script to run before the test
42
43       foo.teardown
44              a shell script to run after test
45
46       Usually, a single test is not enough. All tests are put into  the  same
47       directory, and they may share some setup and teardown code:
48
49       setup-once
50              a shell script to run once, before any tests
51
52       setup  a shell script to run before each test
53
54       teardown
55              a shell script to run after each test
56
57       teardown-once
58              a shell script to run once, after all tests
59
60       cmdtest  is given the name of the directory with all the tests, or sev‐
61       eral such directories, and it does the following:
62
63       · execute setup-once
64
65       · for each test case (unique prefix foo):
66
67              — execute setup
68
69              — execute foo.setup
70
71              — execute the command, by running foo.script, and redirecting
72                standard input to come from foo.stdin, and capturing standard
73                output and error and exit codes
74
75              — execute foo.teardown
76
77              — execute teardown
78
79              — report result of test: does exit code match foo.exit, standard
80                output match foo.stdout, and standard error match foo.stderr?
81
82       · execute teardown-once
83
84       Except for foo.script, all of these files are optional.  If a setup or
85       teardown script is missing, it is simply not executed.  If one of the
86       standard input, output, or error files is missing, it is treated as if
87       it were empty.  If the exit code file is missing, it is treated as if
88       it specified an exit code of zero.
89
90       The shell scripts may use the following environment variables:
91
92       DATADIR
93              a temporary directory where files may be created by the test
94
95       TESTNAME
96              name of the current test (will be empty for setup-once and tear‐
97              down-once)
98
99       SRCDIR directory from which cmdtest was launched
100

OPTIONS

102       -c, --command=COMMAND
103              ignored for backwards compatibility
104
105       --generate-manpage=TEMPLATE
106              fill in manual page TEMPLATE
107
108       -h, --help
109              show this help message and exit
110
111       -k, --keep
112              keep temporary data on failure
113
114       --no-keep
115              opposite of --keep
116
117       --output=FILE
118              write output to FILE, instead of standard output
119
120       -t, --test=TEST
121              run only TEST (can be given many times)
122
123       --timings
124              report how long each test takes
125
126       --no-timings
127              opposite of --timings
128
129       --version
130              show program's version number and exit
131
132   Configuration files and settings
133       --config=FILE
134              add FILE to config files
135
136       --dump-config
137              write out the entire current configuration
138
139       --dump-setting-names
140              write out all names of settings and quit
141
142       --help-all
143              show all options
144
145       --list-config-files
146              list all possible config files
147
148       --no-default-configs
149              clear list of configuration files to read
150
151   Logging
152       --log=FILE
153              write log entries to FILE (default is to not write log files at
154              all); use "syslog" to log to system log, "stderr" to log to the
155              standard error output, or "none" to disable logging
156
157       --log-keep=N
158              keep last N logs (10)
159
160       --log-level=LEVEL
161              log at LEVEL, one of debug, info, warning, error, critical, fa‐
162              tal (default: debug)
163
164       --log-max=SIZE
165              rotate logs larger than SIZE, zero for never (default: 0)
166
167       --log-mode=MODE
168              set permissions of new log files to MODE (octal; default 0600)
169
170   Peformance
171       --dump-memory-profile=METHOD
172              make memory profiling dumps using METHOD, which is one of: none,
173              or simple (no meliae support anymore)(default: simple)
174
175       --memory-dump-interval=SECONDS
176              make memory profiling dumps at least SECONDS apart
177

EXAMPLE

179       To test that the echo(1) command outputs the expected string, create a
180       file called echo-tests/hello.script containing the following content:
181
182              #!/bin/sh
183              echo hello, world
184
185       Also create the file echo-tests/hello.stdout containing:
186
187              hello, world
188
189       Then you can run the tests:
190
191              $ cmdtest echo-tests
192              test 1/1
193              1/1 tests OK, 0 failures
194
195       If you change the stdout file to be something else, cmdtest will report
196       the differences:
197
198              $ cmdtest echo-tests
199              FAIL: hello: stdout diff:
200              --- echo-tests/hello.stdout   2011-09-11 19:14:47 +0100
201              +++ echo-tests/hello.stdout-actual 2011-09-11 19:14:49 +0100
202              @@ -1 +1 @@
203              -something else
204              +hello, world
205
206              test 1/1
207              0/1 tests OK, 1 failures
208
209       Furthermore, the echo-tests directory will contain the actual output
210       files, and diffs from the expected files.  If one of the actual output
211       files is actually correct, you can actualy rename it to be the expected
212       file.  Actually, that's a very convenient way of creating the expected
213       output files: you run the test, fixing things, until you've manually
214       checked the actual output is correct, then you rename the file.
215

SEE ALSO

217       cliapp(5).
218
219
220
221                                                                    CMDTEST(1)
Impressum