1YARN(1) General Commands Manual YARN(1)
2
6 yarn - scenario testing of Unix command line tools
7
9 yarn [--allow-missing-steps] [--no-allow-missing-steps] [--cd-datadir]
10 [--no-cd-datadir] [--config=FILE] [--dump-config]
11 [--dump-setting-names] [--generate-manpage=TEMPLATE] [-h] [--help]
12 [--help-all] [--list-config-files] [--version] [--no-default-configs]
13 [--dump-memory-profile=METHOD] [--env=NAME=VALUE] [--log=FILE]
14 [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE]
15 [--memory-dump-interval=SECONDS] [--output=FILE] [-q] [--quiet]
16 [--no-quiet] [--require-assumptions] [--no-require-assumptions]
17 [-rSCENARIO] [--run=SCENARIO] [--shell=SHELL] [--shell-arg=ARG]
18 [-sSHELL-LIBRARY] [--shell-library=SHELL-LIBRARY] [--snapshot]
19 [--no-snapshot] [--stop-on-first-fail] [--no-stop-on-first-fail]
20 [--tempdir=DIR] [--timings] [--no-timings] [-v] [--verbose]
21 [--no-verbose] [-n] [--no-act] [--dry-run] [--pretend] [--no-no-act]
22 [--no-dry-run] [--no-pretend] [FILE]...
23
25 yarn is a scenario testing tool: you write a scenario describing how a
26 user uses your software and what should happen, and express, using very
27 lightweight syntax, the scenario in such a way that it can be tested
28 automatically. The scenario has a simple, but strict structure:
29 GIVEN some setup for the test
31 WHEN thing that is to be tested happens
32 THEN the post-conditions must be true
33 As an example, consider a very short test scenario for verifying that a
35 backup program works, at least for one simple case.
36 SCENARIO backups can be restored
38 GIVEN some live data in a directory
39 AND an empty backup repository
40 WHEN a backup is made
41 THEN the data case be restored
42 FINALLY cleanup
43 Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN
45 statements. The AND keyword makes the text be more readable. SCENARIO
46 is also necessary, and gives the title.
47 FINALLY is for cleanups. The FINALLY steps will be run regardless of
49 whether the scenario succeeds or not.
50 Scenarios are meant to be written in somewhat human readable language.
52 However, they are not free form text. In addition to the GIV‐
53 EN/WHEN/THEN structure, the text for each of the steps needs a comput‐
54 er-executable implementation. This is done by using IMPLEMENTS. The
55 backup scenario from above might be implemented as follows:
56 IMPLEMENTS GIVEN some live data in a directory
58 rm -rf "$DATADIR/data"
59 mkdir "$DATADIR/data"
60 echo foo > "$DATADIR/data/foo"
61 IMPLEMENTS GIVEN an empty backup repository
63 rm -rf "$DATADIR/repo"
64 mkdir "$DATADIR/repo"
65 IMPLEMENTS WHEN a backup is made
67 backup-program -r "$DATADIR/repo" "$DATADIR/data"
68 IMPLEMENTS THEN the data can be restored
70 mkdir "$DATADIR/restored"
71 restore-program -r "$DATADIR/repo" "$DATADIR/restored"
72 diff -rq "$DATADIR/data" "$DATADIR/restored"
73 IMPLEMENTS FINALLY cleanup
75 echo nothing to do, actually
76 Each "IMPLEMENTS GIVEN" (or WHEN, THEN, FINALLY) is followed by a regu‐
78 lar expression on the same line, and then a shell script that gets exe‐
79 cuted to implement any step that matches the regular expression. The
80 implementation can extract data from the match as well: for example,
81 the regular expression might allow a file size to be specified.
82 The above example is a bit silly, of course: why go to the effort to
84 obfuscate the various steps? The answer is that the various steps, im‐
85 plemented using IMPLEMENTS, can be combined in many ways, to test dif‐
86 ferent aspects of the program being tested.
87 Moreover, by making the step descriptions be human language text,
89 matched by regular expressions, most of the test can hopefully be writ‐
90 ten, and understood, by non-programmers. Someone who understands what
91 a program should do, could write tests to verify its behaviour. The
92 implementations of the various steps need to be implemented by a pro‐
93 grammer, but given a well-designed set of steps, with enough flexibili‐
94 ty in their implementation, that quite a good test suite can be writ‐
95 ten.
96 The shell commands in an IMPLEMENTS section are run in the directory in
98 which the user ran yarn. The environment variable SRCDIR is set to the
99 fully qualified path to that directory.
100
102 --allow-missing-steps
103 allow scenarios to reference steps that do not exist, by warning
104 about them, but otherwise ignoring the scenarios
105 --no-allow-missing-steps
107 opposite of --allow-missing-steps
108 --cd-datadir
110 change to DATADIR when running commands
111 --no-cd-datadir
113 opposite of --cd-datadir
114 --env=NAME=VALUE
116 add NAME=VALUE to the environment when tests are run
117 --generate-manpage=TEMPLATE
119 fill in manual page TEMPLATE
120 -h, --help
122 show this help message and exit
123 --output=FILE
125 write output to FILE, instead of standard output
126 -q, --quiet
128 be quiet, avoid progress reporting, only show errors
129 --no-quiet
131 opposite of --quiet
132 --require-assumptions
134 require ASSUMING to always pass
135 --no-require-assumptions
137 opposite of --require-assumptions
138 -r, --run=SCENARIO
140 run only SCENARIO (this option can be repeated)
141 --shell=SHELL
143 run IMPLEMENTS using SHELL
144 --shell-arg=ARG
146 use ARG when running shell
147 -s, --shell-library=SHELL-LIBRARY
149 include a shell library for the IMPLEMENTS sections to use
150 --snapshot
152 make snapshots of test working directory after each scenario
153 step; you probably want to use this with --tempdir
154 --no-snapshot
156 opposite of --snapshot
157 --stop-on-first-fail
159 stop if any scenario step fails, don't run more scenarios
160 --no-stop-on-first-fail
162 opposite of --stop-on-first-fail
163 --tempdir=DIR
165 use DIR as the temporary directory for tests; it should be empty
166 or not exist
167 --timings
169 report wall clock time for each scenario and step
170 --no-timings
172 opposite of --timings
173 -v, --verbose
175 make progress reporting be more verbose ("wall of text"), in‐
176 stead of a one-line status info; this is turned automatically if
177 there is not terminal
178 --no-verbose
180 opposite of --verbose
181 --version
183 show program's version number and exit
184 -n, --no-act, --dry-run, --pretend
186 do not actually run any tests, merely print what would be run
187 --no-no-act, --no-dry-run, --no-pretend
189 opposite of --no-act
190 Configuration files and settings
192 --config=FILE
193 add FILE to config files
194 --dump-config
196 write out the entire current configuration
197 --dump-setting-names
199 write out all names of settings and quit
200 --help-all
202 show all options
203 --list-config-files
205 list all possible config files
206 --no-default-configs
208 clear list of configuration files to read
209 Logging
211 --log=FILE
212 write log entries to FILE (default is to not write log files at
213 all); use "syslog" to log to system log, "stderr" to log to the
214 standard error output, or "none" to disable logging
215 --log-keep=N
217 keep last N logs (10)
218 --log-level=LEVEL
220 log at LEVEL, one of debug, info, warning, error, critical, fa‐
221 tal (default: debug)
222 --log-max=SIZE
224 rotate logs larger than SIZE, zero for never (default: 0)
225 --log-mode=MODE
227 set permissions of new log files to MODE (octal; default 0600)
228 Peformance
230 --dump-memory-profile=METHOD
231 make memory profiling dumps using METHOD, which is one of: none,
232 simple, or meliae (default: simple)
233 --memory-dump-interval=SECONDS
235 make memory profiling dumps at least SECONDS apart
236
238 DATADIR
239 Fully qualified pathname to a temporary directory, in which the
240 tests can use files. The temporary directory is removed at the
241 end of the test execution, unless the user specifies otherwise
242 with --snapshot.
243 SRCDIR Fully qualitifed pathname to the directory in which the user ran
245 yarn. This is useful when the tests want to change the directo‐
246 ry.
247
249 To run yarn on all the scenarios in your current directory:
250 yarn *.scenario
252 All the files will be treated together as if they had been one file.
254 To add a shell library to be included when running any IMPLEMENTS sec‐
256 tion:
257 yarn --shell-library mylib.sh *.scenario
259 You can repeat --shell-library as many times as necessary.
261
263 cmdtest(1), cliapp(5).
264 The README.yarn file has more details on the scenario testing language.
266 YARN(1)