1KYUA-TEST(1) BSD General Commands Manual KYUA-TEST(1)
2
4 kyua test — Runs tests
5
7 kyua test [--build-root path] [--kyuafile file] [--results-file file]
8 [test_filter1 .. test_filterN]
9
11 The kyua test command loads a test suite definition from a kyuafile(5),
12 runs the tests defined in it, and records the results into a new results
13 file. By default, all tests in the test suite are executed but the
14 optional arguments to kyua test can be used to select which test programs
15 or test cases to run. These are filters and are described below in Test
16 filters.
17 Every test executed by kyua test is run under a controlled environment as
19 described in Test isolation.
20 The following subcommand options are recognized:
22 --build-root path
24 Specifies the build root in which to find the test programs refer‐
25 enced by the Kyuafile, if different from the Kyuafile's directory.
26 See Build directories below for more information.
27 --kyuafile path, -k path
29 Specifies the Kyuafile to process. Defaults to a Kyuafile file in
30 the current directory.
31 --results-file path, -s path
33 Specifies the results file to create. Defaults to ‘LATEST’, which
34 causes kyua test to automatically generate a new results file for the
35 test run.
36 The following values are accepted:
38 ‘NEW’
40 Requests the automatic generation of a new results file name
41 based on the test suite being run and the current time.
42 Explicit file name (aka everything else)
44 Store the results file where indicated.
45 See Results files for more details.
47 You can later inspect the results of the test run in more detail by using
49 kyua-report(1) or you can execute a single test case with debugging func‐
50 tionality by using kyua-debug(1).
51 Build directories
53 Build directories (or object directories, target directories, product
54 directories, etc.) is the concept that allows a developer to keep the
55 source tree clean from build products by asking the build system to place
56 such build products under a separate subtree.
57 Most build systems today support build directories. For example, the GNU
59 Automake/Autoconf build system exposes such concept when invoked as fol‐
60 lows:
61 $ cd my-project-1.0
63 $ mkdir build
64 $ cd build
65 $ ../configure
66 $ make
67 Under such invocation, all the results of the build are left in the
69 my-project-1.0/build/ subdirectory while maintaining the contents of
70 my-project-1.0/ intact.
71 Because build directories are an integral part of most build systems, and
73 because they are a tool that developers use frequently, kyua test sup‐
74 ports build directories too. This manifests in the form of kyua test
75 being able to run tests from build directories while reading the (often
76 immutable) test suite definition from the source tree.
77 One important property of build directories is that they follow (or need
79 to follow) the exact same layout as the source tree. For example, con‐
80 sider the following directory listings:
81 src/Kyuafile
83 src/bin/ls/
84 src/bin/ls/Kyuafile
85 src/bin/ls/ls.c
86 src/bin/ls/ls_test.c
87 src/sbin/su/
88 src/sbin/su/Kyuafile
89 src/sbin/su/su.c
90 src/sbin/su/su_test.c
91 obj/bin/ls/
93 obj/bin/ls/ls*
94 obj/bin/ls/ls_test*
95 obj/sbin/su/
96 obj/sbin/su/su*
97 obj/sbin/su/su_test*
98 Note how the directory layout within src/ matches that of obj/. The src/
100 directory contains only source files and the definition of the test suite
101 (the Kyuafiles), while the obj/ directory contains only the binaries gen‐
102 erated during a build.
103 All commands that deal with the workspace support the --build-root path
105 option. When this option is provided, the directory specified by the
106 option is considered to be the root of the build directory. For example,
107 considering our previous fake tree layout, we could invoke kyua test as
108 any of the following:
109 $ kyua test --kyuafile=src/Kyuafile --build-root=obj
111 $ cd src && kyua test --build-root=../obj
112 Results files
114 Results files contain, as their name implies, the results of the execu‐
115 tion of a test suite. Each test suite executed by kyua-test(1) generates
116 a new results file, and such results files can be loaded later on by
117 inspection commands such as kyua-report(1) to analyze their contents.
118 Results files support identifier-based lookups and also path name
120 lookups. The differences between the two are described below.
121 The default naming scheme for the results files provides simple support
123 for identifier-based lookups and historical recording of test suite runs.
124 Each results file is given an identifier derived from the test suite that
125 generated it and the time the test suite was run. Kyua can later look up
126 results files by these fileds.
127 The identifier follows this pattern:
129 <test_suite>.<YYYYMMDD>-<HHMMSS>-<uuuuuu>
131 where ‘test_suite’ is the path to the root of the test suite that was run
133 with all slashes replaced by underscores and ‘YYYYMMDD-HHMMSS-uuuuuu’ is
134 a timestamp with microsecond resolution.
135 When using the default naming scheme, results files are stored in the
137 ~/.kyua/store/ subdirectory and each file holds a name of the form:
138 ~/.kyua/store/results.<identifier>.db
140 Results files are simple SQLite databases with the schema described in
142 the /usr/share/kyua/store/schema_v?.sql files. For details on the
143 schema, please refer to the heavily commented SQL file.
144 Test filters
146 A test filter is a string that is used to match test cases or test pro‐
147 grams in a test suite. Filters have the following form:
148 test_program_name[:test_case_name]
150 Where ‘test_program_name’ is the name of a test program or a subdirectory
152 in the test suite, and ‘test_case_name’ is the name of a test case.
153 Test isolation
155 The test programs and test cases run by kyua test are all executed in a
156 deterministic environment. This known, clean environment serves to make
157 the test execution as reproducible as possible and also to prevent
158 clashes between tests that may, for example, create auxiliary files with
159 overlapping names.
160 For plain test programs and for TAP test programs, the whole test program
162 is run under a single instance of the environment described in this page.
163 For ATF test programs (see atf(7)), each individual test case and test
164 cleanup routine are executed in separate environments.
165 Process space
167 Each test is executed in an independent processes. Corollary: the
168 test can do whatever it wants to the current process (such as modify
169 global variables) without having to undo such changes.
170 Session and process group
172 The test is executed in its own session and its own process group.
173 There is no controlling terminal attached to the session.
174 Should the test spawn any children, the children should maintain the
176 same session and process group. Modifying any of these settings pre‐
177 vents kyua test from being able to kill any stray subprocess as part
178 of the cleanup phase. If modifying these settings is necessary, or
179 if any subprocess started by the test decides to use a different
180 process group or session, it is the responsibility of the test to
181 ensure those subprocesses are forcibly terminated during cleanup.
182 Work directory
184 The test is executed in a temporary directory automatically created
185 by the runtime engine. Corollary: the test can write to its current
186 directory without needing to clean any files and/or directories it
187 creates. The runtime engine takes care to recursively delete the
188 temporary directories after the execution of a test case. Any file
189 systems mounted within the temporary directory are also unmounted.
190 Home directory
192 The HOME environment variable is set to the absolute path of the work
193 directory.
194 Umask
196 The value of the umask is set to 0022.
197 Environment
199 The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY,
200 LC_NUMERIC and LC_TIME variables are unset.
201 The TZ variable is set to ‘UTC’.
203 The TMPDIR variable is set to the absolute path of the work direc‐
205 tory. This is to prevent the test from mistakenly using a temporary
206 directory outside of the automatically-managed work directory, should
207 the test use the mktemp(3) familiy of functions.
208 Process limits
210 The maximum soft core size limit is raised to its corresponding hard
211 limit. This is a simple, best-effort attempt at allowing tests to
212 dump core for further diagnostic purposes.
213 Configuration varibles
215 The test engine may pass run-time configuration variables to the test
216 program via the environment. The name of the configuration variable
217 is prefixed with ‘TEST_ENV_’ so that a configuration variable of the
218 form ‘foo=bar’ becomes accessible in the environment as
219 ‘TEST_ENV_foo=bar’.
220
222 The kyua test command returns 0 if all executed test cases pass or 1 if
223 any of the executed test cases fails or if any of the given test case
224 filters does not match any test case.
225 Additional exit codes may be returned as described in kyua(1).
227
229 Workflow with results files
230 Let's say you run the following command twice in a row:
231 kyua test -k /usr/tests/Kyuafile
233 The two executions will generate two files with names like these:
235 ~/.kyua/store/results.usr_tests.20140731-150500-196784.db
237 ~/.kyua/store/results.usr_tests.20140731-151730-997451.db
238 Taking advantage of the default naming scheme, the following commands
240 would all generate a report for the results of the latest execution of
241 the test suite:
242 cd /usr/tests && kyua report
244 cd /usr/tests && kyua report --results-file=LATEST
245 kyua report --results-file=/usr/tests
246 kyua report --results-file=usr_tests
247 kyua report --results-file=usr_tests.20140731-151730-997451
248 But it is also possible to explicitly load data for older runs or from
250 explicitly-named files:
251 kyua report \
253 --results-file=usr_tests.20140731-150500-196784
254 kyua report \
255 --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
256
258 kyua(1), kyua-report(1), kyuafile(5)
259BSD October 13, 2014 BSD