1KYUA-TEST(1)              BSD General Commands Manual             KYUA-TEST(1)
2

NAME

4     kyua test — Runs tests
5

SYNOPSIS

7     kyua test [--build-root path] [--kyuafile file] [--results-file file]
8          [test_filter1 .. test_filterN]
9

DESCRIPTION

11     The kyua test command loads a test suite definition from a kyuafile(5),
12     runs the tests defined in it, and records the results into a new results
13     file.  By default, all tests in the test suite are executed but the
14     optional arguments to kyua test can be used to select which test programs
15     or test cases to run.  These are filters and are described below in Test
16     filters.
17
18     Every test executed by kyua test is run under a controlled environment as
19     described in Test isolation.
20
21     The following subcommand options are recognized:
22
23     --build-root path
24         Specifies the build root in which to find the test programs refer‐
25         enced by the Kyuafile, if different from the Kyuafile's directory.
26         See Build directories below for more information.
27
28     --kyuafile path, -k path
29         Specifies the Kyuafile to process.  Defaults to a Kyuafile file in
30         the current directory.
31
32     --results-file path, -s path
33         Specifies the results file to create.  Defaults to ‘LATEST’, which
34         causes kyua test to automatically generate a new results file for the
35         test run.
36
37         The following values are accepted:
38
39         ‘NEW’
40             Requests the automatic generation of a new results file name
41             based on the test suite being run and the current time.
42
43         Explicit file name (aka everything else)
44             Store the results file where indicated.
45
46         See Results files for more details.
47
48     You can later inspect the results of the test run in more detail by using
49     kyua-report(1) or you can execute a single test case with debugging func‐
50     tionality by using kyua-debug(1).
51
52   Build directories
53     Build directories (or object directories, target directories, product
54     directories, etc.) is the concept that allows a developer to keep the
55     source tree clean from build products by asking the build system to place
56     such build products under a separate subtree.
57
58     Most build systems today support build directories.  For example, the GNU
59     Automake/Autoconf build system exposes such concept when invoked as fol‐
60     lows:
61
62           $ cd my-project-1.0
63           $ mkdir build
64           $ cd build
65           $ ../configure
66           $ make
67
68     Under such invocation, all the results of the build are left in the
69     my-project-1.0/build/ subdirectory while maintaining the contents of
70     my-project-1.0/ intact.
71
72     Because build directories are an integral part of most build systems, and
73     because they are a tool that developers use frequently, kyua test sup‐
74     ports build directories too.  This manifests in the form of kyua test
75     being able to run tests from build directories while reading the (often
76     immutable) test suite definition from the source tree.
77
78     One important property of build directories is that they follow (or need
79     to follow) the exact same layout as the source tree.  For example, con‐
80     sider the following directory listings:
81
82           src/Kyuafile
83           src/bin/ls/
84           src/bin/ls/Kyuafile
85           src/bin/ls/ls.c
86           src/bin/ls/ls_test.c
87           src/sbin/su/
88           src/sbin/su/Kyuafile
89           src/sbin/su/su.c
90           src/sbin/su/su_test.c
91
92           obj/bin/ls/
93           obj/bin/ls/ls*
94           obj/bin/ls/ls_test*
95           obj/sbin/su/
96           obj/sbin/su/su*
97           obj/sbin/su/su_test*
98
99     Note how the directory layout within src/ matches that of obj/.  The src/
100     directory contains only source files and the definition of the test suite
101     (the Kyuafiles), while the obj/ directory contains only the binaries gen‐
102     erated during a build.
103
104     All commands that deal with the workspace support the --build-root path
105     option.  When this option is provided, the directory specified by the
106     option is considered to be the root of the build directory.  For example,
107     considering our previous fake tree layout, we could invoke kyua test as
108     any of the following:
109
110           $ kyua test --kyuafile=src/Kyuafile --build-root=obj
111           $ cd src && kyua test --build-root=../obj
112
113   Results files
114     Results files contain, as their name implies, the results of the execu‐
115     tion of a test suite.  Each test suite executed by kyua-test(1) generates
116     a new results file, and such results files can be loaded later on by
117     inspection commands such as kyua-report(1) to analyze their contents.
118
119     Results files support identifier-based lookups and also path name
120     lookups.  The differences between the two are described below.
121
122     The default naming scheme for the results files provides simple support
123     for identifier-based lookups and historical recording of test suite runs.
124     Each results file is given an identifier derived from the test suite that
125     generated it and the time the test suite was run.  Kyua can later look up
126     results files by these fileds.
127
128     The identifier follows this pattern:
129
130           <test_suite>.<YYYYMMDD>-<HHMMSS>-<uuuuuu>
131
132     where ‘test_suite’ is the path to the root of the test suite that was run
133     with all slashes replaced by underscores and ‘YYYYMMDD-HHMMSS-uuuuuu’ is
134     a timestamp with microsecond resolution.
135
136     When using the default naming scheme, results files are stored in the
137     ~/.kyua/store/ subdirectory and each file holds a name of the form:
138
139           ~/.kyua/store/results.<identifier>.db
140
141     Results files are simple SQLite databases with the schema described in
142     the /usr/share/kyua/store/schema_v?.sql files.  For details on the
143     schema, please refer to the heavily commented SQL file.
144
145   Test filters
146     A test filter is a string that is used to match test cases or test pro‐
147     grams in a test suite.  Filters have the following form:
148
149           test_program_name[:test_case_name]
150
151     Where ‘test_program_name’ is the name of a test program or a subdirectory
152     in the test suite, and ‘test_case_name’ is the name of a test case.
153
154   Test isolation
155     The test programs and test cases run by kyua test are all executed in a
156     deterministic environment.  This known, clean environment serves to make
157     the test execution as reproducible as possible and also to prevent
158     clashes between tests that may, for example, create auxiliary files with
159     overlapping names.
160
161     For plain test programs and for TAP test programs, the whole test program
162     is run under a single instance of the environment described in this page.
163     For ATF test programs (see atf(7)), each individual test case and test
164     cleanup routine are executed in separate environments.
165
166     Process space
167         Each test is executed in an independent processes.  Corollary: the
168         test can do whatever it wants to the current process (such as modify
169         global variables) without having to undo such changes.
170
171     Session and process group
172         The test is executed in its own session and its own process group.
173         There is no controlling terminal attached to the session.
174
175         Should the test spawn any children, the children should maintain the
176         same session and process group.  Modifying any of these settings pre‐
177         vents kyua test from being able to kill any stray subprocess as part
178         of the cleanup phase.  If modifying these settings is necessary, or
179         if any subprocess started by the test decides to use a different
180         process group or session, it is the responsibility of the test to
181         ensure those subprocesses are forcibly terminated during cleanup.
182
183     Work directory
184         The test is executed in a temporary directory automatically created
185         by the runtime engine.  Corollary: the test can write to its current
186         directory without needing to clean any files and/or directories it
187         creates.  The runtime engine takes care to recursively delete the
188         temporary directories after the execution of a test case.  Any file
189         systems mounted within the temporary directory are also unmounted.
190
191     Home directory
192         The HOME environment variable is set to the absolute path of the work
193         directory.
194
195     Umask
196         The value of the umask is set to 0022.
197
198     Environment
199         The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY,
200         LC_NUMERIC and LC_TIME variables are unset.
201
202         The TZ variable is set to ‘UTC’.
203
204         The TMPDIR variable is set to the absolute path of the work direc‐
205         tory.  This is to prevent the test from mistakenly using a temporary
206         directory outside of the automatically-managed work directory, should
207         the test use the mktemp(3) familiy of functions.
208
209     Process limits
210         The maximum soft core size limit is raised to its corresponding hard
211         limit.  This is a simple, best-effort attempt at allowing tests to
212         dump core for further diagnostic purposes.
213
214     Configuration varibles
215         The test engine may pass run-time configuration variables to the test
216         program via the environment.  The name of the configuration variable
217         is prefixed with ‘TEST_ENV_’ so that a configuration variable of the
218         form ‘foo=bar’ becomes accessible in the environment as
219         ‘TEST_ENV_foo=bar’.
220

EXIT STATUS

222     The kyua test command returns 0 if all executed test cases pass or 1 if
223     any of the executed test cases fails or if any of the given test case
224     filters does not match any test case.
225
226     Additional exit codes may be returned as described in kyua(1).
227

EXAMPLES

229   Workflow with results files
230     Let's say you run the following command twice in a row:
231
232           kyua test -k /usr/tests/Kyuafile
233
234     The two executions will generate two files with names like these:
235
236           ~/.kyua/store/results.usr_tests.20140731-150500-196784.db
237           ~/.kyua/store/results.usr_tests.20140731-151730-997451.db
238
239     Taking advantage of the default naming scheme, the following commands
240     would all generate a report for the results of the latest execution of
241     the test suite:
242
243           cd /usr/tests && kyua report
244           cd /usr/tests && kyua report --results-file=LATEST
245           kyua report --results-file=/usr/tests
246           kyua report --results-file=usr_tests
247           kyua report --results-file=usr_tests.20140731-151730-997451
248
249     But it is also possible to explicitly load data for older runs or from
250     explicitly-named files:
251
252           kyua report \
253               --results-file=usr_tests.20140731-150500-196784
254           kyua report \
255               --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
256

SEE ALSO

258     kyua(1), kyua-report(1), kyuafile(5)
259
260BSD                            October 13, 2014                            BSD
Impressum