1KYUAFILE(5) BSD File Formats Manual KYUAFILE(5)
2
4 Kyuafile — Test suite description files
5
7 atf_test_program(string name, [string metadata]);
8 current_kyuafile();
10 fs.basename(string path);
12 fs.dirname(string path);
14 fs.exists(string path);
16 fs.files(string path);
18 fs.is_absolute(string path);
20 fs.join(string path, string path);
22 include(string path);
24 plain_test_program(string name, [string metadata]);
26 syntax(int version);
28 tap_test_program(string name, [string metadata]);
30 test_suite(string name);
32
34 A test suite is a collection of test programs and is represented by a
35 hierarchical layout of test binaries on the file system. Any subtree of
36 the file system can represent a test suite, provided that it includes one
37 or more Kyuafiles, which are the test suite definition files.
38 A Kyuafile is a Lua script whose purpose is to describe the structure of
40 the test suite it belongs to. To do so, the script has access to a col‐
41 lection of special functions provided by kyua(1) as described in Helper
42 functions.
43 File versioning
45 Every Kyuafile file starts with a call to syntax(int version). This call
46 determines the specific schema used by the file so that future backwards-
47 incompatible modifications to the file can be introduced.
48 Any new Kyuafile file should set version to ‘2’.
50 Test suite definition
52 If the Kyuafile registers any test programs, the Kyuafile must define the
53 name of the test suite the test programs belong to by using the
54 test_suite() function at the very beginning of the file.
55 The test suite name provided in the test_suite() call tells kyua(1) which
57 set of configuration variables from kyua.conf(5) to pass to the test pro‐
58 grams at run time.
59 Test program registration
61 A Kyuafile can register test programs by means of a variety of
62 *_test_program() functions, all of which take the name of a test program
63 and a set of optional metadata properties that describe such test pro‐
64 gram.
65 The test programs to be registered must live in the current directory; in
67 other words, the various *_test_program() calls cannot reference test
68 programs in other directories. The rationale for this is to force all
69 Kyuafile files to be self-contained, and to simplify their internal rep‐
70 resentation.
71 ATF test programs are those that use the atf(7) libraries. They can be
73 registered with the atf_test_program() table constructor. This function
74 takes the name of the test program and a collection of optional metadata
75 settings for all the test cases in the test program. Any metadata prop‐
76 erties defined by the test cases themselves override the metadata values
77 defined here.
78 Plain test programs are those that return 0 on success and non-0 on fail‐
80 ure; in general, most test programs (even those that use fancy unit-test‐
81 ing libraries) behave this way and thus also qualify as plain test pro‐
82 grams. They can be registered with the plain_test_program() table con‐
83 structor. This function takes the name of the test program, an optional
84 test_suite name that overrides the global test suite name, and a collec‐
85 tion of optional metadata settings for the test program.
86 TAP test programs are those that implement the Test Anything Protocol.
88 They can be registered with the tap_test_program() table constructor.
89 This function takes the name of the test program and a collection of
90 optional metadata settings for the test program.
91 The following metadata properties can be passed to any test program defi‐
93 nition:
94 allowed_architectures
96 Whitespace-separated list of machine architecture names allowed
97 by the test. If empty or not defined, the test is allowed to
98 run on any machine architecture.
99 allowed_platforms
101 Whitespace-separated list of machine platform names allowed by
102 the test. If empty or not defined, the test is allowed to run
103 on any machine platform.
104 custom.NAME
106 Custom variable defined by the test where ‘NAME’ denotes the
107 name of the variable. These variables are useful to tag your
108 tests with information specific to your project. The values of
109 such variables are propagated all the way from the tests to the
110 results files and later to any generated reports.
111 Note that if the name happens to have dashes or any other spe‐
113 cial characters in it, you will have to use a special Lua syn‐
114 tax to define the property. Refer to the EXAMPLES section
115 below for clarification.
116 description
118 Textual description of the test.
119 is_exclusive
121 If true, indicates that this test program cannot be executed
122 along any other programs at the same time. Test programs that
123 affect global system state, such as those that modify the value
124 of a sysctl(8) setting, must set themselves as exclusive to
125 prevent failures due to race conditions. Defaults to false.
126 required_configs
128 Whitespace-separated list of configuration variables that the
129 test requires to be defined before it can run.
130 required_disk_space
132 Amount of available disk space that the test needs to run suc‐
133 cessfully.
134 required_files
136 Whitespace-separated list of paths that the test requires to
137 exist before it can run.
138 required_memory
140 Amount of physical memory that the test needs to run success‐
141 fully.
142 required_programs
144 Whitespace-separated list of basenames or absolute paths point‐
145 ing to executable binaries that the test requires to exist
146 before it can run.
147 required_user
149 If empty, the test has no restrictions on the calling user for
150 it to run. If set to ‘unprivileged’, the test needs to not run
151 as root. If set to ‘root’, the test must run as root.
152 timeout
154 Amount of seconds that the test is allowed to execute before
155 being killed.
156 Recursion
158 To reference test programs in another subdirectory, a different Kyuafile
159 must be created in that directory and it must be included into the origi‐
160 nal Kyuafile by means of the include() function.
161 include() may only be called with a relative path and with at most one
163 directory component. This is by design: Kyua uses the file system struc‐
164 ture as the layout of the test suite definition. Therefore, each subdi‐
165 rectory in a test suite must include its own Kyuafile and each Kyuafile
166 can only descend into the Kyuafiles of immediate subdirectories.
167 If you need to source a Kyuafile located in disjoint parts of your file
169 system namespace, you will have to create a ‘shadow tree’ using symbolic
170 links and possibly helper Kyuafiles to plug the various subdirectories
171 together. See the EXAMPLES section below for details.
172 Note that each file is processed in its own Lua environment: there is no
174 mechanism to pass state from one file to the other. The reason for this
175 is that there is no such thing as a “top-level” Kyuafile in a test suite:
176 the user has to be able to run the test suite from any directory in a
177 given hierarchy, and this execution must not depend on files that live in
178 parent directories.
179 Top-level Kyuafile
181 Every system has a top directory into which test suites get installed.
182 The default is /usr/libexec/kyua/tests. Within this directory live test
183 suites, each of which is in an independent subdirectory. Each subdirec‐
184 tory can be provided separately by independent third-party packages.
185 Kyua allows running all the installed test suites at once in order to
187 provide comprehensive cross-component reports. In order to do this,
188 there is a special file in the top directory that knows how to inspect
189 the subdirectories in search for other Kyuafiles and include them.
190 The FILES section includes more details on where this file lives.
192 Helper functions
194 The ‘base’, ‘string’, and ‘table’ Lua modules are fully available in the
195 context of a Kyuafile.
196 The following extra functions are provided by Kyua:
198 string current_kyuafile()
200 Returns the absolute path to the current Kyuafile.
201 string fs.basename(string path)
203 Returns the last component of the given path.
204 string fs.dirname(string path)
206 Returns the given path without its last component or a dot if
207 the path has a single component.
208 bool fs.exists(string path)
210 Checks if the given path exists. If the path is not absolute,
211 it is relative to the directory containing the Kyuafile in
212 which the call to this function occurs.
213 iterator fs.files(string path)
215 Opens a directory for scanning of its entries. The returned
216 iterator yields an entry on each call, and the entry is simply
217 the file name. If the path is not absolute, it is relative to
218 the directory containing the Kyuafile in which the call to this
219 function occurs.
220 is_absolute fs.is_absolute(string path)
222 Returns true if the given path is absolute; false otherwise.
223 join fs.join(string path, string path)
225 Concatenates the two paths. The second path cannot be abso‐
226 lute.
227
229 /usr/libexec/kyua/tests/Kyuafile.
230 Top-level Kyuafile for the current system.
231 /usr/share/kyua/examples/Kyuafile.top.
233 Sample file to serve as a top-level Kyuafile.
234
236 The following Kyuafile is the simplest you can define. It provides a
237 test suite definition and registers a couple of different test programs
238 using different interfaces:
239 syntax(2)
241 test_suite('first')
243 atf_test_program{name='integration_test'}
245 plain_test_program{name='legacy_test'}
246 The following example is a bit more elaborate. It introduces some meta‐
248 data properties to the test program definitions and recurses into a cou‐
249 ple of subdirectories:
250 syntax(2)
252 test_suite('second')
254 plain_test_program{name='legacy_test',
256 allowed_architectures='amd64 i386',
257 required_files='/bin/ls',
258 timeout=30}
259 tap_test_program{name='privileged_test',
261 required_user='root'}
262 include('module-1/Kyuafile')
264 include('module-2/Kyuafile')
265 The syntax to define custom properties may be not obvious if their names
267 have any characters that make the property name not be a valid Lua iden‐
268 tifier. Dashes are just one example. To set such properties, do some‐
269 thing like this:
270 syntax(2)
272 test_suite('FreeBSD')
274 plain_test_program{name='the_test',
276 ['custom.FreeBSD-Bug-Id']='category/12345'}
277 Connecting disjoint test suites
279 Now suppose you had various test suites on your file system and you would
280 like to connect them together so that they could be executed and treated
281 as a single unit. The test suites we would like to connect live under
282 /usr/tests, /usr/local/tests and ~/local/tests.
283 We cannot create a Kyuafile that references these because the include()
285 directive does not support absolute paths. Instead, what we can do is
286 create a shadow tree using symbolic links:
287 $ mkdir ~/everything
289 $ ln -s /usr/tests ~/everything/system-tests
290 $ ln -s /usr/local/tests ~/everything/local-tests
291 $ ln -s ~/local/tests ~/everything/home-tests
292 And then we create an ~/everything/Kyuafile file to drive the execution
294 of the integrated test suite:
295 syntax(2)
297 test_suite('test-all-the-things')
299 include('system-tests/Kyuafile')
301 include('local-tests/Kyuafile')
302 include('home-tests/Kyuafile')
303 Or, simply, you could reuse the sample top-level Kyuafile to avoid having
305 to manually craft the list of directories into which to recurse:
306 $ cp /usr/share/kyua/examples/Kyuafile.top ~/everything/Kyuafile
308
310 kyua(1)
311BSD July 3, 2015 BSD