1KYUAFILE(5)                 BSD File Formats Manual                KYUAFILE(5)
2

NAME

4     Kyuafile — Test suite description files
5

SYNOPSIS

7     atf_test_program(string name, [string metadata]);
8
9     current_kyuafile();
10
11     fs.basename(string path);
12
13     fs.dirname(string path);
14
15     fs.exists(string path);
16
17     fs.files(string path);
18
19     fs.is_absolute(string path);
20
21     fs.join(string path, string path);
22
23     include(string path);
24
25     plain_test_program(string name, [string metadata]);
26
27     syntax(int version);
28
29     tap_test_program(string name, [string metadata]);
30
31     test_suite(string name);
32

DESCRIPTION

34     A test suite is a collection of test programs and is represented by a
35     hierarchical layout of test binaries on the file system.  Any subtree of
36     the file system can represent a test suite, provided that it includes one
37     or more Kyuafiles, which are the test suite definition files.
38
39     A Kyuafile is a Lua script whose purpose is to describe the structure of
40     the test suite it belongs to.  To do so, the script has access to a col‐
41     lection of special functions provided by kyua(1) as described in Helper
42     functions.
43
44   File versioning
45     Every Kyuafile file starts with a call to syntax(int version).  This call
46     determines the specific schema used by the file so that future backwards-
47     incompatible modifications to the file can be introduced.
48
49     Any new Kyuafile file should set version to ‘2’.
50
51   Test suite definition
52     If the Kyuafile registers any test programs, the Kyuafile must define the
53     name of the test suite the test programs belong to by using the
54     test_suite() function at the very beginning of the file.
55
56     The test suite name provided in the test_suite() call tells kyua(1) which
57     set of configuration variables from kyua.conf(5) to pass to the test pro‐
58     grams at run time.
59
60   Test program registration
61     A Kyuafile can register test programs by means of a variety of
62     *_test_program() functions, all of which take the name of a test program
63     and a set of optional metadata properties that describe such test pro‐
64     gram.
65
66     The test programs to be registered must live in the current directory; in
67     other words, the various *_test_program() calls cannot reference test
68     programs in other directories.  The rationale for this is to force all
69     Kyuafile files to be self-contained, and to simplify their internal rep‐
70     resentation.
71
72     ATF test programs are those that use the atf(7) libraries.  They can be
73     registered with the atf_test_program() table constructor.  This function
74     takes the name of the test program and a collection of optional metadata
75     settings for all the test cases in the test program.  Any metadata prop‐
76     erties defined by the test cases themselves override the metadata values
77     defined here.
78
79     Plain test programs are those that return 0 on success and non-0 on fail‐
80     ure; in general, most test programs (even those that use fancy unit-test‐
81     ing libraries) behave this way and thus also qualify as plain test pro‐
82     grams.  They can be registered with the plain_test_program() table con‐
83     structor.  This function takes the name of the test program, an optional
84     test_suite name that overrides the global test suite name, and a collec‐
85     tion of optional metadata settings for the test program.
86
87     TAP test programs are those that implement the Test Anything Protocol.
88     They can be registered with the tap_test_program() table constructor.
89     This function takes the name of the test program and a collection of
90     optional metadata settings for the test program.
91
92     The following metadata properties can be passed to any test program defi‐
93     nition:
94
95           allowed_architectures
96               Whitespace-separated list of machine architecture names allowed
97               by the test.  If empty or not defined, the test is allowed to
98               run on any machine architecture.
99
100           allowed_platforms
101               Whitespace-separated list of machine platform names allowed by
102               the test.  If empty or not defined, the test is allowed to run
103               on any machine platform.
104
105           custom.NAME
106               Custom variable defined by the test where ‘NAME’ denotes the
107               name of the variable.  These variables are useful to tag your
108               tests with information specific to your project.  The values of
109               such variables are propagated all the way from the tests to the
110               results files and later to any generated reports.
111
112               Note that if the name happens to have dashes or any other spe‐
113               cial characters in it, you will have to use a special Lua syn‐
114               tax to define the property.  Refer to the EXAMPLES section
115               below for clarification.
116
117           description
118               Textual description of the test.
119
120           is_exclusive
121               If true, indicates that this test program cannot be executed
122               along any other programs at the same time.  Test programs that
123               affect global system state, such as those that modify the value
124               of a sysctl(8) setting, must set themselves as exclusive to
125               prevent failures due to race conditions.  Defaults to false.
126
127           required_configs
128               Whitespace-separated list of configuration variables that the
129               test requires to be defined before it can run.
130
131           required_disk_space
132               Amount of available disk space that the test needs to run suc‐
133               cessfully.
134
135           required_files
136               Whitespace-separated list of paths that the test requires to
137               exist before it can run.
138
139           required_memory
140               Amount of physical memory that the test needs to run success‐
141               fully.
142
143           required_programs
144               Whitespace-separated list of basenames or absolute paths point‐
145               ing to executable binaries that the test requires to exist
146               before it can run.
147
148           required_user
149               If empty, the test has no restrictions on the calling user for
150               it to run.  If set to ‘unprivileged’, the test needs to not run
151               as root.  If set to ‘root’, the test must run as root.
152
153           timeout
154               Amount of seconds that the test is allowed to execute before
155               being killed.
156
157   Recursion
158     To reference test programs in another subdirectory, a different Kyuafile
159     must be created in that directory and it must be included into the origi‐
160     nal Kyuafile by means of the include() function.
161
162     include() may only be called with a relative path and with at most one
163     directory component.  This is by design: Kyua uses the file system struc‐
164     ture as the layout of the test suite definition.  Therefore, each subdi‐
165     rectory in a test suite must include its own Kyuafile and each Kyuafile
166     can only descend into the Kyuafiles of immediate subdirectories.
167
168     If you need to source a Kyuafile located in disjoint parts of your file
169     system namespace, you will have to create a ‘shadow tree’ using symbolic
170     links and possibly helper Kyuafiles to plug the various subdirectories
171     together.  See the EXAMPLES section below for details.
172
173     Note that each file is processed in its own Lua environment: there is no
174     mechanism to pass state from one file to the other.  The reason for this
175     is that there is no such thing as a “top-level” Kyuafile in a test suite:
176     the user has to be able to run the test suite from any directory in a
177     given hierarchy, and this execution must not depend on files that live in
178     parent directories.
179
180   Top-level Kyuafile
181     Every system has a top directory into which test suites get installed.
182     The default is /usr/libexec/kyua/tests.  Within this directory live test
183     suites, each of which is in an independent subdirectory.  Each subdirec‐
184     tory can be provided separately by independent third-party packages.
185
186     Kyua allows running all the installed test suites at once in order to
187     provide comprehensive cross-component reports.  In order to do this,
188     there is a special file in the top directory that knows how to inspect
189     the subdirectories in search for other Kyuafiles and include them.
190
191     The FILES section includes more details on where this file lives.
192
193   Helper functions
194     The ‘base’, ‘string’, and ‘table’ Lua modules are fully available in the
195     context of a Kyuafile.
196
197     The following extra functions are provided by Kyua:
198
199           string current_kyuafile()
200               Returns the absolute path to the current Kyuafile.
201
202           string fs.basename(string path)
203               Returns the last component of the given path.
204
205           string fs.dirname(string path)
206               Returns the given path without its last component or a dot if
207               the path has a single component.
208
209           bool fs.exists(string path)
210               Checks if the given path exists.  If the path is not absolute,
211               it is relative to the directory containing the Kyuafile in
212               which the call to this function occurs.
213
214           iterator fs.files(string path)
215               Opens a directory for scanning of its entries.  The returned
216               iterator yields an entry on each call, and the entry is simply
217               the file name.  If the path is not absolute, it is relative to
218               the directory containing the Kyuafile in which the call to this
219               function occurs.
220
221           is_absolute fs.is_absolute(string path)
222               Returns true if the given path is absolute; false otherwise.
223
224           join fs.join(string path, string path)
225               Concatenates the two paths.  The second path cannot be abso‐
226               lute.
227

FILES

229     /usr/libexec/kyua/tests/Kyuafile.
230         Top-level Kyuafile for the current system.
231
232     /usr/share/kyua/examples/Kyuafile.top.
233         Sample file to serve as a top-level Kyuafile.
234

EXAMPLES

236     The following Kyuafile is the simplest you can define.  It provides a
237     test suite definition and registers a couple of different test programs
238     using different interfaces:
239
240           syntax(2)
241
242           test_suite('first')
243
244           atf_test_program{name='integration_test'}
245           plain_test_program{name='legacy_test'}
246
247     The following example is a bit more elaborate.  It introduces some meta‐
248     data properties to the test program definitions and recurses into a cou‐
249     ple of subdirectories:
250
251           syntax(2)
252
253           test_suite('second')
254
255           plain_test_program{name='legacy_test',
256                              allowed_architectures='amd64 i386',
257                              required_files='/bin/ls',
258                              timeout=30}
259
260           tap_test_program{name='privileged_test',
261                            required_user='root'}
262
263           include('module-1/Kyuafile')
264           include('module-2/Kyuafile')
265
266     The syntax to define custom properties may be not obvious if their names
267     have any characters that make the property name not be a valid Lua iden‐
268     tifier.  Dashes are just one example.  To set such properties, do some‐
269     thing like this:
270
271           syntax(2)
272
273           test_suite('FreeBSD')
274
275           plain_test_program{name='the_test',
276                              ['custom.FreeBSD-Bug-Id']='category/12345'}
277
278   Connecting disjoint test suites
279     Now suppose you had various test suites on your file system and you would
280     like to connect them together so that they could be executed and treated
281     as a single unit.  The test suites we would like to connect live under
282     /usr/tests, /usr/local/tests and ~/local/tests.
283
284     We cannot create a Kyuafile that references these because the include()
285     directive does not support absolute paths.  Instead, what we can do is
286     create a shadow tree using symbolic links:
287
288           $ mkdir ~/everything
289           $ ln -s /usr/tests ~/everything/system-tests
290           $ ln -s /usr/local/tests ~/everything/local-tests
291           $ ln -s ~/local/tests ~/everything/home-tests
292
293     And then we create an ~/everything/Kyuafile file to drive the execution
294     of the integrated test suite:
295
296           syntax(2)
297
298           test_suite('test-all-the-things')
299
300           include('system-tests/Kyuafile')
301           include('local-tests/Kyuafile')
302           include('home-tests/Kyuafile')
303
304     Or, simply, you could reuse the sample top-level Kyuafile to avoid having
305     to manually craft the list of directories into which to recurse:
306
307           $ cp /usr/share/kyua/examples/Kyuafile.top ~/everything/Kyuafile
308

SEE ALSO

310     kyua(1)
311
312BSD                              July 3, 2015                              BSD
Impressum