1faltest(1) Falcon User Manuals faltest(1)
2
6 faltest - Falcon unit test interface.
7
10 faltest [-d unit_test_dir] [options] [unit_list] module_file.fam
11
14 The faltest command line tool is a powerful interface to the Falcon
15 unit testing system.
16 The basic working principle of faltest is that of taking all the .fal
18 script files contained in a directory, compile and execute them, even‐
19 tually keeping track of errors, elapsed times and execution failures.
20 After running all the scripts, faltest may print a report on what hap‐
21 pened if requested to do so.
22 A list of one or more unit test may be indicated in the faltest command
24 line after the options. Also, the executed tests can be limited to
25 named subsets.
26 The unit test directory is added to the module load path, so load
28 directives will be resolved searching the required scripts in the test
29 path.
30
33 Scripts being part of unit test have to start with a common header
34 indicating some information about them. The header is actually a for‐
35 matted Falcon comment which is read by the faltest utility.
36 This is a typical header:
38 /**********************************************
40 * Falcon test suite
41 *
42 * ID: 10a
43 * Category: cat-name
44 * Subcategory: subcat-name
45 * Short: The short name
46 * Description:
47 * Long description of this test
48 * Spanning on many lines...
49 * [/Description]
50 **********************************************/
51 The header has a free form; faltest recognizes the following fields,
53 being inside a comment and eventually preceded by a "*".
54 ID:
56 The only mandatory field, it declares the ID under which this
57 unit test is known. It will be used in listing the tests and in
58 selecting them as argument of the faltest command line. The id
59 must be an integer number, eventually followed by a single low‐
60 ercase letter. Similar tests should be filed under the same ID
61 with a different specification letter.
62 Scripts not having this field will be ignored by faltest.
64 Category:
66 The name of the category under which this test is filed. Fal‐
67 test can select a subset of scripts to be executed to a certain
68 category.
69 Subcategory:
71 The name of the subcategory under which this test is filed.
72 Faltest can select a subset of scripts to be executed to a cer‐
73 tain subcategory.
74 Short:
76 Short description (or symbolic name) for this unit test.
77 Description:
79 A longer description explaining what this test is supposed to
80 do. The description always spans on more lines, and is closed by
81 a [/Description] tag.
82
85 Falcon system provides a loadable module called "testsuite". The module
86 is actually embedded in faltest , and is provided to all the scripts it
87 runs. The module provides the following functions:
88 success()
90 The script is terminated and recorded as a success.
91 failure( reason )
93 The script is terminated and recorded as a failure. An optional
94 parameter containing a description of the failure condition may
95 be optionally provided; it will be written as part of the report
96 and may be used to track which part of the test wasn't working.
97 testReflect( item )
99 Returns the passed item. This is used to test for engine respon‐
100 siveness to external calls and item copy through external func‐
101 tions.
102 alive( percent )
104 In tests running for some human sensible time, this function
105 should be called periodically to inform the system and the user
106 that the test is proceeding.
107 An optional "percent" parameter can be provided. It will be
109 interpreted as a value between 1 and 100 representing the amount
110 of test that has been performed up to this moment.
111 timings( total_time, performed_ops )
113 In case the execution time is relevant for the test, like in
114 benchmarks, this function can be used to communicate back to
115 faltest the time elapsed in the operations being tested and the
116 number of operations performed. Those parameters will be
117 recorded and eventually saved in the report file, to be used as
118 benchmarks against falcon engine modifications.
119 timeFactor()
121 Lengthy tests are often performed by looping over the operation
122 to be tested for a certain time. Benchmarks and performance
123 tests should be written so that they can normally complete in a
124 reasonable time, between one and ten seconds. In case the user
125 wants the tests to perform longer, in order to obtain better
126 statistical data, it can pass the -f (time factor) option to
127 faltest command line. The time factor will be a number greater
128 than 1, and should be used by tests that may perform lengthy
129 operation to customize the number of performed tests. This
130 function returns as an integer value of the -f option given to
131 faltest or 1 if the -f option was not given.
132
135 -c <cat>
136 Select this category and ignore the rest.
137 -C <subcat>
140 Select this subcategory and ignore the rest.
141 -d Directory from where to load the unit tests. If not specified,
144 it will be the current directory.
145 -f <n> Set time factor to N. Some scripts may use this information to
148 perform more loops or lengthy operations.
149 -h Show version and a short help.
152 -l List the selected tests and exit. Combine with -v to have the
155 path of the tests, as -l only lists the script ID, its short
156 name and the category/subcategory pair.
157 -m Do not compile in memory. Use disk temporary files.
160 -M Checks for memory leaks. This option makes faltest to report
163 unclaimed memory after each script execution, and a final report
164 at the end of the tests. The check is extended to all the engine
165 operations, so errors in the engine are detected too.
166 -o <file>
169 Write final report to the given output file.
170 -s Perform module serialization test. Other than compiling the
173 file, the module is also saved and then restored before being
174 executed. This allows to check for errors in module serializa‐
175 tion (that is, loading of .fam files). The operation is per‐
176 formed in memory, unless the option -m is also specified.
177 -S Compile via assembly. This test the correct behavior of the
180 assembler generator and compiler instead of the binary module
181 generator.
182 -t Records and display timings. The statistics of compilation,
185 linking and execution overall times are recorded and written as
186 part of the report.
187 -T Records timings() calls from scripts. This allows the scripts to
190 declare their own performance ratings, and collects the results
191 in the final report.
192 -v Be verbose. Normally, execution and failures are sparsely
195 reported. This is because the normal execution mode is meant for
196 automated runs. Tests can be executed by automated utilities and
197 errors can be reported to system administrator by simple checks
198 on the output data.
199 A developer willing to fix a broken test must run that test
201 alone with the -v enabled. A more complete error report (includ‐
202 ing compilation or execution errors, if they were the cause for
203 the failure) will be then visualized. The -v options also allows
204 to see the path of the original script, which is otherwise hid‐
205 den (masked by the testsuite ID).
206 -V Prints version number and exits.
209
212 This is a simple and complete example from the Falcon benchmark suite.
213 /*******************************************
215 * Falcon direct benchmarks
216 *
217 * ID: 2a
218 * Category: benchmark
219 * Subcategory: calls
220 * Short: Benchmark on function calls
221 * Description:
222 * Performing repeated function calls and returns.
223 * This test calls a function without parameters.
224 *
225 * [/Description]
226 ******************************************/
227 loops = 1000000 * timeFactor()
229 each = int(loops/10)
230 function toBeCalled()
232 end
233 // getting time
235 time = seconds()
236 for i = 1 to loops
237 // perform the call
239 toBeCalled()
240 if i % each == 0
242 alive( i/loops*100 )
243 end
244 end
245 // taking end time
246 time = seconds() - time
247 // subtract alive time
249 timeAlive = seconds()
250 for i = 1 to loops
251 if i % each == 0
252 alive( i/loops*100 )
253 end
254 end
255 timeAlive = seconds() - timeAlive
256 time -= timeAlive
257 timings( time, loops )
259 /* end of file */
261
265 /usr/lib/libfalcon_engine.so
266 Default location of the Falcon Engine loadable module.
267
270 Giancarlo Niccolai <gc@falconpl.org>
271
274 falcon(1)
275
278 This document is released under the "GNU Free Documentation License,
279 version 1.2". On Debian systems, the complete text of the Free Docu‐
280 mentation License, version 1.2, can be found in /usr/share/com‐
281 mon-licenses/.
282Falcon toolset April 2007 faltest(1)