1faltest(1)                    Falcon User Manuals                   faltest(1)
2
3
4

NAME

6       faltest - Falcon unit test interface.
7
8

SYNOPSIS

10       faltest [-d unit_test_dir] [options] [unit_list] module_file.fam
11
12

DESCRIPTION

14       The  faltest  command  line  tool is a powerful interface to the Falcon
15       unit testing system.
16
17       The basic working principle of faltest is that of taking all  the  .fal
18       script  files contained in a directory, compile and execute them, even‐
19       tually keeping track of errors, elapsed times and  execution  failures.
20       After  running all the scripts, faltest may print a report on what hap‐
21       pened if requested to do so.
22
23       A list of one or more unit test may be indicated in the faltest command
24       line  after  the  options.  Also,  the executed tests can be limited to
25       named subsets.
26
27       The unit test directory is added to  the  module  load  path,  so  load
28       directives  will be resolved searching the required scripts in the test
29       path.
30
31

UNIT TEST SCRIPTS

33       Scripts being part of unit test have to  start  with  a  common  header
34       indicating  some  information about them. The header is actually a for‐
35       matted Falcon comment which is read by the faltest utility.
36
37       This is a typical header:
38
39       /**********************************************
40       * Falcon test suite
41       *
42       * ID: 10a
43       * Category: cat-name
44       * Subcategory: subcat-name
45       * Short: The short name
46       * Description:
47       *   Long description of this test
48       *   Spanning on many lines...
49       * [/Description]
50       **********************************************/
51
52       The header has a free form; faltest recognizes  the  following  fields,
53       being inside a comment and eventually preceded by a "*".
54
55       ID:
56              The  only  mandatory  field, it declares the ID under which this
57              unit test is known. It will be used in listing the tests and  in
58              selecting  them  as argument of the faltest command line. The id
59              must be an integer number, eventually followed by a single  low‐
60              ercase  letter.  Similar tests should be filed under the same ID
61              with a different specification letter.
62
63              Scripts not having this field will be ignored by faltest.
64
65       Category:
66              The name of the category under which this test is  filed.   Fal‐
67              test  can select a subset of scripts to be executed to a certain
68              category.
69
70       Subcategory:
71              The name of the subcategory under  which  this  test  is  filed.
72              Faltest  can select a subset of scripts to be executed to a cer‐
73              tain subcategory.
74
75       Short:
76              Short description (or symbolic name) for this unit test.
77
78       Description:
79              A longer description explaining what this test  is  supposed  to
80              do. The description always spans on more lines, and is closed by
81              a [/Description] tag.
82
83

THE TESTSUITE MODULE

85       Falcon system provides a loadable module called "testsuite". The module
86       is actually embedded in faltest , and is provided to all the scripts it
87       runs. The module provides the following functions:
88
89       success()
90              The script is terminated and recorded as a success.
91
92       failure( reason )
93              The script is terminated and recorded as a failure. An  optional
94              parameter  containing a description of the failure condition may
95              be optionally provided; it will be written as part of the report
96              and may be used to track which part of the test wasn't working.
97
98       testReflect( item )
99              Returns the passed item. This is used to test for engine respon‐
100              siveness to external calls and item copy through external  func‐
101              tions.
102
103       alive( percent )
104              In  tests  running  for  some human sensible time, this function
105              should be called periodically to inform the system and the  user
106              that the test is proceeding.
107
108              An  optional  "percent"  parameter  can  be provided. It will be
109              interpreted as a value between 1 and 100 representing the amount
110              of test that has been performed up to this moment.
111
112       timings( total_time, performed_ops )
113              In  case  the  execution  time is relevant for the test, like in
114              benchmarks, this function can be used  to  communicate  back  to
115              faltest  the time elapsed in the operations being tested and the
116              number  of  operations  performed.  Those  parameters  will   be
117              recorded  and eventually saved in the report file, to be used as
118              benchmarks against falcon engine modifications.
119
120       timeFactor()
121              Lengthy tests are often performed by looping over the  operation
122              to  be  tested  for  a  certain time. Benchmarks and performance
123              tests should be written so that they can normally complete in  a
124              reasonable  time,  between one and ten seconds. In case the user
125              wants the tests to perform longer, in  order  to  obtain  better
126              statistical  data,  it  can  pass the -f (time factor) option to
127              faltest command line. The time factor will be a  number  greater
128              than  1,  and  should  be used by tests that may perform lengthy
129              operation to customize the  number  of  performed  tests.   This
130              function  returns  as an integer value of the -f option given to
131              faltest or 1 if the -f option was not given.
132
133

OPTIONS

135       -c <cat>
136              Select this category and ignore the rest.
137
138
139       -C <subcat>
140              Select this subcategory and ignore the rest.
141
142
143       -d     Directory from where to load the unit tests. If  not  specified,
144              it will be the current directory.
145
146
147       -f <n> Set  time  factor to N. Some scripts may use this information to
148              perform more loops or lengthy operations.
149
150
151       -h     Show version and a short help.
152
153
154       -l     List the selected tests and exit. Combine with -v  to  have  the
155              path  of  the  tests,  as -l only lists the script ID, its short
156              name and the category/subcategory pair.
157
158
159       -m     Do not compile in memory. Use disk temporary files.
160
161
162       -M     Checks for memory leaks. This option  makes  faltest  to  report
163              unclaimed memory after each script execution, and a final report
164              at the end of the tests. The check is extended to all the engine
165              operations, so errors in the engine are detected too.
166
167
168       -o <file>
169              Write final report to the given output file.
170
171
172       -s     Perform  module  serialization  test.  Other  than compiling the
173              file, the module is also saved and then  restored  before  being
174              executed.   This allows to check for errors in module serializa‐
175              tion (that is, loading of .fam files).  The  operation  is  per‐
176              formed in memory, unless the option -m is also specified.
177
178
179       -S     Compile  via  assembly.  This  test  the correct behavior of the
180              assembler generator and compiler instead of  the  binary  module
181              generator.
182
183
184       -t     Records  and  display  timings.  The  statistics of compilation,
185              linking and execution overall times are recorded and written  as
186              part of the report.
187
188
189       -T     Records timings() calls from scripts. This allows the scripts to
190              declare their own performance ratings, and collects the  results
191              in the final report.
192
193
194       -v     Be  verbose.  Normally,  execution  and  failures  are  sparsely
195              reported. This is because the normal execution mode is meant for
196              automated runs. Tests can be executed by automated utilities and
197              errors can be reported to system administrator by simple  checks
198              on the output data.
199
200              A  developer  willing  to  fix  a broken test must run that test
201              alone with the -v enabled. A more complete error report (includ‐
202              ing  compilation or execution errors, if they were the cause for
203              the failure) will be then visualized. The -v options also allows
204              to  see the path of the original script, which is otherwise hid‐
205              den (masked by the testsuite ID).
206
207
208       -V     Prints version number and exits.
209
210

SAMPLE

212       This is a simple and complete example from the Falcon benchmark suite.
213
214       /*******************************************
215       * Falcon direct benchmarks
216       *
217       * ID: 2a
218       * Category: benchmark
219       * Subcategory: calls
220       * Short: Benchmark on function calls
221       * Description:
222       *    Performing repeated function calls and returns.
223       *    This test calls a function without parameters.
224       *
225       * [/Description]
226       ******************************************/
227
228       loops = 1000000 * timeFactor()
229       each = int(loops/10)
230
231       function toBeCalled()
232       end
233
234       // getting time
235       time = seconds()
236       for i = 1 to loops
237
238          // perform the call
239          toBeCalled()
240
241          if i % each == 0
242             alive( i/loops*100 )
243          end
244       end
245       // taking end time
246       time = seconds() - time
247
248       // subtract alive time
249       timeAlive = seconds()
250       for i = 1 to loops
251          if i % each == 0
252             alive( i/loops*100 )
253          end
254       end
255       timeAlive = seconds() - timeAlive
256       time -= timeAlive
257
258       timings( time, loops )
259
260       /* end of file */
261
262
263

FILES

265       /usr/lib/libfalcon_engine.so
266              Default location of the Falcon Engine loadable module.
267
268

AUTHOR

270       Giancarlo Niccolai <gc@falconpl.org>
271
272

SEE ALSO

274       falcon(1)
275
276

LICENSE

278       This document is released under the "GNU  Free  Documentation  License,
279       version  1.2".   On Debian systems, the complete text of the Free Docu‐
280       mentation  License,  version  1.2,  can  be  found  in  /usr/share/com‐
281       mon-licenses/.
282
283
284
285Falcon toolset                    April 2007                        faltest(1)
Impressum