1common_test(3) Erlang Module Definition common_test(3)
2
6 common_test - A framework for automated testing of any target nodes.
7
9 The Common Test framework is an environment for implementing and per‐
10 forming automatic and semi-automatic execution of test cases.
11 In brief, Common Test supports:
13 * Automated execution of test suites (sets of test cases)
15 * Logging of events during execution
17 * HTML presentation of test suite results
19 * HTML presentation of test suite code
21 * Support functions for test suite authors
23 * Step-by-step execution of test cases
25 The following section describes the mandatory and optional test suite
27 functions that Common Test calls during test execution. For more
28 details, see section Writing Test Suites in the User's Guide.
29
31 The following functions define the callback interface for a test suite.
32
34 Module:all() -> Tests | {skip,Reason}
35 Types:
37 Tests = [TestCase | {testcase,TestCase,TCRepeatProps} |
39 {group,GroupName} | {group,GroupName,Properties} |
40 {group,GroupName,Properties,SubGroups}]
41 TestCase = atom()
42 TCRepeatProps = [{repeat,N} | {repeat_until_ok,N} |
43 {repeat_until_fail,N}]
44 GroupName = atom()
45 Properties = [parallel | sequence | Shuffle | {GroupRepeat‐
46 Type,N}] | default
47 SubGroups = [{GroupName,Properties} | {GroupName,Proper‐
48 ties,SubGroups}]
49 Shuffle = shuffle | {shuffle,Seed}
50 Seed = {integer(),integer(),integer()}
51 GroupRepeatType = repeat | repeat_until_all_ok |
52 repeat_until_all_fail | repeat_until_any_ok |
53 repeat_until_any_fail
54 N = integer() | forever
55 Reason = term()
56 MANDATORY
58 Returns the list of all test cases and test case groups in the
60 test suite module to be executed. This list also specifies the
61 order the cases and groups are executed by Common Test. A test
62 case is represented by an atom, the name of the test case func‐
63 tion, or a testcase tuple indicating that the test case shall be
64 repeated. A test case group is represented by a group tuple,
65 where GroupName, an atom, is the name of the group (defined in
66 groups/0). Execution properties for groups can also be speci‐
67 fied, both for a top-level group and for any of its subgroups.
68 Group execution properties specified here override properties in
69 the group definition (see groups/0). (With value default, the
70 group definition properties are used).
71 If {skip,Reason} is returned, all test cases in the module are
73 skipped and Reason is printed on the HTML result page.
74 For details on groups, see section Test Case Groups in the
76 User's Guide.
77 Module:groups() -> GroupDefs
79 Types:
81 GroupDefs = [Group]
83 Group = {GroupName,Properties,GroupsAndTestCases}
84 GroupName = atom()
85 Properties = [parallel | sequence | Shuffle | {GroupRepeat‐
86 Type,N}]
87 GroupsAndTestCases = [Group | {group,GroupName} | TestCase |
88 {testcase,TestCase,TCRepeatProps}]
89 TestCase = atom()
90 TCRepeatProps = [{repeat,N} | {repeat_until_ok,N} |
91 {repeat_until_fail,N}]
92 Shuffle = shuffle | {shuffle,Seed}
93 Seed = {integer(),integer(),integer()}
94 GroupRepeatType = repeat | repeat_until_all_ok |
95 repeat_until_all_fail | repeat_until_any_ok |
96 repeat_until_any_fail
97 N = integer() | forever
98 OPTIONAL
100 Defines test case groups. For details, see section Test Case
102 Groups in the User's Guide.
103 Module:suite() -> [Info]
105 Types:
107 Info = {timetrap,Time} | {require,Required} |
109 {require,Name,Required} | {userdata,UserData} | {silent_con‐
110 nections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}
111 Time = TimeVal | TimeFunc
112 TimeVal = MilliSec | {seconds,integer()} | {minutes,inte‐
113 ger()} | {hours,integer()}
114 TimeFunc = {Mod,Func,Args} | Fun
115 MilliSec = integer()
116 Mod = atom()
117 Func = atom()
118 Args = list()
119 Fun = fun()
120 Required = Key | {Key,SubKeys} | {Key,SubKey} | {Key,Sub‐
121 Key,SubKeys}
122 Key = atom()
123 SubKeys = SubKey | [SubKey]
124 SubKey = atom()
125 Name = atom()
126 UserData = term()
127 Conns = [atom()]
128 CSSFile = string()
129 CTHs = [CTHModule |
130 {CTHModule, CTHInitArgs} |
131 {CTHModule, CTHInitArgs, CTHPriority}]
132 CTHModule = atom()
133 CTHInitArgs = term()
134 OPTIONAL
136 The test suite information function. Returns a list of tagged
138 tuples specifying various properties related to the execution of
139 this test suite (common for all test cases in the suite).
140 Tag timetrap sets the maximum time that each test case is
142 allowed to execute (including init_per_testcase/2 and
143 end_per_testcase/2). If the timetrap time is exceeded, the test
144 case fails with reason timetrap_timeout. A TimeFunc function can
145 be used to set a new timetrap by returning a TimeVal. It can
146 also be used to trigger a timetrap time-out by, at some point,
147 returning a value other than a TimeVal. For details, see section
148 Timetrap Time-Outs in the User's Guide.
149 Tag require specifies configuration variables required by test
151 cases (or configuration functions) in the suite. If the required
152 configuration variables are not found in any of the configura‐
153 tion files, all test cases are skipped. For details about the
154 require functionality, see funtion ct:require/1,2.
155 With userdata, the user can specify any test suite-related
157 information, which can be read by calling ct:userdata/2.
158 Tag ct_hooks specifies the Common Test Hooks to be run with this
160 suite.
161 Other tuples than the ones defined are ignored.
163 For details about the test suite information function, see sec‐
165 tion Test Suite Information Function in the User's Guide.
166 Module:init_per_suite(Config) -> NewConfig | {skip,Reason} |
168 {skip_and_save,Reason,SaveConfig}
169 Types:
171 Config = NewConfig = SaveConfig = [{Key,Value}]
173 Key = atom()
174 Value = term()
175 Reason = term()
176 OPTIONAL; if this function is defined, then end_per_suite/1 must
178 also be defined.
179 This configuration function is called as the first function in
181 the suite. It typically contains initializations that are common
182 for all test cases in the suite, and that must only be done
183 once. Parameter Config is the configuration data that can be
184 modified. Whatever is returned from this function is specified
185 as Config to all configuration functions and test cases in the
186 suite.
187 If {skip,Reason} is returned, all test cases in the suite are
189 skipped and Reason is printed in the overview log for the suite.
190 For information on save_config and skip_and_save, see section
192 Saving Configuration Data in the User's Guide.
193 Module:end_per_suite(Config) -> term() | {save_config,SaveConfig}
195 Types:
197 Config = SaveConfig = [{Key,Value}]
199 Key = atom()
200 Value = term()
201 OPTIONAL; if this function is defined, then init_per_suite/1
203 must also be defined.
204 This function is called as the last test case in the suite. It
206 is meant to be used for cleaning up after init_per_suite/1.
207 For information on save_config, see section Saving Configuration
209 Data in the User's Guide.
210 Module:group(GroupName) -> [Info]
212 Types:
214 Info = {timetrap,Time} | {require,Required} |
216 {require,Name,Required} | {userdata,UserData} | {silent_con‐
217 nections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}
218 Time = TimeVal | TimeFunc
219 TimeVal = MilliSec | {seconds,integer()} | {minutes,inte‐
220 ger()} | {hours,integer()}
221 TimeFunc = {Mod,Func,Args} | Fun
222 MilliSec = integer()
223 Mod = atom()
224 Func = atom()
225 Args = list()
226 Fun = fun()
227 Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Sub‐
228 key,SubKeys}
229 Key = atom()
230 SubKeys = SubKey | [SubKey]
231 SubKey = atom()
232 Name = atom()
233 UserData = term()
234 Conns = [atom()]
235 CSSFile = string()
236 CTHs = [CTHModule |
237 {CTHModule, CTHInitArgs} |
238 {CTHModule, CTHInitArgs, CTHPriority}]
239 CTHModule = atom()
240 CTHInitArgs = term()
241 OPTIONAL
243 The test case group information function. It is supposed to
245 return a list of tagged tuples that specify various properties
246 related to the execution of a test case group (that is, its test
247 cases and subgroups). Properties set by group/1 override proper‐
248 ties with the same key that have been set previously by suite/0.
249 Tag timetrap sets the maximum time that each test case is
251 allowed to execute (including init_per_testcase/2 and
252 end_per_testcase/2). If the timetrap time is exceeded, the test
253 case fails with reason timetrap_timeout. A TimeFunc function can
254 be used to set a new timetrap by returning a TimeVal. It can
255 also be used to trigger a timetrap time-out by, at some point,
256 returning a value other than a TimeVal. For details, see section
257 Timetrap Time-Outs in the User's Guide.
258 Tag require specifies configuration variables required by test
260 cases (or configuration functions) in the suite. If the required
261 configuration variables are not found in any of the configura‐
262 tion files, all test cases in this group are skipped. For
263 details about the require functionality, see function
264 ct:require/1,2.
265 With userdata, the user can specify any test case group related
267 information that can be read by calling ct:userdata/2.
268 Tag ct_hooks specifies the Common Test Hooks to be run with this
270 suite.
271 Other tuples than the ones defined are ignored.
273 For details about the test case group information function, see
275 section Group Information Function in the User's Guide.
276 Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason}
278 Types:
280 GroupName = atom()
282 Config = NewConfig = [{Key,Value}]
283 Key = atom()
284 Value = term()
285 Reason = term()
286 OPTIONAL; if this function is defined, then end_per_group/2 must
288 also be defined.
289 This configuration function is called before execution of a test
291 case group. It typically contains initializations that are com‐
292 mon for all test cases and subgroups in the group, and that must
293 only be performed once. GroupName is the name of the group, as
294 specified in the group definition (see groups/0). Parameter Con‐
295 fig is the configuration data that can be modified. The return
296 value of this function is given as Config to all test cases and
297 subgroups in the group.
298 If {skip,Reason} is returned, all test cases in the group are
300 skipped and Reason is printed in the overview log for the group.
301 For information about test case groups, see section Test Case
303 Groups in the User's Guide.
304 Module:end_per_group(GroupName, Config) -> term() |
306 {return_group_result,Status}
307 Types:
309 GroupName = atom()
311 Config = [{Key,Value}]
312 Key = atom()
313 Value = term()
314 Status = ok | skipped | failed
315 OPTIONAL; if this function is defined, then init_per_group/2
317 must also be defined.
318 This function is called after the execution of a test case group
320 is finished. It is meant to be used for cleaning up after
321 init_per_group/2. A status value for a nested subgroup can be
322 returned with {return_group_result,Status}. The status can be
323 retrieved in end_per_group/2 for the group on the level above.
324 The status is also used by Common Test for deciding if execution
325 of a group is to proceed if property sequence or repeat_until_*
326 is set.
327 For details about test case groups, see section Test Case Groups
329 in the User's Guide.
330 Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason}
332 | {skip,Reason}
333 Types:
335 TestCase = atom()
337 Config = NewConfig = [{Key,Value}]
338 Key = atom()
339 Value = term()
340 Reason = term()
341 OPTIONAL; if this function is defined, then end_per_testcase/2
343 must also be defined.
344 This function is called before each test case. Argument TestCase
346 is the test case name, and Config (list of key-value tuples) is
347 the configuration data that can be modified. The NewConfig list
348 returned from this function is given as Config to the test case.
349 If {fail,Reason} is returned, the test case is marked as failed
350 without being executed.
351 If {skip,Reason} is returned, the test case is skipped and Rea‐
353 son is printed in the overview log for the suite.
354 Module:end_per_testcase(TestCase, Config) -> term() | {fail,Reason} |
356 {save_config,SaveConfig}
357 Types:
359 TestCase = atom()
361 Config = SaveConfig = [{Key,Value}]
362 Key = atom()
363 Value = term()
364 Reason = term()
365 OPTIONAL; if this function is defined, then init_per_testcase/2
367 must also be defined.
368 This function is called after each test case, and can be used to
370 clean up after init_per_testcase/2 and the test case. Any return
371 value (besides {fail,Reason} and {save_config,SaveConfig}) is
372 ignored. By returning {fail,Reason}, TestCase is marked as
373 faulty (even though it was successful in the sense that it
374 returned a value instead of terminating).
375 For information on save_config, see section Saving Configuration
377 Data in the User's Guide.
378 Module:Testcase() -> [Info]
380 Types:
382 Info = {timetrap,Time} | {require,Required} |
384 {require,Name,Required} | {userdata,UserData} | {silent_con‐
385 nections,Conns}
386 Time = TimeVal | TimeFunc
387 TimeVal = MilliSec | {seconds,integer()} | {minutes,inte‐
388 ger()} | {hours,integer()}
389 TimeFunc = {Mod,Func,Args} | Fun
390 MilliSec = integer()
391 Mod = atom()
392 Func = atom()
393 Args = list()
394 Fun = fun()
395 Required = Key | {Key,SubKeys} | {Key,Subkey} | {Key,Sub‐
396 key,SubKeys}
397 Key = atom()
398 SubKeys = SubKey | [SubKey]
399 SubKey = atom()
400 Name = atom()
401 UserData = term()
402 Conns = [atom()]
403 OPTIONAL
405 The test case information function. It is supposed to return a
407 list of tagged tuples that specify various properties related to
408 the execution of this particular test case. Properties set by
409 Testcase/0 override properties set previously for the test case
410 by group/1 or suite/0.
411 Tag timetrap sets the maximum time that the test case is allowed
413 to execute. If the timetrap time is exceeded, the test case
414 fails with reason timetrap_timeout. init_per_testcase/2 and
415 end_per_testcase/2 are included in the timetrap time. A TimeFunc
416 function can be used to set a new timetrap by returning a
417 TimeVal. It can also be used to trigger a timetrap time-out by,
418 at some point, returning a value other than a TimeVal. For
419 details, see section Timetrap Time-Outs in the User's Guide.
420 Tag require specifies configuration variables that are required
422 by the test case (or init_per_testcase/2 or end_per_testcase/2).
423 If the required configuration variables are not found in any of
424 the configuration files, the test case is skipped. For details
425 about the require functionality, see function ct:require/1,2.
426 If timetrap or require is not set, the default values specified
428 by suite/0 (or group/1) are used.
429 With userdata, the user can specify any test case-related infor‐
431 mation that can be read by calling ct:userdata/3.
432 Other tuples than the ones defined are ignored.
434 For details about the test case information function, see sec‐
436 tion Test Case Information Function in the User's Guide.
437 Module:Testcase(Config) -> term() | {skip,Reason} | {comment,Comment} |
439 {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit()
440 Types:
442 Config = SaveConfig = [{Key,Value}]
444 Key = atom()
445 Value = term()
446 Reason = term()
447 Comment = string()
448 MANDATORY
450 The implementation of a test case. Call the functions to test
452 and check the result. If something fails, ensure the function
453 causes a runtime error or call ct:fail/1,2 (which also causes
454 the test case process to terminate).
455 Elements from the Config list can, for example, be read with
457 proplists:get_value/2 in STDLIB (or the macro ?config defined in
458 ct.hrl).
459 If you decide not to run the test case after all, return
461 {skip,Reason}. Reason is then printed in field Comment on the
462 HTML result page.
463 To print some information in field Comment on the HTML result
465 page, return {comment,Comment}.
466 If the function returns anything else, the test case is consid‐
468 ered successful. The return value always gets printed in the
469 test case log file.
470 For details about test case implementation, see section Test
472 Cases in the User's Guide.
473 For information on save_config and skip_and_save, see section
475 Saving Configuration Data in the User's Guide.
476Ericsson AB common_test 1.18 common_test(3)