1common_test(3)             Erlang Module Definition             common_test(3)
2
3
4

NAME

6       common_test - A framework for automated testing of any target nodes.
7

DESCRIPTION

9       The  Common  Test framework is an environment for implementing and per‐
10       forming automatic and semi-automatic execution of test cases.
11
12       In brief, Common Test supports:
13
14         * Automated execution of test suites (sets of test cases)
15
16         * Logging of events during execution
17
18         * HTML presentation of test suite results
19
20         * HTML presentation of test suite code
21
22         * Support functions for test suite authors
23
24         * Step-by-step execution of test cases
25
26       The following section describes the mandatory and optional  test  suite
27       functions  that  Common  Test  calls  during  test  execution. For more
28       details, see section Writing Test Suites in the User's Guide.
29

TEST CASE CALLBACK FUNCTIONS

31       The following functions define the callback interface for a test suite.
32

EXPORTS

34       Module:all() -> Tests | {skip,Reason}
35
36              Types:
37
38                 Tests  =  [TestCase  |  {testcase,TestCase,TCRepeatProps}   |
39                 {group,GroupName}     |     {group,GroupName,Properties}    |
40                 {group,GroupName,Properties,SubGroups}]
41                 TestCase = atom()
42                 TCRepeatProps   =   [{repeat,N}   |   {repeat_until_ok,N}   |
43                 {repeat_until_fail,N}]
44                 GroupName = atom()
45                 Properties  =  [parallel | sequence | Shuffle | {GroupRepeat‐
46                 Type,N}] | default
47                 SubGroups  =  [{GroupName,Properties}  |   {GroupName,Proper‐
48                 ties,SubGroups}]
49                 Shuffle = shuffle | {shuffle,Seed}
50                 Seed = {integer(),integer(),integer()}
51                 GroupRepeatType    =    repeat    |   repeat_until_all_ok   |
52                 repeat_until_all_fail      |      repeat_until_any_ok       |
53                 repeat_until_any_fail
54                 N = integer() | forever
55                 Reason = term()
56
57              MANDATORY
58
59              Returns  the  list of all test cases and test case groups in the
60              test suite module to be executed. This list also  specifies  the
61              order  the  cases and groups are executed by Common Test. A test
62              case is represented by an atom, the name of the test case  func‐
63              tion, or a testcase tuple indicating that the test case shall be
64              repeated. A test case group is represented  by  a  group  tuple,
65              where  GroupName,  an atom, is the name of the group (defined in
66              groups/0). Execution properties for groups can  also  be  speci‐
67              fied,  both  for a top-level group and for any of its subgroups.
68              Group execution properties specified here override properties in
69              the  group  definition  (see groups/0). (With value default, the
70              group definition properties are used).
71
72              If {skip,Reason} is returned, all test cases in the  module  are
73              skipped and Reason is printed on the HTML result page.
74
75              For  details  on  groups,  see  section  Test Case Groups in the
76              User's Guide.
77
78       Module:groups() -> GroupDefs
79
80              Types:
81
82                 GroupDefs = [Group]
83                 Group = {GroupName,Properties,GroupsAndTestCases}
84                 GroupName = atom()
85                 Properties = [parallel | sequence | Shuffle  |  {GroupRepeat‐
86                 Type,N}]
87                 GroupsAndTestCases  = [Group | {group,GroupName} | TestCase |
88                 {testcase,TestCase,TCRepeatProps}]
89                 TestCase = atom()
90                 TCRepeatProps   =   [{repeat,N}   |   {repeat_until_ok,N}   |
91                 {repeat_until_fail,N}]
92                 Shuffle = shuffle | {shuffle,Seed}
93                 Seed = {integer(),integer(),integer()}
94                 GroupRepeatType    =    repeat    |   repeat_until_all_ok   |
95                 repeat_until_all_fail      |      repeat_until_any_ok       |
96                 repeat_until_any_fail
97                 N = integer() | forever
98
99              OPTIONAL
100
101              Defines  test  case  groups.  For details, see section Test Case
102              Groups in the User's Guide.
103
104       Module:suite() -> [Info]
105
106              Types:
107
108                 Info    =    {timetrap,Time}    |    {require,Required}     |
109                 {require,Name,Required}  | {userdata,UserData} | {silent_con‐
110                 nections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}
111                 Time = TimeVal | TimeFunc
112                 TimeVal = MilliSec  |  {seconds,integer()}  |  {minutes,inte‐
113                 ger()} | {hours,integer()}
114                 TimeFunc = {Mod,Func,Args} | Fun
115                 MilliSec = integer()
116                 Mod = atom()
117                 Func = atom()
118                 Args = list()
119                 Fun = fun()
120                 Required  =  Key  |  {Key,SubKeys} | {Key,SubKey} | {Key,Sub‐
121                 Key,SubKeys}
122                 Key = atom()
123                 SubKeys = SubKey | [SubKey]
124                 SubKey = atom()
125                 Name = atom()
126                 UserData = term()
127                 Conns = [atom()]
128                 CSSFile = string()
129                 CTHs = [CTHModule |
130                  {CTHModule, CTHInitArgs} |
131                  {CTHModule, CTHInitArgs, CTHPriority}]
132                 CTHModule = atom()
133                 CTHInitArgs = term()
134
135              OPTIONAL
136
137              The test suite information function. Returns a  list  of  tagged
138              tuples specifying various properties related to the execution of
139              this test suite (common for all test cases in the suite).
140
141              Tag timetrap sets the  maximum  time  that  each  test  case  is
142              allowed    to   execute   (including   init_per_testcase/2   and
143              end_per_testcase/2). If the timetrap time is exceeded, the  test
144              case fails with reason timetrap_timeout. A TimeFunc function can
145              be used to set a new timetrap by returning  a  TimeVal.  It  can
146              also  be  used to trigger a timetrap time-out by, at some point,
147              returning a value other than a TimeVal. For details, see section
148              Timetrap Time-Outs in the User's Guide.
149
150              Tag  require  specifies configuration variables required by test
151              cases (or configuration functions) in the suite. If the required
152              configuration  variables  are not found in any of the configura‐
153              tion files, all test cases are skipped. For  details  about  the
154              require functionality, see funtion ct:require/1,2.
155
156              With  userdata,  the  user  can  specify  any test suite-related
157              information, which can be read by calling ct:userdata/2.
158
159              Tag ct_hooks specifies the Common Test Hooks to be run with this
160              suite.
161
162              Other tuples than the ones defined are ignored.
163
164              For  details about the test suite information function, see sec‐
165              tion Test Suite Information Function in the User's Guide.
166
167       Module:init_per_suite(Config)   ->   NewConfig   |   {skip,Reason}    |
168       {skip_and_save,Reason,SaveConfig}
169
170              Types:
171
172                 Config = NewConfig = SaveConfig = [{Key,Value}]
173                 Key = atom()
174                 Value = term()
175                 Reason = term()
176
177              OPTIONAL; if this function is defined, then end_per_suite/1 must
178              also be defined.
179
180              This configuration function is called as the first  function  in
181              the suite. It typically contains initializations that are common
182              for all test cases in the suite, and  that  must  only  be  done
183              once.  Parameter  Config  is  the configuration data that can be
184              modified. Whatever is returned from this function  is  specified
185              as  Config  to all configuration functions and test cases in the
186              suite.
187
188              If {skip,Reason} is returned, all test cases in  the  suite  are
189              skipped and Reason is printed in the overview log for the suite.
190
191              For  information  on  save_config and skip_and_save, see section
192              Saving Configuration Data in the User's Guide.
193
194       Module:end_per_suite(Config) -> term() | {save_config,SaveConfig}
195
196              Types:
197
198                 Config = SaveConfig = [{Key,Value}]
199                 Key = atom()
200                 Value = term()
201
202              OPTIONAL; if this function  is  defined,  then  init_per_suite/1
203              must also be defined.
204
205              This  function  is called as the last test case in the suite. It
206              is meant to be used for cleaning up after init_per_suite/1.
207
208              For information on save_config, see section Saving Configuration
209              Data in the User's Guide.
210
211       Module:group(GroupName) -> [Info]
212
213              Types:
214
215                 Info     =    {timetrap,Time}    |    {require,Required}    |
216                 {require,Name,Required} | {userdata,UserData} |  {silent_con‐
217                 nections,Conns} | {stylesheet,CSSFile} | {ct_hooks, CTHs}
218                 Time = TimeVal | TimeFunc
219                 TimeVal  =  MilliSec  |  {seconds,integer()} | {minutes,inte‐
220                 ger()} | {hours,integer()}
221                 TimeFunc = {Mod,Func,Args} | Fun
222                 MilliSec = integer()
223                 Mod = atom()
224                 Func = atom()
225                 Args = list()
226                 Fun = fun()
227                 Required = Key | {Key,SubKeys}  |  {Key,Subkey}  |  {Key,Sub‐
228                 key,SubKeys}
229                 Key = atom()
230                 SubKeys = SubKey | [SubKey]
231                 SubKey = atom()
232                 Name = atom()
233                 UserData = term()
234                 Conns = [atom()]
235                 CSSFile = string()
236                 CTHs = [CTHModule |
237                  {CTHModule, CTHInitArgs} |
238                  {CTHModule, CTHInitArgs, CTHPriority}]
239                 CTHModule = atom()
240                 CTHInitArgs = term()
241
242              OPTIONAL
243
244              The  test  case  group  information  function. It is supposed to
245              return a list of tagged tuples that specify  various  properties
246              related to the execution of a test case group (that is, its test
247              cases and subgroups). Properties set by group/1 override proper‐
248              ties with the same key that have been set previously by suite/0.
249
250              Tag  timetrap  sets  the  maximum  time  that  each test case is
251              allowed   to   execute   (including   init_per_testcase/2    and
252              end_per_testcase/2).  If the timetrap time is exceeded, the test
253              case fails with reason timetrap_timeout. A TimeFunc function can
254              be  used  to  set  a new timetrap by returning a TimeVal. It can
255              also be used to trigger a timetrap time-out by, at  some  point,
256              returning a value other than a TimeVal. For details, see section
257              Timetrap Time-Outs in the User's Guide.
258
259              Tag require specifies configuration variables required  by  test
260              cases (or configuration functions) in the suite. If the required
261              configuration variables are not found in any of  the  configura‐
262              tion  files,  all  test  cases  in  this  group are skipped. For
263              details  about   the   require   functionality,   see   function
264              ct:require/1,2.
265
266              With  userdata, the user can specify any test case group related
267              information that can be read by calling ct:userdata/2.
268
269              Tag ct_hooks specifies the Common Test Hooks to be run with this
270              suite.
271
272              Other tuples than the ones defined are ignored.
273
274              For  details about the test case group information function, see
275              section Group Information Function in the User's Guide.
276
277       Module:init_per_group(GroupName, Config) -> NewConfig | {skip,Reason}
278
279              Types:
280
281                 GroupName = atom()
282                 Config = NewConfig = [{Key,Value}]
283                 Key = atom()
284                 Value = term()
285                 Reason = term()
286
287              OPTIONAL; if this function is defined, then end_per_group/2 must
288              also be defined.
289
290              This configuration function is called before execution of a test
291              case group. It typically contains initializations that are  com‐
292              mon for all test cases and subgroups in the group, and that must
293              only be performed once. GroupName is the name of the  group,  as
294              specified in the group definition (see groups/0). Parameter Con‐
295              fig is the configuration data that can be modified.  The  return
296              value  of this function is given as Config to all test cases and
297              subgroups in the group.
298
299              If {skip,Reason} is returned, all test cases in  the  group  are
300              skipped and Reason is printed in the overview log for the group.
301
302              For  information  about  test case groups, see section Test Case
303              Groups in the User's Guide.
304
305       Module:end_per_group(GroupName,      Config)      ->      term()      |
306       {return_group_result,Status}
307
308              Types:
309
310                 GroupName = atom()
311                 Config = [{Key,Value}]
312                 Key = atom()
313                 Value = term()
314                 Status = ok | skipped | failed
315
316              OPTIONAL;  if  this  function  is defined, then init_per_group/2
317              must also be defined.
318
319              This function is called after the execution of a test case group
320              is  finished.  It  is  meant  to  be  used for cleaning up after
321              init_per_group/2. A status value for a nested  subgroup  can  be
322              returned  with  {return_group_result,Status}.  The status can be
323              retrieved in end_per_group/2 for the group on the  level  above.
324              The status is also used by Common Test for deciding if execution
325              of a group is to proceed if property sequence or  repeat_until_*
326              is set.
327
328              For details about test case groups, see section Test Case Groups
329              in the User's Guide.
330
331       Module:init_per_testcase(TestCase, Config) -> NewConfig | {fail,Reason}
332       | {skip,Reason}
333
334              Types:
335
336                  TestCase = atom()
337                  Config = NewConfig = [{Key,Value}]
338                  Key = atom()
339                  Value = term()
340                  Reason = term()
341
342              OPTIONAL;  if  this function is defined, then end_per_testcase/2
343              must also be defined.
344
345              This function is called before each test case. Argument TestCase
346              is  the test case name, and Config (list of key-value tuples) is
347              the configuration data that can be modified. The NewConfig  list
348              returned from this function is given as Config to the test case.
349              If {fail,Reason} is returned, the test case is marked as  failed
350              without being executed.
351
352              If  {skip,Reason} is returned, the test case is skipped and Rea‐
353              son is printed in the overview log for the suite.
354
355       Module:end_per_testcase(TestCase, Config) -> term() |  {fail,Reason}  |
356       {save_config,SaveConfig}
357
358              Types:
359
360                 TestCase = atom()
361                 Config = SaveConfig = [{Key,Value}]
362                 Key = atom()
363                 Value = term()
364                 Reason = term()
365
366              OPTIONAL;  if this function is defined, then init_per_testcase/2
367              must also be defined.
368
369              This function is called after each test case, and can be used to
370              clean up after init_per_testcase/2 and the test case. Any return
371              value (besides {fail,Reason}  and  {save_config,SaveConfig})  is
372              ignored.  By  returning  {fail,Reason},  TestCase  is  marked as
373              faulty (even though it was  successful  in  the  sense  that  it
374              returned a value instead of terminating).
375
376              For information on save_config, see section Saving Configuration
377              Data in the User's Guide.
378
379       Module:Testcase() -> [Info]
380
381              Types:
382
383                 Info    =    {timetrap,Time}    |    {require,Required}     |
384                 {require,Name,Required}  | {userdata,UserData} | {silent_con‐
385                 nections,Conns}
386                 Time = TimeVal | TimeFunc
387                 TimeVal = MilliSec  |  {seconds,integer()}  |  {minutes,inte‐
388                 ger()} | {hours,integer()}
389                 TimeFunc = {Mod,Func,Args} | Fun
390                 MilliSec = integer()
391                 Mod = atom()
392                 Func = atom()
393                 Args = list()
394                 Fun = fun()
395                 Required  =  Key  |  {Key,SubKeys} | {Key,Subkey} | {Key,Sub‐
396                 key,SubKeys}
397                 Key = atom()
398                 SubKeys = SubKey | [SubKey]
399                 SubKey = atom()
400                 Name = atom()
401                 UserData = term()
402                 Conns = [atom()]
403
404              OPTIONAL
405
406              The test case information function. It is supposed to  return  a
407              list of tagged tuples that specify various properties related to
408              the execution of this particular test case.  Properties  set  by
409              Testcase/0  override properties set previously for the test case
410              by group/1 or suite/0.
411
412              Tag timetrap sets the maximum time that the test case is allowed
413              to  execute.  If  the  timetrap  time is exceeded, the test case
414              fails  with  reason  timetrap_timeout.  init_per_testcase/2  and
415              end_per_testcase/2 are included in the timetrap time. A TimeFunc
416              function can be used to  set  a  new  timetrap  by  returning  a
417              TimeVal.  It can also be used to trigger a timetrap time-out by,
418              at some point, returning a  value  other  than  a  TimeVal.  For
419              details, see section Timetrap Time-Outs in the User's Guide.
420
421              Tag  require specifies configuration variables that are required
422              by the test case (or init_per_testcase/2 or end_per_testcase/2).
423              If  the required configuration variables are not found in any of
424              the configuration files, the test case is skipped.  For  details
425              about the require functionality, see function ct:require/1,2.
426
427              If  timetrap or require is not set, the default values specified
428              by suite/0 (or group/1) are used.
429
430              With userdata, the user can specify any test case-related infor‐
431              mation that can be read by calling ct:userdata/3.
432
433              Other tuples than the ones defined are ignored.
434
435              For  details  about the test case information function, see sec‐
436              tion Test Case Information Function in the User's Guide.
437
438       Module:Testcase(Config) -> term() | {skip,Reason} | {comment,Comment} |
439       {save_config,SaveConfig} | {skip_and_save,Reason,SaveConfig} | exit()
440
441              Types:
442
443                 Config = SaveConfig = [{Key,Value}]
444                 Key = atom()
445                 Value = term()
446                 Reason = term()
447                 Comment = string()
448
449              MANDATORY
450
451              The  implementation  of  a test case. Call the functions to test
452              and check the result. If something fails,  ensure  the  function
453              causes  a  runtime  error or call ct:fail/1,2 (which also causes
454              the test case process to terminate).
455
456              Elements from the Config list can, for  example,  be  read  with
457              proplists:get_value/2 in STDLIB (or the macro ?config defined in
458              ct.hrl).
459
460              If you decide not  to  run  the  test  case  after  all,  return
461              {skip,Reason}.  Reason  is  then printed in field Comment on the
462              HTML result page.
463
464              To print some information in field Comment on  the  HTML  result
465              page, return {comment,Comment}.
466
467              If  the function returns anything else, the test case is consid‐
468              ered successful. The return value always  gets  printed  in  the
469              test case log file.
470
471              For  details  about  test  case implementation, see section Test
472              Cases in the User's Guide.
473
474              For information on save_config and  skip_and_save,  see  section
475              Saving Configuration Data in the User's Guide.
476
477
478
479Ericsson AB                    common_test 1.18                 common_test(3)
Impressum