1ct_hooks(3)                Erlang Module Definition                ct_hooks(3)
2
3
4

NAME

6       ct_hooks - A callback interface on top of Common Test.
7

DESCRIPTION

9       The  Common  Test Hook (CTH) framework allows extensions of the default
10       behavior of Common Test by callbacks before and after  all  test  suite
11       calls. It is intended for advanced users of Common Test who want to ab‐
12       stract out behavior that is common to multiple test suites.
13
14       In brief, CTH allows you to:
15
16         * Manipulate the runtime configuration before each  suite  configura‐
17           tion call.
18
19         * Manipulate  the  return of all suite configuration calls and by ex‐
20           tension the result of the test themselves.
21
22       The following sections describe the mandatory and  optional  CTH  func‐
23       tions  that  Common Test calls during test execution. For more details,
24       see section Common Test Hooks in the User's Guide.
25
26       For information about how to add a CTH to your suite, see  section  In‐
27       stalling a CTH in the User's Guide.
28
29   Note:
30       For  a  minimal example of a CTH, see section Example CTH in the User's
31       Guide.
32
33

CALLBACK FUNCTIONS

35       The following functions define the callback interface for a CTH.
36

EXPORTS

38       Module:init(Id, Opts) -> {ok, State} | {ok, State, Priority}
39
40              Types:
41
42                 Id = reference() | term()
43                 Opts = term()
44                 State = term()
45                 Priority = integer()
46
47              MANDATORY
48
49              This function is always called before any other  callback  func‐
50              tion.  Use  it  to  initiate any common state. It is to return a
51              state for this CTH.
52
53              Id is either the return value of ct_hooks:id/1, or  a  reference
54              (created  using  erlang:make_ref/0  in ERTS) if ct_hooks:id/1 is
55              not implemented.
56
57              Priority is the relative priority of this  hook.  Hooks  with  a
58              lower  priority are executed first. If no priority is specified,
59              it is set to 0.
60
61              For details about hook execution order, see section  CTH  Execu‐
62              tion Order in the User's Guide.
63
64              For  details about when init is called, see section CTH Scope in
65              the User's Guide.
66
67       Module:post_groups(SuiteName, GroupDefs) -> NewGroupDefs
68
69              Types:
70
71                 SuiteName = atom()
72                 GroupDefs = NewGroupDefs = [Group]
73                 Group = {GroupName,Properties,GroupsAndTestCases}
74                 GroupName = atom()
75                 Properties = [parallel | sequence | Shuffle  |  {GroupRepeat‐
76                 Type,N}]
77                 GroupsAndTestCases  = [Group | {group,GroupName} | TestCase |
78                 {testcase,TestCase,TCRepeatProps}]
79                 TestCase = atom()
80                 TCRepeatProps =  [{repeat,N}  |  {repeat_until_ok,N}  |  {re‐
81                 peat_until_fail,N}]
82                 Shuffle = shuffle | {shuffle,Seed}
83                 Seed = {integer(),integer(),integer()}
84                 GroupRepeatType  =  repeat | repeat_until_all_ok | repeat_un‐
85                 til_all_fail | repeat_until_any_ok | repeat_until_any_fail
86                 N = integer() | forever
87
88              OPTIONAL
89
90              This function is called after groups/0. It is used to modify the
91              test  group definitions, for instance to add or remove groups or
92              change group properties.
93
94              GroupDefs is what groups/0 returned, that is, a  list  of  group
95              definitions.
96
97              NewGroupDefs is the possibly modified version of this list.
98
99              This  function  is  called  only  if  the  CTH  is  added before
100              init_per_suite is run. For details, see section CTH Scope in the
101              User's Guide.
102
103              Notice  that for CTHs that are installed by means of the suite/0
104              function, post_groups/2 is called before the init/2  hook  func‐
105              tion.  However,  for  CTHs that are installed by means of the CT
106              start flag, the init/2 function is called first.
107
108          Note:
109              Prior to each test execution, Common Test does a simulated  test
110              run  in order to count test suites, groups and cases for logging
111              purposes. This causes the post_groups/2 hook function to  always
112              be  called twice. For this reason, side effects are best avoided
113              in this callback.
114
115
116       Module:post_all(SuiteName, Return, GroupDefs) -> NewReturn
117
118              Types:
119
120                 SuiteName = atom()
121                 Return = NewReturn = Tests | {skip,Reason}
122                 Tests  =  [TestCase  |  {testcase,TestCase,TCRepeatProps}   |
123                 {group,GroupName}     |     {group,GroupName,Properties}    |
124                 {group,GroupName,Properties,SubGroups}]
125                 TestCase = atom()
126                 TCRepeatProps =  [{repeat,N}  |  {repeat_until_ok,N}  |  {re‐
127                 peat_until_fail,N}]
128                 GroupName = atom()
129                 Properties = GroupProperties | default
130                 SubGroups   =  [{GroupName,Properties}  |  {GroupName,Proper‐
131                 ties,SubGroups}]
132                 Shuffle = shuffle | {shuffle,Seed}
133                 Seed = {integer(),integer(),integer()}
134                 GroupRepeatType = repeat | repeat_until_all_ok  |  repeat_un‐
135                 til_all_fail | repeat_until_any_ok | repeat_until_any_fail
136                 N = integer() | forever
137                 GroupDefs = NewGroupDefs = [Group]
138                 Group = {GroupName,GroupProperties,GroupsAndTestCases}
139                 GroupProperties  = [parallel | sequence | Shuffle | {GroupRe‐
140                 peatType,N}]
141                 GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
142                 Reason = term()
143
144              OPTIONAL
145
146              This function is called after all/0. It is used  to  modify  the
147              set of test cases and test group to be executed, for instance to
148              add or remove test cases and groups, change group properties, or
149              even skip all tests in the suite.
150
151              Return is what all/0 returned, that is, a list of test cases and
152              groups to be executed, or a tuple {skip,Reason}.
153
154              GroupDefs is what groups/0 or the post_groups/2  hook  returned,
155              that is, a list of group definitions.
156
157              NewReturn is the possibly modified version of Return.
158
159              This  function  is  called  only  if  the  CTH  is  added before
160              init_per_suite is run. For details, see section CTH Scope in the
161              User's Guide.
162
163              Notice  that for CTHs that are installed by means of the suite/0
164              function, post_all/2 is called before the init/2 hook  function.
165              However,  for  CTHs  that are installed by means of the CT start
166              flag, the init/2 function is called first.
167
168          Note:
169              Prior to each test execution, Common Test does a simulated  test
170              run  in order to count test suites, groups and cases for logging
171              purposes. This causes the post_all/3 hook function to always  be
172              called  twice. For this reason, side effects are best avoided in
173              this callback.
174
175
176       Module:pre_init_per_suite(SuiteName, InitData, CTHState) -> Result
177
178              Types:
179
180                 SuiteName = atom()
181                 InitData = Config | SkipOrFail
182                 Config = NewConfig = [{Key,Value}]
183                 CTHState = NewCTHState = term()
184                 Result = {Return, NewCTHState}
185                 Return = NewConfig | SkipOrFail
186                 SkipOrFail = {fail, Reason} | {skip, Reason}
187                 Key = atom()
188                 Value = term()
189                 Reason = term()
190
191              OPTIONAL
192
193              This function is called before init_per_suite if it  exists.  It
194              typically  contains initialization/logging that must be done be‐
195              fore init_per_suite is called. If {skip,Reason} or {fail,Reason}
196              is  returned, init_per_suite and all test cases of the suite are
197              skipped and Reason printed in the overview log of the suite.
198
199              SuiteName is the name of the suite to be run.
200
201              InitData is the original configuration list of the  test  suite,
202              or a SkipOrFail tuple if a previous CTH has returned this.
203
204              CTHState is the current internal state of the CTH.
205
206              Return  is  the  result of the init_per_suite function. If it is
207              {skip,Reason} or {fail,Reason}, init_per_suite is never  called,
208              instead  the  initiation  is considered to be skipped or failed,
209              respectively. If a NewConfig list is returned, init_per_suite is
210              called  with  that NewConfig list. For more details, see section
211              Pre Hooks in the User's Guide.
212
213              This function  is  called  only  if  the  CTH  is  added  before
214              init_per_suite is run. For details, see section CTH Scope in the
215              User's Guide.
216
217       Module:post_init_per_suite(SuiteName, Config, Return, CTHState) ->  Re‐
218       sult
219
220              Types:
221
222                 SuiteName = atom()
223                 Config = [{Key,Value}]
224                 Return = NewReturn = Config | SkipOrFail | term()
225                 SkipOrFail = {fail, Reason} | {skip, Reason} | term()
226                 CTHState = NewCTHState = term()
227                 Result = {NewReturn, NewCTHState}
228                 Key = atom()
229                 Value = term()
230                 Reason = term()
231
232              OPTIONAL
233
234              This  function  is  called after init_per_suite if it exists. It
235              typically contains extra checks to ensure that all  the  correct
236              dependencies are started correctly.
237
238              Return  is what init_per_suite returned, that is, {fail,Reason},
239              {skip,Reason},  a  Config  list,  or  a  term   describing   how
240              init_per_suite failed.
241
242              NewReturn   is   the   possibly   modified   return   value   of
243              init_per_suite. To recover from a failure in init_per_suite, re‐
244              turn ConfigList with the tc_status element removed. For more de‐
245              tails, see  Post Hooks in section "Manipulating  Tests"  in  the
246              User's Guide.
247
248              CTHState is the current internal state of the CTH.
249
250              This  function  is  called only if the CTH is added before or in
251              init_per_suite. For details, see section CTH Scope in the User's
252              Guide.
253
254       Module:pre_init_per_group(SuiteName,  GroupName, InitData, CTHState) ->
255       Result
256
257              Types:
258
259                 SuiteName = atom()
260                 GroupName = atom()
261                 InitData = Config | SkipOrFail
262                 Config = NewConfig = [{Key,Value}]
263                 CTHState = NewCTHState = term()
264                 Result = {NewConfig | SkipOrFail, NewCTHState}
265                 SkipOrFail = {fail,Reason} | {skip, Reason}
266                 Key = atom()
267                 Value = term()
268                 Reason = term()
269
270              OPTIONAL
271
272              This function is called before init_per_group if it  exists.  It
273              behaves  the  same  way  as pre_init_per_suite, but for function
274              init_per_group instead.
275
276              If Module:pre_init_per_group/4 is not exported, common_test will
277              attempt  to  call Module:pre_init_per_group(GroupName, InitData,
278              CTHState) instead. This is for backwards compatibility.
279
280       Module:post_init_per_group(SuiteName, GroupName, Config,  Return,  CTH‐
281       State) -> Result
282
283              Types:
284
285                 SuiteName = atom()
286                 GroupName = atom()
287                 Config = [{Key,Value}]
288                 Return = NewReturn = Config | SkipOrFail | term()
289                 SkipOrFail = {fail,Reason} | {skip, Reason}
290                 CTHState = NewCTHState = term()
291                 Result = {NewReturn, NewCTHState}
292                 Key = atom()
293                 Value = term()
294                 Reason = term()
295
296              OPTIONAL
297
298              This  function  is  called after init_per_group if it exists. It
299              behaves the same way as post_init_per_suite,  but  for  function
300              init_per_group instead.
301
302              If  Module:post_init_per_group/5  is  not  exported, common_test
303              will attempt to call Module:post_init_per_group(GroupName,  Con‐
304              fig,  Return,  CTHState) instead. This is for backwards compati‐
305              bility.
306
307       Module:pre_init_per_testcase(SuiteName,  TestcaseName,  InitData,  CTH‐
308       State) -> Result
309
310              Types:
311
312                 SuiteName = atom()
313                 TestcaseName = atom()
314                 InitData = Config | SkipOrFail
315                 Config = NewConfig = [{Key,Value}]
316                 CTHState = NewCTHState = term()
317                 Result = {NewConfig | SkipOrFail, NewCTHState}
318                 SkipOrFail = {fail,Reason} | {skip, Reason}
319                 Key = atom()
320                 Value = term()
321                 Reason = term()
322
323              OPTIONAL
324
325              This  function  is called before init_per_testcase if it exists.
326              It behaves the same way as pre_init_per_suite, but for  function
327              init_per_testcase instead.
328
329              If  Module:pre_init_per_testcase/4  is not exported, common_test
330              will attempt to call  Module:pre_init_per_testcase(TestcaseName,
331              InitData,  CTHState)  instead. This is for backwards compatibil‐
332              ity.
333
334              CTHs cannot be added here right now. That feature may  be  added
335              in  a later release, but it would right now break backwards com‐
336              patibility.
337
338       Module:post_init_per_testcase(SuiteName, TestcaseName, Config,  Return,
339       CTHState) -> Result
340
341              Types:
342
343                 SuiteName = atom()
344                 TestcaseName = atom()
345                 Config = [{Key,Value}]
346                 Return = NewReturn = Config | SkipOrFail | term()
347                 SkipOrFail = {fail,Reason} | {skip, Reason}
348                 CTHState = NewCTHState = term()
349                 Result = {NewReturn, NewCTHState}
350                 Key = atom()
351                 Value = term()
352                 Reason = term()
353
354              OPTIONAL
355
356              This function is called after init_per_testcase if it exists. It
357              behaves the same way as post_init_per_suite,  but  for  function
358              init_per_testcase instead.
359
360              If  Module:post_init_per_testcase/5 is not exported, common_test
361              will attempt to call Module:post_init_per_testcase(TestcaseName,
362              Config, Return, CTHState) instead. This is for backwards compat‐
363              ibility.
364
365       Module:pre_end_per_testcase(SuiteName, TestcaseName, EndData, CTHState)
366       -> Result
367
368              Types:
369
370                 SuiteName = atom()
371                 TestcaseName = atom()
372                 EndData = Config
373                 Config = NewConfig = [{Key,Value}]
374                 CTHState = NewCTHState = term()
375                 Result = {NewConfig, NewCTHState}
376                 Key = atom()
377                 Value = term()
378                 Reason = term()
379
380              OPTIONAL
381
382              This function is called before end_per_testcase if it exists. It
383              behaves the same way  as  pre_end_per_suite,  but  for  function
384              end_per_testcase instead.
385
386              This  function  cannot change the result of the test case by re‐
387              turning skip or fail tuples, but it may insert items  in  Config
388              that  can be read in end_per_testcase/2 or in post_end_per_test‐
389              case/5.
390
391              If Module:pre_end_per_testcase/4 is  not  exported,  common_test
392              will  attempt  to call Module:pre_end_per_testcase(TestcaseName,
393              EndData, CTHState) instead. This is for backwards compatibility.
394
395       Module:post_end_per_testcase(SuiteName, TestcaseName,  Config,  Return,
396       CTHState) -> Result
397
398              Types:
399
400                 SuiteName = atom()
401                 TestcaseName = atom()
402                 Config = [{Key,Value}]
403                 Return = NewReturn = Config | SkipOrFail | term()
404                 SkipOrFail = {fail,Reason} | {skip, Reason}
405                 CTHState = NewCTHState = term()
406                 Result = {NewReturn, NewCTHState}
407                 Key = atom()
408                 Value = term()
409                 Reason = term()
410
411              OPTIONAL
412
413              This  function is called after end_per_testcase if it exists. It
414              behaves the same way as  post_end_per_suite,  but  for  function
415              end_per_testcase instead.
416
417              If  Module:post_end_per_testcase/5  is not exported, common_test
418              will attempt to call  Module:post_end_per_testcase(TestcaseName,
419              Config, Return, CTHState) instead. This is for backwards compat‐
420              ibility.
421
422       Module:pre_end_per_group(SuiteName, GroupName,  EndData,  CTHState)  ->
423       Result
424
425              Types:
426
427                 SuiteName = atom()
428                 GroupName = atom()
429                 EndData = Config | SkipOrFail
430                 Config = NewConfig = [{Key,Value}]
431                 CTHState = NewCTHState = term()
432                 Result = {NewConfig | SkipOrFail, NewCTHState}
433                 SkipOrFail = {fail,Reason} | {skip, Reason}
434                 Key = atom()
435                 Value = term()
436                 Reason = term()
437
438              OPTIONAL
439
440              This  function  is  called before end_per_group if it exists. It
441              behaves the same way as  pre_init_per_suite,  but  for  function
442              end_per_group instead.
443
444              If  Module:pre_end_per_group/4 is not exported, common_test will
445              attempt  to  call  Module:pre_end_per_group(GroupName,  EndData,
446              CTHState) instead. This is for backwards compatibility.
447
448       Module:post_end_per_group(SuiteName,  GroupName,  Config,  Return, CTH‐
449       State) -> Result
450
451              Types:
452
453                 SuiteName = atom()
454                 GroupName = atom()
455                 Config = [{Key,Value}]
456                 Return = NewReturn = Config | SkipOrFail | term()
457                 SkipOrFail = {fail,Reason} | {skip, Reason}
458                 CTHState = NewCTHState = term()
459                 Result = {NewReturn, NewCTHState}
460                 Key = atom()
461                 Value = term()
462                 Reason = term()
463
464              OPTIONAL
465
466              This function is called after end_per_group if it exists. It be‐
467              haves  the  same  way  as  post_init_per_suite, but for function
468              end_per_group instead.
469
470              If Module:post_end_per_group/5 is not exported, common_test will
471              attempt to call Module:post_end_per_group(GroupName, Config, Re‐
472              turn, CTHState) instead. This is for backwards compatibility.
473
474       Module:pre_end_per_suite(SuiteName, EndData, CTHState) -> Result
475
476              Types:
477
478                 SuiteName = atom()
479                 EndData = Config | SkipOrFail
480                 Config = NewConfig = [{Key,Value}]
481                 CTHState = NewCTHState = term()
482                 Result = {NewConfig | SkipOrFail, NewCTHState}
483                 SkipOrFail = {fail,Reason} | {skip, Reason}
484                 Key = atom()
485                 Value = term()
486                 Reason = term()
487
488              OPTIONAL
489
490              This function is called before end_per_suite if  it  exists.  It
491              behaves  the  same  way  as pre_init_per_suite, but for function
492              end_per_suite instead.
493
494       Module:post_end_per_suite(SuiteName, Config, Return, CTHState)  ->  Re‐
495       sult
496
497              Types:
498
499                 SuiteName = atom()
500                 Config = [{Key,Value}]
501                 Return = NewReturn = Config | SkipOrFail | term()
502                 SkipOrFail = {fail,Reason} | {skip, Reason}
503                 CTHState = NewCTHState = term()
504                 Result = {NewReturn, NewCTHState}
505                 Key = atom()
506                 Value = term()
507                 Reason = term()
508
509              OPTIONAL
510
511              This function is called after end_per_suite if it exists. It be‐
512              haves the same way  as  post_init_per_suite,  but  for  function
513              end_per_suite instead.
514
515       Module:on_tc_fail(SuiteName, TestName, Reason, CTHState) -> NewCTHState
516
517              Types:
518
519                 SuiteName = atom()
520                 TestName     =     init_per_suite     |    end_per_suite    |
521                 {init_per_group,GroupName}  |   {end_per_group,GroupName}   |
522                 {FuncName,GroupName} | FuncName
523                 FuncName = atom()
524                 GroupName = atom()
525                 Reason = term()
526                 CTHState = NewCTHState = term()
527
528              OPTIONAL
529
530              This  function  is called whenever a test case (or configuration
531              function) fails. It is called after the post function is  called
532              for the failed test case, that is:
533
534                * If  init_per_suite  fails,  this  function  is  called after
535                  post_init_per_suite.
536
537                * If  a  test  case  fails,  this  function  is  called  after
538                  post_end_per_testcase.
539
540              If  the failed test case belongs to a test case group, the first
541              argument is a tuple  {FuncName,GroupName},  otherwise  only  the
542              function name.
543
544              The data that comes with Reason follows the same format as Fail‐
545              Reason in event tc_done. For details, see section Event Handling
546              in the User's Guide.
547
548              If Module:on_tc_fail/4 is not exported, common_test will attempt
549              to call Module:on_tc_fail(TestName, Reason,  CTHState)  instead.
550              This is for backwards compatibility.
551
552       Module:on_tc_skip(SuiteName, TestName, Reason, CTHState) -> NewCTHState
553
554              Types:
555
556                 SuiteName = atom()
557                 TestName     =     init_per_suite     |    end_per_suite    |
558                 {init_per_group,GroupName}  |   {end_per_group,GroupName}   |
559                 {FuncName,GroupName} | FuncName
560                 FuncName = atom()
561                 GroupName = atom()
562                 Reason = {tc_auto_skip | tc_user_skip, term()}
563                 CTHState = NewCTHState = term()
564
565              OPTIONAL
566
567              This  function  is called whenever a test case (or configuration
568              function) is skipped. It is called after the  post  function  is
569              called for the skipped test case, that is:
570
571                * If  init_per_group is skipped, this function is called after
572                  post_init_per_group.
573
574                * If a test case is skipped, this  function  is  called  after
575                  post_end_per_testcase.
576
577              If the skipped test case belongs to a test case group, the first
578              argument is a tuple  {FuncName,GroupName},  otherwise  only  the
579              function name.
580
581              The  data  that  comes  with  Reason  follows the same format as
582              events tc_auto_skip and tc_user_skip For  details,  see  section
583              Event Handling in the User's Guide.
584
585              If Module:on_tc_skip/4 is not exported, common_test will attempt
586              to call Module:on_tc_skip(TestName, Reason,  CTHState)  instead.
587              This is for backwards compatibility.
588
589       Module:terminate(CTHState)
590
591              Types:
592
593                 CTHState = term()
594
595              OPTIONAL
596
597              This function is called at the end of a CTH scope.
598
599       Module:id(Opts) -> Id
600
601              Types:
602
603                 Opts = term()
604                 Id = term()
605
606              OPTIONAL
607
608              The  Id  identifies  a CTH instance uniquely. If two CTHs return
609              the same Id, the second CTH is ignored and subsequent  calls  to
610              the  CTH  are  only made to the first instance. For details, see
611              section Installing a CTH in the User's Guide.
612
613              This function is not to have any side  effects,  as  it  can  be
614              called multiple times by Common Test.
615
616              If  not implemented, the CTH acts as if this function returned a
617              call to make_ref/0.
618
619
620
621Ericsson AB                   common_test 1.25.1                   ct_hooks(3)