1ct_hooks(3)                Erlang Module Definition                ct_hooks(3)
2
3
4

NAME

6       ct_hooks - A callback interface on top of Common Test.
7

DESCRIPTION

9       The  Common  Test Hook (CTH) framework allows extensions of the default
10       behavior of Common Test by callbacks before and after  all  test  suite
11       calls. It is intended for advanced users of Common Test who want to ab‐
12       stract out behavior that is common to multiple test suites.
13
14       In brief, CTH allows you to:
15
16         * Manipulate the runtime configuration before each  suite  configura‐
17           tion call.
18
19         * Manipulate  the  return of all suite configuration calls and by ex‐
20           tension the result of the test themselves.
21
22       The following sections describe the mandatory and  optional  CTH  func‐
23       tions  that  Common Test calls during test execution. For more details,
24       see section Common Test Hooks in the User's Guide.
25
26       For information about how to add a CTH to your suite, see  section  In‐
27       stalling a CTH in the User's Guide.
28
29   Note:
30       For  a  minimal example of a CTH, see section Example CTH in the User's
31       Guide.
32
33

CALLBACK FUNCTIONS

35       The following functions define the callback interface for a CTH.
36

EXPORTS

38       Module:init(Id, Opts) -> {ok, State} | {ok, State, Priority}
39
40              Types:
41
42                 Id = reference() | term()
43                 Opts = term()
44                 State = term()
45                 Priority = integer()
46
47              MANDATORY
48
49              This function is always called before any other  callback  func‐
50              tion.  Use  it  to  initiate any common state. It is to return a
51              state for this CTH.
52
53              Id is either the return value of ct_hooks:id/1, or  a  reference
54              (created  using  erlang:make_ref/0  in ERTS) if ct_hooks:id/1 is
55              not implemented.
56
57              Priority is the relative priority of this  hook.  Hooks  with  a
58              lower  priority are executed first. If no priority is specified,
59              it is set to 0.
60
61              For details about when init is called, see section CTH Scope  in
62              the User's Guide.
63
64       Module:post_groups(SuiteName, GroupDefs) -> NewGroupDefs
65
66              Types:
67
68                 SuiteName = atom()
69                 GroupDefs = NewGroupDefs = [Group]
70                 Group = {GroupName,Properties,GroupsAndTestCases}
71                 GroupName = atom()
72                 Properties  =  [parallel | sequence | Shuffle | {GroupRepeat‐
73                 Type,N}]
74                 GroupsAndTestCases = [Group | {group,GroupName} | TestCase  |
75                 {testcase,TestCase,TCRepeatProps}]
76                 TestCase = atom()
77                 TCRepeatProps  =  [{repeat,N}  |  {repeat_until_ok,N}  | {re‐
78                 peat_until_fail,N}]
79                 Shuffle = shuffle | {shuffle,Seed}
80                 Seed = {integer(),integer(),integer()}
81                 GroupRepeatType = repeat | repeat_until_all_ok  |  repeat_un‐
82                 til_all_fail | repeat_until_any_ok | repeat_until_any_fail
83                 N = integer() | forever
84
85              OPTIONAL
86
87              This function is called after groups/0. It is used to modify the
88              test group definitions, for instance to add or remove groups  or
89              change group properties.
90
91              GroupDefs  is  what  groups/0 returned, that is, a list of group
92              definitions.
93
94              NewGroupDefs is the possibly modified version of this list.
95
96              This function  is  called  only  if  the  CTH  is  added  before
97              init_per_suite is run. For details, see section CTH Scope in the
98              User's Guide.
99
100              Notice that for CTHs that are installed by means of the  suite/0
101              function,  post_groups/2  is called before the init/2 hook func‐
102              tion. However, for CTHs that are installed by means  of  the  CT
103              start flag, the init/2 function is called first.
104
105          Note:
106              Prior  to each test execution, Common Test does a simulated test
107              run in order to count test suites, groups and cases for  logging
108              purposes.  This causes the post_groups/2 hook function to always
109              be called twice. For this reason, side effects are best  avoided
110              in this callback.
111
112
113       Module:post_all(SuiteName, Return, GroupDefs) -> NewReturn
114
115              Types:
116
117                 SuiteName = atom()
118                 Return = NewReturn = Tests | {skip,Reason}
119                 Tests   =  [TestCase  |  {testcase,TestCase,TCRepeatProps}  |
120                 {group,GroupName}    |     {group,GroupName,Properties}     |
121                 {group,GroupName,Properties,SubGroups}]
122                 TestCase = atom()
123                 TCRepeatProps  =  [{repeat,N}  |  {repeat_until_ok,N}  | {re‐
124                 peat_until_fail,N}]
125                 GroupName = atom()
126                 Properties = GroupProperties | default
127                 SubGroups  =  [{GroupName,Properties}  |   {GroupName,Proper‐
128                 ties,SubGroups}]
129                 Shuffle = shuffle | {shuffle,Seed}
130                 Seed = {integer(),integer(),integer()}
131                 GroupRepeatType  =  repeat | repeat_until_all_ok | repeat_un‐
132                 til_all_fail | repeat_until_any_ok | repeat_until_any_fail
133                 N = integer() | forever
134                 GroupDefs = NewGroupDefs = [Group]
135                 Group = {GroupName,GroupProperties,GroupsAndTestCases}
136                 GroupProperties = [parallel | sequence | Shuffle |  {GroupRe‐
137                 peatType,N}]
138                 GroupsAndTestCases = [Group | {group,GroupName} | TestCase]
139                 Reason = term()
140
141              OPTIONAL
142
143              This  function  is  called after all/0. It is used to modify the
144              set of test cases and test group to be executed, for instance to
145              add or remove test cases and groups, change group properties, or
146              even skip all tests in the suite.
147
148              Return is what all/0 returned, that is, a list of test cases and
149              groups to be executed, or a tuple {skip,Reason}.
150
151              GroupDefs  is  what groups/0 or the post_groups/2 hook returned,
152              that is, a list of group definitions.
153
154              NewReturn is the possibly modified version of Return.
155
156              This function  is  called  only  if  the  CTH  is  added  before
157              init_per_suite is run. For details, see section CTH Scope in the
158              User's Guide.
159
160              Notice that for CTHs that are installed by means of the  suite/0
161              function,  post_all/2 is called before the init/2 hook function.
162              However, for CTHs that are installed by means of  the  CT  start
163              flag, the init/2 function is called first.
164
165          Note:
166              Prior  to each test execution, Common Test does a simulated test
167              run in order to count test suites, groups and cases for  logging
168              purposes.  This causes the post_all/3 hook function to always be
169              called twice. For this reason, side effects are best avoided  in
170              this callback.
171
172
173       Module:pre_init_per_suite(SuiteName, InitData, CTHState) -> Result
174
175              Types:
176
177                 SuiteName = atom()
178                 InitData = Config | SkipOrFail
179                 Config = NewConfig = [{Key,Value}]
180                 CTHState = NewCTHState = term()
181                 Result = {Return, NewCTHState}
182                 Return = NewConfig | SkipOrFail
183                 SkipOrFail = {fail, Reason} | {skip, Reason}
184                 Key = atom()
185                 Value = term()
186                 Reason = term()
187
188              OPTIONAL
189
190              This  function  is called before init_per_suite if it exists. It
191              typically contains initialization/logging that must be done  be‐
192              fore init_per_suite is called. If {skip,Reason} or {fail,Reason}
193              is returned, init_per_suite and all test cases of the suite  are
194              skipped and Reason printed in the overview log of the suite.
195
196              SuiteName is the name of the suite to be run.
197
198              InitData  is  the original configuration list of the test suite,
199              or a SkipOrFail tuple if a previous CTH has returned this.
200
201              CTHState is the current internal state of the CTH.
202
203              Return is the result of the init_per_suite function.  If  it  is
204              {skip,Reason}  or {fail,Reason}, init_per_suite is never called,
205              instead the initiation is considered to be  skipped  or  failed,
206              respectively. If a NewConfig list is returned, init_per_suite is
207              called with that NewConfig list. For more details,  see  section
208              Pre Hooks in the User's Guide.
209
210              This  function  is  called  only  if  the  CTH  is  added before
211              init_per_suite is run. For details, see section CTH Scope in the
212              User's Guide.
213
214       Module:post_init_per_suite(SuiteName,  Config, Return, CTHState) -> Re‐
215       sult
216
217              Types:
218
219                 SuiteName = atom()
220                 Config = [{Key,Value}]
221                 Return = NewReturn = Config | SkipOrFail | term()
222                 SkipOrFail = {fail, Reason} | {skip, Reason} | term()
223                 CTHState = NewCTHState = term()
224                 Result = {NewReturn, NewCTHState}
225                 Key = atom()
226                 Value = term()
227                 Reason = term()
228
229              OPTIONAL
230
231              This function is called after init_per_suite if  it  exists.  It
232              typically  contains  extra checks to ensure that all the correct
233              dependencies are started correctly.
234
235              Return is what init_per_suite returned, that is,  {fail,Reason},
236              {skip,Reason},   a   Config  list,  or  a  term  describing  how
237              init_per_suite failed.
238
239              NewReturn   is   the   possibly   modified   return   value   of
240              init_per_suite. To recover from a failure in init_per_suite, re‐
241              turn ConfigList with the tc_status element removed. For more de‐
242              tails,  see   Post  Hooks in section "Manipulating Tests" in the
243              User's Guide.
244
245              CTHState is the current internal state of the CTH.
246
247              This function is called only if the CTH is added  before  or  in
248              init_per_suite. For details, see section CTH Scope in the User's
249              Guide.
250
251       Module:pre_init_per_group(SuiteName, GroupName, InitData, CTHState)  ->
252       Result
253
254              Types:
255
256                 SuiteName = atom()
257                 GroupName = atom()
258                 InitData = Config | SkipOrFail
259                 Config = NewConfig = [{Key,Value}]
260                 CTHState = NewCTHState = term()
261                 Result = {NewConfig | SkipOrFail, NewCTHState}
262                 SkipOrFail = {fail,Reason} | {skip, Reason}
263                 Key = atom()
264                 Value = term()
265                 Reason = term()
266
267              OPTIONAL
268
269              This  function  is called before init_per_group if it exists. It
270              behaves the same way as  pre_init_per_suite,  but  for  function
271              init_per_group instead.
272
273              If Module:pre_init_per_group/4 is not exported, common_test will
274              attempt to call  Module:pre_init_per_group(GroupName,  InitData,
275              CTHState) instead. This is for backwards compatibility.
276
277       Module:post_init_per_group(SuiteName,  GroupName,  Config, Return, CTH‐
278       State) -> Result
279
280              Types:
281
282                 SuiteName = atom()
283                 GroupName = atom()
284                 Config = [{Key,Value}]
285                 Return = NewReturn = Config | SkipOrFail | term()
286                 SkipOrFail = {fail,Reason} | {skip, Reason}
287                 CTHState = NewCTHState = term()
288                 Result = {NewReturn, NewCTHState}
289                 Key = atom()
290                 Value = term()
291                 Reason = term()
292
293              OPTIONAL
294
295              This function is called after init_per_group if  it  exists.  It
296              behaves  the  same  way as post_init_per_suite, but for function
297              init_per_group instead.
298
299              If Module:post_init_per_group/5  is  not  exported,  common_test
300              will  attempt to call Module:post_init_per_group(GroupName, Con‐
301              fig, Return, CTHState) instead. This is for  backwards  compati‐
302              bility.
303
304       Module:pre_init_per_testcase(SuiteName,  TestcaseName,  InitData,  CTH‐
305       State) -> Result
306
307              Types:
308
309                 SuiteName = atom()
310                 TestcaseName = atom()
311                 InitData = Config | SkipOrFail
312                 Config = NewConfig = [{Key,Value}]
313                 CTHState = NewCTHState = term()
314                 Result = {NewConfig | SkipOrFail, NewCTHState}
315                 SkipOrFail = {fail,Reason} | {skip, Reason}
316                 Key = atom()
317                 Value = term()
318                 Reason = term()
319
320              OPTIONAL
321
322              This function is called before init_per_testcase if  it  exists.
323              It  behaves the same way as pre_init_per_suite, but for function
324              init_per_testcase instead.
325
326              If Module:pre_init_per_testcase/4 is not  exported,  common_test
327              will  attempt to call Module:pre_init_per_testcase(TestcaseName,
328              InitData, CTHState) instead. This is for  backwards  compatibil‐
329              ity.
330
331              CTHs  cannot  be added here right now. That feature may be added
332              in a later release, but it would right now break backwards  com‐
333              patibility.
334
335       Module:post_init_per_testcase(SuiteName,  TestcaseName, Config, Return,
336       CTHState) -> Result
337
338              Types:
339
340                 SuiteName = atom()
341                 TestcaseName = atom()
342                 Config = [{Key,Value}]
343                 Return = NewReturn = Config | SkipOrFail | term()
344                 SkipOrFail = {fail,Reason} | {skip, Reason}
345                 CTHState = NewCTHState = term()
346                 Result = {NewReturn, NewCTHState}
347                 Key = atom()
348                 Value = term()
349                 Reason = term()
350
351              OPTIONAL
352
353              This function is called after init_per_testcase if it exists. It
354              behaves  the  same  way as post_init_per_suite, but for function
355              init_per_testcase instead.
356
357              If Module:post_init_per_testcase/5 is not exported,  common_test
358              will attempt to call Module:post_init_per_testcase(TestcaseName,
359              Config, Return, CTHState) instead. This is for backwards compat‐
360              ibility.
361
362       Module:pre_end_per_testcase(SuiteName, TestcaseName, EndData, CTHState)
363       -> Result
364
365              Types:
366
367                 SuiteName = atom()
368                 TestcaseName = atom()
369                 EndData = Config
370                 Config = NewConfig = [{Key,Value}]
371                 CTHState = NewCTHState = term()
372                 Result = {NewConfig, NewCTHState}
373                 Key = atom()
374                 Value = term()
375                 Reason = term()
376
377              OPTIONAL
378
379              This function is called before end_per_testcase if it exists. It
380              behaves  the  same  way  as  pre_end_per_suite, but for function
381              end_per_testcase instead.
382
383              This function cannot change the result of the test case  by  re‐
384              turning  skip  or fail tuples, but it may insert items in Config
385              that can be read in end_per_testcase/2 or in  post_end_per_test‐
386              case/5.
387
388              If  Module:pre_end_per_testcase/4  is  not exported, common_test
389              will attempt to  call  Module:pre_end_per_testcase(TestcaseName,
390              EndData, CTHState) instead. This is for backwards compatibility.
391
392       Module:post_end_per_testcase(SuiteName,  TestcaseName,  Config, Return,
393       CTHState) -> Result
394
395              Types:
396
397                 SuiteName = atom()
398                 TestcaseName = atom()
399                 Config = [{Key,Value}]
400                 Return = NewReturn = Config | SkipOrFail | term()
401                 SkipOrFail = {fail,Reason} | {skip, Reason}
402                 CTHState = NewCTHState = term()
403                 Result = {NewReturn, NewCTHState}
404                 Key = atom()
405                 Value = term()
406                 Reason = term()
407
408              OPTIONAL
409
410              This function is called after end_per_testcase if it exists.  It
411              behaves  the  same  way  as post_end_per_suite, but for function
412              end_per_testcase instead.
413
414              If Module:post_end_per_testcase/5 is not  exported,  common_test
415              will  attempt to call Module:post_end_per_testcase(TestcaseName,
416              Config, Return, CTHState) instead. This is for backwards compat‐
417              ibility.
418
419       Module:pre_end_per_group(SuiteName,  GroupName,  EndData,  CTHState) ->
420       Result
421
422              Types:
423
424                 SuiteName = atom()
425                 GroupName = atom()
426                 EndData = Config | SkipOrFail
427                 Config = NewConfig = [{Key,Value}]
428                 CTHState = NewCTHState = term()
429                 Result = {NewConfig | SkipOrFail, NewCTHState}
430                 SkipOrFail = {fail,Reason} | {skip, Reason}
431                 Key = atom()
432                 Value = term()
433                 Reason = term()
434
435              OPTIONAL
436
437              This function is called before end_per_group if  it  exists.  It
438              behaves  the  same  way  as pre_init_per_suite, but for function
439              end_per_group instead.
440
441              If Module:pre_end_per_group/4 is not exported, common_test  will
442              attempt  to  call  Module:pre_end_per_group(GroupName,  EndData,
443              CTHState) instead. This is for backwards compatibility.
444
445       Module:post_end_per_group(SuiteName, GroupName,  Config,  Return,  CTH‐
446       State) -> Result
447
448              Types:
449
450                 SuiteName = atom()
451                 GroupName = atom()
452                 Config = [{Key,Value}]
453                 Return = NewReturn = Config | SkipOrFail | term()
454                 SkipOrFail = {fail,Reason} | {skip, Reason}
455                 CTHState = NewCTHState = term()
456                 Result = {NewReturn, NewCTHState}
457                 Key = atom()
458                 Value = term()
459                 Reason = term()
460
461              OPTIONAL
462
463              This function is called after end_per_group if it exists. It be‐
464              haves the same way  as  post_init_per_suite,  but  for  function
465              end_per_group instead.
466
467              If Module:post_end_per_group/5 is not exported, common_test will
468              attempt to call Module:post_end_per_group(GroupName, Config, Re‐
469              turn, CTHState) instead. This is for backwards compatibility.
470
471       Module:pre_end_per_suite(SuiteName, EndData, CTHState) -> Result
472
473              Types:
474
475                 SuiteName = atom()
476                 EndData = Config | SkipOrFail
477                 Config = NewConfig = [{Key,Value}]
478                 CTHState = NewCTHState = term()
479                 Result = {NewConfig | SkipOrFail, NewCTHState}
480                 SkipOrFail = {fail,Reason} | {skip, Reason}
481                 Key = atom()
482                 Value = term()
483                 Reason = term()
484
485              OPTIONAL
486
487              This  function  is  called before end_per_suite if it exists. It
488              behaves the same way as  pre_init_per_suite,  but  for  function
489              end_per_suite instead.
490
491       Module:post_end_per_suite(SuiteName,  Config,  Return, CTHState) -> Re‐
492       sult
493
494              Types:
495
496                 SuiteName = atom()
497                 Config = [{Key,Value}]
498                 Return = NewReturn = Config | SkipOrFail | term()
499                 SkipOrFail = {fail,Reason} | {skip, Reason}
500                 CTHState = NewCTHState = term()
501                 Result = {NewReturn, NewCTHState}
502                 Key = atom()
503                 Value = term()
504                 Reason = term()
505
506              OPTIONAL
507
508              This function is called after end_per_suite if it exists. It be‐
509              haves  the  same  way  as  post_init_per_suite, but for function
510              end_per_suite instead.
511
512       Module:on_tc_fail(SuiteName, TestName, Reason, CTHState) -> NewCTHState
513
514              Types:
515
516                 SuiteName = atom()
517                 TestName    =    init_per_suite     |     end_per_suite     |
518                 {init_per_group,GroupName}   |   {end_per_group,GroupName}  |
519                 {FuncName,GroupName} | FuncName
520                 FuncName = atom()
521                 GroupName = atom()
522                 Reason = term()
523                 CTHState = NewCTHState = term()
524
525              OPTIONAL
526
527              This function is called whenever a test case  (or  configuration
528              function)  fails. It is called after the post function is called
529              for the failed test case, that is:
530
531                * If init_per_suite  fails,  this  function  is  called  after
532                  post_init_per_suite.
533
534                * If  a  test  case  fails,  this  function  is  called  after
535                  post_end_per_testcase.
536
537              If the failed test case belongs to a test case group, the  first
538              argument  is  a  tuple  {FuncName,GroupName}, otherwise only the
539              function name.
540
541              The data that comes with Reason follows the same format as Fail‐
542              Reason in event tc_done. For details, see section Event Handling
543              in the User's Guide.
544
545              If Module:on_tc_fail/4 is not exported, common_test will attempt
546              to  call  Module:on_tc_fail(TestName, Reason, CTHState) instead.
547              This is for backwards compatibility.
548
549       Module:on_tc_skip(SuiteName, TestName, Reason, CTHState) -> NewCTHState
550
551              Types:
552
553                 SuiteName = atom()
554                 TestName    =    init_per_suite     |     end_per_suite     |
555                 {init_per_group,GroupName}   |   {end_per_group,GroupName}  |
556                 {FuncName,GroupName} | FuncName
557                 FuncName = atom()
558                 GroupName = atom()
559                 Reason = {tc_auto_skip | tc_user_skip, term()}
560                 CTHState = NewCTHState = term()
561
562              OPTIONAL
563
564              This function is called whenever a test case  (or  configuration
565              function)  is  skipped.  It is called after the post function is
566              called for the skipped test case, that is:
567
568                * If init_per_group is skipped, this function is called  after
569                  post_init_per_group.
570
571                * If  a  test  case  is skipped, this function is called after
572                  post_end_per_testcase.
573
574              If the skipped test case belongs to a test case group, the first
575              argument  is  a  tuple  {FuncName,GroupName}, otherwise only the
576              function name.
577
578              The data that comes with  Reason  follows  the  same  format  as
579              events  tc_auto_skip  and  tc_user_skip For details, see section
580              Event Handling in the User's Guide.
581
582              If Module:on_tc_skip/4 is not exported, common_test will attempt
583              to  call  Module:on_tc_skip(TestName, Reason, CTHState) instead.
584              This is for backwards compatibility.
585
586       Module:terminate(CTHState)
587
588              Types:
589
590                 CTHState = term()
591
592              OPTIONAL
593
594              This function is called at the end of a CTH scope.
595
596       Module:id(Opts) -> Id
597
598              Types:
599
600                 Opts = term()
601                 Id = term()
602
603              OPTIONAL
604
605              The Id identifies a CTH instance uniquely. If  two  CTHs  return
606              the  same  Id, the second CTH is ignored and subsequent calls to
607              the CTH are only made to the first instance.  For  details,  see
608              section Installing a CTH in the User's Guide.
609
610              This  function  is  not  to  have any side effects, as it can be
611              called multiple times by Common Test.
612
613              If not implemented, the CTH acts as if this function returned  a
614              call to make_ref/0.
615
616
617
618Ericsson AB                   common_test 1.23.3                   ct_hooks(3)