1Test2::API(3) User Contributed Perl Documentation Test2::API(3)
2
6 Test2::API - Primary interface for writing Test2 based testing tools.
7
9 The internals of this package are subject to change at any time! The
10 public methods provided will not change in backwards-incompatible ways
11 (once there is a stable release), but the underlying implementation
12 details might. Do not break encapsulation here!
13 Currently the implementation is to create a single instance of the
15 Test2::API::Instance Object. All class methods defer to the single
16 instance. There is no public access to the singleton, and that is
17 intentional. The class methods provided by this package provide the
18 only functionality publicly exposed.
19 This is done primarily to avoid the problems Test::Builder had by
21 exposing its singleton. We do not want anyone to replace this
22 singleton, rebless it, or directly muck with its internals. If you need
23 to do something and cannot because of the restrictions placed here,
24 then please report it as an issue. If possible, we will create a way
25 for you to implement your functionality without exposing things that
26 should not be exposed.
27
29 This package exports all the functions necessary to write and/or verify
30 testing tools. Using these building blocks you can begin writing test
31 tools very quickly. You are also provided with tools that help you to
32 test the tools you write.
33
35 WRITING A TOOL
36 The "context()" method is your primary interface into the Test2
37 framework.
38 package My::Ok;
40 use Test2::API qw/context/;
41 our @EXPORT = qw/my_ok/;
43 use base 'Exporter';
44 # Just like ok() from Test::More
46 sub my_ok($;$) {
47 my ($bool, $name) = @_;
48 my $ctx = context(); # Get a context
49 $ctx->ok($bool, $name);
50 $ctx->release; # Release the context
51 return $bool;
52 }
53 See Test2::API::Context for a list of methods available on the context
55 object.
56 TESTING YOUR TOOLS
58 The "intercept { ... }" tool lets you temporarily intercept all events
59 generated by the test system:
60 use Test2::API qw/intercept/;
62 use My::Ok qw/my_ok/;
64 my $events = intercept {
66 # These events are not displayed
67 my_ok(1, "pass");
68 my_ok(0, "fail");
69 };
70 As of version 1.302178 this now returns an arrayref that is also an
72 instance of Test2::API::InterceptResult. See the
73 Test2::API::InterceptResult documentation for details on how to best
74 use it.
75 OTHER API FUNCTIONS
77 use Test2::API qw{
78 test2_init_done
79 test2_stack
80 test2_set_is_end
81 test2_get_is_end
82 test2_ipc
83 test2_formatter_set
84 test2_formatter
85 test2_is_testing_done
86 };
87 my $init = test2_init_done();
89 my $stack = test2_stack();
90 my $ipc = test2_ipc();
91 test2_formatter_set($FORMATTER)
93 my $formatter = test2_formatter();
94 ... And others ...
96
98 All exports are optional. You must specify subs to import.
99 use Test2::API qw/context intercept run_subtest/;
101 This is the list of exports that are most commonly needed. If you are
103 simply writing a tool, then this is probably all you need. If you need
104 something and you cannot find it here, then you can also look at "OTHER
105 API EXPORTS".
106 These exports lack the 'test2_' prefix because of how important/common
108 they are. Exports in the "OTHER API EXPORTS" section have the 'test2_'
109 prefix to ensure they stand out.
110 context(...)
112 Usage:
113 $ctx = context()
115 $ctx = context(%params)
116 The "context()" function will always return the current context. If
118 there is already a context active, it will be returned. If there is not
119 an active context, one will be generated. When a context is generated
120 it will default to using the file and line number where the currently
121 running sub was called from.
122 Please see "CRITICAL DETAILS" in Test2::API::Context for important
124 rules about what you can and cannot do with a context once it is
125 obtained.
126 Note This function will throw an exception if you ignore the context
128 object it returns.
129 Note On perls 5.14+ a depth check is used to insure there are no
131 context leaks. This cannot be safely done on older perls due to
132 <https://rt.perl.org/Public/Bug/Display.html?id=127774> You can
133 forcefully enable it either by setting "$ENV{T2_CHECK_DEPTH} = 1" or
134 "$Test2::API::DO_DEPTH_CHECK = 1" BEFORE loading Test2::API.
135 OPTIONAL PARAMETERS
137 All parameters to "context" are optional.
139 level => $int
141 If you must obtain a context in a sub deeper than your entry point
142 you can use this to tell it how many EXTRA stack frames to look
143 back. If this option is not provided the default of 0 is used.
144 sub third_party_tool {
146 my $sub = shift;
147 ... # Does not obtain a context
148 $sub->();
149 ...
150 }
151 third_party_tool(sub {
153 my $ctx = context(level => 1);
154 ...
155 $ctx->release;
156 });
157 wrapped => $int
159 Use this if you need to write your own tool that wraps a call to
160 "context()" with the intent that it should return a context object.
161 sub my_context {
163 my %params = ( wrapped => 0, @_ );
164 $params{wrapped}++;
165 my $ctx = context(%params);
166 ...
167 return $ctx;
168 }
169 sub my_tool {
171 my $ctx = my_context();
172 ...
173 $ctx->release;
174 }
175 If you do not do this, then tools you call that also check for a
177 context will notice that the context they grabbed was created at
178 the same stack depth, which will trigger protective measures that
179 warn you and destroy the existing context.
180 stack => $stack
182 Normally "context()" looks at the global hub stack. If you are
183 maintaining your own Test2::API::Stack instance you may pass it in
184 to be used instead of the global one.
185 hub => $hub
187 Use this parameter if you want to obtain the context for a specific
188 hub instead of whatever one happens to be at the top of the stack.
189 on_init => sub { ... }
191 This lets you provide a callback sub that will be called ONLY if
192 your call to "context()" generated a new context. The callback WILL
193 NOT be called if "context()" is returning an existing context. The
194 only argument passed into the callback will be the context object
195 itself.
196 sub foo {
198 my $ctx = context(on_init => sub { 'will run' });
199 my $inner = sub {
201 # This callback is not run since we are getting the existing
202 # context from our parent sub.
203 my $ctx = context(on_init => sub { 'will NOT run' });
204 $ctx->release;
205 }
206 $inner->();
207 $ctx->release;
209 }
210 on_release => sub { ... }
212 This lets you provide a callback sub that will be called when the
213 context instance is released. This callback will be added to the
214 returned context even if an existing context is returned. If
215 multiple calls to context add callbacks, then all will be called in
216 reverse order when the context is finally released.
217 sub foo {
219 my $ctx = context(on_release => sub { 'will run second' });
220 my $inner = sub {
222 my $ctx = context(on_release => sub { 'will run first' });
223 # Neither callback runs on this release
225 $ctx->release;
226 }
227 $inner->();
228 # Both callbacks run here.
230 $ctx->release;
231 }
232 release($;$)
234 Usage:
235 release $ctx;
237 release $ctx, ...;
238 This is intended as a shortcut that lets you release your context and
240 return a value in one statement. This function will get your context,
241 and an optional return value. It will release your context, then return
242 your value. Scalar context is always assumed.
243 sub tool {
245 my $ctx = context();
246 ...
247 return release $ctx, 1;
249 }
250 This tool is most useful when you want to return the value you get from
252 calling a function that needs to see the current context:
253 my $ctx = context();
255 my $out = some_tool(...);
256 $ctx->release;
257 return $out;
258 We can combine the last 3 lines of the above like so:
260 my $ctx = context();
262 release $ctx, some_tool(...);
263 context_do(&;@)
265 Usage:
266 sub my_tool {
268 context_do {
269 my $ctx = shift;
270 my (@args) = @_;
272 $ctx->ok(1, "pass");
274 ...
276 # No need to call $ctx->release, done for you on scope exit.
278 } @_;
279 }
280 Using this inside your test tool takes care of a lot of boilerplate for
282 you. It will ensure a context is acquired. It will capture and rethrow
283 any exception. It will insure the context is released when you are
284 done. It preserves the subroutine call context (array, scalar, void).
285 This is the safest way to write a test tool. The only two downsides to
287 this are a slight performance decrease, and some extra indentation in
288 your source. If the indentation is a problem for you then you can take
289 a peek at the next section.
290 no_context(&;$)
292 Usage:
293 no_context { ... };
295 no_context { ... } $hid;
296 sub my_tool(&) {
297 my $code = shift;
298 my $ctx = context();
299 ...
300 no_context {
302 # Things in here will not see our current context, they get a new
303 # one.
304 $code->();
306 };
307 ...
309 $ctx->release;
310 };
311 This tool will hide a context for the provided block of code. This
313 means any tools run inside the block will get a completely new context
314 if they acquire one. The new context will be inherited by tools nested
315 below the one that acquired it.
316 This will normally hide the current context for the top hub. If you
318 need to hide the context for a different hub you can pass in the
319 optional $hid parameter.
320 intercept(&)
322 Usage:
323 my $events = intercept {
325 ok(1, "pass");
326 ok(0, "fail");
327 ...
328 };
329 This function takes a codeblock as its only argument, and it has a
331 prototype. It will execute the codeblock, intercepting any generated
332 events in the process. It will return an array reference with all the
333 generated event objects. All events should be subclasses of
334 Test2::Event.
335 As of version 1.302178 the events array that is returned is blssed as
337 an Test2::API::InterceptResult instance. Test2::API::InterceptResult
338 Provides a helpful interface for filtering and/or inspecting the events
339 list overall, or individual events within the list.
340 This is intended to help you test your test code. This is not intended
342 for people simply writing tests.
343 run_subtest(...)
345 Usage:
346 run_subtest($NAME, \&CODE, $BUFFERED, @ARGS)
348 # or
350 run_subtest($NAME, \&CODE, \%PARAMS, @ARGS)
352 This will run the provided codeblock with the args in @args. This
354 codeblock will be run as a subtest. A subtest is an isolated test state
355 that is condensed into a single Test2::Event::Subtest event, which
356 contains all events generated inside the subtest.
357 ARGUMENTS:
359 $NAME
361 The name of the subtest.
362 \&CODE
364 The code to run inside the subtest.
365 $BUFFERED or \%PARAMS
367 If this is a simple scalar then it will be treated as a boolean for
368 the 'buffered' setting. If this is a hash reference then it will be
369 used as a parameters hash. The param hash will be used for hub
370 construction (with the specified keys removed).
371 Keys that are removed and used by run_subtest:
373 'buffered' => $bool
375 Toggle buffered status.
376 'inherit_trace' => $bool
378 Normally the subtest hub is pushed and the sub is allowed to
379 generate its own root context for the hub. When this setting is
380 turned on a root context will be created for the hub that
381 shares the same trace as the current context.
382 Set this to true if your tool is producing subtests without
384 user-specified subs.
385 'no_fork' => $bool
387 Defaults to off. Normally forking inside a subtest will
388 actually fork the subtest, resulting in 2 final subtest events.
389 This parameter will turn off that behavior, only the original
390 process/thread will return a final subtest event.
391 @ARGS
393 Any extra arguments you want passed into the subtest code.
394 BUFFERED VS UNBUFFERED (OR STREAMED)
396 Normally all events inside and outside a subtest are sent to the
398 formatter immediately by the hub. Sometimes it is desirable to hold off
399 sending events within a subtest until the subtest is complete. This
400 usually depends on the formatter being used.
401 Things not effected by this flag
403 In both cases events are generated and stored in an array. This
404 array is eventually used to populate the "subevents" attribute on
405 the Test2::Event::Subtest event that is generated at the end of the
406 subtest. This flag has no effect on this part, it always happens.
407 At the end of the subtest, the final Test2::Event::Subtest event is
409 sent to the formatter.
410 Things that are effected by this flag
412 The "buffered" attribute of the Test2::Event::Subtest event will be
413 set to the value of this flag. This means any formatter, listener,
414 etc which looks at the event will know if it was buffered.
415 Things that are formatter dependant
417 Events within a buffered subtest may or may not be sent to the
418 formatter as they happen. If a formatter fails to specify then the
419 default is to NOT SEND the events as they are generated, instead
420 the formatter can pull them from the "subevents" attribute.
421 A formatter can specify by implementing the "hide_buffered()"
423 method. If this method returns true then events generated inside a
424 buffered subtest will not be sent independently of the final
425 subtest event.
426 An example of how this is used is the Test2::Formatter::TAP formatter.
428 For unbuffered subtests the events are rendered as they are generated.
429 At the end of the subtest, the final subtest event is rendered, but the
430 "subevents" attribute is ignored. For buffered subtests the opposite
431 occurs, the events are NOT rendered as they are generated, instead the
432 "subevents" attribute is used to render them all at once. This is
433 useful when running subtests tests in parallel, since without it the
434 output from subtests would be interleaved together.
435
437 Exports in this section are not commonly needed. These all have the
438 'test2_' prefix to help ensure they stand out. You should look at the
439 "MAIN API EXPORTS" section before looking here. This section is one
440 where "Great power comes with great responsibility". It is possible to
441 break things badly if you are not careful with these.
442 All exports are optional. You need to list which ones you want at
444 import time:
445 use Test2::API qw/test2_init_done .../;
447 STATUS AND INITIALIZATION STATE
449 These provide access to internal state and object instances.
450 $bool = test2_init_done()
452 This will return true if the stack and IPC instances have already
453 been initialized. It will return false if they have not. Init
454 happens as late as possible. It happens as soon as a tool requests
455 the IPC instance, the formatter, or the stack.
456 $bool = test2_load_done()
458 This will simply return the boolean value of the loaded flag. If
459 Test2 has finished loading this will be true, otherwise false.
460 Loading is considered complete the first time a tool requests a
461 context.
462 test2_set_is_end()
464 test2_set_is_end($bool)
465 This is used to toggle Test2's belief that the END phase has
466 already started. With no arguments this will set it to true. With
467 arguments it will set it to the first argument's value.
468 This is used to prevent the use of "caller()" in END blocks which
470 can cause segfaults. This is only necessary in some persistent
471 environments that may have multiple END phases.
472 $bool = test2_get_is_end()
474 Check if Test2 believes it is the END phase.
475 $stack = test2_stack()
477 This will return the global Test2::API::Stack instance. If this has
478 not yet been initialized it will be initialized now.
479 $bool = test2_is_testing_done()
481 This will return true if testing is complete and no other events
482 should be sent. This is useful in things like warning handlers
483 where you might want to turn warnings into events, but need them to
484 start acting like normal warnings when testing is done.
485 $SIG{__WARN__} = sub {
487 my ($warning) = @_;
488 if (test2_is_testing_done()) {
490 warn @_;
491 }
492 else {
493 my $ctx = context();
494 ...
495 $ctx->release
496 }
497 }
498 test2_ipc_disable
500 Disable IPC.
501 $bool = test2_ipc_diabled
503 Check if IPC is disabled.
504 test2_ipc_wait_enable()
506 test2_ipc_wait_disable()
507 $bool = test2_ipc_wait_enabled()
508 These can be used to turn IPC waiting on and off, or check the
509 current value of the flag.
510 Waiting is turned on by default. Waiting will cause the parent
512 process/thread to wait until all child processes and threads are
513 finished before exiting. You will almost never want to turn this
514 off.
515 $bool = test2_no_wait()
517 test2_no_wait($bool)
518 DISCOURAGED: This is a confusing interface, it is better to use
519 "test2_ipc_wait_enable()", "test2_ipc_wait_disable()" and
520 "test2_ipc_wait_enabled()".
521 This can be used to get/set the no_wait status. Waiting is turned
523 on by default. Waiting will cause the parent process/thread to wait
524 until all child processes and threads are finished before exiting.
525 You will almost never want to turn this off.
526 $fh = test2_stdout()
528 $fh = test2_stderr()
529 These functions return the filehandles that test output should be
530 written to. They are primarily useful when writing a custom
531 formatter and code that turns events into actual output (TAP,
532 etc.). They will return a dupe of the original filehandles that
533 formatted output can be sent to regardless of whatever state the
534 currently running test may have left STDOUT and STDERR in.
535 test2_reset_io()
537 Re-dupe the internal filehandles returned by "test2_stdout()" and
538 "test2_stderr()" from the current STDOUT and STDERR. You shouldn't
539 need to do this except in very peculiar situations (for example,
540 you're testing a new formatter and you need control over where the
541 formatter is sending its output.)
542 BEHAVIOR HOOKS
544 These are hooks that allow you to add custom behavior to actions taken
545 by Test2 and tools built on top of it.
546 test2_add_callback_exit(sub { ... })
548 This can be used to add a callback that is called after all testing
549 is done. This is too late to add additional results, the main use
550 of this callback is to set the exit code.
551 test2_add_callback_exit(
553 sub {
554 my ($context, $exit, \$new_exit) = @_;
555 ...
556 }
557 );
558 The $context passed in will be an instance of Test2::API::Context.
560 The $exit argument will be the original exit code before anything
561 modified it. $$new_exit is a reference to the new exit code. You
562 may modify this to change the exit code. Please note that
563 $$new_exit may already be different from $exit
564 test2_add_callback_post_load(sub { ... })
566 Add a callback that will be called when Test2 is finished loading.
567 This means the callback will be run once, the first time a context
568 is obtained. If Test2 has already finished loading then the
569 callback will be run immediately.
570 test2_add_callback_testing_done(sub { ... })
572 This adds your coderef as a follow-up to the root hub after Test2
573 is finished loading.
574 This is essentially a helper to do the following:
576 test2_add_callback_post_load(sub {
578 my $stack = test2_stack();
579 $stack->top; # Insure we have a hub
580 my ($hub) = Test2::API::test2_stack->all;
581 $hub->set_active(1);
583 $hub->follow_up(sub { ... }); # <-- Your coderef here
585 });
586 test2_add_callback_context_acquire(sub { ... })
588 Add a callback that will be called every time someone tries to
589 acquire a context. This will be called on EVERY call to
590 "context()". It gets a single argument, a reference to the hash of
591 parameters being used the construct the context. This is your
592 chance to change the parameters by directly altering the hash.
593 test2_add_callback_context_acquire(sub {
595 my $params = shift;
596 $params->{level}++;
597 });
598 This is a very scary API function. Please do not use this unless
600 you need to. This is here for Test::Builder and backwards
601 compatibility. This has you directly manipulate the hash instead of
602 returning a new one for performance reasons.
603 test2_add_callback_context_init(sub { ... })
605 Add a callback that will be called every time a new context is
606 created. The callback will receive the newly created context as its
607 only argument.
608 test2_add_callback_context_release(sub { ... })
610 Add a callback that will be called every time a context is
611 released. The callback will receive the released context as its
612 only argument.
613 test2_add_callback_pre_subtest(sub { ... })
615 Add a callback that will be called every time a subtest is going to
616 be run. The callback will receive the subtest name, coderef, and
617 any arguments.
618 @list = test2_list_context_acquire_callbacks()
620 Return all the context acquire callback references.
621 @list = test2_list_context_init_callbacks()
623 Returns all the context init callback references.
624 @list = test2_list_context_release_callbacks()
626 Returns all the context release callback references.
627 @list = test2_list_exit_callbacks()
629 Returns all the exit callback references.
630 @list = test2_list_post_load_callbacks()
632 Returns all the post load callback references.
633 @list = test2_list_pre_subtest_callbacks()
635 Returns all the pre-subtest callback references.
636 test2_add_uuid_via(sub { ... })
638 $sub = test2_add_uuid_via()
639 This allows you to provide a UUID generator. If provided UUIDs will
640 be attached to all events, hubs, and contexts. This is useful for
641 storing, tracking, and linking these objects.
642 The sub you provide should always return a unique identifier. Most
644 things will expect a proper UUID string, however nothing in
645 Test2::API enforces this.
646 The sub will receive exactly 1 argument, the type of thing being
648 tagged 'context', 'hub', or 'event'. In the future additional
649 things may be tagged, in which case new strings will be passed in.
650 These are purely informative, you can (and usually should) ignore
651 them.
652 IPC AND CONCURRENCY
654 These let you access, or specify, the IPC system internals.
655 $bool = test2_has_ipc()
657 Check if IPC is enabled.
658 $ipc = test2_ipc()
660 This will return the global Test2::IPC::Driver instance. If this
661 has not yet been initialized it will be initialized now.
662 test2_ipc_add_driver($DRIVER)
664 Add an IPC driver to the list. This will add the driver to the
665 start of the list.
666 @drivers = test2_ipc_drivers()
668 Get the list of IPC drivers.
669 $bool = test2_ipc_polling()
671 Check if polling is enabled.
672 test2_ipc_enable_polling()
674 Turn on polling. This will cull events from other processes and
675 threads every time a context is created.
676 test2_ipc_disable_polling()
678 Turn off IPC polling.
679 test2_ipc_enable_shm()
681 Legacy, this is currently a no-op that returns 0;
682 test2_ipc_set_pending($uniq_val)
684 Tell other processes and events that an event is pending. $uniq_val
685 should be a unique value no other thread/process will generate.
686 Note: After calling this "test2_ipc_get_pending()" will return 1.
688 This is intentional, and not avoidable.
689 $pending = test2_ipc_get_pending()
691 This returns -1 if there is no way to check (assume yes)
692 This returns 0 if there are (most likely) no pending events.
694 This returns 1 if there are (likely) pending events. Upon return it
696 will reset, nothing else will be able to see that there were
697 pending events.
698 $timeout = test2_ipc_get_timeout()
700 test2_ipc_set_timeout($timeout)
701 Get/Set the timeout value for the IPC system. This timeout is how
702 long the IPC system will wait for child processes and threads to
703 finish before aborting.
704 The default value is 30 seconds.
706 MANAGING FORMATTERS
708 These let you access, or specify, the formatters that can/should be
709 used.
710 $formatter = test2_formatter
712 This will return the global formatter class. This is not an
713 instance. By default the formatter is set to Test2::Formatter::TAP.
714 You can override this default using the "T2_FORMATTER" environment
716 variable.
717 Normally 'Test2::Formatter::' is prefixed to the value in the
719 environment variable:
720 $ T2_FORMATTER='TAP' perl test.t # Use the Test2::Formatter::TAP formatter
722 $ T2_FORMATTER='Foo' perl test.t # Use the Test2::Formatter::Foo formatter
723 If you want to specify a full module name you use the '+' prefix:
725 $ T2_FORMATTER='+Foo::Bar' perl test.t # Use the Foo::Bar formatter
727 test2_formatter_set($class_or_instance)
729 Set the global formatter class. This can only be set once. Note:
730 This will override anything specified in the 'T2_FORMATTER'
731 environment variable.
732 @formatters = test2_formatters()
734 Get a list of all loaded formatters.
735 test2_formatter_add($class_or_instance)
737 Add a formatter to the list. Last formatter added is used at
738 initialization. If this is called after initialization a warning
739 will be issued.
740
742 See the "/Examples/" directory included in this distribution.
743
745 Test2::API::Context - Detailed documentation of the context object.
746 Test2::IPC - The IPC system used for threading/fork support.
748 Test2::Formatter - Formatters such as TAP live here.
750 Test2::Event - Events live in this namespace.
752 Test2::Hub - All events eventually funnel through a hub. Custom hubs
754 are how "intercept()" and "run_subtest()" are implemented.
755
757 This package has an END block. This END block is responsible for
758 setting the exit code based on the test results. This end block also
759 calls the callbacks that can be added to this package.
760
762 The source code repository for Test2 can be found at
763 http://github.com/Test-More/test-more/.
764
766 Chad Granum <exodist@cpan.org>
767
769 Chad Granum <exodist@cpan.org>
770
772 Copyright 2020 Chad Granum <exodist@cpan.org>.
773 This program is free software; you can redistribute it and/or modify it
775 under the same terms as Perl itself.
776 See http://dev.perl.org/licenses/
778perl v5.32.0 2020-10-15 Test2::API(3)