1Test2::API(3) User Contributed Perl Documentation Test2::API(3)
2
6 Test2::API - Primary interface for writing Test2 based testing tools.
7
9 The internals of this package are subject to change at any time! The
10 public methods provided will not change in backwards-incompatible ways
11 (once there is a stable release), but the underlying implementation
12 details might. Do not break encapsulation here!
13 Currently the implementation is to create a single instance of the
15 Test2::API::Instance Object. All class methods defer to the single
16 instance. There is no public access to the singleton, and that is
17 intentional. The class methods provided by this package provide the
18 only functionality publicly exposed.
19 This is done primarily to avoid the problems Test::Builder had by
21 exposing its singleton. We do not want anyone to replace this
22 singleton, rebless it, or directly muck with its internals. If you need
23 to do something and cannot because of the restrictions placed here,
24 then please report it as an issue. If possible, we will create a way
25 for you to implement your functionality without exposing things that
26 should not be exposed.
27
29 This package exports all the functions necessary to write and/or verify
30 testing tools. Using these building blocks you can begin writing test
31 tools very quickly. You are also provided with tools that help you to
32 test the tools you write.
33
35 WRITING A TOOL
36 The "context()" method is your primary interface into the Test2
37 framework.
38 package My::Ok;
40 use Test2::API qw/context/;
41 our @EXPORT = qw/my_ok/;
43 use base 'Exporter';
44 # Just like ok() from Test::More
46 sub my_ok($;$) {
47 my ($bool, $name) = @_;
48 my $ctx = context(); # Get a context
49 $ctx->ok($bool, $name);
50 $ctx->release; # Release the context
51 return $bool;
52 }
53 See Test2::API::Context for a list of methods available on the context
55 object.
56 TESTING YOUR TOOLS
58 The "intercept { ... }" tool lets you temporarily intercept all events
59 generated by the test system:
60 use Test2::API qw/intercept/;
62 use My::Ok qw/my_ok/;
64 my $events = intercept {
66 # These events are not displayed
67 my_ok(1, "pass");
68 my_ok(0, "fail");
69 };
70 my_ok(@$events == 2, "got 2 events, the pass and the fail");
72 my_ok($events->[0]->pass, "first event passed");
73 my_ok(!$events->[1]->pass, "second event failed");
74 DEEP EVENT INTERCEPTION
76 Normally "intercept { ... }" only intercepts events sent to the main
78 hub (as added by intercept itself). Nested hubs, such as those created
79 by subtests, will not be intercepted. This is normally what you will
80 still see the nested events by inspecting the subtest event. However
81 there are times where you want to verify each event as it is sent, in
82 that case use "intercept_deep { ... }".
83 my $events = intercept_Deep {
85 buffered_subtest foo => sub {
86 ok(1, "pass");
87 };
88 };
89 $events in this case will contain 3 items:
91 The event from "ok(1, "pass")"
93 The plan event for the subtest
94 The subtest event itself, with the first 2 events nested inside it as
95 children.
96 This lets you see the order in which the events were sent, unlike
98 "intercept { ... }" which only lets you see events as the main hub sees
99 them.
100 OTHER API FUNCTIONS
102 use Test2::API qw{
103 test2_init_done
104 test2_stack
105 test2_set_is_end
106 test2_get_is_end
107 test2_ipc
108 test2_formatter_set
109 test2_formatter
110 test2_is_testing_done
111 };
112 my $init = test2_init_done();
114 my $stack = test2_stack();
115 my $ipc = test2_ipc();
116 test2_formatter_set($FORMATTER)
118 my $formatter = test2_formatter();
119 ... And others ...
121
123 All exports are optional. You must specify subs to import.
124 use Test2::API qw/context intercept run_subtest/;
126 This is the list of exports that are most commonly needed. If you are
128 simply writing a tool, then this is probably all you need. If you need
129 something and you cannot find it here, then you can also look at "OTHER
130 API EXPORTS".
131 These exports lack the 'test2_' prefix because of how important/common
133 they are. Exports in the "OTHER API EXPORTS" section have the 'test2_'
134 prefix to ensure they stand out.
135 context(...)
137 Usage:
138 $ctx = context()
140 $ctx = context(%params)
141 The "context()" function will always return the current context. If
143 there is already a context active, it will be returned. If there is not
144 an active context, one will be generated. When a context is generated
145 it will default to using the file and line number where the currently
146 running sub was called from.
147 Please see "CRITICAL DETAILS" in Test2::API::Context for important
149 rules about what you can and cannot do with a context once it is
150 obtained.
151 Note This function will throw an exception if you ignore the context
153 object it returns.
154 Note On perls 5.14+ a depth check is used to insure there are no
156 context leaks. This cannot be safely done on older perls due to
157 <https://rt.perl.org/Public/Bug/Display.html?id=127774> You can
158 forcefully enable it either by setting "$ENV{T2_CHECK_DEPTH} = 1" or
159 "$Test2::API::DO_DEPTH_CHECK = 1" BEFORE loading Test2::API.
160 OPTIONAL PARAMETERS
162 All parameters to "context" are optional.
164 level => $int
166 If you must obtain a context in a sub deeper than your entry point
167 you can use this to tell it how many EXTRA stack frames to look
168 back. If this option is not provided the default of 0 is used.
169 sub third_party_tool {
171 my $sub = shift;
172 ... # Does not obtain a context
173 $sub->();
174 ...
175 }
176 third_party_tool(sub {
178 my $ctx = context(level => 1);
179 ...
180 $ctx->release;
181 });
182 wrapped => $int
184 Use this if you need to write your own tool that wraps a call to
185 "context()" with the intent that it should return a context object.
186 sub my_context {
188 my %params = ( wrapped => 0, @_ );
189 $params{wrapped}++;
190 my $ctx = context(%params);
191 ...
192 return $ctx;
193 }
194 sub my_tool {
196 my $ctx = my_context();
197 ...
198 $ctx->release;
199 }
200 If you do not do this, then tools you call that also check for a
202 context will notice that the context they grabbed was created at
203 the same stack depth, which will trigger protective measures that
204 warn you and destroy the existing context.
205 stack => $stack
207 Normally "context()" looks at the global hub stack. If you are
208 maintaining your own Test2::API::Stack instance you may pass it in
209 to be used instead of the global one.
210 hub => $hub
212 Use this parameter if you want to obtain the context for a specific
213 hub instead of whatever one happens to be at the top of the stack.
214 on_init => sub { ... }
216 This lets you provide a callback sub that will be called ONLY if
217 your call to "context()" generated a new context. The callback WILL
218 NOT be called if "context()" is returning an existing context. The
219 only argument passed into the callback will be the context object
220 itself.
221 sub foo {
223 my $ctx = context(on_init => sub { 'will run' });
224 my $inner = sub {
226 # This callback is not run since we are getting the existing
227 # context from our parent sub.
228 my $ctx = context(on_init => sub { 'will NOT run' });
229 $ctx->release;
230 }
231 $inner->();
232 $ctx->release;
234 }
235 on_release => sub { ... }
237 This lets you provide a callback sub that will be called when the
238 context instance is released. This callback will be added to the
239 returned context even if an existing context is returned. If
240 multiple calls to context add callbacks, then all will be called in
241 reverse order when the context is finally released.
242 sub foo {
244 my $ctx = context(on_release => sub { 'will run second' });
245 my $inner = sub {
247 my $ctx = context(on_release => sub { 'will run first' });
248 # Neither callback runs on this release
250 $ctx->release;
251 }
252 $inner->();
253 # Both callbacks run here.
255 $ctx->release;
256 }
257 release($;$)
259 Usage:
260 release $ctx;
262 release $ctx, ...;
263 This is intended as a shortcut that lets you release your context and
265 return a value in one statement. This function will get your context,
266 and an optional return value. It will release your context, then return
267 your value. Scalar context is always assumed.
268 sub tool {
270 my $ctx = context();
271 ...
272 return release $ctx, 1;
274 }
275 This tool is most useful when you want to return the value you get from
277 calling a function that needs to see the current context:
278 my $ctx = context();
280 my $out = some_tool(...);
281 $ctx->release;
282 return $out;
283 We can combine the last 3 lines of the above like so:
285 my $ctx = context();
287 release $ctx, some_tool(...);
288 context_do(&;@)
290 Usage:
291 sub my_tool {
293 context_do {
294 my $ctx = shift;
295 my (@args) = @_;
297 $ctx->ok(1, "pass");
299 ...
301 # No need to call $ctx->release, done for you on scope exit.
303 } @_;
304 }
305 Using this inside your test tool takes care of a lot of boilerplate for
307 you. It will ensure a context is acquired. It will capture and rethrow
308 any exception. It will insure the context is released when you are
309 done. It preserves the subroutine call context (array, scalar, void).
310 This is the safest way to write a test tool. The only two downsides to
312 this are a slight performance decrease, and some extra indentation in
313 your source. If the indentation is a problem for you then you can take
314 a peek at the next section.
315 no_context(&;$)
317 Usage:
318 no_context { ... };
320 no_context { ... } $hid;
321 sub my_tool(&) {
322 my $code = shift;
323 my $ctx = context();
324 ...
325 no_context {
327 # Things in here will not see our current context, they get a new
328 # one.
329 $code->();
331 };
332 ...
334 $ctx->release;
335 };
336 This tool will hide a context for the provided block of code. This
338 means any tools run inside the block will get a completely new context
339 if they acquire one. The new context will be inherited by tools nested
340 below the one that acquired it.
341 This will normally hide the current context for the top hub. If you
343 need to hide the context for a different hub you can pass in the
344 optional $hid parameter.
345 intercept(&)
347 Usage:
348 my $events = intercept {
350 ok(1, "pass");
351 ok(0, "fail");
352 ...
353 };
354 This function takes a codeblock as its only argument, and it has a
356 prototype. It will execute the codeblock, intercepting any generated
357 events in the process. It will return an array reference with all the
358 generated event objects. All events should be subclasses of
359 Test2::Event.
360 This is a very low-level subtest tool. This is useful for writing tools
362 which produce subtests. This is not intended for people simply writing
363 tests.
364 run_subtest(...)
366 Usage:
367 run_subtest($NAME, \&CODE, $BUFFERED, @ARGS)
369 # or
371 run_subtest($NAME, \&CODE, \%PARAMS, @ARGS)
373 This will run the provided codeblock with the args in @args. This
375 codeblock will be run as a subtest. A subtest is an isolated test state
376 that is condensed into a single Test2::Event::Subtest event, which
377 contains all events generated inside the subtest.
378 ARGUMENTS:
380 $NAME
382 The name of the subtest.
383 \&CODE
385 The code to run inside the subtest.
386 $BUFFERED or \%PARAMS
388 If this is a simple scalar then it will be treated as a boolean for
389 the 'buffered' setting. If this is a hash reference then it will be
390 used as a parameters hash. The param hash will be used for hub
391 construction (with the specified keys removed).
392 Keys that are removed and used by run_subtest:
394 'buffered' => $bool
396 Toggle buffered status.
397 'inherit_trace' => $bool
399 Normally the subtest hub is pushed and the sub is allowed to
400 generate its own root context for the hub. When this setting is
401 turned on a root context will be created for the hub that
402 shares the same trace as the current context.
403 Set this to true if your tool is producing subtests without
405 user-specified subs.
406 'no_fork' => $bool
408 Defaults to off. Normally forking inside a subtest will
409 actually fork the subtest, resulting in 2 final subtest events.
410 This parameter will turn off that behavior, only the original
411 process/thread will return a final subtest event.
412 @ARGS
414 Any extra arguments you want passed into the subtest code.
415 BUFFERED VS UNBUFFERED (OR STREAMED)
417 Normally all events inside and outside a subtest are sent to the
419 formatter immediately by the hub. Sometimes it is desirable to hold off
420 sending events within a subtest until the subtest is complete. This
421 usually depends on the formatter being used.
422 Things not effected by this flag
424 In both cases events are generated and stored in an array. This
425 array is eventually used to populate the "subevents" attribute on
426 the Test2::Event::Subtest event that is generated at the end of the
427 subtest. This flag has no effect on this part, it always happens.
428 At the end of the subtest, the final Test2::Event::Subtest event is
430 sent to the formatter.
431 Things that are effected by this flag
433 The "buffered" attribute of the Test2::Event::Subtest event will be
434 set to the value of this flag. This means any formatter, listener,
435 etc which looks at the event will know if it was buffered.
436 Things that are formatter dependant
438 Events within a buffered subtest may or may not be sent to the
439 formatter as they happen. If a formatter fails to specify then the
440 default is to NOT SEND the events as they are generated, instead
441 the formatter can pull them from the "subevents" attribute.
442 A formatter can specify by implementing the "hide_buffered()"
444 method. If this method returns true then events generated inside a
445 buffered subtest will not be sent independently of the final
446 subtest event.
447 An example of how this is used is the Test2::Formatter::TAP formatter.
449 For unbuffered subtests the events are rendered as they are generated.
450 At the end of the subtest, the final subtest event is rendered, but the
451 "subevents" attribute is ignored. For buffered subtests the opposite
452 occurs, the events are NOT rendered as they are generated, instead the
453 "subevents" attribute is used to render them all at once. This is
454 useful when running subtests tests in parallel, since without it the
455 output from subtests would be interleaved together.
456
458 Exports in this section are not commonly needed. These all have the
459 'test2_' prefix to help ensure they stand out. You should look at the
460 "MAIN API EXPORTS" section before looking here. This section is one
461 where "Great power comes with great responsibility". It is possible to
462 break things badly if you are not careful with these.
463 All exports are optional. You need to list which ones you want at
465 import time:
466 use Test2::API qw/test2_init_done .../;
468 STATUS AND INITIALIZATION STATE
470 These provide access to internal state and object instances.
471 $bool = test2_init_done()
473 This will return true if the stack and IPC instances have already
474 been initialized. It will return false if they have not. Init
475 happens as late as possible. It happens as soon as a tool requests
476 the IPC instance, the formatter, or the stack.
477 $bool = test2_load_done()
479 This will simply return the boolean value of the loaded flag. If
480 Test2 has finished loading this will be true, otherwise false.
481 Loading is considered complete the first time a tool requests a
482 context.
483 test2_set_is_end()
485 test2_set_is_end($bool)
486 This is used to toggle Test2's belief that the END phase has
487 already started. With no arguments this will set it to true. With
488 arguments it will set it to the first argument's value.
489 This is used to prevent the use of "caller()" in END blocks which
491 can cause segfaults. This is only necessary in some persistent
492 environments that may have multiple END phases.
493 $bool = test2_get_is_end()
495 Check if Test2 believes it is the END phase.
496 $stack = test2_stack()
498 This will return the global Test2::API::Stack instance. If this has
499 not yet been initialized it will be initialized now.
500 $bool = test2_is_testing_done()
502 This will return true if testing is complete and no other events
503 should be sent. This is useful in things like warning handlers
504 where you might want to turn warnings into events, but need them to
505 start acting like normal warnings when testing is done.
506 $SIG{__WARN__} = sub {
508 my ($warning) = @_;
509 if (test2_is_testing_done()) {
511 warn @_;
512 }
513 else {
514 my $ctx = context();
515 ...
516 $ctx->release
517 }
518 }
519 test2_ipc_disable
521 Disable IPC.
522 $bool = test2_ipc_diabled
524 Check if IPC is disabled.
525 test2_ipc_wait_enable()
527 test2_ipc_wait_disable()
528 $bool = test2_ipc_wait_enabled()
529 These can be used to turn IPC waiting on and off, or check the
530 current value of the flag.
531 Waiting is turned on by default. Waiting will cause the parent
533 process/thread to wait until all child processes and threads are
534 finished before exiting. You will almost never want to turn this
535 off.
536 $bool = test2_no_wait()
538 test2_no_wait($bool)
539 DISCOURAGED: This is a confusing interface, it is better to use
540 "test2_ipc_wait_enable()", "test2_ipc_wait_disable()" and
541 "test2_ipc_wait_enabled()".
542 This can be used to get/set the no_wait status. Waiting is turned
544 on by default. Waiting will cause the parent process/thread to wait
545 until all child processes and threads are finished before exiting.
546 You will almost never want to turn this off.
547 $fh = test2_stdout()
549 $fh = test2_stderr()
550 These functions return the filehandles that test output should be
551 written to. They are primarily useful when writing a custom
552 formatter and code that turns events into actual output (TAP,
553 etc.). They will return a dupe of the original filehandles that
554 formatted output can be sent to regardless of whatever state the
555 currently running test may have left STDOUT and STDERR in.
556 test2_reset_io()
558 Re-dupe the internal filehandles returned by "test2_stdout()" and
559 "test2_stderr()" from the current STDOUT and STDERR. You shouldn't
560 need to do this except in very peculiar situations (for example,
561 you're testing a new formatter and you need control over where the
562 formatter is sending its output.)
563 BEHAVIOR HOOKS
565 These are hooks that allow you to add custom behavior to actions taken
566 by Test2 and tools built on top of it.
567 test2_add_callback_exit(sub { ... })
569 This can be used to add a callback that is called after all testing
570 is done. This is too late to add additional results, the main use
571 of this callback is to set the exit code.
572 test2_add_callback_exit(
574 sub {
575 my ($context, $exit, \$new_exit) = @_;
576 ...
577 }
578 );
579 The $context passed in will be an instance of Test2::API::Context.
581 The $exit argument will be the original exit code before anything
582 modified it. $$new_exit is a reference to the new exit code. You
583 may modify this to change the exit code. Please note that
584 $$new_exit may already be different from $exit
585 test2_add_callback_post_load(sub { ... })
587 Add a callback that will be called when Test2 is finished loading.
588 This means the callback will be run once, the first time a context
589 is obtained. If Test2 has already finished loading then the
590 callback will be run immediately.
591 test2_add_callback_testing_done(sub { ... })
593 This adds your coderef as a follow-up to the root hub after Test2
594 is finished loading.
595 This is essentially a helper to do the following:
597 test2_add_callback_post_load(sub {
599 my $stack = test2_stack();
600 $stack->top; # Insure we have a hub
601 my ($hub) = Test2::API::test2_stack->all;
602 $hub->set_active(1);
604 $hub->follow_up(sub { ... }); # <-- Your coderef here
606 });
607 test2_add_callback_context_acquire(sub { ... })
609 Add a callback that will be called every time someone tries to
610 acquire a context. This will be called on EVERY call to
611 "context()". It gets a single argument, a reference to the hash of
612 parameters being used the construct the context. This is your
613 chance to change the parameters by directly altering the hash.
614 test2_add_callback_context_acquire(sub {
616 my $params = shift;
617 $params->{level}++;
618 });
619 This is a very scary API function. Please do not use this unless
621 you need to. This is here for Test::Builder and backwards
622 compatibility. This has you directly manipulate the hash instead of
623 returning a new one for performance reasons.
624 test2_add_callback_context_init(sub { ... })
626 Add a callback that will be called every time a new context is
627 created. The callback will receive the newly created context as its
628 only argument.
629 test2_add_callback_context_release(sub { ... })
631 Add a callback that will be called every time a context is
632 released. The callback will receive the released context as its
633 only argument.
634 test2_add_callback_pre_subtest(sub { ... })
636 Add a callback that will be called every time a subtest is going to
637 be run. The callback will receive the subtest name, coderef, and
638 any arguments.
639 @list = test2_list_context_acquire_callbacks()
641 Return all the context acquire callback references.
642 @list = test2_list_context_init_callbacks()
644 Returns all the context init callback references.
645 @list = test2_list_context_release_callbacks()
647 Returns all the context release callback references.
648 @list = test2_list_exit_callbacks()
650 Returns all the exit callback references.
651 @list = test2_list_post_load_callbacks()
653 Returns all the post load callback references.
654 @list = test2_list_pre_subtest_callbacks()
656 Returns all the pre-subtest callback references.
657 test2_add_uuid_via(sub { ... })
659 $sub = test2_add_uuid_via()
660 This allows you to provide a UUID generator. If provided UUIDs will
661 be attached to all events, hubs, and contexts. This is useful for
662 storing, tracking, and linking these objects.
663 The sub you provide should always return a unique identifier. Most
665 things will expect a proper UUID string, however nothing in
666 Test2::API enforces this.
667 The sub will receive exactly 1 argument, the type of thing being
669 tagged 'context', 'hub', or 'event'. In the future additional
670 things may be tagged, in which case new strings will be passed in.
671 These are purely informative, you can (and usually should) ignore
672 them.
673 IPC AND CONCURRENCY
675 These let you access, or specify, the IPC system internals.
676 $bool = test2_has_ipc()
678 Check if IPC is enabled.
679 $ipc = test2_ipc()
681 This will return the global Test2::IPC::Driver instance. If this
682 has not yet been initialized it will be initialized now.
683 test2_ipc_add_driver($DRIVER)
685 Add an IPC driver to the list. This will add the driver to the
686 start of the list.
687 @drivers = test2_ipc_drivers()
689 Get the list of IPC drivers.
690 $bool = test2_ipc_polling()
692 Check if polling is enabled.
693 test2_ipc_enable_polling()
695 Turn on polling. This will cull events from other processes and
696 threads every time a context is created.
697 test2_ipc_disable_polling()
699 Turn off IPC polling.
700 test2_ipc_enable_shm()
702 Legacy, this is currently a no-op that returns 0;
703 test2_ipc_set_pending($uniq_val)
705 Tell other processes and events that an event is pending. $uniq_val
706 should be a unique value no other thread/process will generate.
707 Note: After calling this "test2_ipc_get_pending()" will return 1.
709 This is intentional, and not avoidable.
710 $pending = test2_ipc_get_pending()
712 This returns -1 if there is no way to check (assume yes)
713 This returns 0 if there are (most likely) no pending events.
715 This returns 1 if there are (likely) pending events. Upon return it
717 will reset, nothing else will be able to see that there were
718 pending events.
719 $timeout = test2_ipc_get_timeout()
721 test2_ipc_set_timeout($timeout)
722 Get/Set the timeout value for the IPC system. This timeout is how
723 long the IPC system will wait for child processes and threads to
724 finish before aborting.
725 The default value is 30 seconds.
727 MANAGING FORMATTERS
729 These let you access, or specify, the formatters that can/should be
730 used.
731 $formatter = test2_formatter
733 This will return the global formatter class. This is not an
734 instance. By default the formatter is set to Test2::Formatter::TAP.
735 You can override this default using the "T2_FORMATTER" environment
737 variable.
738 Normally 'Test2::Formatter::' is prefixed to the value in the
740 environment variable:
741 $ T2_FORMATTER='TAP' perl test.t # Use the Test2::Formatter::TAP formatter
743 $ T2_FORMATTER='Foo' perl test.t # Use the Test2::Formatter::Foo formatter
744 If you want to specify a full module name you use the '+' prefix:
746 $ T2_FORMATTER='+Foo::Bar' perl test.t # Use the Foo::Bar formatter
748 test2_formatter_set($class_or_instance)
750 Set the global formatter class. This can only be set once. Note:
751 This will override anything specified in the 'T2_FORMATTER'
752 environment variable.
753 @formatters = test2_formatters()
755 Get a list of all loaded formatters.
756 test2_formatter_add($class_or_instance)
758 Add a formatter to the list. Last formatter added is used at
759 initialization. If this is called after initialization a warning
760 will be issued.
761
763 See the "/Examples/" directory included in this distribution.
764
766 Test2::API::Context - Detailed documentation of the context object.
767 Test2::IPC - The IPC system used for threading/fork support.
769 Test2::Formatter - Formatters such as TAP live here.
771 Test2::Event - Events live in this namespace.
773 Test2::Hub - All events eventually funnel through a hub. Custom hubs
775 are how "intercept()" and "run_subtest()" are implemented.
776
778 This package has an END block. This END block is responsible for
779 setting the exit code based on the test results. This end block also
780 calls the callbacks that can be added to this package.
781
783 The source code repository for Test2 can be found at
784 http://github.com/Test-More/test-more/.
785
787 Chad Granum <exodist@cpan.org>
788
790 Chad Granum <exodist@cpan.org>
791
793 Copyright 2019 Chad Granum <exodist@cpan.org>.
794 This program is free software; you can redistribute it and/or modify it
796 under the same terms as Perl itself.
797 See http://dev.perl.org/licenses/
799perl v5.30.2 2020-03-31 Test2::API(3)