1TAP::Harness::Beyond(3)User Contributed Perl DocumentatioTnAP::Harness::Beyond(3)
2
6 Test::Harness::Beyond - Beyond make test
7
9 Test::Harness is responsible for running test scripts, analysing their
10 output and reporting success or failure. When I type make test (or
11 ./Build test) for a module, Test::Harness is usually used to run the
12 tests (not all modules use Test::Harness but the majority do).
13 To start exploring some of the features of Test::Harness I need to
15 switch from make test to the prove command (which ships with
16 Test::Harness). For the following examples I'll also need a recent
17 version of Test::Harness installed; 3.14 is current as I write.
18 For the examples I'm going to assume that we're working with a 'normal'
20 Perl module distribution. Specifically I'll assume that typing make or
21 ./Build causes the built, ready-to-install module code to be available
22 below ./blib/lib and ./blib/arch and that there's a directory called
23 't' that contains our tests. Test::Harness isn't hardwired to that
24 configuration but it saves me from explaining which files live where
25 for each example.
26 Back to prove; like make test it runs a test suite - but it provides
28 far more control over which tests are executed, in what order and how
29 their results are reported. Typically make test runs all the test
30 scripts below the 't' directory. To do the same thing with prove I
31 type:
32 prove -rb t
34 The switches here are -r to recurse into any directories below 't' and
36 -b which adds ./blib/lib and ./blib/arch to Perl's include path so that
37 the tests can find the code they will be testing. If I'm testing a
38 module of which an earlier version is already installed I need to be
39 careful about the include path to make sure I'm not running my tests
40 against the installed version rather than the new one that I'm working
41 on.
42 Unlike make test, typing prove doesn't automatically rebuild my module.
44 If I forget to make before prove I will be testing against older
45 versions of those files - which inevitably leads to confusion. I
46 either get into the habit of typing
47 make && prove -rb t
49 or - if I have no XS code that needs to be built I use the modules
51 below lib instead
52 prove -Ilib -r t
54 So far I've shown you nothing that make test doesn't do. Let's fix
56 that.
57 Saved State
59 If I have failing tests in a test suite that consists of more than a
60 handful of scripts and takes more than a few seconds to run it rapidly
61 becomes tedious to run the whole test suite repeatedly as I track down
62 the problems.
63 I can tell prove just to run the tests that are failing like this:
65 prove -b t/this_fails.t t/so_does_this.t
67 That speeds things up but I have to make a note of which tests are
69 failing and make sure that I run those tests. Instead I can use prove's
70 --state switch and have it keep track of failing tests for me. First I
71 do a complete run of the test suite and tell prove to save the results:
72 prove -rb --state=save t
74 That stores a machine readable summary of the test run in a file called
76 '.prove' in the current directory. If I have failures I can then run
77 just the failing scripts like this:
78 prove -b --state=failed
80 I can also tell prove to save the results again so that it updates its
82 idea of which tests failed:
83 prove -b --state=failed,save
85 As soon as one of my failing tests passes it will be removed from the
87 list of failed tests. Eventually I fix them all and prove can find no
88 failing tests to run:
89 Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
91 Result: NOTESTS
92 As I work on a particular part of my module it's most likely that the
94 tests that cover that code will fail. I'd like to run the whole test
95 suite but have it prioritize these 'hot' tests. I can tell prove to do
96 this:
97 prove -rb --state=hot,save t
99 All the tests will run but those that failed most recently will be run
101 first. If no tests have failed since I started saving state all tests
102 will run in their normal order. This combines full test coverage with
103 early notification of failures.
104 The --state switch supports a number of options; for example to run
106 failed tests first followed by all remaining tests ordered by the
107 timestamps of the test scripts - and save the results - I can use
108 prove -rb --state=failed,new,save t
110 See the prove documentation (type prove --man) for the full list of
112 state options.
113 When I tell prove to save state it writes a file called '.prove'
115 ('_prove' on Windows) in the current directory. It's a YAML document so
116 it's quite easy to write tools of your own that work on the saved test
117 state - but the format isn't officially documented so it might change
118 without (much) warning in the future.
119 Parallel Testing
121 If my tests take too long to run I may be able to speed them up by
122 running multiple test scripts in parallel. This is particularly
123 effective if the tests are I/O bound or if I have multiple CPU cores. I
124 tell prove to run my tests in parallel like this:
125 prove -rb -j 9 t
127 The -j switch enables parallel testing; the number that follows it is
129 the maximum number of tests to run in parallel. Sometimes tests that
130 pass when run sequentially will fail when run in parallel. For example
131 if two different test scripts use the same temporary file or attempt to
132 listen on the same socket I'll have problems running them in parallel.
133 If I see unexpected failures I need to check my tests to work out which
134 of them are trampling on the same resource and rename temporary files
135 or add locks as appropriate.
136 To get the most performance benefit I want to have the test scripts
138 that take the longest to run start first - otherwise I'll be waiting
139 for the one test that takes nearly a minute to complete after all the
140 others are done. I can use the --state switch to run the tests in
141 slowest to fastest order:
142 prove -rb -j 9 --state=slow,save t
144 Non-Perl Tests
146 The Test Anything Protocol (http://testanything.org/) isn't just for
147 Perl. Just about any language can be used to write tests that output
148 TAP. There are TAP based testing libraries for C, C++, PHP, Python and
149 many others. If I can't find a TAP library for my language of choice
150 it's easy to generate valid TAP. It looks like this:
151 1..3
153 ok 1 - init OK
154 ok 2 - opened file
155 not ok 3 - appended to file
156 The first line is the plan - it specifies the number of tests I'm going
158 to run so that it's easy to check that the test script didn't exit
159 before running all the expected tests. The following lines are the test
160 results - 'ok' for pass, 'not ok' for fail. Each test has a number and,
161 optionally, a description. And that's it. Any language that can produce
162 output like that on STDOUT can be used to write tests.
163 Recently I've been rekindling a two-decades-old interest in Forth.
165 Evidently I have a masochistic streak that even Perl can't satisfy. I
166 want to write tests in Forth and run them using prove (you can find my
167 gforth TAP experiments at https://svn.hexten.net/andy/Forth/Testing/).
168 I can use the --exec switch to tell prove to run the tests using gforth
169 like this:
170 prove -r --exec gforth t
172 Alternately, if the language used to write my tests allows a shebang
174 line I can use that to specify the interpreter. Here's a test written
175 in PHP:
176 #!/usr/bin/php
178 <?php
179 print "1..2\n";
180 print "ok 1\n";
181 print "not ok 2\n";
182 ?>
183 If I save that as t/phptest.t the shebang line will ensure that it runs
185 correctly along with all my other tests.
186 Mixing it up
188 Subtle interdependencies between test programs can mask problems - for
189 example an earlier test may neglect to remove a temporary file that
190 affects the behaviour of a later test. To find this kind of problem I
191 use the --shuffle and --reverse options to run my tests in random or
192 reversed order.
193 Rolling My Own
195 If I need a feature that prove doesn't provide I can easily write my
196 own.
197 Typically you'll want to change how TAP gets input into and output from
199 the parser. App::Prove supports arbitrary plugins, and TAP::Harness
200 supports custom formatters and source handlers that you can load using
201 either prove or Module::Build; there are many examples to base mine on.
202 For more details see App::Prove, TAP::Parser::SourceHandler, and
203 TAP::Formatter::Base.
204 If writing a plugin is not enough, you can write your own test harness;
206 one of the motives for the 3.00 rewrite of Test::Harness was to make it
207 easier to subclass and extend.
208 The Test::Harness module is a compatibility wrapper around
210 TAP::Harness. For new applications I should use TAP::Harness directly.
211 As we'll see, prove uses TAP::Harness.
212 When I run prove it processes its arguments, figures out which test
214 scripts to run and then passes control to TAP::Harness to run the
215 tests, parse, analyse and present the results. By subclassing
216 TAP::Harness I can customise many aspects of the test run.
217 I want to log my test results in a database so I can track them over
219 time. To do this I override the summary method in TAP::Harness. I
220 start with a simple prototype that dumps the results as a YAML
221 document:
222 package My::TAP::Harness;
224 use base 'TAP::Harness';
226 use YAML;
227 sub summary {
229 my ( $self, $aggregate ) = @_;
230 print Dump( $aggregate );
231 $self->SUPER::summary( $aggregate );
232 }
233 1;
235 I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness is
237 on Perl's @INC include path I can
238 prove --harness=My::TAP::Harness -rb t
240 If I don't have My::TAP::Harness installed on @INC I need to provide
242 the correct path to perl when I run prove:
243 perl -Ilib `which prove` --harness=My::TAP::Harness -rb t
245 I can incorporate these options into my own version of prove. It's
247 pretty simple. Most of the work of prove is handled by App::Prove. The
248 important code in prove is just:
249 use App::Prove;
251 my $app = App::Prove->new;
253 $app->process_args(@ARGV);
254 exit( $app->run ? 0 : 1 );
255 If I write a subclass of App::Prove I can customise any aspect of the
257 test runner while inheriting all of prove's behaviour. Here's myprove:
258 #!/usr/bin/env perl use lib qw( lib ); # Add ./lib to @INC
260 use App::Prove;
261 my $app = App::Prove->new;
263 # Use custom TAP::Harness subclass
265 $app->harness( 'My::TAP::Harness' );
266 $app->process_args( @ARGV ); exit( $app->run ? 0 : 1 );
268 Now I can run my tests like this
270 ./myprove -rb t
272 Deeper Customisation
274 Now that I know how to subclass and replace TAP::Harness I can replace
275 any other part of the harness. To do that I need to know which classes
276 are responsible for which functionality. Here's a brief guided tour;
277 the default class for each component is shown in parentheses. Normally
278 any replacements I write will be subclasses of these default classes.
279 When I run my tests TAP::Harness creates a scheduler
281 (TAP::Parser::Scheduler) to work out the running order for the tests,
282 an aggregator (TAP::Parser::Aggregator) to collect and analyse the test
283 results and a formatter (TAP::Formatter::Console) to display those
284 results.
285 If I'm running my tests in parallel there may also be a multiplexer
287 (TAP::Parser::Multiplexer) - the component that allows multiple tests
288 to run simultaneously.
289 Once it has created those helpers TAP::Harness starts running the
291 tests. For each test it creates a new parser (TAP::Parser) which is
292 responsible for running the test script and parsing its output.
293 To replace any of these components I call one of these harness methods
295 with the name of the replacement class:
296 aggregator_class
298 formatter_class
299 multiplexer_class
300 parser_class
301 scheduler_class
302 For example, to replace the aggregator I would
304 $harness->aggregator_class( 'My::Aggregator' );
306 Alternately I can supply the names of my substitute classes to the
308 TAP::Harness constructor:
309 my $harness = TAP::Harness->new(
311 { aggregator_class => 'My::Aggregator' }
312 );
313 If I need to reach even deeper into the internals of the harness I can
315 replace the classes that TAP::Parser uses to execute test scripts and
316 tokenise their output. Before running a test script TAP::Parser creates
317 a grammar (TAP::Parser::Grammar) to decode the raw TAP into tokens, a
318 result factory (TAP::Parser::ResultFactory) to turn the decoded TAP
319 results into objects and, depending on whether it's running a test
320 script or reading TAP from a file, scalar or array a source or an
321 iterator (TAP::Parser::IteratorFactory).
322 Each of these objects may be replaced by calling one of these parser
324 methods:
325 source_class
327 perl_source_class
328 grammar_class
329 iterator_factory_class
330 result_factory_class
331 Callbacks
333 As an alternative to subclassing the components I need to change I can
334 attach callbacks to the default classes. TAP::Harness exposes these
335 callbacks:
336 parser_args Tweak the parameters used to create the parser
338 made_parser Just made a new parser
339 before_runtests About to run tests
340 after_runtests Have run all tests
341 after_test Have run an individual test script
342 TAP::Parser also supports callbacks; bailout, comment, plan, test,
344 unknown, version and yaml are called for the corresponding TAP result
345 types, ALL is called for all results, ELSE is called for all results
346 for which a named callback is not installed and EOF is called once at
347 the end of each TAP stream.
348 To install a callback I pass the name of the callback and a subroutine
350 reference to TAP::Harness or TAP::Parser's callback method:
351 $harness->callback( after_test => sub {
353 my ( $script, $desc, $parser ) = @_;
354 } );
355 I can also pass callbacks to the constructor:
357 my $harness = TAP::Harness->new({
359 callbacks => {
360 after_test => sub {
361 my ( $script, $desc, $parser ) = @_;
362 # Do something interesting here
363 }
364 }
365 });
366 When it comes to altering the behaviour of the test harness there's
368 more than one way to do it. Which way is best depends on my
369 requirements. In general if I only want to observe test execution
370 without changing the harness' behaviour (for example to log test
371 results to a database) I choose callbacks. If I want to make the
372 harness behave differently subclassing gives me more control.
373 Parsing TAP
375 Perhaps I don't need a complete test harness. If I already have a TAP
376 test log that I need to parse all I need is TAP::Parser and the various
377 classes it depends upon. Here's the code I need to run a test and parse
378 its TAP output
379 use TAP::Parser;
381 my $parser = TAP::Parser->new( { source => 't/simple.t' } );
383 while ( my $result = $parser->next ) {
384 print $result->as_string, "\n";
385 }
386 Alternately I can pass an open filehandle as source and have the parser
388 read from that rather than attempting to run a test script:
389 open my $tap, '<', 'tests.tap'
391 or die "Can't read TAP transcript ($!)\n";
392 my $parser = TAP::Parser->new( { source => $tap } );
393 while ( my $result = $parser->next ) {
394 print $result->as_string, "\n";
395 }
396 This approach is useful if I need to convert my TAP based test results
398 into some other representation. See TAP::Convert::TET
399 (http://search.cpan.org/dist/TAP-Convert-TET/) for an example of this
400 approach.
401 Getting Support
403 The Test::Harness developers hang out on the tapx-dev mailing list[1].
404 For discussion of general, language independent TAP issues there's the
405 tap-l[2] list. Finally there's a wiki dedicated to the Test Anything
406 Protocol[3]. Contributions to the wiki, patches and suggestions are all
407 welcome.
408 [1] <http://www.hexten.net/mailman/listinfo/tapx-dev> [2]
410 <http://testanything.org/mailman/listinfo/tap-l> [3]
411 <http://testanything.org/>
412perl v5.30.0 2019-07-26 TAP::Harness::Beyond(3)