1TAP::Harness::Beyond(3)User Contributed Perl DocumentatioTnAP::Harness::Beyond(3)
2
6 Test::Harness is responsible for running test scripts, analysing their
7 output and reporting success or failure. When I type make test (or
8 ./Build test) for a module, Test::Harness is usually used to run the
9 tests (not all modules use Test::Harness but the majority do).
10 To start exploring some of the features of Test::Harness I need to
12 switch from make test to the prove command (which ships with
13 Test::Harness). For the following examples I'll also need a recent
14 version of Test::Harness installed; 3.14 is current as I write.
15 For the examples I'm going to assume that we're working with a 'normal'
17 Perl module distribution. Specifically I'll assume that typing make or
18 ./Build causes the built, ready-to-install module code to be available
19 below ./blib/lib and ./blib/arch and that there's a directory called
20 't' that contains our tests. Test::Harness isn't hardwired to that
21 configuration but it saves me from explaining which files live where
22 for each example.
23 Back to prove; like make test it runs a test suite - but it provides
25 far more control over which tests are executed, in what order and how
26 their results are reported. Typically make test runs all the test
27 scripts below the 't' directory. To do the same thing with prove I
28 type:
29 prove -rb t
31 The switches here are -r to recurse into any directories below 't' and
33 -b which adds ./blib/lib and ./blib/arch to Perl's include path so that
34 the tests can find the code they will be testing. If I'm testing a
35 module of which an earlier version is already installed I need to be
36 careful about the include path to make sure I'm not running my tests
37 against the installed version rather than the new one that I'm working
38 on.
39 Unlike make test, typing prove doesn't automatically rebuild my module.
41 If I forget to make before prove I will be testing against older
42 versions of those files - which inevitably leads to confusion. I
43 either get into the habit of typing
44 make && prove -rb t
46 or - if I have no XS code that needs to be built I use the modules
48 below lib instead
49 prove -Ilib -r t
51 So far I've shown you nothing that make test doesn't do. Let's fix
53 that.
54 Saved State
56 If I have failing tests in a test suite that consists of more than a
57 handful of scripts and takes more than a few seconds to run it rapidly
58 becomes tedious to run the whole test suite repeatedly as I track down
59 the problems.
60 I can tell prove just to run the tests that are failing like this:
62 prove -b t/this_fails.t t/so_does_this.t
64 That speeds things up but I have to make a note of which tests are
66 failing and make sure that I run those tests. Instead I can use prove's
67 --state switch and have it keep track of failing tests for me. First I
68 do a complete run of the test suite and tell prove to save the results:
69 prove -rb --state=save t
71 That stores a machine readable summary of the test run in a file called
73 '.prove' in the current directory. If I have failures I can then run
74 just the failing scripts like this:
75 prove -b --state=failed
77 I can also tell prove to save the results again so that it updates its
79 idea of which tests failed:
80 prove -b --state=failed,save
82 As soon as one of my failing tests passes it will be removed from the
84 list of failed tests. Eventually I fix them all and prove can find no
85 failing tests to run:
86 Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
88 Result: NOTESTS
89 As I work on a particular part of my module it's most likely that the
91 tests that cover that code will fail. I'd like to run the whole test
92 suite but have it prioritize these 'hot' tests. I can tell prove to do
93 this:
94 prove -rb --state=hot,save t
96 All the tests will run but those that failed most recently will be run
98 first. If no tests have failed since I started saving state all tests
99 will run in their normal order. This combines full test coverage with
100 early notification of failures.
101 The --state switch supports a number of options; for example to run
103 failed tests first followed by all remaining tests ordered by the
104 timestamps of the test scripts - and save the results - I can use
105 prove -rb --state=failed,new,save t
107 See the prove documentation (type prove --man) for the full list of
109 state options.
110 When I tell prove to save state it writes a file called '.prove'
112 ('_prove' on Windows) in the current directory. It's a YAML document so
113 it's quite easy to write tools of your own that work on the saved test
114 state - but the format isn't officially documented so it might change
115 without (much) warning in the future.
116 Parallel Testing
118 If my tests take too long to run I may be able to speed them up by
119 running multiple test scripts in parallel. This is particularly
120 effective if the tests are I/O bound or if I have multiple CPU cores. I
121 tell prove to run my tests in parallel like this:
122 prove -rb -j 9 t
124 The -j switch enables parallel testing; the number that follows it is
126 the maximum number of tests to run in parallel. Sometimes tests that
127 pass when run sequentially will fail when run in parallel. For example
128 if two different test scripts use the same temporary file or attempt to
129 listen on the same socket I'll have problems running them in parallel.
130 If I see unexpected failures I need to check my tests to work out which
131 of them are trampling on the same resource and rename temporary files
132 or add locks as appropriate.
133 To get the most performance benefit I want to have the test scripts
135 that take the longest to run start first - otherwise I'll be waiting
136 for the one test that takes nearly a minute to complete after all the
137 others are done. I can use the --state switch to run the tests in
138 slowest to fastest order:
139 prove -rb -j 9 --state=slow,save t
141 Non-Perl Tests
143 The Test Anything Protocol (http://testanything.org/) isn't just for
144 Perl. Just about any language can be used to write tests that output
145 TAP. There are TAP based testing libraries for C, C++, PHP, Python and
146 many others. If I can't find a TAP library for my language of choice
147 it's easy to generate valid TAP. It looks like this:
148 1..3
150 ok 1 - init OK
151 ok 2 - opened file
152 not ok 3 - appended to file
153 The first line is the plan - it specifies the number of tests I'm going
155 to run so that it's easy to check that the test script didn't exit
156 before running all the expected tests. The following lines are the test
157 results - 'ok' for pass, 'not ok' for fail. Each test has a number and,
158 optionally, a description. And that's it. Any language that can produce
159 output like that on STDOUT can be used to write tests.
160 Recently I've been rekindling a two-decades-old interest in Forth.
162 Evidently I have a masochistic streak that even Perl can't satisfy. I
163 want to write tests in Forth and run them using prove (you can find my
164 gforth TAP experiments at https://svn.hexten.net/andy/Forth/Testing/).
165 I can use the --exec switch to tell prove to run the tests using gforth
166 like this:
167 prove -r --exec gforth t
169 Alternately, if the language used to write my tests allows a shebang
171 line I can use that to specify the interpreter. Here's a test written
172 in PHP:
173 #!/usr/bin/php
175 <?php
176 print "1..2\n";
177 print "ok 1\n";
178 print "not ok 2\n";
179 ?>
180 If I save that as t/phptest.t the shebang line will ensure that it runs
182 correctly along with all my other tests.
183 Mixing it up
185 Subtle interdependencies between test programs can mask problems - for
186 example an earlier test may neglect to remove a temporary file that
187 affects the behaviour of a later test. To find this kind of problem I
188 use the --shuffle and --reverse options to run my tests in random or
189 reversed order.
190 Rolling My Own
192 If I need a feature that prove doesn't provide I can easily write my
193 own.
194 Typically you'll want to change how TAP gets input into and output from
196 the parser. App::Prove supports arbitrary plugins, and TAP::Harness
197 supports custom formatters and source handlers that you can load using
198 either prove or Module::Build; there are many examples to base mine on.
199 For more details see App::Prove, TAP::Parser::SourceHandler, and
200 TAP::Formatter::Base.
201 If writing a plugin is not enough, you can write your own test harness;
203 one of the motives for the 3.00 rewrite of Test::Harness was to make it
204 easier to subclass and extend.
205 The Test::Harness module is a compatibility wrapper around
207 TAP::Harness. For new applications I should use TAP::Harness directly.
208 As we'll see, prove uses TAP::Harness.
209 When I run prove it processes its arguments, figures out which test
211 scripts to run and then passes control to TAP::Harness to run the
212 tests, parse, analyse and present the results. By subclassing
213 TAP::Harness I can customise many aspects of the test run.
214 I want to log my test results in a database so I can track them over
216 time. To do this I override the summary method in TAP::Harness. I
217 start with a simple prototype that dumps the results as a YAML
218 document:
219 package My::TAP::Harness;
221 use base qw( TAP::Harness ); use YAML;
223 sub summary {
225 my ( $self, $aggregate ) = @_;
226 print Dump( $aggregate );
227 $self->SUPER::summary( $aggregate );
228 }
229 1;
231 I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness is
233 on Perl's @INC include path I can
234 prove --harness=My::TAP::Harness -rb t
236 If I don't have My::TAP::Harness installed on @INC I need to provide
238 the correct path to perl when I run prove:
239 perl -Ilib `which prove` --harness=My::TAP::Harness -rb t
241 I can incorporate these options into my own version of prove. It's
243 pretty simple. Most of the work of prove is handled by App::Prove. The
244 important code in prove is just:
245 use App::Prove;
247 my $app = App::Prove->new;
249 $app->process_args(@ARGV);
250 exit( $app->run ? 0 : 1 );
251 If I write a subclass of App::Prove I can customise any aspect of the
253 test runner while inheriting all of prove's behaviour. Here's myprove:
254 #!/usr/bin/env perl use lib qw( lib ); # Add ./lib to @INC
256 use App::Prove;
257 my $app = App::Prove->new;
259 # Use custom TAP::Harness subclass
261 $app->harness( 'My::TAP::Harness' );
262 $app->process_args( @ARGV ); exit( $app->run ? 0 : 1 );
264 Now I can run my tests like this
266 ./myprove -rb t
268 Deeper Customisation
270 Now that I know how to subclass and replace TAP::Harness I can replace
271 any other part of the harness. To do that I need to know which classes
272 are responsible for which functionality. Here's a brief guided tour;
273 the default class for each component is shown in parentheses. Normally
274 any replacements I write will be subclasses of these default classes.
275 When I run my tests TAP::Harness creates a scheduler
277 (TAP::Parser::Scheduler) to work out the running order for the tests,
278 an aggregator (TAP::Parser::Aggregator) to collect and analyse the test
279 results and a formatter (TAP::Formatter::Console) to display those
280 results.
281 If I'm running my tests in parallel there may also be a multiplexer
283 (TAP::Parser::Multiplexer) - the component that allows multiple tests
284 to run simultaneously.
285 Once it has created those helpers TAP::Harness starts running the
287 tests. For each test it creates a new parser (TAP::Parser) which is
288 responsible for running the test script and parsing its output.
289 To replace any of these components I call one of these harness methods
291 with the name of the replacement class:
292 aggregator_class
294 formatter_class
295 multiplexer_class
296 parser_class
297 scheduler_class
298 For example, to replace the aggregator I would
300 $harness->aggregator_class( 'My::Aggregator' );
302 Alternately I can supply the names of my substitute classes to the
304 TAP::Harness constructor:
305 my $harness = TAP::Harness->new(
307 { aggregator_class => 'My::Aggregator' }
308 );
309 If I need to reach even deeper into the internals of the harness I can
311 replace the classes that TAP::Parser uses to execute test scripts and
312 tokenise their output. Before running a test script TAP::Parser creates
313 a grammar (TAP::Parser::Grammar) to decode the raw TAP into tokens, a
314 result factory (TAP::Parser::ResultFactory) to turn the decoded TAP
315 results into objects and, depending on whether it's running a test
316 script or reading TAP from a file, scalar or array a source or an
317 iterator (TAP::Parser::IteratorFactory).
318 Each of these objects may be replaced by calling one of these parser
320 methods:
321 source_class
323 perl_source_class
324 grammar_class
325 iterator_factory_class
326 result_factory_class
327 Callbacks
329 As an alternative to subclassing the components I need to change I can
330 attach callbacks to the default classes. TAP::Harness exposes these
331 callbacks:
332 parser_args Tweak the parameters used to create the parser
334 made_parser Just made a new parser
335 before_runtests About to run tests
336 after_runtests Have run all tests
337 after_test Have run an individual test script
338 TAP::Parser also supports callbacks; bailout, comment, plan, test,
340 unknown, version and yaml are called for the corresponding TAP result
341 types, ALL is called for all results, ELSE is called for all results
342 for which a named callback is not installed and EOF is called once at
343 the end of each TAP stream.
344 To install a callback I pass the name of the callback and a subroutine
346 reference to TAP::Harness or TAP::Parser's callback method:
347 $harness->callback( after_test => sub {
349 my ( $script, $desc, $parser ) = @_;
350 } );
351 I can also pass callbacks to the constructor:
353 my $harness = TAP::Harness->new({
355 callbacks => {
356 after_test => sub {
357 my ( $script, $desc, $parser ) = @_;
358 # Do something interesting here
359 }
360 }
361 });
362 When it comes to altering the behaviour of the test harness there's
364 more than one way to do it. Which way is best depends on my
365 requirements. In general if I only want to observe test execution
366 without changing the harness' behaviour (for example to log test
367 results to a database) I choose callbacks. If I want to make the
368 harness behave differently subclassing gives me more control.
369 Parsing TAP
371 Perhaps I don't need a complete test harness. If I already have a TAP
372 test log that I need to parse all I need is TAP::Parser and the various
373 classes it depends upon. Here's the code I need to run a test and parse
374 its TAP output
375 use TAP::Parser;
377 my $parser = TAP::Parser->new( { source => 't/simple.t' } );
379 while ( my $result = $parser->next ) {
380 print $result->as_string, "\n";
381 }
382 Alternately I can pass an open filehandle as source and have the parser
384 read from that rather than attempting to run a test script:
385 open my $tap, '<', 'tests.tap'
387 or die "Can't read TAP transcript ($!)\n";
388 my $parser = TAP::Parser->new( { source => $tap } );
389 while ( my $result = $parser->next ) {
390 print $result->as_string, "\n";
391 }
392 This approach is useful if I need to convert my TAP based test results
394 into some other representation. See TAP::Convert::TET
395 (http://search.cpan.org/dist/TAP-Convert-TET/) for an example of this
396 approach.
397 Getting Support
399 The Test::Harness developers hang out on the tapx-dev mailing list[1].
400 For discussion of general, language independent TAP issues there's the
401 tap-l[2] list. Finally there's a wiki dedicated to the Test Anything
402 Protocol[3]. Contributions to the wiki, patches and suggestions are all
403 welcome.
404 [1] <http://www.hexten.net/mailman/listinfo/tapx-dev> [2]
406 <http://testanything.org/mailman/listinfo/tap-l> [3]
407 <http://testanything.org/>
408perl v5.16.3 2013-05-02 TAP::Harness::Beyond(3)