1TAP::Harness::Beyond(3)User Contributed Perl DocumentatioTnAP::Harness::Beyond(3)
2
3
4

NAME

6       Test::Harness::Beyond - Beyond make test
7

Beyond make test

9       Test::Harness is responsible for running test scripts, analysing their
10       output and reporting success or failure. When I type make test (or
11       ./Build test) for a module, Test::Harness is usually used to run the
12       tests (not all modules use Test::Harness but the majority do).
13
14       To start exploring some of the features of Test::Harness I need to
15       switch from make test to the prove command (which ships with
16       Test::Harness). For the following examples I'll also need a recent
17       version of Test::Harness installed; 3.14 is current as I write.
18
19       For the examples I'm going to assume that we're working with a 'normal'
20       Perl module distribution. Specifically I'll assume that typing make or
21       ./Build causes the built, ready-to-install module code to be available
22       below ./blib/lib and ./blib/arch and that there's a directory called
23       't' that contains our tests. Test::Harness isn't hardwired to that
24       configuration but it  saves me from explaining which files live where
25       for each example.
26
27       Back to prove; like make test it runs a test suite - but it provides
28       far more control over which tests are executed, in what order and how
29       their results are reported. Typically make test runs all the test
30       scripts below the 't' directory. To do the same thing with prove I
31       type:
32
33         prove -rb t
34
35       The switches here are -r to recurse into any directories below 't' and
36       -b which adds ./blib/lib and ./blib/arch to Perl's include path so that
37       the tests can find the code they will be testing. If I'm testing a
38       module of which an earlier version is already installed I need to be
39       careful about the include path to make sure I'm not running my tests
40       against the installed version rather than the new one that I'm working
41       on.
42
43       Unlike make test, typing prove doesn't automatically rebuild my module.
44       If I forget to make before prove I will be testing against older
45       versions of those files - which inevitably leads to confusion.  I
46       either get into the habit of typing
47
48         make && prove -rb t
49
50       or - if I have no XS code that needs to be built I use the modules
51       below lib instead
52
53         prove -Ilib -r t
54
55       So far I've shown you nothing that make test doesn't do. Let's fix
56       that.
57
58   Saved State
59       If I have failing tests in a test suite that consists of more than a
60       handful of scripts and takes more than a few seconds to run it rapidly
61       becomes tedious to run the whole test suite repeatedly as I track down
62       the problems.
63
64       I can tell prove just to run the tests that are failing like this:
65
66         prove -b t/this_fails.t t/so_does_this.t
67
68       That speeds things up but I have to make a note of which tests are
69       failing and make sure that I run those tests. Instead I can use prove's
70       --state switch and have it keep track of failing tests for me. First I
71       do a complete run of the test suite and tell prove to save the results:
72
73         prove -rb --state=save t
74
75       That stores a machine readable summary of the test run in a file called
76       '.prove' in the current directory. If I have failures I can then run
77       just the failing scripts like this:
78
79         prove -b --state=failed
80
81       I can also tell prove to save the results again so that it updates its
82       idea of which tests failed:
83
84         prove -b --state=failed,save
85
86       As soon as one of my failing tests passes it will be removed from the
87       list of failed tests. Eventually I fix them all and prove can find no
88       failing tests to run:
89
90         Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
91         Result: NOTESTS
92
93       As I work on a particular part of my module it's most likely that the
94       tests that cover that code will fail. I'd like to run the whole test
95       suite but have it prioritize these 'hot' tests. I can tell prove to do
96       this:
97
98         prove -rb --state=hot,save t
99
100       All the tests will run but those that failed most recently will be run
101       first. If no tests have failed since I started saving state all tests
102       will run in their normal order. This combines full test coverage with
103       early notification of failures.
104
105       The --state switch supports a number of options; for example to run
106       failed tests first followed by all remaining tests ordered by the
107       timestamps of the test scripts - and save the results - I can use
108
109         prove -rb --state=failed,new,save t
110
111       See the prove documentation (type prove --man) for the full list of
112       state options.
113
114       When I tell prove to save state it writes a file called '.prove'
115       ('_prove' on Windows) in the current directory. It's a YAML document so
116       it's quite easy to write tools of your own that work on the saved test
117       state - but the format isn't officially documented so it might change
118       without (much) warning in the future.
119
120   Parallel Testing
121       If my tests take too long to run I may be able to speed them up by
122       running multiple test scripts in parallel. This is particularly
123       effective if the tests are I/O bound or if I have multiple CPU cores. I
124       tell prove to run my tests in parallel like this:
125
126         prove -rb -j 9 t
127
128       The -j switch enables parallel testing; the number that follows it is
129       the maximum number of tests to run in parallel. Sometimes tests that
130       pass when run sequentially will fail when run in parallel. For example
131       if two different test scripts use the same temporary file or attempt to
132       listen on the same socket I'll have problems running them in parallel.
133       If I see unexpected failures I need to check my tests to work out which
134       of them are trampling on the same resource and rename temporary files
135       or add locks as appropriate.
136
137       To get the most performance benefit I want to have the test scripts
138       that take the longest to run start first - otherwise I'll be waiting
139       for the one test that takes nearly a minute to complete after all the
140       others are done. I can use the --state switch to run the tests in
141       slowest to fastest order:
142
143         prove -rb -j 9 --state=slow,save t
144
145   Non-Perl Tests
146       The Test Anything Protocol (http://testanything.org/) isn't just for
147       Perl. Just about any language can be used to write tests that output
148       TAP. There are TAP based testing libraries for C, C++, PHP, Python and
149       many others. If I can't find a TAP library for my language of choice
150       it's easy to generate valid TAP. It looks like this:
151
152         1..3
153         ok 1 - init OK
154         ok 2 - opened file
155         not ok 3 - appended to file
156
157       The first line is the plan - it specifies the number of tests I'm going
158       to run so that it's easy to check that the test script didn't exit
159       before running all the expected tests. The following lines are the test
160       results - 'ok' for pass, 'not ok' for fail. Each test has a number and,
161       optionally, a description. And that's it. Any language that can produce
162       output like that on STDOUT can be used to write tests.
163
164       Recently I've been rekindling a two-decades-old interest in Forth.
165       Evidently I have a masochistic streak that even Perl can't satisfy.  I
166       want to write tests in Forth and run them using prove (you can find my
167       gforth TAP experiments at https://svn.hexten.net/andy/Forth/Testing/).
168       I can use the --exec switch to tell prove to run the tests using gforth
169       like this:
170
171         prove -r --exec gforth t
172
173       Alternately, if the language used to write my tests allows a shebang
174       line I can use that to specify the interpreter. Here's a test written
175       in PHP:
176
177         #!/usr/bin/php
178         <?php
179           print "1..2\n";
180           print "ok 1\n";
181           print "not ok 2\n";
182         ?>
183
184       If I save that as t/phptest.t the shebang line will ensure that it runs
185       correctly along with all my other tests.
186
187   Mixing it up
188       Subtle interdependencies between test programs can mask problems - for
189       example an earlier test may neglect to remove a temporary file that
190       affects the behaviour of a later test. To find this kind of problem I
191       use the --shuffle and --reverse options to run my tests in random or
192       reversed order.
193
194   Rolling My Own
195       If I need a feature that prove doesn't provide I can easily write my
196       own.
197
198       Typically you'll want to change how TAP gets input into and output from
199       the parser.  App::Prove supports arbitrary plugins, and TAP::Harness
200       supports custom formatters and source handlers that you can load using
201       either prove or Module::Build; there are many examples to base mine on.
202       For more details see App::Prove, TAP::Parser::SourceHandler, and
203       TAP::Formatter::Base.
204
205       If writing a plugin is not enough, you can write your own test harness;
206       one of the motives for the 3.00 rewrite of Test::Harness was to make it
207       easier to subclass and extend.
208
209       The Test::Harness module is a compatibility wrapper around
210       TAP::Harness.  For new applications I should use TAP::Harness directly.
211       As we'll see, prove uses TAP::Harness.
212
213       When I run prove it processes its arguments, figures out which test
214       scripts to run and then passes control to TAP::Harness to run the
215       tests, parse, analyse and present the results. By subclassing
216       TAP::Harness I can customise many aspects of the test run.
217
218       I want to log my test results in a database so I can track them over
219       time. To do this I override the summary method in TAP::Harness.  I
220       start with a simple prototype that dumps the results as a YAML
221       document:
222
223         package My::TAP::Harness;
224
225         use base 'TAP::Harness';
226         use YAML;
227
228         sub summary {
229           my ( $self, $aggregate ) = @_;
230           print Dump( $aggregate );
231           $self->SUPER::summary( $aggregate );
232         }
233
234         1;
235
236       I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness is
237       on Perl's @INC include path I can
238
239         prove --harness=My::TAP::Harness -rb t
240
241       If I don't have My::TAP::Harness installed on @INC I need to provide
242       the correct path to perl when I run prove:
243
244         perl -Ilib `which prove` --harness=My::TAP::Harness -rb t
245
246       I can incorporate these options into my own version of prove. It's
247       pretty simple. Most of the work of prove is handled by App::Prove.  The
248       important code in prove is just:
249
250         use App::Prove;
251
252         my $app = App::Prove->new;
253         $app->process_args(@ARGV);
254         exit( $app->run ? 0 : 1 );
255
256       If I write a subclass of App::Prove I can customise any aspect of the
257       test runner while inheriting all of prove's behaviour. Here's myprove:
258
259         #!/usr/bin/env perl use lib qw( lib );      # Add ./lib to @INC
260         use App::Prove;
261
262         my $app = App::Prove->new;
263
264         # Use custom TAP::Harness subclass
265         $app->harness( 'My::TAP::Harness' );
266
267         $app->process_args( @ARGV ); exit( $app->run ? 0 : 1 );
268
269       Now I can run my tests like this
270
271         ./myprove -rb t
272
273   Deeper Customisation
274       Now that I know how to subclass and replace TAP::Harness I can replace
275       any other part of the harness. To do that I need to know which classes
276       are responsible for which functionality. Here's a brief guided tour;
277       the default class for each component is shown in parentheses. Normally
278       any replacements I write will be subclasses of these default classes.
279
280       When I run my tests TAP::Harness creates a scheduler
281       (TAP::Parser::Scheduler) to work out the running order for the tests,
282       an aggregator (TAP::Parser::Aggregator) to collect and analyse the test
283       results and a formatter (TAP::Formatter::Console) to display those
284       results.
285
286       If I'm running my tests in parallel there may also be a multiplexer
287       (TAP::Parser::Multiplexer) - the component that allows multiple tests
288       to run simultaneously.
289
290       Once it has created those helpers TAP::Harness starts running the
291       tests. For each test it creates a new parser (TAP::Parser) which is
292       responsible for running the test script and parsing its output.
293
294       To replace any of these components I call one of these harness methods
295       with the name of the replacement class:
296
297         aggregator_class
298         formatter_class
299         multiplexer_class
300         parser_class
301         scheduler_class
302
303       For example, to replace the aggregator I would
304
305         $harness->aggregator_class( 'My::Aggregator' );
306
307       Alternately I can supply the names of my substitute classes to the
308       TAP::Harness constructor:
309
310         my $harness = TAP::Harness->new(
311           { aggregator_class => 'My::Aggregator' }
312         );
313
314       If I need to reach even deeper into the internals of the harness I can
315       replace the classes that TAP::Parser uses to execute test scripts and
316       tokenise their output. Before running a test script TAP::Parser creates
317       a grammar (TAP::Parser::Grammar) to decode the raw TAP into tokens, a
318       result factory (TAP::Parser::ResultFactory) to turn the decoded TAP
319       results into objects and, depending on whether it's running a test
320       script or reading TAP from a file, scalar or array a source or an
321       iterator (TAP::Parser::IteratorFactory).
322
323       Each of these objects may be replaced by calling one of these parser
324       methods:
325
326         source_class
327         perl_source_class
328         grammar_class
329         iterator_factory_class
330         result_factory_class
331
332   Callbacks
333       As an alternative to subclassing the components I need to change I can
334       attach callbacks to the default classes. TAP::Harness exposes these
335       callbacks:
336
337         parser_args      Tweak the parameters used to create the parser
338         made_parser      Just made a new parser
339         before_runtests  About to run tests
340         after_runtests   Have run all tests
341         after_test       Have run an individual test script
342
343       TAP::Parser also supports callbacks; bailout, comment, plan, test,
344       unknown, version and yaml are called for the corresponding TAP result
345       types, ALL is called for all results, ELSE is called for all results
346       for which a named callback is not installed and EOF is called once at
347       the end of each TAP stream.
348
349       To install a callback I pass the name of the callback and a subroutine
350       reference to TAP::Harness or TAP::Parser's callback method:
351
352         $harness->callback( after_test => sub {
353           my ( $script, $desc, $parser ) = @_;
354         } );
355
356       I can also pass callbacks to the constructor:
357
358         my $harness = TAP::Harness->new({
359           callbacks => {
360                   after_test => sub {
361               my ( $script, $desc, $parser ) = @_;
362               # Do something interesting here
363                   }
364           }
365         });
366
367       When it comes to altering the behaviour of the test harness there's
368       more than one way to do it. Which way is best depends on my
369       requirements. In general if I only want to observe test execution
370       without changing the harness' behaviour (for example to log test
371       results to a database) I choose callbacks. If I want to make the
372       harness behave differently subclassing gives me more control.
373
374   Parsing TAP
375       Perhaps I don't need a complete test harness. If I already have a TAP
376       test log that I need to parse all I need is TAP::Parser and the various
377       classes it depends upon. Here's the code I need to run a test and parse
378       its TAP output
379
380         use TAP::Parser;
381
382         my $parser = TAP::Parser->new( { source => 't/simple.t' } );
383         while ( my $result = $parser->next ) {
384           print $result->as_string, "\n";
385         }
386
387       Alternately I can pass an open filehandle as source and have the parser
388       read from that rather than attempting to run a test script:
389
390         open my $tap, '<', 'tests.tap'
391           or die "Can't read TAP transcript ($!)\n";
392         my $parser = TAP::Parser->new( { source => $tap } );
393         while ( my $result = $parser->next ) {
394           print $result->as_string, "\n";
395         }
396
397       This approach is useful if I need to convert my TAP based test results
398       into some other representation. See TAP::Convert::TET
399       (http://search.cpan.org/dist/TAP-Convert-TET/) for an example of this
400       approach.
401
402   Getting Support
403       The Test::Harness developers hang out on the tapx-dev mailing list[1].
404       For discussion of general, language independent TAP issues there's the
405       tap-l[2] list. Finally there's a wiki dedicated to the Test Anything
406       Protocol[3]. Contributions to the wiki, patches and suggestions are all
407       welcome.
408
409       [1] <http://www.hexten.net/mailman/listinfo/tapx-dev> [2]
410       <http://testanything.org/mailman/listinfo/tap-l> [3]
411       <http://testanything.org/>
412
413
414
415perl v5.32.1                      2021-01-27           TAP::Harness::Beyond(3)
Impressum