TAP::Harness(3pm)

1TAP::Harness(3pm)      Perl Programmers Reference Guide      TAP::Harness(3pm)
2
3
4

NAME

6       TAP::Harness - Run test scripts with statistics
7

VERSION

9       Version 3.17
10

DESCRIPTION

12       This is a simple test harness which allows tests to be run and results
13       automatically aggregated and output to STDOUT.
14

SYNOPSIS

16        use TAP::Harness;
17        my $harness = TAP::Harness->new( \%args );
18        $harness->runtests(@tests);
19

METHODS

21   Class Methods
22       "new"
23
24        my %args = (
25           verbosity => 1,
26           lib     => [ 'lib', 'blib/lib', 'blib/arch' ],
27        )
28        my $harness = TAP::Harness->new( \%args );
29
30       The constructor returns a new "TAP::Harness" object. It accepts an
31       optional hashref whose allowed keys are:
32
33       ·   "verbosity"
34
35           Set the verbosity level:
36
37                1   verbose        Print individual test results to STDOUT.
38                0   normal
39               -1   quiet          Suppress some test output (mostly failures
40                                   while tests are running).
41               -2   really quiet   Suppress everything but the tests summary.
42               -3   silent         Suppress everything.
43
44       ·   "timer"
45
46           Append run time for each test to output. Uses Time::HiRes if
47           available.
48
49       ·   "failures"
50
51           Show test failures (this is a no-op if "verbose" is selected).
52
53       ·   "comments"
54
55           Show test comments (this is a no-op if "verbose" is selected).
56
57       ·   "show_count"
58
59           Update the running test count during testing.
60
61       ·   "normalize"
62
63           Set to a true value to normalize the TAP that is emitted in verbose
64           modes.
65
66       ·   "lib"
67
68           Accepts a scalar value or array ref of scalar values indicating
69           which paths to allowed libraries should be included if Perl tests
70           are executed. Naturally, this only makes sense in the context of
71           tests written in Perl.
72
73       ·   "switches"
74
75           Accepts a scalar value or array ref of scalar values indicating
76           which switches should be included if Perl tests are executed.
77           Naturally, this only makes sense in the context of tests written in
78           Perl.
79
80       ·   "test_args"
81
82           A reference to an @INC style array of arguments to be passed to
83           each test program.
84
85       ·   "color"
86
87           Attempt to produce color output.
88
89       ·   "exec"
90
91           Typically, Perl tests are run through this. However, anything which
92           spits out TAP is fine. You can use this argument to specify the
93           name of the program (and optional switches) to run your tests with:
94
95             exec => ['/usr/bin/ruby', '-w']
96
97           You can also pass a subroutine reference in order to determine and
98           return the proper program to run based on a given test script. The
99           subroutine reference should expect the TAP::Harness object itself
100           as the first argument, and the file name as the second argument. It
101           should return an array reference containing the command to be run
102           and including the test file name. It can also simply return
103           "undef", in which case TAP::Harness will fall back on executing the
104           test script in Perl:
105
106               exec => sub {
107                   my ( $harness, $test_file ) = @_;
108
109                   # Let Perl tests run.
110                   return undef if $test_file =~ /[.]t$/;
111                   return [ qw( /usr/bin/ruby -w ), $test_file ]
112                     if $test_file =~ /[.]rb$/;
113                 }
114
115           If the subroutine returns a scalar with a newline or a filehandle,
116           it will be interpreted as raw TAP or as a TAP stream, respectively.
117
118       ·   "merge"
119
120           If "merge" is true the harness will create parsers that merge
121           STDOUT and STDERR together for any processes they start.
122
123       ·   "aggregator_class"
124
125           The name of the class to use to aggregate test results. The default
126           is TAP::Parser::Aggregator.
127
128       ·   "formatter_class"
129
130           The name of the class to use to format output. The default is
131           TAP::Formatter::Console, or TAP::Formatter::File if the output
132           isn't a TTY.
133
134       ·   "multiplexer_class"
135
136           The name of the class to use to multiplex tests during parallel
137           testing.  The default is TAP::Parser::Multiplexer.
138
139       ·   "parser_class"
140
141           The name of the class to use to parse TAP. The default is
142           TAP::Parser.
143
144       ·   "scheduler_class"
145
146           The name of the class to use to schedule test execution. The
147           default is TAP::Parser::Scheduler.
148
149       ·   "formatter"
150
151           If set "formatter" must be an object that is capable of formatting
152           the TAP output. See TAP::Formatter::Console for an example.
153
154       ·   "errors"
155
156           If parse errors are found in the TAP output, a note of this will be
157           made in the summary report. To see all of the parse errors, set
158           this argument to true:
159
160             errors => 1
161
162       ·   "directives"
163
164           If set to a true value, only test results with directives will be
165           displayed. This overrides other settings such as "verbose" or
166           "failures".
167
168       ·   "ignore_exit"
169
170           If set to a true value instruct "TAP::Parser" to ignore exit and
171           wait status from test scripts.
172
173       ·   "jobs"
174
175           The maximum number of parallel tests to run at any time.  Which
176           tests can be run in parallel is controlled by "rules".  The default
177           is to run only one test at a time.
178
179       ·   "rules"
180
181           A reference to a hash of rules that control which tests may be
182           executed in parallel. This is an experimental feature and the
183           interface may change.
184
185               $harness->rules(
186                   {   par => [
187                           { seq => '../ext/DB_File/t/*' },
188                           { seq => '../ext/IO_Compress_Zlib/t/*' },
189                           { seq => '../lib/CPANPLUS/*' },
190                           { seq => '../lib/ExtUtils/t/*' },
191                           '*'
192                       ]
193                   }
194               );
195
196       ·   "stdout"
197
198           A filehandle for catching standard output.
199
200       Any keys for which the value is "undef" will be ignored.
201
202   Instance Methods
203       "runtests"
204
205           $harness->runtests(@tests);
206
207       Accepts and array of @tests to be run. This should generally be the
208       names of test files, but this is not required. Each element in @tests
209       will be passed to "TAP::Parser::new()" as a "source". See TAP::Parser
210       for more information.
211
212       It is possible to provide aliases that will be displayed in place of
213       the test name by supplying the test as a reference to an array
214       containing "[ $test, $alias ]":
215
216           $harness->runtests( [ 't/foo.t', 'Foo Once' ],
217                               [ 't/foo.t', 'Foo Twice' ] );
218
219       Normally it is an error to attempt to run the same test twice. Aliases
220       allow you to overcome this limitation by giving each run of the test a
221       unique name.
222
223       Tests will be run in the order found.
224
225       If the environment variable "PERL_TEST_HARNESS_DUMP_TAP" is defined it
226       should name a directory into which a copy of the raw TAP for each test
227       will be written. TAP is written to files named for each test.
228       Subdirectories will be created as needed.
229
230       Returns a TAP::Parser::Aggregator containing the test results.
231
232       "summary"
233
234       Output the summary for a TAP::Parser::Aggregator.
235
236       "aggregate_tests"
237
238         $harness->aggregate_tests( $aggregate, @tests );
239
240       Run the named tests and display a summary of result. Tests will be run
241       in the order found.
242
243       Test results will be added to the supplied TAP::Parser::Aggregator.
244       "aggregate_tests" may be called multiple times to run several sets of
245       tests. Multiple "Test::Harness" instances may be used to pass results
246       to a single aggregator so that different parts of a complex test suite
247       may be run using different "TAP::Harness" settings. This is useful, for
248       example, in the case where some tests should run in parallel but others
249       are unsuitable for parallel execution.
250
251           my $formatter   = TAP::Formatter::Console->new;
252           my $ser_harness = TAP::Harness->new( { formatter => $formatter } );
253           my $par_harness = TAP::Harness->new(
254               {   formatter => $formatter,
255                   jobs      => 9
256               }
257           );
258           my $aggregator = TAP::Parser::Aggregator->new;
259
260           $aggregator->start();
261           $ser_harness->aggregate_tests( $aggregator, @ser_tests );
262           $par_harness->aggregate_tests( $aggregator, @par_tests );
263           $aggregator->stop();
264           $formatter->summary($aggregator);
265
266       Note that for simpler testing requirements it will often be possible to
267       replace the above code with a single call to "runtests".
268
269       Each elements of the @tests array is either
270
271       ·   the file name of a test script to run
272
273       ·   a reference to a [ file name, display name ] array
274
275       When you supply a separate display name it becomes possible to run a
276       test more than once; the display name is effectively the alias by which
277       the test is known inside the harness. The harness doesn't care if it
278       runs the same script more than once when each invocation uses a
279       different name.
280
281       "make_scheduler"
282
283       Called by the harness when it needs to create a TAP::Parser::Scheduler.
284       Override in a subclass to provide an alternative scheduler.
285       "make_scheduler" is passed the list of tests that was passed to
286       "aggregate_tests".
287
288       "jobs"
289
290       Gets or sets the number of concurrent test runs the harness is
291       handling.  By default, this value is 1 -- for parallel testing, this
292       should be set higher.
293

SUBCLASSING

295       "TAP::Harness" is designed to be (mostly) easy to subclass. If you
296       don't like how a particular feature functions, just override the
297       desired methods.
298
299   Methods
300       TODO: This is out of date
301
302       The following methods are ones you may wish to override if you want to
303       subclass "TAP::Harness".
304
305       "summary"
306
307         $harness->summary( \%args );
308
309       "summary" prints the summary report after all tests are run. The
310       argument is a hashref with the following keys:
311
312       ·   "start"
313
314           This is created with "Benchmark->new" and it the time the tests
315           started. You can print a useful summary time, if desired, with:
316
317               $self->output(
318                   timestr( timediff( Benchmark->new, $start_time ), 'nop' ) );
319
320       ·   "tests"
321
322           This is an array reference of all test names. To get the
323           TAP::Parser object for individual tests:
324
325            my $aggregate = $args->{aggregate};
326            my $tests     = $args->{tests};
327
328            for my $name ( @$tests ) {
329                my ($parser) = $aggregate->parsers($test);
330                ... do something with $parser
331            }
332
333           This is a bit clunky and will be cleaned up in a later release.
334
335       "make_parser"
336
337       Make a new parser and display formatter session. Typically used and/or
338       overridden in subclasses.
339
340           my ( $parser, $session ) = $harness->make_parser;
341
342       "finish_parser"
343
344       Terminate use of a parser. Typically used and/or overridden in
345       subclasses. The parser isn't destroyed as a result of this.
346

REPLACING

348       If you like the "prove" utility and TAP::Parser but you want your own
349       harness, all you need to do is write one and provide "new" and
350       "runtests" methods. Then you can use the "prove" utility like so:
351
352        prove --harness My::Test::Harness
353
354       Note that while "prove" accepts a list of tests (or things to be
355       tested), "new" has a fairly rich set of arguments. You'll probably want
356       to read over this code carefully to see how all of them are being used.
357