1TAP::Harness(3)       User Contributed Perl Documentation      TAP::Harness(3)
2
3
4

NAME

6       TAP::Harness - Run test scripts with statistics
7

VERSION

9       Version 3.42
10

DESCRIPTION

12       This is a simple test harness which allows tests to be run and results
13       automatically aggregated and output to STDOUT.
14

SYNOPSIS

16        use TAP::Harness;
17        my $harness = TAP::Harness->new( \%args );
18        $harness->runtests(@tests);
19

METHODS

21   Class Methods
22       "new"
23
24        my %args = (
25           verbosity => 1,
26           lib     => [ 'lib', 'blib/lib', 'blib/arch' ],
27        )
28        my $harness = TAP::Harness->new( \%args );
29
30       The constructor returns a new "TAP::Harness" object. It accepts an
31       optional hashref whose allowed keys are:
32
33       ·   "verbosity"
34
35           Set the verbosity level:
36
37                1   verbose        Print individual test results to STDOUT.
38                0   normal
39               -1   quiet          Suppress some test output (mostly failures
40                                   while tests are running).
41               -2   really quiet   Suppress everything but the tests summary.
42               -3   silent         Suppress everything.
43
44       ·   "timer"
45
46           Append run time for each test to output. Uses Time::HiRes if
47           available.
48
49       ·   "failures"
50
51           Show test failures (this is a no-op if "verbose" is selected).
52
53       ·   "comments"
54
55           Show test comments (this is a no-op if "verbose" is selected).
56
57       ·   "show_count"
58
59           Update the running test count during testing.
60
61       ·   "normalize"
62
63           Set to a true value to normalize the TAP that is emitted in verbose
64           modes.
65
66       ·   "lib"
67
68           Accepts a scalar value or array ref of scalar values indicating
69           which paths to allowed libraries should be included if Perl tests
70           are executed. Naturally, this only makes sense in the context of
71           tests written in Perl.
72
73       ·   "switches"
74
75           Accepts a scalar value or array ref of scalar values indicating
76           which switches should be included if Perl tests are executed.
77           Naturally, this only makes sense in the context of tests written in
78           Perl.
79
80       ·   "test_args"
81
82           A reference to an @INC style array of arguments to be passed to
83           each test program.
84
85             test_args => ['foo', 'bar'],
86
87           if you want to pass different arguments to each test then you
88           should pass a hash of arrays, keyed by the alias for each test:
89
90             test_args => {
91               my_test    => ['foo', 'bar'],
92               other_test => ['baz'],
93             }
94
95       ·   "color"
96
97           Attempt to produce color output.
98
99       ·   "exec"
100
101           Typically, Perl tests are run through this. However, anything which
102           spits out TAP is fine. You can use this argument to specify the
103           name of the program (and optional switches) to run your tests with:
104
105             exec => ['/usr/bin/ruby', '-w']
106
107           You can also pass a subroutine reference in order to determine and
108           return the proper program to run based on a given test script. The
109           subroutine reference should expect the TAP::Harness object itself
110           as the first argument, and the file name as the second argument. It
111           should return an array reference containing the command to be run
112           and including the test file name. It can also simply return
113           "undef", in which case TAP::Harness will fall back on executing the
114           test script in Perl:
115
116               exec => sub {
117                   my ( $harness, $test_file ) = @_;
118
119                   # Let Perl tests run.
120                   return undef if $test_file =~ /[.]t$/;
121                   return [ qw( /usr/bin/ruby -w ), $test_file ]
122                     if $test_file =~ /[.]rb$/;
123                 }
124
125           If the subroutine returns a scalar with a newline or a filehandle,
126           it will be interpreted as raw TAP or as a TAP stream, respectively.
127
128       ·   "merge"
129
130           If "merge" is true the harness will create parsers that merge
131           STDOUT and STDERR together for any processes they start.
132
133       ·   "sources"
134
135           NEW to 3.18.
136
137           If set, "sources" must be a hashref containing the names of the
138           TAP::Parser::SourceHandlers to load and/or configure.  The values
139           are a hash of configuration that will be accessible to the source
140           handlers via "config_for" in TAP::Parser::Source.
141
142           For example:
143
144             sources => {
145               Perl => { exec => '/path/to/custom/perl' },
146               File => { extensions => [ '.tap', '.txt' ] },
147               MyCustom => { some => 'config' },
148             }
149
150           The "sources" parameter affects how "source", "tap" and "exec"
151           parameters are handled.
152
153           For more details, see the "sources" parameter in "new" in
154           TAP::Parser, TAP::Parser::Source, and TAP::Parser::IteratorFactory.
155
156       ·   "aggregator_class"
157
158           The name of the class to use to aggregate test results. The default
159           is TAP::Parser::Aggregator.
160
161       ·   "version"
162
163           NEW to 3.22.
164
165           Assume this TAP version for TAP::Parser instead of default TAP
166           version 12.
167
168       ·   "formatter_class"
169
170           The name of the class to use to format output. The default is
171           TAP::Formatter::Console, or TAP::Formatter::File if the output
172           isn't a TTY.
173
174       ·   "multiplexer_class"
175
176           The name of the class to use to multiplex tests during parallel
177           testing.  The default is TAP::Parser::Multiplexer.
178
179       ·   "parser_class"
180
181           The name of the class to use to parse TAP. The default is
182           TAP::Parser.
183
184       ·   "scheduler_class"
185
186           The name of the class to use to schedule test execution. The
187           default is TAP::Parser::Scheduler.
188
189       ·   "formatter"
190
191           If set "formatter" must be an object that is capable of formatting
192           the TAP output. See TAP::Formatter::Console for an example.
193
194       ·   "errors"
195
196           If parse errors are found in the TAP output, a note of this will be
197           made in the summary report. To see all of the parse errors, set
198           this argument to true:
199
200             errors => 1
201
202       ·   "directives"
203
204           If set to a true value, only test results with directives will be
205           displayed. This overrides other settings such as "verbose" or
206           "failures".
207
208       ·   "ignore_exit"
209
210           If set to a true value instruct "TAP::Parser" to ignore exit and
211           wait status from test scripts.
212
213       ·   "jobs"
214
215           The maximum number of parallel tests to run at any time.  Which
216           tests can be run in parallel is controlled by "rules".  The default
217           is to run only one test at a time.
218
219       ·   "rules"
220
221           A reference to a hash of rules that control which tests may be
222           executed in parallel. If no rules are declared and CPAN::Meta::YAML
223           is available, "TAP::Harness" attempts to load rules from a YAML
224           file specified by the "rulesfile" parameter. If no rules file
225           exists, the default is for all tests to be eligible to be run in
226           parallel.
227
228           Here some simple examples. For the full details of the data
229           structure and the related glob-style pattern matching, see "Rules
230           data structure" in TAP::Parser::Scheduler.
231
232               # Run all tests in sequence, except those starting with "p"
233               $harness->rules({
234                   par => 't/p*.t'
235               });
236
237               # Equivalent YAML file
238               ---
239               par: t/p*.t
240
241               # Run all tests in parallel, except those starting with "p"
242               $harness->rules({
243                   seq => [
244                             { seq => 't/p*.t' },
245                             { par => '**'     },
246                          ],
247               });
248
249               # Equivalent YAML file
250               ---
251               seq:
252                   - seq: t/p*.t
253                   - par: **
254
255               # Run some  startup tests in sequence, then some parallel tests than some
256               # teardown tests in sequence.
257               $harness->rules({
258                   seq => [
259                       { seq => 't/startup/*.t' },
260                       { par => ['t/a/*.t','t/b/*.t','t/c/*.t'], }
261                       { seq => 't/shutdown/*.t' },
262                   ],
263
264               });
265
266               # Equivalent YAML file
267               ---
268               seq:
269                   - seq: t/startup/*.t
270                   - par:
271                       - t/a/*.t
272                       - t/b/*.t
273                       - t/c/*.t
274                   - seq: t/shutdown/*.t
275
276           This is an experimental feature and the interface may change.
277
278       ·   "rulesfiles"
279
280           This specifies where to find a YAML file of test scheduling rules.
281           If not provided, it looks for a default file to use.  It first
282           checks for a file given in the "HARNESS_RULESFILE" environment
283           variable, then it checks for testrules.yml and then
284           t/testrules.yml.
285
286       ·   "stdout"
287
288           A filehandle for catching standard output.
289
290       ·   "trap"
291
292           Attempt to print summary information if run is interrupted by
293           SIGINT (Ctrl-C).
294
295       Any keys for which the value is "undef" will be ignored.
296
297   Instance Methods
298       "runtests"
299
300           $harness->runtests(@tests);
301
302       Accepts an array of @tests to be run. This should generally be the
303       names of test files, but this is not required. Each element in @tests
304       will be passed to "TAP::Parser::new()" as a "source". See TAP::Parser
305       for more information.
306
307       It is possible to provide aliases that will be displayed in place of
308       the test name by supplying the test as a reference to an array
309       containing "[ $test, $alias ]":
310
311           $harness->runtests( [ 't/foo.t', 'Foo Once' ],
312                               [ 't/foo.t', 'Foo Twice' ] );
313
314       Normally it is an error to attempt to run the same test twice. Aliases
315       allow you to overcome this limitation by giving each run of the test a
316       unique name.
317
318       Tests will be run in the order found.
319
320       If the environment variable "PERL_TEST_HARNESS_DUMP_TAP" is defined it
321       should name a directory into which a copy of the raw TAP for each test
322       will be written. TAP is written to files named for each test.
323       Subdirectories will be created as needed.
324
325       Returns a TAP::Parser::Aggregator containing the test results.
326
327       "summary"
328
329         $harness->summary( $aggregator );
330
331       Output the summary for a TAP::Parser::Aggregator.
332
333       "aggregate_tests"
334
335         $harness->aggregate_tests( $aggregate, @tests );
336
337       Run the named tests and display a summary of result. Tests will be run
338       in the order found.
339
340       Test results will be added to the supplied TAP::Parser::Aggregator.
341       "aggregate_tests" may be called multiple times to run several sets of
342       tests. Multiple "Test::Harness" instances may be used to pass results
343       to a single aggregator so that different parts of a complex test suite
344       may be run using different "TAP::Harness" settings. This is useful, for
345       example, in the case where some tests should run in parallel but others
346       are unsuitable for parallel execution.
347
348           my $formatter   = TAP::Formatter::Console->new;
349           my $ser_harness = TAP::Harness->new( { formatter => $formatter } );
350           my $par_harness = TAP::Harness->new(
351               {   formatter => $formatter,
352                   jobs      => 9
353               }
354           );
355           my $aggregator = TAP::Parser::Aggregator->new;
356
357           $aggregator->start();
358           $ser_harness->aggregate_tests( $aggregator, @ser_tests );
359           $par_harness->aggregate_tests( $aggregator, @par_tests );
360           $aggregator->stop();
361           $formatter->summary($aggregator);
362
363       Note that for simpler testing requirements it will often be possible to
364       replace the above code with a single call to "runtests".
365
366       Each element of the @tests array is either:
367
368       ·   the source name of a test to run
369
370       ·   a reference to a [ source name, display name ] array
371
372       In the case of a perl test suite, typically source names are simply the
373       file names of the test scripts to run.
374
375       When you supply a separate display name it becomes possible to run a
376       test more than once; the display name is effectively the alias by which
377       the test is known inside the harness. The harness doesn't care if it
378       runs the same test more than once when each invocation uses a different
379       name.
380
381       "make_scheduler"
382
383       Called by the harness when it needs to create a TAP::Parser::Scheduler.
384       Override in a subclass to provide an alternative scheduler.
385       "make_scheduler" is passed the list of tests that was passed to
386       "aggregate_tests".
387
388       "jobs"
389
390       Gets or sets the number of concurrent test runs the harness is
391       handling.  By default, this value is 1 -- for parallel testing, this
392       should be set higher.
393
394       "make_parser"
395
396       Make a new parser and display formatter session. Typically used and/or
397       overridden in subclasses.
398
399           my ( $parser, $session ) = $harness->make_parser;
400
401       "finish_parser"
402
403       Terminate use of a parser. Typically used and/or overridden in
404       subclasses. The parser isn't destroyed as a result of this.
405

CONFIGURING

407       "TAP::Harness" is designed to be easy to configure.
408
409   Plugins
410       "TAP::Parser" plugins let you change the way TAP is input to and output
411       from the parser.
412
413       TAP::Parser::SourceHandlers handle TAP input.  You can configure them
414       and load custom handlers using the "sources" parameter to "new".
415
416       TAP::Formatters handle TAP output.  You can load custom formatters by
417       using the "formatter_class" parameter to "new".  To configure a
418       formatter, you currently need to instantiate it outside of TAP::Harness
419       and pass it in with the "formatter" parameter to "new".  This may be
420       addressed by adding a formatters parameter to "new" in the future.
421
422   "Module::Build"
423       Module::Build version 0.30 supports "TAP::Harness".
424
425       To load "TAP::Harness" plugins, you'll need to use the
426       "tap_harness_args" parameter to "new", typically from your "Build.PL".
427       For example:
428
429         Module::Build->new(
430             module_name        => 'MyApp',
431             test_file_exts     => [qw(.t .tap .txt)],
432             use_tap_harness    => 1,
433             tap_harness_args   => {
434                 sources => {
435                     MyCustom => {},
436                     File => {
437                         extensions => ['.tap', '.txt'],
438                     },
439                 },
440                 formatter_class => 'TAP::Formatter::HTML',
441             },
442             build_requires     => {
443                 'Module::Build' => '0.30',
444                 'TAP::Harness'  => '3.18',
445             },
446         )->create_build_script;
447
448       See "new"
449
450   "ExtUtils::MakeMaker"
451       ExtUtils::MakeMaker does not support TAP::Harness out-of-the-box.
452
453   "prove"
454       prove supports "TAP::Harness" plugins, and has a plugin system of its
455       own.  See "FORMATTERS" in prove, "SOURCE HANDLERS" in prove and
456       App::Prove for more details.
457

WRITING PLUGINS

459       If you can't configure "TAP::Harness" to do what you want, and you
460       can't find an existing plugin, consider writing one.
461
462       The two primary use cases supported by TAP::Harness for plugins are
463       input and output:
464
465       Customize how TAP gets into the parser
466         To do this, you can either extend an existing
467         TAP::Parser::SourceHandler, or write your own.  It's a pretty simple
468         API, and they can be loaded and configured using the "sources"
469         parameter to "new".
470
471       Customize how TAP results are output from the parser
472         To do this, you can either extend an existing TAP::Formatter, or
473         write your own.  Writing formatters are a bit more involved than
474         writing a SourceHandler, as you'll need to understand the TAP::Parser
475         API.  A good place to start is by understanding how "aggregate_tests"
476         works.
477
478         Custom formatters can be loaded configured using the
479         "formatter_class" parameter to "new".
480

SUBCLASSING

482       If you can't configure "TAP::Harness" to do exactly what you want, and
483       writing a plugin isn't an option, consider extending it.  It is
484       designed to be (mostly) easy to subclass, though the cases when sub-
485       classing is necessary should be few and far between.
486
487   Methods
488       The following methods are ones you may wish to override if you want to
489       subclass "TAP::Harness".
490
491       "new"
492       "runtests"
493       "summary"
494

REPLACING

496       If you like the "prove" utility and TAP::Parser but you want your own
497       harness, all you need to do is write one and provide "new" and
498       "runtests" methods. Then you can use the "prove" utility like so:
499
500        prove --harness My::Test::Harness
501
502       Note that while "prove" accepts a list of tests (or things to be
503       tested), "new" has a fairly rich set of arguments. You'll probably want
504       to read over this code carefully to see how all of them are being used.
505

SEE ALSO

507       Test::Harness
508
509
510
511perl v5.30.1                      2020-01-30                   TAP::Harness(3)
Impressum