1App::Yath(3) User Contributed Perl Documentation App::Yath(3)
2
6 App::Yath - Yet Another Test Harness (Test2-Harness) Command Line
7 Interface (CLI)
8
10 This is the primary documentation for "yath", App::Yath,
11 Test2::Harness.
12 The canonical source of up-to-date command options are the help output
14 when using "$ yath help" and "$ yath help COMMAND".
15 This document is mainly an overview of "yath" usage and common recipes.
17 App::Yath is an alternative to App::Prove, and Test2::Harness is an
19 alternative to Test::Harness. It is not designed to replace
20 Test::Harness/prove. Test2::Harness is designed to take full advantage
21 of the rich data Test2 can provide. Test2::Harness is also able to use
22 non-core modules and provide more functionality than prove can achieve
23 with its restrictions.
24
26 Test2::Harness/App::Yath is is focused on unix-like platforms. Most
27 development happens on linux, but bsd, macos, etc should work fine as
28 well.
29 Patches are welcome for any/all platforms, but the primary author (Chad
31 'Exodist' Granum) does not directly develop against non-unix platforms.
32 WINDOWS
34 Currently windows is not supported, and it is known that the package
35 will not install on windows. Patches are be welcome, and it would be
36 great if someone wanted to take on the windows-support role, but it is
37 not a primary goal for the project.
38
40 To use Test2::Harness, you use the "yath" command. Yath will find the
41 tests (or use the ones you specify) and run them. As it runs, it will
42 output diagnostic information such as failures. At the end, yath will
43 print a summary of the test run.
44 "yath" can be thought of as a more powerful alternative to "prove"
46 (Test::Harness)
47
49 These are common recipes for using "yath".
50 RUN PROJECT TESTS
52 $ yath
53 Simply running yath with no arguments means "Run all tests for the
55 current project". Yath will look for tests in "./t", "./t2", and
56 "./test.pl" and run any which are found.
57 Normally this implies the "test" command but will instead imply the
59 "run" command if a persistent test runner is detected.
60 PRELOAD MODULES
62 Yath has the ability to preload modules. Yath normally forks to start
63 new tests, so preloading can reduce the time spent loading modules over
64 and over in each test.
65 Note that some tests may depend on certain modules not being loaded. In
67 these cases you can add the "# HARNESS-NO-PRELOAD" directive to the top
68 of the test files that cannot use preload.
69 SIMPLE PRELOAD
71 Any module can be preloaded:
73 $ yath -PMoose
75 You can preload as many modules as you want:
77 $ yath -PList::Util -PScalar::Util
79 COMPLEX PRELOAD
81 If your preload is a subclass of Test2::Harness::Runner::Preload then
83 more complex preload behavior is possible. See those docs for more
84 info.
85 LOGGING
87 RECORDING A LOG
88 You can turn on logging with a flag. The filename of the log will be
90 printed at the end.
91 $ yath -L
93 ...
94 Wrote log file: test-logs/2017-09-12~22:44:34~1505281474~25709.jsonl
95 The event log can be quite large. It can be compressed with bzip2.
97 $ yath -B
99 ...
100 Wrote log file: test-logs/2017-09-12~22:44:34~1505281474~25709.jsonl.bz2
101 gzip compression is also supported.
103 $ yath -G
105 ...
106 Wrote log file: test-logs/2017-09-12~22:44:34~1505281474~25709.jsonl.gz
107 "-B" and "-G" both imply "-L".
109 REPLAYING FROM A LOG
111 You can replay a test run from a log file:
113 $ yath test-logs/2017-09-12~22:44:34~1505281474~25709.jsonl.bz2
115 This will be significantly faster than the initial run as no tests are
117 actually being executed. All events are simply read from the log, and
118 processed by the harness.
119 You can change display options and limit rendering/processing to
121 specific test jobs from the run:
122 $ yath test-logs/2017-09-12~22:44:34~1505281474~25709.jsonl.bz2 -v [TEST UUID(S)]
124 Note: This is done using the "$ yath replay ..." command. The "replay"
126 command is implied if the first argument is a log file.
127 PER-TEST TIMING DATA
129 The "-T" option will cause each test file to report how long it took to
130 run.
131 $ yath -T
133 ( PASSED ) job 1 t/yath_script.t
135 ( TIME ) job 1 Startup: 0.07692s | Events: 0.01170s | Cleanup: 0.00190s | Total: 0.09052s
136 PERSISTENT RUNNER
138 yath supports starting a yath session that waits for tests to run. This
139 is very useful when combined with preload.
140 STARTING
142 This starts the server. Many options available to the 'test' command
144 will work here but not all. See "$ yath help start" for more info.
145 $ yath start
147 RUNNING
149 This will run tests using the persistent runner. By default, it will
151 search for tests just like the 'test' command. Many options available
152 to the "test" command will work for this as well. See "$ yath help run"
153 for more details.
154 $ yath run
156 STOPPING
158 Stopping a persistent runner is easy.
160 $ yath stop
162 INFORMATIONAL
164 The "which" command will tell you which persistent runner will be used.
166 Yath searches for the persistent runner in the current directory, then
167 searches in parent directories until it either hits the root directory,
168 or finds the persistent runner tracking file.
169 $ yath which
171 The "watch" command will tail the runner's log files.
173 $ yath watch
175 PRELOAD + PERSISTENT RUNNER
177 You can use preloads with the "yath start" command. In this case, yath
179 will track all the modules pulled in during preload. If any of them
180 change, the server will reload itself to bring in the changes. Further,
181 modified modules will be blacklisted so that they are not preloaded on
182 subsequent reloads. This behavior is useful if you are actively working
183 on a module that is normally preloaded.
184 MAKING YOUR PROJECT ALWAYS USE YATH
186 $ yath init
187 The above command will create "test.pl". "test.pl" is automatically run
189 by most build utils, in which case only the exit value matters. The
190 generated "test.pl" will run "yath" and execute all tests in the "./t"
191 and/or "./t2" directories. Tests in "./t" will ALSO be run by prove but
192 tests in "./t2" will only be run by yath.
193 PROJECT-SPECIFIC YATH CONFIG
195 You can write a ".yath.rc" file. The file format is very simple. Create
196 a "[COMMAND]" section to start the configuration for a command and then
197 provide any options normally allowed by it. When "yath" is run inside
198 your project, it will use the config specified in the rc file, unless
199 overridden by command line options.
200 Note: You can also add pre-command options by placing them at the top
202 of your config file BEFORE any "[cmd]" markers.
203 Comments start with a semi-colon.
205 Example .yath.rc:
207 -pFoo ; Load the 'foo' plugin before dealing with commands.
209 [test]
211 -B ;Always write a bzip2-compressed log
212 [start]
214 -PMoose ;Always preload Moose with a persistent runner
215 This file is normally committed into the project's repo.
217 SPECIAL PATH PSEUDO-FUNCTIONS
219 Sometimes you want to specify files relative to the .yath.rc so that
221 the config option works from any subdirectory of the project. Other
222 times you may wish to use a shell expansion. Sometimes you want both!
223 rel(path/to/file)
225 -I rel(path/to/extra_lib)
226 -I=rel(path/to/extra_lib)
227 This will take the path to ".yath.rc" and prefix it to the path
229 inside "rel(...)". If for example you have "/project/.yath.rc" then
230 the path would become "/project/path/to/extra_lib".
231 glob(path/*/file)
233 --default-search glob(subprojects/*/t)
234 --default-search=glob(subprojects/*/t)
235 This will add a "--default-search $_" for every item found in the
237 glob. This uses the perl builtin function "glob()" under the hood.
238 relglob(path/*/file)
240 --default-search relglob(subprojects/*/t)
241 --default-search=relglob(subprojects/*/t)
242 Same as "glob()" except paths are relative to the ".yath.rc" file.
244 PROJECT-SPECIFIC YATH CONFIG USER OVERRIDES
246 You can add a ".yath.user.rc" file. Format is the same as the regular
247 ".yath.rc" file. This file will be read in addition to the regular
248 config file. Directives in this file will come AFTER the directives in
249 the primary config so it may be used to override config.
250 This file should not normally be committed to the project repo.
252 HARNESS DIRECTIVES INSIDE TESTS
254 "yath" will recognise a number of directive comments placed near the
255 top of test files. These directives should be placed after the "#!"
256 line but before any real code.
257 Real code is defined as any line that does not start with use, require,
259 BEGIN, package, or #
260 good example 1
262 #!/usr/bin/perl
263 # HARNESS-NO-FORK
264 ...
266 good example 2
268 #!/usr/bin/perl
269 use strict;
270 use warnings;
271 # HARNESS-NO-FORK
273 ...
275 bad example 1
277 #!/usr/bin/perl
278 # blah
280 # HARNESS-NO-FORK
282 ...
284 bad example 2
286 #!/usr/bin/perl
287 print "hi\n";
289 # HARNESS-NO-FORK
291 ...
293 HARNESS-NO-PRELOAD
295 #!/usr/bin/perl
297 # HARNESS-NO-PRELOAD
298 Use this if your test will fail when modules are preloaded. This will
300 tell yath to start a new perl process to run the script instead of
301 forking with preloaded modules.
302 Currently this implies HARNESS-NO-FORK, but that may not always be the
304 case.
305 HARNESS-NO-FORK
307 #!/usr/bin/perl
309 # HARNESS-NO-FORK
310 Use this if your test file cannot run in a forked process, but instead
312 must be run directly with a new perl process.
313 This implies HARNESS-NO-PRELOAD.
315 HARNESS-NO-STREAM
317 "yath" usually uses the Test2::Formatter::Stream formatter instead of
319 TAP. Some tests depend on using a TAP formatter. This option will make
320 "yath" use Test2::Formatter::TAP or Test::Builder::Formatter.
321 HARNESS-NO-IO-EVENTS
323 "yath" can be configured to use the Test2::Plugin::IOEvents plugin.
325 This plugin replaces STDERR and STDOUT in your test with tied handles
326 that fire off proper Test2::Event's when they are printed to. Most of
327 the time this is not an issue, but any fancy tests or modules which do
328 anything with STDERR or STDOUT other than print may have really messy
329 errors.
330 Note: This plugin is disabled by default, so you only need this
332 directive if you enable it globally but need to turn it back off for
333 select tests.
334 HARNESS-NO-TIMEOUT
336 "yath" will usually kill a test if no events occur within a timeout
338 (default 60 seconds). You can add this directive to tests that are
339 expected to trip the timeout, but should be allowed to continue.
340 NOTE: you usually are doing the wrong thing if you need to set this.
342 See: "HARNESS-TIMEOUT-EVENT".
343 HARNESS-TIMEOUT-EVENT 60
345 "yath" can be told to alter the default event timeout from 60 seconds
347 to another value. This is the recommended alternative to HARNESS-NO-
348 TIMEOUT
349 HARNESS-TIMEOUT-POSTEXIT 15
351 "yath" can be told to alter the default POSTEXIT timeout from 15
353 seconds to another value.
354 Sometimes a test will fork producing output in the child while the
356 parent is allowed to exit. In these cases we cannot rely on the
357 original process exit to tell us when a test is complete. In cases
358 where we have an exit, and partial output (assertions with no final
359 plan, or a plan that has not been completed) we wait for a timeout
360 period to see if any additional events come into
361 HARNESS-DURATION-LONG
363 This lets you tell "yath" that the test file is long-running. This is
365 primarily used when concurrency is turned on in order to run longer
366 tests earlier, and concurrently with shorter ones. There is also a
367 "yath" option to skip all long tests.
368 This duration is set automatically if HARNESS-NO-TIMEOUT is set.
370 HARNESS-DURATION-MEDIUM
372 This lets you tell "yath" that the test is medium.
374 This is the default duration.
376 HARNESS-DURATION-SHORT
378 This lets you tell "yath" That the test is short.
380 HARNESS-CATEGORY-ISOLATION
382 This lets you tell "yath" that the test cannot be run concurrently with
384 other tests. Yath will hold off and run these tests one at a time after
385 all other tests.
386 HARNESS-CATEGORY-IMMISCIBLE
388 This lets you tell "yath" that the test cannot be run concurrently with
390 other tests of this class. This is helpful when you have multiple tests
391 which would otherwise have to be run sequentially at the end of the
392 run.
393 Yath prioritizes running these tests above HARNESS-CATEGORY-LONG.
395 HARNESS-CATEGORY-GENERAL
397 This is the default category.
399 HARNESS-CONFLICTS-XXX
401 This lets you tell "yath" that no other test of type XXX can be run at
403 the same time as this one. You are able to set multiple conflict types
404 and "yath" will honor them.
405 XXX can be replaced with any type of your choosing.
407 NOTE: This directive does not alter the category of your test. You are
409 free to mark the test with LONG or MEDIUM in addition to this marker.
410 Example with multiple lines.
412 #!/usr/bin/perl
413 # DASH and space are split the same way.
414 # HARNESS-CONFLICTS-DAEMON
415 # HARNESS-CONFLICTS MYSQL
416 ...
418 Or on a single line.
420 #!/usr/bin/perl
421 # HARNESS-CONFLICTS DAEMON MYSQL
422 ...
424 HARNESS-RETRY-n
426 This lets you specify a number (minimum n=1) of retries on test failure
428 for a specific test. HARNESS-RETRY-1 means a failing test will be run
429 twice and is equivalent to HARNESS-RETRY.
430 HARNESS-NO-RETRY
432 Use this to avoid this test being retried regardless of your retry
434 settings.
435
437 This section documents the App::Yath module itself.
438 SYNOPSIS
440 In practice you should never need to write your own yath script, or
441 construct an App::Yath instance, or even access themain instance when
442 yath is running. However some aspects of doing so are documented here
443 for completeness.
444 A minimum yath script looks like this:
446 BEGIN {
448 package App::Yath:Script;
449 require Time::HiRes;
451 require App::Yath;
452 require Test2::Harness::Settings;
453 my $settings = Test2::Harness::Settings->new(
455 harness => {
456 orig_argv => [@ARGV],
457 orig_inc => [@INC],
458 script => __FILE__,
459 start => Time::HiRes::time(),
460 version => $App::Yath::VERSION,
461 },
462 );
463 my $app = App::Yath->new(
465 argv => \@ARGV,
466 config => {},
467 settings => $settings,
468 );
469 $app->generate_run_sub('App::Yath::Script::run');
471 }
472 exit(App::Yath::Script::run());
474 It is important that most logic live in a BEGIN block. This is so that
476 goto::file can be used post-fork to execute a test script.
477 The actual yath script is significantly more complicated with the
479 following behaviors:
480 pre-process essential arguments such as -D and no-scan-plugins
482 re-exec with a different yath script if in developer mode and a local
483 copy is found
484 Parse the yath-rc config files
485 gather and store essential startup information
486 METHODS
488 App::Yath does not provide many methods to use externally.
489 $app->generate_run_sub($symbol_name)
491 This tells App::Yath to generate a subroutine at the specified
492 symbol name which can be run and be expected to return an exit
493 value.
494 $lib_path = $app->app_path()
496 Get the include directory App::Yath was loaded from.
497
499 The source code repository for Test2-Harness can be found at
500 http://github.com/Test-More/Test2-Harness/.
501
503 Chad Granum <exodist@cpan.org>
504
506 Chad Granum <exodist@cpan.org>
507
509 Copyright 2020 Chad Granum <exodist7@gmail.com>.
510 This program is free software; you can redistribute it and/or modify it
512 under the same terms as Perl itself.
513 See http://dev.perl.org/licenses/
515perl v5.34.0 2021-11-05 App::Yath(3)