1Test::More(3pm) Perl Programmers Reference Guide Test::More(3pm)
2
6 Test::More - yet another framework for writing test scripts
7
9 use Test::More tests => $Num_Tests;
10 # or
11 use Test::More qw(no_plan);
12 # or
13 use Test::More skip_all => $reason;
14 BEGIN { use_ok( 'Some::Module' ); }
16 require_ok( 'Some::Module' );
17 # Various ways to say "ok"
19 ok($this eq $that, $test_name);
20 is ($this, $that, $test_name);
22 isnt($this, $that, $test_name);
23 # Rather than print STDERR "# here's what went wrong\n"
25 diag("here's what went wrong");
26 like ($this, qr/that/, $test_name);
28 unlike($this, qr/that/, $test_name);
29 cmp_ok($this, '==', $that, $test_name);
31 is_deeply($complex_structure1, $complex_structure2, $test_name);
33 SKIP: {
35 skip $why, $how_many unless $have_some_feature;
36 ok( foo(), $test_name );
38 is( foo(42), 23, $test_name );
39 };
40 TODO: {
42 local $TODO = $why;
43 ok( foo(), $test_name );
45 is( foo(42), 23, $test_name );
46 };
47 can_ok($module, @methods);
49 isa_ok($object, $class);
50 pass($test_name);
52 fail($test_name);
53 BAIL_OUT($why);
55 # UNIMPLEMENTED!!!
57 my @status = Test::More::status;
58
60 STOP! If you're just getting started writing tests, have a look at
61 Test::Simple first. This is a drop in replacement for Test::Simple
62 which you can switch to once you get the hang of basic testing.
63 The purpose of this module is to provide a wide range of testing utili‐
65 ties. Various ways to say "ok" with better diagnostics, facilities to
66 skip tests, test future features and compare complicated data struc‐
67 tures. While you can do almost anything with a simple "ok()" function,
68 it doesn't provide good diagnostic output.
69 I love it when a plan comes together
71 Before anything else, you need a testing plan. This basically declares
73 how many tests your script is going to run to protect against premature
74 failure.
75 The preferred way to do this is to declare a plan when you "use
77 Test::More".
78 use Test::More tests => $Num_Tests;
80 There are rare cases when you will not know beforehand how many tests
82 your script is going to run. In this case, you can declare that you
83 have no plan. (Try to avoid using this as it weakens your test.)
84 use Test::More qw(no_plan);
86 NOTE: using no_plan requires a Test::Harness upgrade else it will think
88 everything has failed. See "CAVEATS and NOTES").
89 In some cases, you'll want to completely skip an entire testing script.
91 use Test::More skip_all => $skip_reason;
93 Your script will declare a skip with the reason why you skipped and
95 exit immediately with a zero (success). See Test::Harness for details.
96 If you want to control what functions Test::More will export, you have
98 to use the 'import' option. For example, to import everything but
99 'fail', you'd do:
100 use Test::More tests => 23, import => ['!fail'];
102 Alternatively, you can use the plan() function. Useful for when you
104 have to calculate the number of tests.
105 use Test::More;
107 plan tests => keys %Stuff * 3;
108 or for deciding between running the tests at all:
110 use Test::More;
112 if( $^O eq 'MacOS' ) {
113 plan skip_all => 'Test irrelevant on MacOS';
114 }
115 else {
116 plan tests => 42;
117 }
118 Test names
120 By convention, each test is assigned a number in order. This is
122 largely done automatically for you. However, it's often very useful to
123 assign a name to each test. Which would you rather see:
124 ok 4
126 not ok 5
127 ok 6
128 or
130 ok 4 - basic multi-variable
132 not ok 5 - simple exponential
133 ok 6 - force == mass * acceleration
134 The later gives you some idea of what failed. It also makes it easier
136 to find the test in your script, simply search for "simple exponen‐
137 tial".
138 All test functions take a name argument. It's optional, but highly
140 suggested that you use it.
141 I'm ok, you're not ok.
143 The basic purpose of this module is to print out either "ok #" or "not
145 ok #" depending on if a given test succeeded or failed. Everything
146 else is just gravy.
147 All of the following print "ok" or "not ok" depending on if the test
149 succeeded or failed. They all also return true or false, respectively.
150 ok
152 ok($this eq $that, $test_name);
153 This simply evaluates any expression ("$this eq $that" is just a
155 simple example) and uses that to determine if the test succeeded or
156 failed. A true expression passes, a false one fails. Very simple.
157 For example:
159 ok( $exp{9} == 81, 'simple exponential' );
161 ok( Film->can('db_Main'), 'set_db()' );
162 ok( $p->tests == 4, 'saw tests' );
163 ok( !grep !defined $_, @items, 'items populated' );
164 (Mnemonic: "This is ok.")
166 $test_name is a very short description of the test that will be
168 printed out. It makes it very easy to find a test in your script
169 when it fails and gives others an idea of your intentions.
170 $test_name is optional, but we very strongly encourage its use.
171 Should an ok() fail, it will produce some diagnostics:
173 not ok 18 - sufficient mucus
175 # Failed test 'sufficient mucus'
176 # in foo.t at line 42.
177 This is actually Test::Simple's ok() routine.
179 is
181 isnt
182 is ( $this, $that, $test_name );
183 isnt( $this, $that, $test_name );
184 Similar to ok(), is() and isnt() compare their two arguments with
186 "eq" and "ne" respectively and use the result of that to determine
187 if the test succeeded or failed. So these:
188 # Is the ultimate answer 42?
190 is( ultimate_answer(), 42, "Meaning of Life" );
191 # $foo isn't empty
193 isnt( $foo, '', "Got some foo" );
194 are similar to these:
196 ok( ultimate_answer() eq 42, "Meaning of Life" );
198 ok( $foo ne '', "Got some foo" );
199 (Mnemonic: "This is that." "This isn't that.")
201 So why use these? They produce better diagnostics on failure.
203 ok() cannot know what you are testing for (beyond the name), but
204 is() and isnt() know what the test was and why it failed. For
205 example this test:
206 my $foo = 'waffle'; my $bar = 'yarblokos';
208 is( $foo, $bar, 'Is foo the same as bar?' );
209 Will produce something like this:
211 not ok 17 - Is foo the same as bar?
213 # Failed test 'Is foo the same as bar?'
214 # in foo.t at line 139.
215 # got: 'waffle'
216 # expected: 'yarblokos'
217 So you can figure out what went wrong without rerunning the test.
219 You are encouraged to use is() and isnt() over ok() where possible,
221 however do not be tempted to use them to find out if something is
222 true or false!
223 # XXX BAD!
225 is( exists $brooklyn{tree}, 1, 'A tree grows in Brooklyn' );
226 This does not check if "exists $brooklyn{tree}" is true, it checks
228 if it returns 1. Very different. Similar caveats exist for false
229 and 0. In these cases, use ok().
230 ok( exists $brooklyn{tree}, 'A tree grows in Brooklyn' );
232 For those grammatical pedants out there, there's an "isn't()" func‐
234 tion which is an alias of isnt().
235 like
237 like( $this, qr/that/, $test_name );
238 Similar to ok(), like() matches $this against the regex "qr/that/".
240 So this:
242 like($this, qr/that/, 'this is like that');
244 is similar to:
246 ok( $this =~ /that/, 'this is like that');
248 (Mnemonic "This is like that".)
250 The second argument is a regular expression. It may be given as a
252 regex reference (i.e. "qr//") or (for better compatibility with
253 older perls) as a string that looks like a regex (alternative
254 delimiters are currently not supported):
255 like( $this, '/that/', 'this is like that' );
257 Regex options may be placed on the end ('/that/i').
259 Its advantages over ok() are similar to that of is() and isnt().
261 Better diagnostics on failure.
262 unlike
264 unlike( $this, qr/that/, $test_name );
265 Works exactly as like(), only it checks if $this does not match the
267 given pattern.
268 cmp_ok
270 cmp_ok( $this, $op, $that, $test_name );
271 Halfway between ok() and is() lies cmp_ok(). This allows you to
273 compare two arguments using any binary perl operator.
274 # ok( $this eq $that );
276 cmp_ok( $this, 'eq', $that, 'this eq that' );
277 # ok( $this == $that );
279 cmp_ok( $this, '==', $that, 'this == that' );
280 # ok( $this && $that );
282 cmp_ok( $this, '&&', $that, 'this && that' );
283 ...etc...
284 Its advantage over ok() is when the test fails you'll know what
286 $this and $that were:
287 not ok 1
289 # Failed test in foo.t at line 12.
290 # '23'
291 # &&
292 # undef
293 It's also useful in those cases where you are comparing numbers and
295 is()'s use of "eq" will interfere:
296 cmp_ok( $big_hairy_number, '==', $another_big_hairy_number );
298 can_ok
300 can_ok($module, @methods);
301 can_ok($object, @methods);
302 Checks to make sure the $module or $object can do these @methods
304 (works with functions, too).
305 can_ok('Foo', qw(this that whatever));
307 is almost exactly like saying:
309 ok( Foo->can('this') &&
311 Foo->can('that') &&
312 Foo->can('whatever')
313 );
314 only without all the typing and with a better interface. Handy for
316 quickly testing an interface.
317 No matter how many @methods you check, a single can_ok() call
319 counts as one test. If you desire otherwise, use:
320 foreach my $meth (@methods) {
322 can_ok('Foo', $meth);
323 }
324 isa_ok
326 isa_ok($object, $class, $object_name);
327 isa_ok($ref, $type, $ref_name);
328 Checks to see if the given "$object->isa($class)". Also checks to
330 make sure the object was defined in the first place. Handy for
331 this sort of thing:
332 my $obj = Some::Module->new;
334 isa_ok( $obj, 'Some::Module' );
335 where you'd otherwise have to write
337 my $obj = Some::Module->new;
339 ok( defined $obj && $obj->isa('Some::Module') );
340 to safeguard against your test script blowing up.
342 It works on references, too:
344 isa_ok( $array_ref, 'ARRAY' );
346 The diagnostics of this test normally just refer to 'the object'.
348 If you'd like them to be more specific, you can supply an
349 $object_name (for example 'Test customer').
350 pass
352 fail
353 pass($test_name);
354 fail($test_name);
355 Sometimes you just want to say that the tests have passed. Usually
357 the case is you've got some complicated condition that is difficult
358 to wedge into an ok(). In this case, you can simply use pass() (to
359 declare the test ok) or fail (for not ok). They are synonyms for
360 ok(1) and ok(0).
361 Use these very, very, very sparingly.
363 Module tests
365 You usually want to test if the module you're testing loads ok, rather
367 than just vomiting if its load fails. For such purposes we have
368 "use_ok" and "require_ok".
369 use_ok
371 BEGIN { use_ok($module); }
372 BEGIN { use_ok($module, @imports); }
373 These simply use the given $module and test to make sure the load
375 happened ok. It's recommended that you run use_ok() inside a BEGIN
376 block so its functions are exported at compile-time and prototypes
377 are properly honored.
378 If @imports are given, they are passed through to the use. So
380 this:
381 BEGIN { use_ok('Some::Module', qw(foo bar)) }
383 is like doing this:
385 use Some::Module qw(foo bar);
387 Version numbers can be checked like so:
389 # Just like "use Some::Module 1.02"
391 BEGIN { use_ok('Some::Module', 1.02) }
392 Don't try to do this:
394 BEGIN {
396 use_ok('Some::Module');
397 ...some code that depends on the use...
399 ...happening at compile time...
400 }
401 because the notion of "compile-time" is relative. Instead, you
403 want:
404 BEGIN { use_ok('Some::Module') }
406 BEGIN { ...some code that depends on the use... }
407 require_ok
409 require_ok($module);
410 require_ok($file);
411 Like use_ok(), except it requires the $module or $file.
413 Complex data structures
415 Not everything is a simple eq check or regex. There are times you need
417 to see if two data structures are equivalent. For these instances
418 Test::More provides a handful of useful functions.
419 NOTE I'm not quite sure what will happen with filehandles.
421 is_deeply
423 is_deeply( $this, $that, $test_name );
424 Similar to is(), except that if $this and $that are references, it
426 does a deep comparison walking each data structure to see if they
427 are equivalent. If the two structures are different, it will dis‐
428 play the place where they start differing.
429 is_deeply() compares the dereferenced values of references, the
431 references themselves (except for their type) are ignored. This
432 means aspects such as blessing and ties are not considered "differ‐
433 ent".
434 is_deeply() current has very limited handling of function reference
436 and globs. It merely checks if they have the same referent. This
437 may improve in the future.
438 Test::Differences and Test::Deep provide more in-depth functional‐
440 ity along these lines.
441 Diagnostics
443 If you pick the right test function, you'll usually get a good idea of
445 what went wrong when it failed. But sometimes it doesn't work out that
446 way. So here we have ways for you to write your own diagnostic mes‐
447 sages which are safer than just "print STDERR".
448 diag
450 diag(@diagnostic_message);
451 Prints a diagnostic message which is guaranteed not to interfere
453 with test output. Like "print" @diagnostic_message is simply con‐
454 catenated together.
455 Handy for this sort of thing:
457 ok( grep(/foo/, @users), "There's a foo user" ) or
459 diag("Since there's no foo, check that /etc/bar is set up right");
460 which would produce:
462 not ok 42 - There's a foo user
464 # Failed test 'There's a foo user'
465 # in foo.t at line 52.
466 # Since there's no foo, check that /etc/bar is set up right.
467 You might remember "ok() or diag()" with the mnemonic "open() or
469 die()".
470 NOTE The exact formatting of the diagnostic output is still chang‐
472 ing, but it is guaranteed that whatever you throw at it it won't
473 interfere with the test.
474 Conditional tests
476 Sometimes running a test under certain conditions will cause the test
478 script to die. A certain function or method isn't implemented (such as
479 fork() on MacOS), some resource isn't available (like a net connection)
480 or a module isn't available. In these cases it's necessary to skip
481 tests, or declare that they are supposed to fail but will work in the
482 future (a todo test).
483 For more details on the mechanics of skip and todo tests see Test::Har‐
485 ness.
486 The way Test::More handles this is with a named block. Basically, a
488 block of tests which can be skipped over or made todo. It's best if I
489 just show you...
490 SKIP: BLOCK
492 SKIP: {
493 skip $why, $how_many if $condition;
494 ...normal testing code goes here...
496 }
497 This declares a block of tests that might be skipped, $how_many
499 tests there are, $why and under what $condition to skip them. An
500 example is the easiest way to illustrate:
501 SKIP: {
503 eval { require HTML::Lint };
504 skip "HTML::Lint not installed", 2 if $@;
506 my $lint = new HTML::Lint;
508 isa_ok( $lint, "HTML::Lint" );
509 $lint->parse( $html );
511 is( $lint->errors, 0, "No errors found in HTML" );
512 }
513 If the user does not have HTML::Lint installed, the whole block of
515 code won't be run at all. Test::More will output special ok's
516 which Test::Harness interprets as skipped, but passing, tests.
517 It's important that $how_many accurately reflects the number of
519 tests in the SKIP block so the # of tests run will match up with
520 your plan. If your plan is "no_plan" $how_many is optional and
521 will default to 1.
522 It's perfectly safe to nest SKIP blocks. Each SKIP block must have
524 the label "SKIP", or Test::More can't work its magic.
525 You don't skip tests which are failing because there's a bug in
527 your program, or for which you don't yet have code written. For
528 that you use TODO. Read on.
529 TODO: BLOCK
531 TODO: {
532 local $TODO = $why if $condition;
533 ...normal testing code goes here...
535 }
536 Declares a block of tests you expect to fail and $why. Perhaps
538 it's because you haven't fixed a bug or haven't finished a new fea‐
539 ture:
540 TODO: {
542 local $TODO = "URI::Geller not finished";
543 my $card = "Eight of clubs";
545 is( URI::Geller->your_card, $card, 'Is THIS your card?' );
546 my $spoon;
548 URI::Geller->bend_spoon;
549 is( $spoon, 'bent', "Spoon bending, that's original" );
550 }
551 With a todo block, the tests inside are expected to fail.
553 Test::More will run the tests normally, but print out special flags
554 indicating they are "todo". Test::Harness will interpret failures
555 as being ok. Should anything succeed, it will report it as an
556 unexpected success. You then know the thing you had todo is done
557 and can remove the TODO flag.
558 The nice part about todo tests, as opposed to simply commenting out
560 a block of tests, is it's like having a programmatic todo list.
561 You know how much work is left to be done, you're aware of what
562 bugs there are, and you'll know immediately when they're fixed.
563 Once a todo test starts succeeding, simply move it outside the
565 block. When the block is empty, delete it.
566 NOTE: TODO tests require a Test::Harness upgrade else it will treat
568 it as a normal failure. See "CAVEATS and NOTES").
569 todo_skip
571 TODO: {
572 todo_skip $why, $how_many if $condition;
573 ...normal testing code...
575 }
576 With todo tests, it's best to have the tests actually run. That
578 way you'll know when they start passing. Sometimes this isn't pos‐
579 sible. Often a failing test will cause the whole program to die or
580 hang, even inside an "eval BLOCK" with and using "alarm". In these
581 extreme cases you have no choice but to skip over the broken tests
582 entirely.
583 The syntax and behavior is similar to a "SKIP: BLOCK" except the
585 tests will be marked as failing but todo. Test::Harness will
586 interpret them as passing.
587 When do I use SKIP vs. TODO?
589 If it's something the user might not be able to do, use SKIP. This
590 includes optional modules that aren't installed, running under an
591 OS that doesn't have some feature (like fork() or symlinks), or
592 maybe you need an Internet connection and one isn't available.
593 If it's something the programmer hasn't done yet, use TODO. This
595 is for any code you haven't written yet, or bugs you have yet to
596 fix, but want to put tests in your testing script (always a good
597 idea).
598 Test control
600 BAIL_OUT
602 BAIL_OUT($reason);
603 Incidates to the harness that things are going so badly all testing
605 should terminate. This includes the running any additional test
606 scripts.
607 This is typically used when testing cannot continue such as a crit‐
609 ical module failing to compile or a necessary external utility not
610 being available such as a database connection failing.
611 The test will exit with 255.
613 Discouraged comparison functions
615 The use of the following functions is discouraged as they are not actu‐
617 ally testing functions and produce no diagnostics to help figure out
618 what went wrong. They were written before is_deeply() existed because
619 I couldn't figure out how to display a useful diff of two arbitrary
620 data structures.
621 These functions are usually used inside an ok().
623 ok( eq_array(\@this, \@that) );
625 "is_deeply()" can do that better and with diagnostics.
627 is_deeply( \@this, \@that );
629 They may be deprecated in future versions.
631 eq_array
633 my $is_eq = eq_array(\@this, \@that);
634 Checks if two arrays are equivalent. This is a deep check, so
636 multi-level structures are handled correctly.
637 eq_hash
639 my $is_eq = eq_hash(\%this, \%that);
640 Determines if the two hashes contain the same keys and values.
642 This is a deep check.
643 eq_set
645 my $is_eq = eq_set(\@this, \@that);
646 Similar to eq_array(), except the order of the elements is not
648 important. This is a deep check, but the irrelevancy of order only
649 applies to the top level.
650 ok( eq_set(\@this, \@that) );
652 Is better written:
654 is_deeply( [sort @this], [sort @that] );
656 NOTE By historical accident, this is not a true set comparison.
658 While the order of elements does not matter, duplicate elements do.
659 NOTE eq_set() does not know how to deal with references at the top
661 level. The following is an example of a comparison which might not
662 work:
663 eq_set([\1, \2], [\2, \1]);
665 Test::Deep contains much better set comparison functions.
667 Extending and Embedding Test::More
669 Sometimes the Test::More interface isn't quite enough. Fortunately,
671 Test::More is built on top of Test::Builder which provides a single,
672 unified backend for any test library to use. This means two test
673 libraries which both use Test::Builder can be used together in the same
674 program.
675 If you simply want to do a little tweaking of how the tests behave, you
677 can access the underlying Test::Builder object like so:
678 builder
680 my $test_builder = Test::More->builder;
681 Returns the Test::Builder object underlying Test::More for you to
683 play with.
684
686 If all your tests passed, Test::Builder will exit with zero (which is
687 normal). If anything failed it will exit with how many failed. If you
688 run less (or more) tests than you planned, the missing (or extras) will
689 be considered failures. If no tests were ever run Test::Builder will
690 throw a warning and exit with 255. If the test died, even after having
691 successfully completed all its tests, it will still be considered a
692 failure and will exit with 255.
693 So the exit codes are...
695 0 all tests successful
697 255 test died or all passed but wrong # of tests run
698 any other number how many failed (including missing or extras)
699 If you fail more than 254 tests, it will be reported as 254.
701 NOTE This behavior may go away in future versions.
703
705 Backwards compatibility
706 Test::More works with Perls as old as 5.004_05.
707 Overloaded objects
709 String overloaded objects are compared as strings (or in cmp_ok()'s
710 case, strings or numbers as appropriate to the comparison op).
711 This prevents Test::More from piercing an object's interface allow‐
712 ing better blackbox testing. So if a function starts returning
713 overloaded objects instead of bare strings your tests won't notice
714 the difference. This is good.
715 However, it does mean that functions like is_deeply() cannot be
717 used to test the internals of string overloaded objects. In this
718 case I would suggest Test::Deep which contains more flexible test‐
719 ing functions for complex data structures.
720 Threads
722 Test::More will only be aware of threads if "use threads" has been
723 done before Test::More is loaded. This is ok:
724 use threads;
726 use Test::More;
727 This may cause problems:
729 use Test::More
731 use threads;
732 Test::Harness upgrade
734 no_plan and todo depend on new Test::Harness features and fixes.
735 If you're going to distribute tests that use no_plan or todo your
736 end-users will have to upgrade Test::Harness to the latest one on
737 CPAN. If you avoid no_plan and TODO tests, the stock Test::Harness
738 will work fine.
739 Installing Test::More should also upgrade Test::Harness.
741
743 This is a case of convergent evolution with Joshua Pritikin's Test mod‐
744 ule. I was largely unaware of its existence when I'd first written my
745 own ok() routines. This module exists because I can't figure out how
746 to easily wedge test names into Test's interface (along with a few
747 other problems).
748 The goal here is to have a testing utility that's simple to learn,
750 quick to use and difficult to trip yourself up with while still provid‐
751 ing more flexibility than the existing Test.pm. As such, the names of
752 the most common routines are kept tiny, special cases and magic side-
753 effects are kept to a minimum. WYSIWYG.
754
756 Test::Simple if all this confuses you and you just want to write some
757 tests. You can upgrade to Test::More later (it's forward compatible).
758 Test is the old testing module. Its main benefit is that it has been
760 distributed with Perl since 5.004_05.
761 Test::Harness for details on how your test results are interpreted by
763 Perl.
764 Test::Differences for more ways to test complex data structures. And
766 it plays well with Test::More.
767 Test::Class is like XUnit but more perlish.
769 Test::Deep gives you more powerful complex data structure testing.
771 Test::Unit is XUnit style testing.
773 Test::Inline shows the idea of embedded testing.
775 Bundle::Test installs a whole bunch of useful test modules.
777
779 Michael G Schwern <schwern@pobox.com> with much inspiration from Joshua
780 Pritikin's Test module and lots of help from Barrie Slaymaker, Tony
781 Bowden, blackstar.co.uk, chromatic, Fergal Daly and the perl-qa gang.
782
784 See http://rt.cpan.org to report and view bugs.
785
787 Copyright 2001, 2002, 2004 by Michael G Schwern <schwern@pobox.com>.
788 This program is free software; you can redistribute it and/or modify it
790 under the same terms as Perl itself.
791 See http://www.perl.com/perl/misc/Artistic.html
793perl v5.8.8 2001-09-21 Test::More(3pm)