1Test::Tutorial(3pm) Perl Programmers Reference Guide Test::Tutorial(3pm)
2
6 Test::Tutorial - A tutorial about writing really basic tests
7
9 AHHHHHHH!!!! NOT TESTING! Anything but testing! Beat me, whip me,
10 send me to Detroit, but don't make me write tests!
11 *sob*
13 Besides, I don't know how to write the damned things.
15 Is this you? Is writing tests right up there with writing
17 documentation and having your fingernails pulled out? Did you open up
18 a test and read
19 ######## We start with some black magic
21 and decide that's quite enough for you?
23 It's ok. That's all gone now. We've done all the black magic for you.
25 And here are the tricks...
26 Nuts and bolts of testing.
28 Here's the most basic test program.
29 #!/usr/bin/perl -w
31 print "1..1\n";
33 print 1 + 1 == 2 ? "ok 1\n" : "not ok 1\n";
35 since 1 + 1 is 2, it prints:
37 1..1
39 ok 1
40 What this says is: 1..1 "I'm going to run one test." [1] "ok 1" "The
42 first test passed". And that's about all magic there is to testing.
43 Your basic unit of testing is the ok. For each thing you test, an "ok"
44 is printed. Simple. Test::Harness interprets your test results to
45 determine if you succeeded or failed (more on that later).
46 Writing all these print statements rapidly gets tedious. Fortunately,
48 there's Test::Simple. It has one function, "ok()".
49 #!/usr/bin/perl -w
51 use Test::Simple tests => 1;
53 ok( 1 + 1 == 2 );
55 and that does the same thing as the code above. "ok()" is the backbone
57 of Perl testing, and we'll be using it instead of roll-your-own from
58 here on. If "ok()" gets a true value, the test passes. False, it
59 fails.
60 #!/usr/bin/perl -w
62 use Test::Simple tests => 2;
64 ok( 1 + 1 == 2 );
65 ok( 2 + 2 == 5 );
66 from that comes
68 1..2
70 ok 1
71 not ok 2
72 # Failed test (test.pl at line 5)
73 # Looks like you failed 1 tests of 2.
74 1..2 "I'm going to run two tests." This number is used to ensure your
76 test program ran all the way through and didn't die or skip some tests.
77 "ok 1" "The first test passed." "not ok 2" "The second test failed".
78 Test::Simple helpfully prints out some extra commentary about your
79 tests.
80 It's not scary. Come, hold my hand. We're going to give an example of
82 testing a module. For our example, we'll be testing a date library,
83 Date::ICal. It's on CPAN, so download a copy and follow along. [2]
84 Where to start?
86 This is the hardest part of testing, where do you start? People often
87 get overwhelmed at the apparent enormity of the task of testing a whole
88 module. Best place to start is at the beginning. Date::ICal is an
89 object-oriented module, and that means you start by making an object.
90 So we test "new()".
91 #!/usr/bin/perl -w
93 use Test::Simple tests => 2;
95 use Date::ICal;
97 my $ical = Date::ICal->new; # create an object
99 ok( defined $ical ); # check that we got something
100 ok( $ical->isa('Date::ICal') ); # and it's the right class
101 run that and you should get:
103 1..2
105 ok 1
106 ok 2
107 congratulations, you've written your first useful test.
109 Names
111 That output isn't terribly descriptive, is it? When you have two tests
112 you can figure out which one is #2, but what if you have 102?
113 Each test can be given a little descriptive name as the second argument
115 to "ok()".
116 use Test::Simple tests => 2;
118 ok( defined $ical, 'new() returned something' );
120 ok( $ical->isa('Date::ICal'), " and it's the right class" );
121 So now you'd see...
123 1..2
125 ok 1 - new() returned something
126 ok 2 - and it's the right class
127 Test the manual
129 Simplest way to build up a decent testing suite is to just test what
130 the manual says it does. [3] Let's pull something out of the "SYNOPSIS"
131 in Date::ICal and test that all its bits work.
132 #!/usr/bin/perl -w
134 use Test::Simple tests => 8;
136 use Date::ICal;
138 $ical = Date::ICal->new( year => 1964, month => 10, day => 16,
140 hour => 16, min => 12, sec => 47,
141 tz => '0530' );
142 ok( defined $ical, 'new() returned something' );
144 ok( $ical->isa('Date::ICal'), " and it's the right class" );
145 ok( $ical->sec == 47, ' sec()' );
146 ok( $ical->min == 12, ' min()' );
147 ok( $ical->hour == 16, ' hour()' );
148 ok( $ical->day == 17, ' day()' );
149 ok( $ical->month == 10, ' month()' );
150 ok( $ical->year == 1964, ' year()' );
151 run that and you get:
153 1..8
155 ok 1 - new() returned something
156 ok 2 - and it's the right class
157 ok 3 - sec()
158 ok 4 - min()
159 ok 5 - hour()
160 not ok 6 - day()
161 # Failed test (- at line 16)
162 ok 7 - month()
163 ok 8 - year()
164 # Looks like you failed 1 tests of 8.
165 Whoops, a failure! [4] Test::Simple helpfully lets us know on what line
167 the failure occurred, but not much else. We were supposed to get 17,
168 but we didn't. What did we get?? Dunno. We'll have to re-run the
169 test in the debugger or throw in some print statements to find out.
170 Instead, we'll switch from Test::Simple to Test::More. Test::More does
172 everything Test::Simple does, and more! In fact, Test::More does
173 things exactly the way Test::Simple does. You can literally swap
174 Test::Simple out and put Test::More in its place. That's just what
175 we're going to do.
176 Test::More does more than Test::Simple. The most important difference
178 at this point is it provides more informative ways to say "ok".
179 Although you can write almost any test with a generic "ok()", it can't
180 tell you what went wrong. Instead, we'll use the "is()" function,
181 which lets us declare that something is supposed to be the same as
182 something else:
183 #!/usr/bin/perl -w
185 use Test::More tests => 8;
187 use Date::ICal;
189 $ical = Date::ICal->new( year => 1964, month => 10, day => 16,
191 hour => 16, min => 12, sec => 47,
192 tz => '0530' );
193 ok( defined $ical, 'new() returned something' );
195 ok( $ical->isa('Date::ICal'), " and it's the right class" );
196 is( $ical->sec, 47, ' sec()' );
197 is( $ical->min, 12, ' min()' );
198 is( $ical->hour, 16, ' hour()' );
199 is( $ical->day, 17, ' day()' );
200 is( $ical->month, 10, ' month()' );
201 is( $ical->year, 1964, ' year()' );
202 "Is "$ical->sec" 47?" "Is "$ical->min" 12?" With "is()" in place, you
204 get some more information
205 1..8
207 ok 1 - new() returned something
208 ok 2 - and it's the right class
209 ok 3 - sec()
210 ok 4 - min()
211 ok 5 - hour()
212 not ok 6 - day()
213 # Failed test (- at line 16)
214 # got: '16'
215 # expected: '17'
216 ok 7 - month()
217 ok 8 - year()
218 # Looks like you failed 1 tests of 8.
219 letting us know that "$ical->day" returned 16, but we expected 17. A
221 quick check shows that the code is working fine, we made a mistake when
222 writing up the tests. Just change it to:
223 is( $ical->day, 16, ' day()' );
225 and everything works.
227 So any time you're doing a "this equals that" sort of test, use "is()".
229 It even works on arrays. The test is always in scalar context, so you
230 can test how many elements are in a list this way. [5]
231 is( @foo, 5, 'foo has 5 elements' );
233 Sometimes the tests are wrong
235 Which brings us to a very important lesson. Code has bugs. Tests are
236 code. Ergo, tests have bugs. A failing test could mean a bug in the
237 code, but don't discount the possibility that the test is wrong.
238 On the flip side, don't be tempted to prematurely declare a test
240 incorrect just because you're having trouble finding the bug.
241 Invalidating a test isn't something to be taken lightly, and don't use
242 it as a cop out to avoid work.
243 Testing lots of values
245 We're going to be wanting to test a lot of dates here, trying to trick
246 the code with lots of different edge cases. Does it work before 1970?
247 After 2038? Before 1904? Do years after 10,000 give it trouble? Does
248 it get leap years right? We could keep repeating the code above, or we
249 could set up a little try/expect loop.
250 use Test::More tests => 32;
252 use Date::ICal;
253 my %ICal_Dates = (
255 # An ICal string And the year, month, date
256 # hour, minute and second we expect.
257 '19971024T120000' => # from the docs.
258 [ 1997, 10, 24, 12, 0, 0 ],
259 '20390123T232832' => # after the Unix epoch
260 [ 2039, 1, 23, 23, 28, 32 ],
261 '19671225T000000' => # before the Unix epoch
262 [ 1967, 12, 25, 0, 0, 0 ],
263 '18990505T232323' => # before the MacOS epoch
264 [ 1899, 5, 5, 23, 23, 23 ],
265 );
266 while( my($ical_str, $expect) = each %ICal_Dates ) {
269 my $ical = Date::ICal->new( ical => $ical_str );
270 ok( defined $ical, "new(ical => '$ical_str')" );
272 ok( $ical->isa('Date::ICal'), " and it's the right class" );
273 is( $ical->year, $expect->[0], ' year()' );
275 is( $ical->month, $expect->[1], ' month()' );
276 is( $ical->day, $expect->[2], ' day()' );
277 is( $ical->hour, $expect->[3], ' hour()' );
278 is( $ical->min, $expect->[4], ' min()' );
279 is( $ical->sec, $expect->[5], ' sec()' );
280 }
281 So now we can test bunches of dates by just adding them to %ICal_Dates.
283 Now that it's less work to test with more dates, you'll be inclined to
284 just throw more in as you think of them. Only problem is, every time
285 we add to that we have to keep adjusting the "use Test::More tests =>
286 ##" line. That can rapidly get annoying. There's two ways to make
287 this work better.
288 First, we can calculate the plan dynamically using the "plan()"
290 function.
291 use Test::More;
293 use Date::ICal;
294 my %ICal_Dates = (
296 ...same as before...
297 );
298 # For each key in the hash we're running 8 tests.
300 plan tests => keys %ICal_Dates * 8;
301 Or to be even more flexible, we use "no_plan". This means we're just
303 running some tests, don't know how many. [6]
304 use Test::More 'no_plan'; # instead of tests => 32
306 now we can just add tests and not have to do all sorts of math to
308 figure out how many we're running.
309 Informative names
311 Take a look at this line here
312 ok( defined $ical, "new(ical => '$ical_str')" );
314 we've added more detail about what we're testing and the ICal string
316 itself we're trying out to the name. So you get results like:
317 ok 25 - new(ical => '19971024T120000')
319 ok 26 - and it's the right class
320 ok 27 - year()
321 ok 28 - month()
322 ok 29 - day()
323 ok 30 - hour()
324 ok 31 - min()
325 ok 32 - sec()
326 if something in there fails, you'll know which one it was and that will
328 make tracking down the problem easier. So try to put a bit of
329 debugging information into the test names.
330 Describe what the tests test, to make debugging a failed test easier
332 for you or for the next person who runs your test.
333 Skipping tests
335 Poking around in the existing Date::ICal tests, I found this in
336 t/01sanity.t [7]
337 #!/usr/bin/perl -w
339 use Test::More tests => 7;
341 use Date::ICal;
342 # Make sure epoch time is being handled sanely.
344 my $t1 = Date::ICal->new( epoch => 0 );
345 is( $t1->epoch, 0, "Epoch time of 0" );
346 # XXX This will only work on unix systems.
348 is( $t1->ical, '19700101Z', " epoch to ical" );
349 is( $t1->year, 1970, " year()" );
351 is( $t1->month, 1, " month()" );
352 is( $t1->day, 1, " day()" );
353 # like the tests above, but starting with ical instead of epoch
355 my $t2 = Date::ICal->new( ical => '19700101Z' );
356 is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" );
357 is( $t2->epoch, 0, " and back to ICal" );
359 The beginning of the epoch is different on most non-Unix operating
361 systems [8]. Even though Perl smooths out the differences for the most
362 part, certain ports do it differently. MacPerl is one off the top of
363 my head. [9] We know this will never work on MacOS. So rather than
364 just putting a comment in the test, we can explicitly say it's never
365 going to work and skip the test.
366 use Test::More tests => 7;
368 use Date::ICal;
369 # Make sure epoch time is being handled sanely.
371 my $t1 = Date::ICal->new( epoch => 0 );
372 is( $t1->epoch, 0, "Epoch time of 0" );
373 SKIP: {
375 skip('epoch to ICal not working on MacOS', 6)
376 if $^O eq 'MacOS';
377 is( $t1->ical, '19700101Z', " epoch to ical" );
379 is( $t1->year, 1970, " year()" );
381 is( $t1->month, 1, " month()" );
382 is( $t1->day, 1, " day()" );
383 # like the tests above, but starting with ical instead of epoch
385 my $t2 = Date::ICal->new( ical => '19700101Z' );
386 is( $t2->ical, '19700101Z', "Start of epoch in ICal notation" );
387 is( $t2->epoch, 0, " and back to ICal" );
389 }
390 A little bit of magic happens here. When running on anything but
392 MacOS, all the tests run normally. But when on MacOS, "skip()" causes
393 the entire contents of the SKIP block to be jumped over. It's never
394 run. Instead, it prints special output that tells Test::Harness that
395 the tests have been skipped.
396 1..7
398 ok 1 - Epoch time of 0
399 ok 2 # skip epoch to ICal not working on MacOS
400 ok 3 # skip epoch to ICal not working on MacOS
401 ok 4 # skip epoch to ICal not working on MacOS
402 ok 5 # skip epoch to ICal not working on MacOS
403 ok 6 # skip epoch to ICal not working on MacOS
404 ok 7 # skip epoch to ICal not working on MacOS
405 This means your tests won't fail on MacOS. This means less emails from
407 MacPerl users telling you about failing tests that you know will never
408 work. You've got to be careful with skip tests. These are for tests
409 which don't work and never will. It is not for skipping genuine bugs
410 (we'll get to that in a moment).
411 The tests are wholly and completely skipped. [10] This will work.
413 SKIP: {
415 skip("I don't wanna die!");
416 die, die, die, die, die;
418 }
419 Todo tests
421 Thumbing through the Date::ICal man page, I came across this:
422 ical
424 $ical_string = $ical->ical;
426 Retrieves, or sets, the date on the object, using any
428 valid ICal date/time string.
429 "Retrieves or sets". Hmmm, didn't see a test for using "ical()" to set
431 the date in the Date::ICal test suite. So I'll write one.
432 use Test::More tests => 1;
434 use Date::ICal;
435 my $ical = Date::ICal->new;
437 $ical->ical('20201231Z');
438 is( $ical->ical, '20201231Z', 'Setting via ical()' );
439 run that and I get
441 1..1
443 not ok 1 - Setting via ical()
444 # Failed test (- at line 6)
445 # got: '20010814T233649Z'
446 # expected: '20201231Z'
447 # Looks like you failed 1 tests of 1.
448 Whoops! Looks like it's unimplemented. Let's assume we don't have the
450 time to fix this. [11] Normally, you'd just comment out the test and
451 put a note in a todo list somewhere. Instead, we're going to
452 explicitly state "this test will fail" by wrapping it in a "TODO"
453 block.
454 use Test::More tests => 1;
456 TODO: {
458 local $TODO = 'ical($ical) not yet implemented';
459 my $ical = Date::ICal->new;
461 $ical->ical('20201231Z');
462 is( $ical->ical, '20201231Z', 'Setting via ical()' );
464 }
465 Now when you run, it's a little different:
467 1..1
469 not ok 1 - Setting via ical() # TODO ical($ical) not yet implemented
470 # got: '20010822T201551Z'
471 # expected: '20201231Z'
472 Test::More doesn't say "Looks like you failed 1 tests of 1". That '#
474 TODO' tells Test::Harness "this is supposed to fail" and it treats a
475 failure as a successful test. So you can write tests even before
476 you've fixed the underlying code.
477 If a TODO test passes, Test::Harness will report it "UNEXPECTEDLY
479 SUCCEEDED". When that happens, you simply remove the TODO block with
480 "local $TODO" and turn it into a real test.
481 Testing with taint mode.
483 Taint mode is a funny thing. It's the globalest of all global
484 features. Once you turn it on, it affects all code in your program and
485 all modules used (and all the modules they use). If a single piece of
486 code isn't taint clean, the whole thing explodes. With that in mind,
487 it's very important to ensure your module works under taint mode.
488 It's very simple to have your tests run under taint mode. Just throw a
490 "-T" into the "#!" line. Test::Harness will read the switches in "#!"
491 and use them to run your tests.
492 #!/usr/bin/perl -Tw
494 ...test normally here...
496 So when you say "make test" it will be run with taint mode and warnings
498 on.
499
501 1. The first number doesn't really mean anything, but it has to be 1.
502 It's the second number that's important.
503 2. For those following along at home, I'm using version 1.31. It has
505 some bugs, which is good -- we'll uncover them with our tests.
506 3. You can actually take this one step further and test the manual
508 itself. Have a look at Test::Inline (formerly Pod::Tests).
509 4. Yes, there's a mistake in the test suite. What! Me, contrived?
511 5. We'll get to testing the contents of lists later.
513 6. But what happens if your test program dies halfway through?! Since
515 we didn't say how many tests we're going to run, how can we know it
516 failed? No problem, Test::More employs some magic to catch that
517 death and turn the test into a failure, even if every test passed
518 up to that point.
519 7. I cleaned it up a little.
521 8. Most Operating Systems record time as the number of seconds since a
523 certain date. This date is the beginning of the epoch. Unix's
524 starts at midnight January 1st, 1970 GMT.
525 9. MacOS's epoch is midnight January 1st, 1904. VMS's is midnight,
527 November 17th, 1858, but vmsperl emulates the Unix epoch so it's
528 not a problem.
529 10. As long as the code inside the SKIP block at least compiles.
531 Please don't ask how. No, it's not a filter.
532 11. Do NOT be tempted to use TODO tests as a way to avoid fixing simple
534 bugs!
535
537 Michael G Schwern <schwern@pobox.com> and the perl-qa dancers!
538
540 Copyright 2001 by Michael G Schwern <schwern@pobox.com>.
541 This documentation is free; you can redistribute it and/or modify it
543 under the same terms as Perl itself.
544 Irrespective of its distribution, all code examples in these files are
546 hereby placed into the public domain. You are permitted and encouraged
547 to use this code in your own programs for fun or for profit as you see
548 fit. A simple comment in the code giving credit would be courteous but
549 is not required.
550perl v5.10.1 2009-07-07 Test::Tutorial(3pm)