1Test::LectroTest::TestRUusnenrerC(o3n)tributed Perl DocuTmeesntt:a:tLieocntroTest::TestRunner(3)
2
6 Test::LectroTest::TestRunner - Configurable TAP-compatible engine for
7 running LectroTest property checks
8
10 version 0.5001
11
13 use Test::LectroTest::TestRunner;
14 my @args = trials => 1_000, retries => 20_000;
16 my $runner = Test::LectroTest::TestRunner->new( @args );
17 # test a single property and print details upon failure
19 my $result = $runner->run( $a_single_lectrotest_property );
20 print $result->details unless $result->success;
21 # test a suite of properties, w/ Test::Harness::TAP output
23 my $num_successful = $runner->run_suite( @properties );
24 print "# All passed!" if $num_successful == @properties;
25
27 STOP! If you just want to write and run simple tests, see
28 Test::LectroTest. If you really want to learn about the property-
29 checking apparatus or turn its control knobs, read on.
30 This module provides Test::LectroTest::TestRunner, a class of objects
32 that tests properties by running repeated random trials. Create a
33 TestRunner, configure it, and then call its "run" or "run_suite"
34 methods to test properties individually or in groups.
35
37 The following methods are available.
38 new(named-params)
40 my $runner = new Test::LectroTest::TestRunner(
41 trials => 1_000,
42 retries => 20_000,
43 scalefn => sub { $_[0] / 2 + 1 },
44 verbose => 1,
45 regressions => "/path/to/regression_suite.txt",
46 );
47 Creates a new Test::LectroTest::TestRunner and configures it with the
49 given named parameters, if any. Typically, you need only provide the
50 "trials" parameter because the other values are reasonable for almost
51 all situations. Here is what each parameter means:
52 trials
54 The number of trials to run against each property checked. The
55 default is 1_000.
56 retries
58 The number of times to allow a property to retry trials (via
59 "$tcon->retry") during the entire property check before aborting
60 the check. This is used to prevent infinite looping, should the
61 property retry every attempt.
62 scalefn
64 A subroutine that scales the sizing guidance given to input
65 generators.
66 The TestRunner starts with an initial guidance of 1 at the
68 beginning of a property check. For each trial (or retry) of the
69 property, the guidance value is incremented. This causes
70 successive trials to be tried using successively more complex
71 inputs. The "scalefn" subroutine gets to adjust this guidance on
72 the way to the input generators. Typically, you would change the
73 "scalefn" subroutine if you wanted to change the rate and which
74 inputs grow during the course of the trials.
75 verbose
77 If this paramter is set to true (the default) the TestRunner will
78 use verbose output that includes things like label frequencies and
79 counterexamples. Otherwise, only one-line summaries will be
80 output. Unless you have a good reason to do otherwise, leave this
81 parameter alone because verbose output is almost always what you
82 want.
83 record_failures
85 If this parameter is set to a file's pathname (or a FailureRecorder
86 object), the TestRunner will record property-check failures to the
87 file (or recorder). (This is an easy way to build a regression-
88 testing suite.) If the file cannot be created or written to, this
89 parameter will be ignored. Set this parameter to "undef" (the
90 default) to turn off recording.
91 playback_failures
93 If this parameter is set to a file's pathname (or a FailureRecorder
94 object), the TestRunner will load previously recorded failures from
95 the file (or recorder) and use them as additional test cases when
96 checking properties. If the file cannot be read, this option will
97 be ignored. Set this parameter to "undef" (the default) to turn
98 off recording.
99 regressions
101 If this parameter is set to a file's pathname (or a FailureRecorder
102 object), the TestRunner will load failures from and record failures
103 to the file (or recorder). Setting this parameter is a shortcut
104 for, and exactly equivalent to, setting record_failures and
105 <playback_failures> to the same value, which is typically what you
106 want when managing a persistent suite of regression tests.
107 This is a write-only accessor.
109 You can also set and get the values of the configuration properties
111 using accessors of the same name. For example:
112 $runner->trials( 10_000 );
114 run(property)
116 $results = $runner->run( $a_property );
117 print $results->summary, "\n";
118 if ($results->success) {
119 # celebrate!
120 }
121 Checks whether the given property holds by running repeated random
123 trials. The result is a Test::LectroTest::TestRunner::results object,
124 which you can query for fined-grained information about the outcome of
125 the check.
126 The "run" method takes an optional second argument which gives the test
128 number. If it is not provided (usually the case), the next number
129 available from the TestRunner's internal counter is used.
130 $results = $runner->run( $third_property, 3 );
132 Additionally, if the TestRunner's playback_failures parameter is
134 defined, this method will play back any relevant failure cases from the
135 given playback file (or FailureRecorder).
136 Additionally, if the TestRunner's record_failures parameter is defined,
138 this method will record any new failures to the given file (or
139 FailureRecorder).
140 run_suite(properties...)
142 my $num_successful = $runner->run_suite( @properties );
143 if ($num_successful == @properties) {
144 # celebrate most jubilantly!
145 }
146 Checks a suite of properties, sending the results of each property
148 checked to "STDOUT" in a form that is compatible with
149 Test::Harness::TAP. For example:
150 1..5
152 ok 1 - Property->new disallows use of 'tcon' in bindings
153 ok 2 - magic Property syntax disallows use of 'tcon' in bindings
154 ok 3 - exceptions are caught and reported as failures
155 ok 4 - pre-flight check catches new w/ no args
156 ok 5 - pre-flight check catches unbalanced arguments list
157 By default, labeling statistics and counterexamples (if any) are
159 included in the output if the TestRunner's "verbose" property is true.
160 You may override the default by passing the "verbose" named parameter
161 after all of the properties in the argument list:
162 my $num_successes = $runner->run_suite( @properties,
164 verbose => 1 );
165 my $num_failed = @properties - $num_successes;
166
168 There are two kinds of objects that TestRunner uses as helpers.
169 Neither is meant to be created by you. Rather, a TestRunner will
170 create them on your behalf when they are needed.
171 The objects are described in the following subsections.
173 Test::LectroTest::TestRunner::results
175 my $results = $runner->run( $a_property );
176 print "Property name: ", $results->name, ": ";
177 print $results->success ? "Winner!" : "Loser!";
178 This is the object that you get back from "run". It contains all of
180 the information available about the outcome of a property check and
181 provides the following methods:
182 success
184 Boolean value: True if the property checked out successfully;
185 false otherwise.
186 summary
188 Returns a one line summary of the property-check outcome. It does
189 not end with a newline. Example:
190 ok 1 - Property->new disallows use of 'tcon' in bindings
192 details
194 Returns all relevant information about the property-check outcome
195 as a series of lines. The last line is terminated with a newline.
196 The details are identical to the summary (except for the
197 terminating newline) unless label frequencies are present or a
198 counterexample is present, in which case the details will have
199 these extras (the summary does not). Example:
200 1..1
202 not ok 1 - 'my_sqrt meets defn of sqrt' falsified in 1 attempts
203 # Counterexample:
204 # $x = '0.546384454460178';
205 name
207 Returns the name of the property to which the results pertain.
208 number
210 The number assigned to the property that was checked.
211 counterexample
213 Returns the counterexample that "broke" the code being tested, if
214 there is one. Otherwise, returns an empty string. If any notes
215 have been attached to the failing trial, they will be included.
216 labels
218 Label counts. If any labels were applied to trials during the
219 property check, this value will be a reference to a hash mapping
220 each combination of labels to the count of trials that had that
221 particular combination. Otherwise, it will be undefined.
222 Note that each trial is counted only once -- for the most-specific
224 combination of labels that was applied to it. For example,
225 consider the following labeling logic:
226 Property {
228 ##[ x <- Int ]##
229 $tcon->label("negative") if $x < 0;
230 $tcon->label("odd") if $x % 2;
231 1;
232 }, name => "negative/odd labeling example";
233 For a particular trial, if x was 2 (positive and even), the trial
235 would receive no labels. If x was 3 (positive and odd), the trial
236 would be labeled "odd". If x was -2 (negative and even), the trial
237 would be labeled "negative". If x was -3 (negative and odd), the
238 trial would be labeled "negative & odd".
239 label_frequencies
241 Returns a string containing a line-by-line accounting of labels
242 applied during the series of trials:
243 print $results->label_frequencies;
245 The corresponding output looks like this:
247 25% negative
249 25% negative & odd
250 25% odd
251 If no labels were applied, an empty string is returned.
253 exception
255 Returns the text of the exception or error that caused the series
256 of trials to be aborted, if the trials were aborted because an
257 exception or error was intercepted by LectroTest. Otherwise,
258 returns an empty string.
259 attempts
261 Returns the count of trials performed.
262 incomplete
264 In the event that the series of trials was halted before it was
265 completed (such as when the retry count was exhausted), this method
266 will return the reason. Otherwise, it returns an empty string.
267 Note that a series of trials is complete if a counterexample was
269 found.
270 Test::LectroTest::TestRunner::testcontroller
272 During a live property-check trial, the variable $tcon is available to
273 your Properties. It lets you label the current trial or request that
274 it be re-tried with new inputs.
275 The following methods are available.
277 retry
279 Property {
280 ##[ x <- Int ]##
281 return $tcon->retry if $x == 0;
282 }, ... ;
283 Stops the current trial and tells the TestRunner to re-try it with
285 new inputs. Typically used to reject a particular case of inputs
286 that doesn't make for a good or valid test. While not required,
287 you will probably want to call "$tcon->retry" as part of a "return"
288 statement to prevent further execution of your property's logic,
289 the results of which will be thrown out should it run to
290 completion.
291 The return value of "$tcon->retry" is itself meaningless; it is the
293 side-effect of calling it that causes the current trial to be
294 thrown out and re-tried.
295 label(string)
297 Property {
298 ##[ x <- Int ]##
299 $tcon->label("negative") if $x < 0;
300 $tcon->label("odd") if $x % 2;
301 }, ... ;
302 Applies a label to the current trial. At the end of the trial, all
304 of the labels are gathered together, and the trial is dropped into
305 a bucket bearing the combined label. See the discussion of
306 "labels" for more.
307 trivial
309 Property {
310 ##[ x <- Int ]##
311 $tcon->trivial if $x == 0;
312 }, ... ;
313 Applies the label "trivial" to the current trial. It is identical
315 to calling "label" with "trivial" as the argument.
316 note(string...)
318 Property {
319 ##[ s <- String( charset=>"A-Za-z0-9" ) ]##
320 my $s_enc = encode($s);
321 my $s_enc_dec = decode($s_enc);
322 $tcon->note("s_enc = $s_enc",
323 "s_enc_dec = $s_enc_dec");
324 $s eq $s_enc_dec;
325 }, name => "decode is encode's inverse" ;
326 Adds a note (or notes) to the current trial. In the event that the
328 trial fails, these notes will be emitted as part of the
329 counterexample. For example:
330 1..1
332 not ok 1 - property 'decode is encode's inverse' \
333 falsified in 68 attempts
334 # Counterexample:
335 # $s = "0";
336 # Notes:
337 # $s_enc = "";
338 # $s_enc_dec = "";
339 Notes can help you debug your code when something goes wrong. Use
341 them as debugging hints to yourself. For example, you can use
342 notes to record the output of each stage of a multi-stage test.
343 That way, if the test fails, you can see what happened in each
344 stage without having to plug the counterexample into your code
345 under a debugger.
346 If you want to include complicated values or data structures in
348 your notes, see the "dump" method, next, which may be more
349 appropriate.
350 dump(value, name)
352 Property {
353 ##[ s <- String ]##
354 my $s_enc = encode($s);
355 my $s_enc_dec = decode($s_enc);
356 $tcon->dump($s_enc, "s_enc");
357 $tcon->dump($s_enc_dec, "s_enc_dec");
358 $s eq $s_enc_dec;
359 }, name => "decode is encode's inverse" ;
360 Adds a note to the current trial in which the given value is
362 dumped. The value will be dumped via Data::Dumper and thus may be
363 complex and contain weird control characters and so on. If you
364 supply a name, it will be used to name the dumped value. Returns
365 value as its result.
366 In the event that the trial fails, the note (and any others) will
368 be emitted as part of the counterexample.
369 See "note" above for more.
371
373 Test::LectroTest::Property explains in detail what you can put inside
374 of your property specifications.
375 Test::LectroTest::RegressionTesting explains how to test for
377 regressions and corner cases using LectroTest.
378 Test::Harness:TAP documents the Test Anything Protocol, Perl's simple
380 text-based interface between testing modules such as Test::LectroTest
381 and the test harness Test::Harness.
382
384 Tom Moertel (tom@moertel.com)
385
387 The LectroTest project was inspired by Haskell's QuickCheck module by
388 Koen Claessen and John Hughes:
389 http://www.cs.chalmers.se/~rjmh/QuickCheck/.
390
392 Copyright (c) 2004-13 by Thomas G Moertel. All rights reserved.
393 This program is free software; you can redistribute it and/or modify it
395 under the same terms as Perl itself.
396perl v5.30.1 2020-01-30 Test::LectroTest::TestRunner(3)