1Test::LectroTest::TestRUusnenrerC(o3n)tributed Perl DocuTmeesntt:a:tLieocntroTest::TestRunner(3)
2
3
4

NAME

6       Test::LectroTest::TestRunner - Configurable TAP-compatible engine for
7       running LectroTest property checks
8

VERSION

10       version 0.5001
11

SYNOPSIS

13        use Test::LectroTest::TestRunner;
14
15        my @args = trials => 1_000, retries => 20_000;
16        my $runner = Test::LectroTest::TestRunner->new( @args );
17
18        # test a single property and print details upon failure
19        my $result = $runner->run( $a_single_lectrotest_property );
20        print $result->details unless $result->success;
21
22        # test a suite of properties, w/ Test::Harness::TAP output
23        my $num_successful = $runner->run_suite( @properties );
24        print "# All passed!" if $num_successful == @properties;
25

DESCRIPTION

27       STOP! If you just want to write and run simple tests, see
28       Test::LectroTest.  If you really want to learn about the property-
29       checking apparatus or turn its control knobs, read on.
30
31       This module provides Test::LectroTest::TestRunner, a class of objects
32       that tests properties by running repeated random trials.  Create a
33       TestRunner, configure it, and then call its "run" or "run_suite"
34       methods to test properties individually or in groups.
35

METHODS

37       The following methods are available.
38
39   new(named-params)
40         my $runner = new Test::LectroTest::TestRunner(
41           trials      => 1_000,
42           retries     => 20_000,
43           scalefn     => sub { $_[0] / 2 + 1 },
44           verbose     => 1,
45           regressions => "/path/to/regression_suite.txt",
46         );
47
48       Creates a new Test::LectroTest::TestRunner and configures it with the
49       given named parameters, if any.  Typically, you need only provide the
50       "trials" parameter because the other values are reasonable for almost
51       all situations.  Here is what each parameter means:
52
53       trials
54           The number of trials to run against each property checked.  The
55           default is 1_000.
56
57       retries
58           The number of times to allow a property to retry trials (via
59           "$tcon->retry") during the entire property check before aborting
60           the check.  This is used to prevent infinite looping, should the
61           property retry every attempt.
62
63       scalefn
64           A subroutine that scales the sizing guidance given to input
65           generators.
66
67           The TestRunner starts with an initial guidance of 1 at the
68           beginning of a property check.  For each trial (or retry) of the
69           property, the guidance value is incremented.  This causes
70           successive trials to be tried using successively more complex
71           inputs.  The "scalefn" subroutine gets to adjust this guidance on
72           the way to the input generators.  Typically, you would change the
73           "scalefn" subroutine if you wanted to change the rate and which
74           inputs grow during the course of the trials.
75
76       verbose
77           If this paramter is set to true (the default) the TestRunner will
78           use verbose output that includes things like label frequencies and
79           counterexamples.  Otherwise, only one-line summaries will be
80           output.  Unless you have a good reason to do otherwise, leave this
81           parameter alone because verbose output is almost always what you
82           want.
83
84       record_failures
85           If this parameter is set to a file's pathname (or a FailureRecorder
86           object), the TestRunner will record property-check failures to the
87           file (or recorder).  (This is an easy way to build a regression-
88           testing suite.)  If the file cannot be created or written to, this
89           parameter will be ignored.  Set this parameter to "undef" (the
90           default) to turn off recording.
91
92       playback_failures
93           If this parameter is set to a file's pathname (or a FailureRecorder
94           object), the TestRunner will load previously recorded failures from
95           the file (or recorder) and use them as additional test cases when
96           checking properties.  If the file cannot be read, this option will
97           be ignored.  Set this parameter to "undef" (the default) to turn
98           off recording.
99
100       regressions
101           If this parameter is set to a file's pathname (or a FailureRecorder
102           object), the TestRunner will load failures from and record failures
103           to the file (or recorder).  Setting this parameter is a shortcut
104           for, and exactly equivalent to, setting record_failures and
105           <playback_failures> to the same value, which is typically what you
106           want when managing a persistent suite of regression tests.
107
108           This is a write-only accessor.
109
110       You can also set and get the values of the configuration properties
111       using accessors of the same name.  For example:
112
113         $runner->trials( 10_000 );
114
115   run(property)
116         $results = $runner->run( $a_property );
117         print $results->summary, "\n";
118         if ($results->success) {
119             # celebrate!
120         }
121
122       Checks whether the given property holds by running repeated random
123       trials.  The result is a Test::LectroTest::TestRunner::results object,
124       which you can query for fined-grained information about the outcome of
125       the check.
126
127       The "run" method takes an optional second argument which gives the test
128       number.  If it is not provided (usually the case), the next number
129       available from the TestRunner's internal counter is used.
130
131         $results = $runner->run( $third_property, 3 );
132
133       Additionally, if the TestRunner's playback_failures parameter is
134       defined, this method will play back any relevant failure cases from the
135       given playback file (or FailureRecorder).
136
137       Additionally, if the TestRunner's record_failures parameter is defined,
138       this method will record any new failures to the given file (or
139       FailureRecorder).
140
141   run_suite(properties...)
142         my $num_successful = $runner->run_suite( @properties );
143         if ($num_successful == @properties) {
144             # celebrate most jubilantly!
145         }
146
147       Checks a suite of properties, sending the results of each property
148       checked to "STDOUT" in a form that is compatible with
149       Test::Harness::TAP.  For example:
150
151         1..5
152         ok 1 - Property->new disallows use of 'tcon' in bindings
153         ok 2 - magic Property syntax disallows use of 'tcon' in bindings
154         ok 3 - exceptions are caught and reported as failures
155         ok 4 - pre-flight check catches new w/ no args
156         ok 5 - pre-flight check catches unbalanced arguments list
157
158       By default, labeling statistics and counterexamples (if any) are
159       included in the output if the TestRunner's "verbose" property is true.
160       You may override the default by passing the "verbose" named parameter
161       after all of the properties in the argument list:
162
163         my $num_successes = $runner->run_suite( @properties,
164                                                 verbose => 1 );
165         my $num_failed = @properties - $num_successes;
166

HELPER OBJECTS

168       There are two kinds of objects that TestRunner uses as helpers.
169       Neither is meant to be created by you.  Rather, a TestRunner will
170       create them on your behalf when they are needed.
171
172       The objects are described in the following subsections.
173
174   Test::LectroTest::TestRunner::results
175         my $results = $runner->run( $a_property );
176         print "Property name: ", $results->name, ": ";
177         print $results->success ? "Winner!" : "Loser!";
178
179       This is the object that you get back from "run".  It contains all of
180       the information available about the outcome of a property check and
181       provides the following methods:
182
183       success
184           Boolean value:  True if the property checked out successfully;
185           false otherwise.
186
187       summary
188           Returns a one line summary of the property-check outcome.  It does
189           not end with a newline.  Example:
190
191             ok 1 - Property->new disallows use of 'tcon' in bindings
192
193       details
194           Returns all relevant information about the property-check outcome
195           as a series of lines.  The last line is terminated with a newline.
196           The details are identical to the summary (except for the
197           terminating newline) unless label frequencies are present or a
198           counterexample is present, in which case the details will have
199           these extras (the summary does not).  Example:
200
201             1..1
202             not ok 1 - 'my_sqrt meets defn of sqrt' falsified in 1 attempts
203             # Counterexample:
204             # $x = '0.546384454460178';
205
206       name
207           Returns the name of the property to which the results pertain.
208
209       number
210           The number assigned to the property that was checked.
211
212       counterexample
213           Returns the counterexample that "broke" the code being tested, if
214           there is one.  Otherwise, returns an empty string.  If any notes
215           have been attached to the failing trial, they will be included.
216
217       labels
218           Label counts.  If any labels were applied to trials during the
219           property check, this value will be a reference to a hash mapping
220           each combination of labels to the count of trials that had that
221           particular combination.  Otherwise, it will be undefined.
222
223           Note that each trial is counted only once -- for the most-specific
224           combination of labels that was applied to it.  For example,
225           consider the following labeling logic:
226
227             Property {
228               ##[ x <- Int ]##
229               $tcon->label("negative") if $x < 0;
230               $tcon->label("odd")      if $x % 2;
231               1;
232             }, name => "negative/odd labeling example";
233
234           For a particular trial, if x was 2 (positive and even), the trial
235           would receive no labels.  If x was 3 (positive and odd), the trial
236           would be labeled "odd".  If x was -2 (negative and even), the trial
237           would be labeled "negative".  If x was -3 (negative and odd), the
238           trial would be labeled "negative & odd".
239
240       label_frequencies
241           Returns a string containing a line-by-line accounting of labels
242           applied during the series of trials:
243
244             print $results->label_frequencies;
245
246           The corresponding output looks like this:
247
248             25% negative
249             25% negative & odd
250             25% odd
251
252           If no labels were applied, an empty string is returned.
253
254       exception
255           Returns the text of the exception or error that caused the series
256           of trials to be aborted, if the trials were aborted because an
257           exception or error was intercepted by LectroTest.  Otherwise,
258           returns an empty string.
259
260       attempts
261           Returns the count of trials performed.
262
263       incomplete
264           In the event that the series of trials was halted before it was
265           completed (such as when the retry count was exhausted), this method
266           will return the reason.  Otherwise, it returns an empty string.
267
268           Note that a series of trials is complete if a counterexample was
269           found.
270
271   Test::LectroTest::TestRunner::testcontroller
272       During a live property-check trial, the variable $tcon is available to
273       your Properties.  It lets you label the current trial or request that
274       it be re-tried with new inputs.
275
276       The following methods are available.
277
278       retry
279               Property {
280                 ##[ x <- Int ]##
281                 return $tcon->retry if $x == 0;
282               }, ... ;
283
284           Stops the current trial and tells the TestRunner to re-try it with
285           new inputs.  Typically used to reject a particular case of inputs
286           that doesn't make for a good or valid test.  While not required,
287           you will probably want to call "$tcon->retry" as part of a "return"
288           statement to prevent further execution of your property's logic,
289           the results of which will be thrown out should it run to
290           completion.
291
292           The return value of "$tcon->retry" is itself meaningless; it is the
293           side-effect of calling it that causes the current trial to be
294           thrown out and re-tried.
295
296       label(string)
297               Property {
298                 ##[ x <- Int ]##
299                 $tcon->label("negative") if $x < 0;
300                 $tcon->label("odd")      if $x % 2;
301               }, ... ;
302
303           Applies a label to the current trial.  At the end of the trial, all
304           of the labels are gathered together, and the trial is dropped into
305           a bucket bearing the combined label.  See the discussion of
306           "labels" for more.
307
308       trivial
309               Property {
310                 ##[ x <- Int ]##
311                 $tcon->trivial if $x == 0;
312               }, ... ;
313
314           Applies the label "trivial" to the current trial.  It is identical
315           to calling "label" with "trivial" as the argument.
316
317       note(string...)
318               Property {
319                 ##[ s <- String( charset=>"A-Za-z0-9" ) ]##
320                 my $s_enc     = encode($s);
321                 my $s_enc_dec = decode($s_enc);
322                 $tcon->note("s_enc     = $s_enc",
323                             "s_enc_dec = $s_enc_dec");
324                 $s eq $s_enc_dec;
325               }, name => "decode is encode's inverse" ;
326
327           Adds a note (or notes) to the current trial.  In the event that the
328           trial fails, these notes will be emitted as part of the
329           counterexample.  For example:
330
331               1..1
332               not ok 1 - property 'decode is encode's inverse' \
333                   falsified in 68 attempts
334               #     Counterexample:
335               #     $s = "0";
336               #     Notes:
337               #     $s_enc     = "";
338               #     $s_enc_dec = "";
339
340           Notes can help you debug your code when something goes wrong.  Use
341           them as debugging hints to yourself.  For example, you can use
342           notes to record the output of each stage of a multi-stage test.
343           That way, if the test fails, you can see what happened in each
344           stage without having to plug the counterexample into your code
345           under a debugger.
346
347           If you want to include complicated values or data structures in
348           your notes, see the "dump" method, next, which may be more
349           appropriate.
350
351       dump(value, name)
352               Property {
353                 ##[ s <- String ]##
354                 my $s_enc     = encode($s);
355                 my $s_enc_dec = decode($s_enc);
356                 $tcon->dump($s_enc, "s_enc");
357                 $tcon->dump($s_enc_dec, "s_enc_dec");
358                 $s eq $s_enc_dec;
359               }, name => "decode is encode's inverse" ;
360
361           Adds a note to the current trial in which the given value is
362           dumped.  The value will be dumped via Data::Dumper and thus may be
363           complex and contain weird control characters and so on.  If you
364           supply a name, it will be used to name the dumped value.  Returns
365           value as its result.
366
367           In the event that the trial fails, the note (and any others) will
368           be emitted as part of the counterexample.
369
370           See "note" above for more.
371

SEE ALSO

373       Test::LectroTest::Property explains in detail what you can put inside
374       of your property specifications.
375
376       Test::LectroTest::RegressionTesting explains how to test for
377       regressions and corner cases using LectroTest.
378
379       Test::Harness:TAP documents the Test Anything Protocol, Perl's simple
380       text-based interface between testing modules such as Test::LectroTest
381       and the test harness Test::Harness.
382

AUTHOR

384       Tom Moertel (tom@moertel.com)
385

INSPIRATION

387       The LectroTest project was inspired by Haskell's QuickCheck module by
388       Koen Claessen and John Hughes:
389       http://www.cs.chalmers.se/~rjmh/QuickCheck/.
390
392       Copyright (c) 2004-13 by Thomas G Moertel.  All rights reserved.
393
394       This program is free software; you can redistribute it and/or modify it
395       under the same terms as Perl itself.
396
397
398
399perl v5.30.1                      2020-01-30   Test::LectroTest::TestRunner(3)
Impressum