1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
3
4

NAME

6       Config::Model::Tester - Test framework for Config::Model
7

VERSION

9       version 4.006
10

SYNOPSIS

12       In your test file (typically "t/model_test.t"):
13
14        use warnings;
15        use strict;
16
17        use Config::Model::Tester ;
18        use ExtUtils::testlib;
19
20        run_tests() ;
21
22       Run tests with:
23
24        perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25

DESCRIPTION

27       This class provides a way to test configuration models with tests
28       files.  This class was designed to tests several models and run several
29       tests cases per model.
30
31       A specific layout for test files must be followed.
32
33   Sub test specification
34       Each subtest is defined in a file like:
35
36        t/model_tests.d/<app-name>-test-conf.pl
37
38       This file specifies that "app-name" (which is defined in
39       "lib/Config/Model/*.d" directory) is used for the test cases defined in
40       the "*-test-conf.pl" file. The model to test is inferred from the
41       application name to test.
42
43       This file contains a list of test case (explained below) and expects a
44       set of files used as test data. The layout of these test data files is
45       explained in next section.
46
47   Simple test file layout
48       Each test case is represented by a configuration file (not a directory)
49       in the "*-examples" directory. This configuration file is used by the
50       model to test and is copied as "$confdir/$conf_file_name" using the
51       test data structure explained below.
52
53       In the example below, we have 1 app model to test: "lcdproc" and 2
54       tests cases. The app name matches the file specified in
55       "lib/Config/Model/*.d" directory. In this case, the app name matches
56       "lib/Config/Model/system.d/lcdproc"
57
58        t
59        |-- model_test.t
60        \-- model_tests.d           # do not change directory name
61            |-- lcdproc-test-conf.pl   # subtest specification for lcdproc app
62            \-- lcdproc-examples
63                |-- t0              # test case t0
64                \-- LCDD-0.5.5      # test case for older LCDproc
65
66       Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67       this module looks for files named like "<app-name>-test-conf.pl>").
68
69       Subtests data is provided in files in directory "lcdproc-examples" (
70       i.e. this modules looks for test data in directory
71       "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72       so that each file is used as a "/etc/LCDd.conf" file during each test
73       case.
74
75       "lcdproc-test-conf.pl" can contain specifications for more test cases.
76       Each test case requires a new file in "lcdproc-examples" directory.
77
78       See "Examples" for a link to the actual LCDproc model tests
79
80   Test file layout for multi-file configuration
81       When a configuration is spread over several files, each test case is
82       provided in a sub-directory. This sub-directory is copied in "conf_dir"
83       (a test parameter as explained below)
84
85       In the example below, the test specification is written in
86       "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87       "dpkg-test-conf.pl" contains instructions so that each directory under
88       "dpkg-examples" is used.
89
90        t/model_tests.d
91        \-- dpkg-test-conf.pl         # subtest specification
92        \-- dpkg-examples
93            \-- libversion            # example subdir, used as test case name
94                \-- debian            # directory for used by test case
95                    |-- changelog
96                    |-- compat
97                    |-- control
98                    |-- copyright
99                    |-- rules
100                    |-- source
101                    |   \-- format
102                    \-- watch
103
104       See "Examples" for a link to the (many) Dpkg model tests
105
106   More complex file layout
107       Each test case is a sub-directory on the "*-examples" directory and
108       contains several files. The destination of the test files may depend on
109       the system (e.g. the OS). For instance, system wide "ssh_config" is
110       stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111
112       These files are copied in a test directory using a "setup" parameter in
113       test case specification:
114
115         setup => {
116           test_file_in_example_dir => 'destination'
117         }
118
119       Let's consider this example of 2 tests cases for ssh:
120
121        t/model_tests.d/
122        |-- ssh-test-conf.pl
123        |-- ssh-examples
124            \-- basic
125                |-- system_ssh_config
126                \-- user_ssh_config
127
128       Unfortunately, "user_ssh_config" is a user file, so you need to specify
129       where is located the home directory of the test with another global
130       parameter:
131
132         home_for_test => '/home/joe' ;
133
134       For Linux only, the "setup" parameter is:
135
136        setup => {
137          system_ssh_config => '/etc/ssh/ssh_config',
138          user_ssh_config   => "~/.ssh/config"
139        }
140
141       On the other hand, system wide config file is different on MacOS and
142       the test file must be copied in the correct location. When the value of
143       the "setup" hash is another hash, the key of this other hash is used as
144       to specify the target location for other OS (as returned by Perl $^O
145       variable:
146
147             setup => {
148               'system_ssh_config' => {
149                   'darwin' => '/etc/ssh_config',
150                   'default' => '/etc/ssh/ssh_config',
151               },
152               'user_ssh_config' => "~/.ssh/config"
153             }
154
155       "systemd" is another beast where configuration files can be symlinks to
156       "/dev/null" or other files. To emulate this situation, use an array as
157       setup target:
158
159         setup => {
160             # test data file => [ link (may be repeated), ..       link(s) target contains test data ]
161             'ssh.service' => [ '/etc/systemd/system/sshd.service', '/lib/systemd/system/ssh.service' ]
162         }
163
164       This will result in a symlink like:
165
166          wr_root/model_tests/test-sshd-service/etc/systemd/system/sshd.service
167          -> /absolute_path_to/wr_root/model_tests/test-sshd-service/lib/systemd/system/ssh.service
168
169       See the actual Ssh and Sshd model tests
170       <https://github.com/dod38fr/config-model-
171       openssh/tree/master/t/model_tests.d>
172
173   Basic test specification
174       Each model subtest is specified in "<app>-test-conf.pl". This file must
175       return a data structure containing the test specifications. Each test
176       data structure contains global parameters (Applied to all tests cases)
177       and test cases parameters (parameters are applied to the test case)
178
179        use strict;
180        use warnings;
181        {
182          # global parameters
183
184          # config file name (used to copy test case into test wr_root/model_tests directory)
185          conf_file_name => "fstab",
186          # config dir where to copy the file (optional)
187          conf_dir => "etc",
188          # home directory for this test
189          home_for_test => '/home/joe'
190
191          tests =>  [
192            {
193              # test case 1
194              name => 'my_first_test',
195              # other test case parameters
196            },
197            {
198              # test case 2
199              name => 'my_second_test',
200              # other test case parameters
201            },
202            # ...
203          ],
204        };
205
206        # do not add 1; at the end of the file
207
208       In the example below, "t0" file is copied in
209       "wr_root/model_tests/test-t0/etc/fstab".
210
211        use strict;
212        use warnings;
213        {
214          # list of tests.
215          tests => [
216            {
217              # test name
218              name => 't0',
219              # add optional specification here for t0 test
220            },
221            {
222              name => 't1',
223              # add optional specification here for t1 test
224            },
225          ]
226        };
227
228       You can suppress warnings by specifying "no_warnings => 1" in each test
229       case. On the other hand, you may also want to check for warnings
230       specified to your model. In this case, you should avoid specifying
231       "no_warnings" here and specify warning tests or warning filters as
232       mentioned below.
233
234       See actual fstab test <https://github.com/dod38fr/config-
235       model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
236
237   Skip a test
238       A test file can be skipped using "skip" global test parameter.
239
240       In this example, test is skipped when not running on a Debian system:
241
242        eval { require AptPkg::Config; };
243        my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
244
245        {
246          skip => $skip,
247          tests => [ ] ,
248        };
249
250   Internal tests or backend tests
251       Some tests require the creation of a configuration class dedicated for
252       test (typically to test corner cases on a backend).
253
254       This test class can be created directly in the test specification by
255       specifying tests classes in "config_classes" global test parameter in
256       an array ref. Each array element is a data structure that use
257       create_config_class parameters.  See for instance the layer test
258       <https://github.com/dod38fr/config-
259       model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
260       shellvar backend <https://github.com/dod38fr/config-
261       model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
262
263       In this case, no application exist for such classes so the model to
264       test must be specified in a global test parameter:
265
266         return {
267           config_classes => [ { name => "Foo", element => ... } , ... ],
268           model_to_test => "Foo",
269           tests => [ ... ]
270         };
271
272   Test specification with arbitrary file names
273       In some models, like "Multistrap", the config file is chosen by the
274       user. In this case, the file name must be specified for each tests
275       case:
276
277        {
278          tests => [ {
279              name        => 'arm',
280              config_file => '/home/foo/my_arm.conf',
281              check       => {},
282           }]
283        };
284
285       See the actual multistrap test <https://github.com/dod38fr/config-
286       model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
287
288   Backend argument
289       Some application like systemd requires a backend argument specified by
290       user (e.g. a service name for systemd). The parameter "backend_arg" can
291       be specified to emulate this behavior.
292
293   Re-use test data
294       When the input data for test is quite complex (several files), it may
295       be interesting to re-use these data for other test cases. Knowing that
296       test names must be unique, you can re-use test data with "data_from"
297       parameter. For instance:
298
299         tests => [
300           {
301               name  => 'some-test',
302               # ...
303           },
304           {
305               name  => 'some-other-test',
306               data_from  => 'some-test',    # re-use data from test above
307               # ...
308           },
309         ]
310
311       See plainfile backend test <https://github.com/dod38fr/config-
312       model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
313       real life example.
314
315   Test scenario
316       Each subtest follow a sequence explained below. Each step of this
317       sequence may be altered by adding test case parameters in
318       "<model-to-test>-test-conf.pl":
319
320       •   Setup test in "wr_root/model_tests/<subtest name>/". If your
321           configuration file layout depend on the target system, you will
322           have to specify the path using "setup" parameter.  See "Test file
323           layout depending on system".
324
325       •   Create configuration instance, load config data and check its
326           validity. Use "load_check => 'no'" if your file is not valid.
327
328       •   Check for config data warnings. You should pass the list of
329           expected warnings that are emitted through Log::Log4Perl. The array
330           ref is passed as is to the "expect" function of "expect" in
331           Test::Log::Lo4Perl. E.g:
332
333               log4perl_load_warnings => [
334                    [ 'Tree.Node', (warn => qr/deprecated/) x 2 ]  ,
335                    [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
336               ]
337
338           The Log classes are specified in "cme/Logging".
339
340           Log levels below "warn" are ignored.
341
342           Note that log tests are disabled when "--log" option is used, hence
343           all warnings triggered by the tests are shown.
344
345           Config::Model is currently transitioning from traditional "warn" to
346           warn logs. To avoid breaking all tests based on this module, the
347           warnings are emitted through Log::Log4Perl only when
348           $::_use_log4perl_to_warn is set. This hack will be removed once all
349           warnings checks in tests are ported to log4perl checks.
350
351       •   DEPRECATED. Check for config data warning. You should pass the list
352           of expected warnings.  E.g.
353
354               load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
355
356           Use an empty array_ref to mask load warnings.
357
358       •   Optionally run update command:
359
360               update => {
361                    [ returns => 'foo' , ]
362                    no_warnings => [ 0 | 1 ], # default 0
363                    quiet => [ 0 | 1], # default 0, passed to update method
364                    loag4perl_update_warnings => [ ... ] # Test::Log::Log4Perl::expect arguments
365            }
366
367           Where:
368
369           •   "returns" is the expected return value (optional).
370
371           •   "no_warnings" to suppress the warnings coming from
372               Config::Model::Value. Note that "no_warnings => 1" may be
373               useful for less verbose test.
374
375           •   "quiet" to suppress progress messages during update.
376
377           •   "log4perl_update_warnings" is used to check the warnings
378               produced during update. The argument is passed to "expect"
379               function of Test::Log::Log4Perl. See "load_warnings" parameter
380               above for more details.
381
382           •   DEPRECATED. "update_warnings" is an array ref of quoted regexp
383               (See qr operator) to check the warnings produced during update.
384               use "update => []" to check that no warnings are issued during
385               update.
386
387           All other arguments are passed to "update" method.
388
389       •   Optionally load configuration data. You should design this config
390           data to suppress any error or warning mentioned above. E.g:
391
392               load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
393
394           See Config::Model::Loader for the syntax of the string accepted by
395           "load" parameter.
396
397       •   Optionally, run a check before running apply_fix (if any). This
398           step is useful to check warning messages:
399
400              check_before_fix => {
401                 dump_errors   => [ ... ] # optional, see below
402                 load4perl_dump_warnings => [ ... ] # optional, see below
403              }
404
405           Use "dump_errors" if you expect issues:
406
407             check_before_fix => {
408               dump_errors =>  [
409                   # the issues  and a way to fix the issue using Config::Model::Node::load
410                   qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
411                   qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
412               ],
413             }
414
415           Likewise, specify any expected warnings:
416
417             check_before_fix => {
418                   log4perl_dump_warnings => [ ... ],
419             }
420
421           "log4perl_dump_warnings" passes the array ref content to "expect"
422           function of Test::Log::Log4Perl.
423
424           Both "log4perl_dump_warnings" and "dump_errors" can be specified in
425           "check_before_fix" hash.
426
427       •   Optionally, call apply_fixes:
428
429               apply_fix => 1,
430
431       •   Call dump_tree to check the validity of the data after optional
432           "apply_fix". This step is not optional.
433
434           As with "check_before_fix", both "dump_errors" or "dump_warnings"
435           can be used.
436
437       •   Run specific content check to verify that configuration data was
438           retrieved correctly:
439
440               check => {
441                   'fs:/proc fs_spec' => "proc",
442                   'fs:/proc fs_file' => "/proc",
443                   'fs:/home fs_file' => "/home",
444               },
445
446           The keys of the hash points to the value to be checked using the
447           syntax described in "grab" in Config::Model::Role::Grab.
448
449           Multiple check on the same item can be applied with a array ref:
450
451               check => [
452                   Synopsis => 'fix undefined path_max for st_size zero',
453                   Description => [ qr/^The downstream/,  qr/yada yada/ ]
454               ]
455
456           You can run check using different check modes (See "fetch" in
457           Config::Model::Value) by passing a hash ref instead of a scalar :
458
459               check  => {
460                   'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
461                   'sections:base packages:0'   => { mode => 'layered', value => "gcc-4.2-base' },
462               },
463
464           The whole hash content (except "value") is passed to  grab and
465           fetch
466
467           A regexp can also be used to check value:
468
469              check => {
470                 "License text" => qr/gnu/i,
471              }
472
473           And specification can nest hash or array style:
474
475              check => {
476                 "License:0 text" => qr/gnu/i,
477                 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
478                 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
479                 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
480              }
481
482       •   Verify if a hash contains one or more keys (or keys matching a
483           regexp):
484
485            has_key => [
486               'sections' => 'debian', # sections must point to a hash element
487               'control' => [qw/source binary/],
488               'copyright Files' => qr/.c$/,
489               'copyright Files' => [qr/\.h$/], qr/\.c$/],
490            ],
491
492       •   Verify that a hash does not have a key (or a key matching a
493           regexp):
494
495            has_not_key => [
496               'copyright Files' => qr/.virus$/ # silly, isn't ?
497            ],
498
499       •   Verify annotation extracted from the configuration file comments:
500
501               verify_annotation => {
502                       'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
503                       'source Maintainer' => "what a fine\nteam this one is",
504                   },
505
506       •   Write back the config data in "wr_root/model_tests/<subtest
507           name>/".  Note that write back is forced, so the tested
508           configuration files are written back even if the configuration
509           values were not changed during the test.
510
511           You can skip warning when writing back with the global :
512
513               no_warnings => 1,
514
515       •   Check the content of the written files(s) with
516           Test::File::Contents. Tests can be grouped in an array ref:
517
518              file_contents => {
519                       "/home/foo/my_arm.conf" => "really big string" ,
520                       "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
521                   }
522
523              file_contents_like => {
524                       "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
525              }
526
527              file_contents_unlike => {
528                       "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
529              }
530
531       •   Check the mode of the written files:
532
533             file_mode => {
534                "~/.ssh/ssh_config"     => oct(600), # better than 0600
535                "debian/stuff.postinst" => oct(755),
536             }
537
538           Only the last four octets of the mode are tested. I.e. the test is
539           done with " $file_mode & oct(7777) "
540
541           Note: this test is skipped on Windows
542
543       •   Check added or removed configuration files. If you expect changes,
544           specify a subref to alter the file list:
545
546               file_check_sub => sub {
547                   my $list_ref = shift ;
548                   # file added during tests
549                   push @$list_ref, "/debian/source/format" ;
550               },
551
552           Note that actual and expected file lists are sorted before check,
553           adding a file can be done with "push".
554
555       •   Copy all config data from "wr_root/model_tests/<subtest name>/" to
556           "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
557           check that configuration written back has the same content as the
558           original configuration.
559
560       •   Create a second configuration instance to read the conf file that
561           was just copied (configuration data is checked.)
562
563       •   You can skip the load check if the written file still contain
564           errors (e.g.  some errors were ignored and cannot be fixed) with
565           "load_check2 => 'no'"
566
567       •   Optionally load configuration data in the second instance. You
568           should design this config data to suppress any error or warning
569           that occur in the step below. E.g:
570
571               load2 => 'binary:seaview',
572
573           See Config::Model::Loader for the syntax of the string accepted by
574           "load2" parameter.
575
576       •   Compare data read from original data.
577
578       •   Run specific content check on the written config file to verify
579           that configuration data was written and retrieved correctly:
580
581               wr_check => {
582                   'fs:/proc fs_spec' =>          "proc" ,
583                   'fs:/proc fs_file' =>          "/proc",
584                   'fs:/home fs_file' =>          "/home",
585               },
586
587           Like the "check" item explained above, you can run "wr_check" using
588           different check modes.
589
590   Running the test
591       Run all tests with one of these commands:
592
593        prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
594        perl -Ilib t/model_test.t  [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
595
596       By default, all tests are run on all models.
597
598       You can pass arguments to "t/model_test.t":
599
600       •   Optional parameters: "--trace" to get test traces. "--error" to get
601           stack trace in case of errors, "--log" to have logs. E.g.
602
603             # run with log and error traces
604             prove -lv t/model_test.t :: --error --logl
605
606       •   The model name to tests. E.g.:
607
608             # run only fstab tests
609             prove -lv t/model_test.t :: fstab
610
611       •   A regexp to filter subtest E.g.:
612
613             # run only fstab tests foobar subtest
614             prove -lv t/model_test.t :: fstab foobar
615
616             # run only fstab tests foo subtest
617             prove -lv t/model_test.t :: fstab '^foo$'
618

Examples

620       Some of these examples may still use global variables (which is
621       deprecated). Such files may be considered as buggy after Aug 2019.
622       Please warn the author if you find one.
623
624       •   LCDproc <http://lcdproc.org> has a single configuration file:
625           "/etc/LCDd.conf". Here's LCDproc test layout
626           <https://github.com/dod38fr/config-model-
627           lcdproc/tree/master/t/model_tests.d> and the test specification
628           <https://github.com/dod38fr/config-model-
629           lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
630
631       •   Dpkg packages are constructed from several files. These files are
632           handled like configuration files by Config::Model::Dpkg. The test
633           layout <http://anonscm.debian.org/gitweb/?p=pkg-
634           perl/packages/libconfig-model-dpkg-
635           perl.git;a=tree;f=t/model_tests.d;hb=HEAD> features test with
636           multiple file in dpkg-examples
637           <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
638           model-dpkg-perl.git;a=tree;f=t/model_tests.d/dpkg-
639           examples;hb=HEAD>.  The test is specified in dpkg-test-conf.pl
640           <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
641           model-dpkg-perl.git;a=blob_plain;f=t/model_tests.d/dpkg-test-
642           conf.pl;hb=HEAD>
643
644       •   multistrap-test-conf.pl <https://github.com/dod38fr/config-
645           model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
646           multistrap-examples <https://github.com/dod38fr/config-
647           model/tree/master/t/model_tests.d/multistrap-examples> specify a
648           test where the configuration file name is not imposed by the
649           application. The file name must then be set in the test
650           specification.
651
652       •   backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
653           model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
654           a more complex example showing how to test a backend. The test is
655           done creating a dummy model within the test specification.
656

CREDITS

658       In alphabetical order:
659
660       •   Cyrille Bollu
661
662       Many thanks for your help.
663

SEE ALSO

665       •   Config::Model
666
667       •   Test::More
668

AUTHOR

670       Dominique Dumont
671
673       This software is Copyright (c) 2013-2020 by Dominique Dumont.
674
675       This is free software, licensed under:
676
677         The GNU Lesser General Public License, Version 2.1, February 1999
678

SUPPORT

680   Websites
681       The following websites have more information about this module, and may
682       be of help to you. As always, in addition to those websites please use
683       your favorite search engine to discover more resources.
684
685       •   CPANTS
686
687           The CPANTS is a website that analyzes the Kwalitee ( code metrics )
688           of a distribution.
689
690           <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
691
692       •   CPAN Testers
693
694           The CPAN Testers is a network of smoke testers who run automated
695           tests on uploaded CPAN distributions.
696
697           <http://www.cpantesters.org/distro/C/Config-Model-Tester>
698
699       •   CPAN Testers Matrix
700
701           The CPAN Testers Matrix is a website that provides a visual
702           overview of the test results for a distribution on various
703           Perls/platforms.
704
705           <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
706
707       •   CPAN Testers Dependencies
708
709           The CPAN Testers Dependencies is a website that shows a chart of
710           the test results of all dependencies for a distribution.
711
712           <http://deps.cpantesters.org/?module=Config::Model::Tester>
713
714   Bugs / Feature Requests
715       Please report any bugs or feature requests by email to "ddumont at
716       cpan.org", or through the web interface at
717       <https://github.com/dod38fr/config-model-tester/issues>. You will be
718       automatically notified of any progress on the request by the system.
719
720   Source Code
721       The code is open to the world, and available for you to hack on. Please
722       feel free to browse it and play with it, or whatever. If you want to
723       contribute patches, please send me a diff or prod me to pull from your
724       repository :)
725
726       <http://github.com/dod38fr/config-model-tester.git>
727
728         git clone git://github.com/dod38fr/config-model-tester.git
729
730
731
732perl v5.34.0                      2021-05-24          Config::Model::Tester(3)
Impressum