1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
6 Config::Model::Tester - Test framework for Config::Model
7
9 version 4.007
10
12 In your test file (typically "t/model_test.t"):
13 use warnings;
15 use strict;
16 use Config::Model::Tester ;
18 use ExtUtils::testlib;
19 run_tests() ;
21 Run tests with:
23 perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25
27 This class provides a way to test configuration models with tests
28 files. This class was designed to tests several models and run several
29 tests cases per model.
30 A specific layout for test files must be followed.
32 Sub test specification
34 Each subtest is defined in a file like:
35 t/model_tests.d/<app-name>-test-conf.pl
37 This file specifies that "app-name" (which is defined in
39 "lib/Config/Model/*.d" directory) is used for the test cases defined in
40 the "*-test-conf.pl" file. The model to test is inferred from the
41 application name to test.
42 This file contains a list of test case (explained below) and expects a
44 set of files used as test data. The layout of these test data files is
45 explained in next section.
46 Simple test file layout
48 Each test case is represented by a configuration file (not a directory)
49 in the "*-examples" directory. This configuration file is used by the
50 model to test and is copied as "$confdir/$conf_file_name" using the
51 test data structure explained below.
52 In the example below, we have 1 app model to test: "lcdproc" and 2
54 tests cases. The app name matches the file specified in
55 "lib/Config/Model/*.d" directory. In this case, the app name matches
56 "lib/Config/Model/system.d/lcdproc"
57 t
59 |-- model_test.t
60 \-- model_tests.d # do not change directory name
61 |-- lcdproc-test-conf.pl # subtest specification for lcdproc app
62 \-- lcdproc-examples
63 |-- t0 # test case t0
64 \-- LCDD-0.5.5 # test case for older LCDproc
65 Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67 this module looks for files named like "<app-name>-test-conf.pl>").
68 Subtests data is provided in files in directory "lcdproc-examples" (
70 i.e. this modules looks for test data in directory
71 "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72 so that each file is used as a "/etc/LCDd.conf" file during each test
73 case.
74 "lcdproc-test-conf.pl" can contain specifications for more test cases.
76 Each test case requires a new file in "lcdproc-examples" directory.
77 See "Examples" for a link to the actual LCDproc model tests
79 Test file layout for multi-file configuration
81 When a configuration is spread over several files, each test case is
82 provided in a sub-directory. This sub-directory is copied in "conf_dir"
83 (a test parameter as explained below)
84 In the example below, the test specification is written in
86 "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87 "dpkg-test-conf.pl" contains instructions so that each directory under
88 "dpkg-examples" is used.
89 t/model_tests.d
91 \-- dpkg-test-conf.pl # subtest specification
92 \-- dpkg-examples
93 \-- libversion # example subdir, used as test case name
94 \-- debian # directory for used by test case
95 |-- changelog
96 |-- compat
97 |-- control
98 |-- copyright
99 |-- rules
100 |-- source
101 | \-- format
102 \-- watch
103 See "Examples" for a link to the (many) Dpkg model tests
105 More complex file layout
107 Each test case is a sub-directory on the "*-examples" directory and
108 contains several files. The destination of the test files may depend on
109 the system (e.g. the OS). For instance, system wide "ssh_config" is
110 stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111 These files are copied in a test directory using a "setup" parameter in
113 test case specification.
114 Let's consider this example of 2 tests cases for ssh:
116 t/model_tests.d/
118 |-- ssh-test-conf.pl
119 |-- ssh-examples
120 \-- basic
121 |-- system_ssh_config
122 \-- user_ssh_config
123 Unfortunately, "user_ssh_config" is a user file, so you need to specify
125 where is located the home directory of the test with another global
126 parameter:
127 home_for_test => '/home/joe' ;
129 For Linux only, the "setup" parameter is:
131 setup => {
133 system_ssh_config => '/etc/ssh/ssh_config',
134 user_ssh_config => "~/.ssh/config"
135 }
136 On the other hand, system wide config file is different on MacOS and
138 the test file must be copied in the correct location. When the value of
139 the "setup" hash is another hash, the key of this other hash is used as
140 to specify the target location for other OS (as returned by Perl $^O
141 variable:
142 setup => {
144 'system_ssh_config' => {
145 'darwin' => '/etc/ssh_config',
146 'default' => '/etc/ssh/ssh_config',
147 },
148 'user_ssh_config' => "~/.ssh/config"
149 }
150 "systemd" is another beast where configuration files can be symlinks to
152 "/dev/null" or other files. To emulate this situation, use an array as
153 setup target:
154 setup => {
156 # test data file => [ link (may be repeated), .. link(s) target contains test data ]
157 'ssh.service' => [ '/etc/systemd/system/sshd.service', '/lib/systemd/system/ssh.service' ]
158 }
159 This will result in a symlink like:
161 wr_root/model_tests/test-sshd-service/etc/systemd/system/sshd.service
163 -> /absolute_path_to/wr_root/model_tests/test-sshd-service/lib/systemd/system/ssh.service
164 See the actual Ssh and Sshd model tests
166 <https://github.com/dod38fr/config-model-
167 openssh/tree/master/t/model_tests.d>
168 Basic test specification
170 Each model subtest is specified in "<app>-test-conf.pl". This file must
171 return a data structure containing the test specifications. Each test
172 data structure contains global parameters (Applied to all tests cases)
173 and test cases parameters (parameters are applied to the test case)
174 use strict;
176 use warnings;
177 {
178 # global parameters
179 # config file name (used to copy test case into test wr_root/model_tests directory)
181 conf_file_name => "fstab",
182 # config dir where to copy the file (optional)
183 conf_dir => "etc",
184 # home directory for this test
185 home_for_test => '/home/joe'
186 tests => [
188 {
189 # test case 1
190 name => 'my_first_test',
191 # other test case parameters
192 },
193 {
194 # test case 2
195 name => 'my_second_test',
196 # other test case parameters
197 },
198 # ...
199 ],
200 };
201 # do not add 1; at the end of the file
203 In the example below, "t0" file is copied in
205 "wr_root/model_tests/test-t0/etc/fstab".
206 use strict;
208 use warnings;
209 {
210 # list of tests.
211 tests => [
212 {
213 # test name
214 name => 't0',
215 # add optional specification here for t0 test
216 },
217 {
218 name => 't1',
219 # add optional specification here for t1 test
220 },
221 ]
222 };
223 You can suppress warnings by specifying "no_warnings => 1" in each test
225 case. On the other hand, you may also want to check for warnings
226 specified to your model. In this case, you should avoid specifying
227 "no_warnings" here and specify warning tests or warning filters as
228 mentioned below.
229 See actual fstab test <https://github.com/dod38fr/config-
231 model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
232 Skip a test
234 A test file can be skipped using "skip" global test parameter.
235 In this example, test is skipped when not running on a Debian system:
237 eval { require AptPkg::Config; };
239 my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
240 {
242 skip => $skip,
243 tests => [ ] ,
244 };
245 Internal tests or backend tests
247 Some tests require the creation of a configuration class dedicated for
248 test (typically to test corner cases on a backend).
249 This test class can be created directly in the test specification by
251 specifying tests classes in "config_classes" global test parameter in
252 an array ref. Each array element is a data structure that use
253 create_config_class parameters. See for instance the layer test
254 <https://github.com/dod38fr/config-
255 model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
256 shellvar backend <https://github.com/dod38fr/config-
257 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
258 In this case, no application exist for such classes so the model to
260 test must be specified in a global test parameter:
261 return {
263 config_classes => [ { name => "Foo", element => ... } , ... ],
264 model_to_test => "Foo",
265 tests => [ ... ]
266 };
267 Test specification with arbitrary file names
269 In some models, like "Multistrap", the config file is chosen by the
270 user. In this case, the file name must be specified for each tests
271 case:
272 {
274 tests => [ {
275 name => 'arm',
276 config_file => '/home/foo/my_arm.conf',
277 check => {},
278 }]
279 };
280 See the actual multistrap test <https://github.com/dod38fr/config-
282 model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
283 Backend argument
285 Some application like systemd requires a backend argument specified by
286 user (e.g. a service name for systemd). The parameter "backend_arg" can
287 be specified to emulate this behavior.
288 Re-use test data
290 When the input data for test is quite complex (several files), it may
291 be interesting to re-use these data for other test cases. Knowing that
292 test names must be unique, you can re-use test data with "data_from"
293 parameter. For instance:
294 tests => [
296 {
297 name => 'some-test',
298 # ...
299 },
300 {
301 name => 'some-other-test',
302 data_from => 'some-test', # re-use data from test above
303 # ...
304 },
305 ]
306 See plainfile backend test <https://github.com/dod38fr/config-
308 model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
309 real life example.
310 Likewise, it may be useful to re-use test data from another group of
312 test. Lets see this example from "systemd-service-test-conf.pl":
313 {
315 name => 'transmission',
316 data_from_group => 'systemd', # i.e from ../systemd-examples
317 }
318 "data_from" and "data_from_group" can be together.
320 Test scenario
322 Each subtest follow a sequence explained below. Each step of this
323 sequence may be altered by adding test case parameters in
324 "<model-to-test>-test-conf.pl":
325 • Setup test in "wr_root/model_tests/<subtest name>/". If your
327 configuration file layout depend on the target system, you will
328 have to specify the path using "setup" parameter. See "More
329 complex file layout".
330 • Create configuration instance, load config data and check its
332 validity. Use "load_check => 'no'" if your file is not valid.
333 • Check for config data warnings. You should pass the list of
335 expected warnings that are emitted through Log::Log4perl. The array
336 ref is passed as is to the "expect" function of "expect" in
337 Test::Log::Lo4Perl. E.g:
338 log4perl_load_warnings => [
340 [ 'Tree.Node', (warn => qr/deprecated/) x 2 ] ,
341 [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
342 ]
343 The Log classes are specified in "cme/Logging".
345 Log levels below "warn" are ignored.
347 Note that log tests are disabled when "--log" option is used, hence
349 all warnings triggered by the tests are shown.
350 Config::Model is currently transitioning from traditional "warn" to
352 warn logs. To avoid breaking all tests based on this module, the
353 warnings are emitted through Log::Log4perl only when
354 $::_use_log4perl_to_warn is set. This hack will be removed once all
355 warnings checks in tests are ported to log4perl checks.
356 • DEPRECATED. Check for config data warning. You should pass the list
358 of expected warnings. E.g.
359 load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
361 Use an empty array_ref to mask load warnings.
363 • Optionally run update command:
365 update => {
367 returns => 'foo' , # optional
368 no_warnings => [ 0 | 1 ], # default 0
369 quiet => [ 0 | 1], # default 0, passed to update method
370 load4perl_update_warnings => [ ... ] # Test::Log::Log4perl::expect arguments
371 }
372 Where:
374 • "returns" is the expected return value (optional).
376 • "no_warnings" can be used to suppress the warnings coming from
378 Config::Model::Value. Note that "no_warnings => 1" may be
379 useful for less verbose test.
380 • "quiet" to suppress progress messages during update.
382 • "log4perl_update_warnings" is used to check the warnings
384 produced during update. The argument is passed to "expect"
385 function of Test::Log::Log4perl. See "load_warnings" parameter
386 above for more details.
387 • DEPRECATED. "update_warnings" is an array ref of quoted regexp
389 (See qr operator) to check the warnings produced during update.
390 Please use "log4perl_update_warnings" instead.
391 All other arguments are passed to "update" method.
393 • Optionally load configuration data. You should design this config
395 data to suppress any error or warning mentioned above. E.g:
396 load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
398 See Config::Model::Loader for the syntax of the string accepted by
400 "load" parameter.
401 • Optionally, run a check before running apply_fix (if any). This
403 step is useful to check warning messages:
404 check_before_fix => {
406 dump_errors => [ ... ] # optional, see below
407 log4perl_dump_warnings => [ ... ] # optional, see below
408 }
409 Use "dump_errors" if you expect issues:
411 check_before_fix => {
413 dump_errors => [
414 # the issues and a way to fix the issue using Config::Model::Node::load
415 qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
416 qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
417 ],
418 }
419 Likewise, specify any expected warnings:
421 check_before_fix => {
423 log4perl_dump_warnings => [ ... ],
424 }
425 "log4perl_dump_warnings" passes the array ref content to "expect"
427 function of Test::Log::Log4perl.
428 Both "log4perl_dump_warnings" and "dump_errors" can be specified in
430 "check_before_fix" hash.
431 • Optionally, call apply_fixes:
433 apply_fix => 1,
435 • Call dump_tree to check the validity of the data after optional
437 "apply_fix". This step is not optional.
438 As with "check_before_fix", both "dump_errors" or
440 "log4perl_dump_warnings" can be specified in "full_dump" parameter:
441 full_dump => {
443 log4perl_dump_warnings => [ ... ], # optional
444 dump_errors => [ ... ], # optional
445 }
446 • Run specific content check to verify that configuration data was
448 retrieved correctly:
449 check => {
451 'fs:/proc fs_spec' => "proc",
452 'fs:/proc fs_file' => "/proc",
453 'fs:/home fs_file' => "/home",
454 },
455 The keys of the hash points to the value to be checked using the
457 syntax described in "grab" in Config::Model::Role::Grab.
458 Multiple check on the same item can be applied with a array ref:
460 check => [
462 Synopsis => 'fix undefined path_max for st_size zero',
463 Description => [ qr/^The downstream/, qr/yada yada/ ]
464 ]
465 You can run check using different check modes (See "fetch" in
467 Config::Model::Value) by passing a hash ref instead of a scalar :
468 check => {
470 'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
471 'sections:base packages:0' => { mode => 'layered', value => "gcc-4.2-base' },
472 },
473 The whole hash content (except "value") is passed to grab and
475 fetch
476 A regexp can also be used to check value:
478 check => {
480 "License text" => qr/gnu/i,
481 }
482 And specification can nest hash or array style:
484 check => {
486 "License:0 text" => qr/gnu/i,
487 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
488 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
489 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
490 }
491 • Verify if a hash contains one or more keys (or keys matching a
493 regexp):
494 has_key => [
496 'sections' => 'debian', # sections must point to a hash element
497 'control' => [qw/source binary/],
498 'copyright Files' => qr/.c$/,
499 'copyright Files' => [qr/\.h$/], qr/\.c$/],
500 ],
501 • Verify that a hash does not have a key (or a key matching a
503 regexp):
504 has_not_key => [
506 'copyright Files' => qr/.virus$/ # silly, isn't ?
507 ],
508 • Verify annotation extracted from the configuration file comments:
510 verify_annotation => {
512 'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
513 'source Maintainer' => "what a fine\nteam this one is",
514 },
515 • Write back the config data in "wr_root/model_tests/<subtest
517 name>/". Note that write back is forced, so the tested
518 configuration files are written back even if the configuration
519 values were not changed during the test.
520 You can skip warning when writing back with the global :
522 no_warnings => 1,
524 • Check the content of the written files(s) with
526 Test::File::Contents. Tests can be grouped in an array ref:
527 file_contents => {
529 "/home/foo/my_arm.conf" => "really big string" ,
530 "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
531 }
532 file_contents_like => {
534 "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
535 }
536 file_contents_unlike => {
538 "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
539 }
540 • Check the mode of the written files:
542 file_mode => {
544 "~/.ssh/ssh_config" => oct(600), # better than 0600
545 "debian/stuff.postinst" => oct(755),
546 }
547 Only the last four octets of the mode are tested. I.e. the test is
549 done with " $file_mode & oct(7777) "
550 Note: this test is skipped on Windows
552 • Check added or removed configuration files. If you expect changes,
554 specify a subref to alter the file list:
555 file_check_sub => sub {
557 my $list_ref = shift ;
558 # file added during tests
559 push @$list_ref, "/debian/source/format" ;
560 },
561 Note that actual and expected file lists are sorted before check,
563 adding a file can be done with "push".
564 • Copy all config data from "wr_root/model_tests/<subtest name>/" to
566 "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
567 check that configuration written back has the same content as the
568 original configuration.
569 • Create a second configuration instance to read the conf file that
571 was just copied (configuration data is checked.)
572 • You can skip the load check if the written file still contain
574 errors (e.g. some errors were ignored and cannot be fixed) with
575 "load_check2 => 'no'"
576 • Optionally load configuration data in the second instance. You
578 should design this config data to suppress any error or warning
579 that occur in the step below. E.g:
580 load2 => 'binary:seaview',
582 See Config::Model::Loader for the syntax of the string accepted by
584 "load2" parameter.
585 • Compare data read from original data.
587 • Run specific content check on the written config file to verify
589 that configuration data was written and retrieved correctly:
590 wr_check => {
592 'fs:/proc fs_spec' => "proc" ,
593 'fs:/proc fs_file' => "/proc",
594 'fs:/home fs_file' => "/home",
595 },
596 Like the "check" item explained above, you can run "wr_check" using
598 different check modes.
599 Running the test
601 Run all tests with one of these commands:
602 prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
604 perl -Ilib t/model_test.t [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
605 By default, all tests are run on all models.
607 You can pass arguments to "t/model_test.t":
609 • Optional parameters: "--trace" to get test traces. "--error" to get
611 stack trace in case of errors, "--log" to have logs. E.g.
612 # run with log and error traces
614 prove -lv t/model_test.t :: --error --logl
615 • The model name to tests. E.g.:
617 # run only fstab tests
619 prove -lv t/model_test.t :: fstab
620 • A regexp to filter subtest E.g.:
622 # run only fstab tests foobar subtest
624 prove -lv t/model_test.t :: fstab foobar
625 # run only fstab tests foo subtest
627 prove -lv t/model_test.t :: fstab '^foo$'
628
630 Some of these examples may still use global variables (which is
631 deprecated). Such files may be considered as buggy after Aug 2019.
632 Please warn the author if you find one.
633 • LCDproc <http://lcdproc.org> has a single configuration file:
635 "/etc/LCDd.conf". Here's LCDproc test layout
636 <https://github.com/dod38fr/config-model-
637 lcdproc/tree/master/t/model_tests.d> and the test specification
638 <https://github.com/dod38fr/config-model-
639 lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
640 • Dpkg packages are constructed from several files. These files are
642 handled like configuration files by Config::Model::Dpkg
643 <https://salsa.debian.org/perl-team/modules/packages/libconfig-
644 model-dpkg-perl>. The test layout <https://salsa.debian.org/perl-
645 team/modules/packages/libconfig-model-dpkg-
646 perl/-/tree/master/t/model_tests.d> features test with multiple
647 file in dpkg-examples <https://salsa.debian.org/perl-
648 team/modules/packages/libconfig-model-dpkg-
649 perl/-/tree/master/t/model_tests.d/dpkg-examples>. The test is
650 specified in
651 <https://salsa.debian.org/perl-team/modules/packages/libconfig-model-dpkg-perl/-/blob/master/t/model_tests.d/dpkg-test-conf.pl>
652 • multistrap-test-conf.pl <https://github.com/dod38fr/config-
654 model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
655 multistrap-examples <https://github.com/dod38fr/config-
656 model/tree/master/t/model_tests.d/multistrap-examples> specify a
657 test where the configuration file name is not imposed by the
658 application. The file name must then be set in the test
659 specification.
660 • backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
662 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
663 a more complex example showing how to test a backend. The test is
664 done creating a dummy model within the test specification.
665
667 In alphabetical order:
668 • Cyrille Bollu
670 Many thanks for your help.
672
674 • Config::Model
675 • Test::More
677
679 Dominique Dumont
680
682 This software is Copyright (c) 2013-2020 by Dominique Dumont.
683 This is free software, licensed under:
685 The GNU Lesser General Public License, Version 2.1, February 1999
687
689 Websites
690 The following websites have more information about this module, and may
691 be of help to you. As always, in addition to those websites please use
692 your favorite search engine to discover more resources.
693 • CPANTS
695 The CPANTS is a website that analyzes the Kwalitee ( code metrics )
697 of a distribution.
698 <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
700 • CPAN Testers
702 The CPAN Testers is a network of smoke testers who run automated
704 tests on uploaded CPAN distributions.
705 <http://www.cpantesters.org/distro/C/Config-Model-Tester>
707 • CPAN Testers Matrix
709 The CPAN Testers Matrix is a website that provides a visual
711 overview of the test results for a distribution on various
712 Perls/platforms.
713 <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
715 • CPAN Testers Dependencies
717 The CPAN Testers Dependencies is a website that shows a chart of
719 the test results of all dependencies for a distribution.
720 <http://deps.cpantesters.org/?module=Config::Model::Tester>
722 Bugs / Feature Requests
724 Please report any bugs or feature requests by email to "ddumont at
725 cpan.org", or through the web interface at
726 <https://github.com/dod38fr/config-model-tester/issues>. You will be
727 automatically notified of any progress on the request by the system.
728 Source Code
730 The code is open to the world, and available for you to hack on. Please
731 feel free to browse it and play with it, or whatever. If you want to
732 contribute patches, please send me a diff or prod me to pull from your
733 repository :)
734 <http://github.com/dod38fr/config-model-tester.git>
736 git clone git://github.com/dod38fr/config-model-tester.git
738perl v5.34.0 2022-01-21 Config::Model::Tester(3)