1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
6 Config::Model::Tester - Test framework for Config::Model
7
9 version 4.006
10
12 In your test file (typically "t/model_test.t"):
13 use warnings;
15 use strict;
16 use Config::Model::Tester ;
18 use ExtUtils::testlib;
19 run_tests() ;
21 Run tests with:
23 perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25
27 This class provides a way to test configuration models with tests
28 files. This class was designed to tests several models and run several
29 tests cases per model.
30 A specific layout for test files must be followed.
32 Sub test specification
34 Each subtest is defined in a file like:
35 t/model_tests.d/<app-name>-test-conf.pl
37 This file specifies that "app-name" (which is defined in
39 "lib/Config/Model/*.d" directory) is used for the test cases defined in
40 the "*-test-conf.pl" file. The model to test is inferred from the
41 application name to test.
42 This file contains a list of test case (explained below) and expects a
44 set of files used as test data. The layout of these test data files is
45 explained in next section.
46 Simple test file layout
48 Each test case is represented by a configuration file (not a directory)
49 in the "*-examples" directory. This configuration file is used by the
50 model to test and is copied as "$confdir/$conf_file_name" using the
51 test data structure explained below.
52 In the example below, we have 1 app model to test: "lcdproc" and 2
54 tests cases. The app name matches the file specified in
55 "lib/Config/Model/*.d" directory. In this case, the app name matches
56 "lib/Config/Model/system.d/lcdproc"
57 t
59 |-- model_test.t
60 \-- model_tests.d # do not change directory name
61 |-- lcdproc-test-conf.pl # subtest specification for lcdproc app
62 \-- lcdproc-examples
63 |-- t0 # test case t0
64 \-- LCDD-0.5.5 # test case for older LCDproc
65 Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67 this module looks for files named like "<app-name>-test-conf.pl>").
68 Subtests data is provided in files in directory "lcdproc-examples" (
70 i.e. this modules looks for test data in directory
71 "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72 so that each file is used as a "/etc/LCDd.conf" file during each test
73 case.
74 "lcdproc-test-conf.pl" can contain specifications for more test cases.
76 Each test case requires a new file in "lcdproc-examples" directory.
77 See "Examples" for a link to the actual LCDproc model tests
79 Test file layout for multi-file configuration
81 When a configuration is spread over several files, each test case is
82 provided in a sub-directory. This sub-directory is copied in "conf_dir"
83 (a test parameter as explained below)
84 In the example below, the test specification is written in
86 "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87 "dpkg-test-conf.pl" contains instructions so that each directory under
88 "dpkg-examples" is used.
89 t/model_tests.d
91 \-- dpkg-test-conf.pl # subtest specification
92 \-- dpkg-examples
93 \-- libversion # example subdir, used as test case name
94 \-- debian # directory for used by test case
95 |-- changelog
96 |-- compat
97 |-- control
98 |-- copyright
99 |-- rules
100 |-- source
101 | \-- format
102 \-- watch
103 See "Examples" for a link to the (many) Dpkg model tests
105 More complex file layout
107 Each test case is a sub-directory on the "*-examples" directory and
108 contains several files. The destination of the test files may depend on
109 the system (e.g. the OS). For instance, system wide "ssh_config" is
110 stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111 These files are copied in a test directory using a "setup" parameter in
113 test case specification:
114 setup => {
116 test_file_in_example_dir => 'destination'
117 }
118 Let's consider this example of 2 tests cases for ssh:
120 t/model_tests.d/
122 |-- ssh-test-conf.pl
123 |-- ssh-examples
124 \-- basic
125 |-- system_ssh_config
126 \-- user_ssh_config
127 Unfortunately, "user_ssh_config" is a user file, so you need to specify
129 where is located the home directory of the test with another global
130 parameter:
131 home_for_test => '/home/joe' ;
133 For Linux only, the "setup" parameter is:
135 setup => {
137 system_ssh_config => '/etc/ssh/ssh_config',
138 user_ssh_config => "~/.ssh/config"
139 }
140 On the other hand, system wide config file is different on MacOS and
142 the test file must be copied in the correct location. When the value of
143 the "setup" hash is another hash, the key of this other hash is used as
144 to specify the target location for other OS (as returned by Perl $^O
145 variable:
146 setup => {
148 'system_ssh_config' => {
149 'darwin' => '/etc/ssh_config',
150 'default' => '/etc/ssh/ssh_config',
151 },
152 'user_ssh_config' => "~/.ssh/config"
153 }
154 "systemd" is another beast where configuration files can be symlinks to
156 "/dev/null" or other files. To emulate this situation, use an array as
157 setup target:
158 setup => {
160 # test data file => [ link (may be repeated), .. link(s) target contains test data ]
161 'ssh.service' => [ '/etc/systemd/system/sshd.service', '/lib/systemd/system/ssh.service' ]
162 }
163 This will result in a symlink like:
165 wr_root/model_tests/test-sshd-service/etc/systemd/system/sshd.service
167 -> /absolute_path_to/wr_root/model_tests/test-sshd-service/lib/systemd/system/ssh.service
168 See the actual Ssh and Sshd model tests
170 <https://github.com/dod38fr/config-model-
171 openssh/tree/master/t/model_tests.d>
172 Basic test specification
174 Each model subtest is specified in "<app>-test-conf.pl". This file must
175 return a data structure containing the test specifications. Each test
176 data structure contains global parameters (Applied to all tests cases)
177 and test cases parameters (parameters are applied to the test case)
178 use strict;
180 use warnings;
181 {
182 # global parameters
183 # config file name (used to copy test case into test wr_root/model_tests directory)
185 conf_file_name => "fstab",
186 # config dir where to copy the file (optional)
187 conf_dir => "etc",
188 # home directory for this test
189 home_for_test => '/home/joe'
190 tests => [
192 {
193 # test case 1
194 name => 'my_first_test',
195 # other test case parameters
196 },
197 {
198 # test case 2
199 name => 'my_second_test',
200 # other test case parameters
201 },
202 # ...
203 ],
204 };
205 # do not add 1; at the end of the file
207 In the example below, "t0" file is copied in
209 "wr_root/model_tests/test-t0/etc/fstab".
210 use strict;
212 use warnings;
213 {
214 # list of tests.
215 tests => [
216 {
217 # test name
218 name => 't0',
219 # add optional specification here for t0 test
220 },
221 {
222 name => 't1',
223 # add optional specification here for t1 test
224 },
225 ]
226 };
227 You can suppress warnings by specifying "no_warnings => 1" in each test
229 case. On the other hand, you may also want to check for warnings
230 specified to your model. In this case, you should avoid specifying
231 "no_warnings" here and specify warning tests or warning filters as
232 mentioned below.
233 See actual fstab test <https://github.com/dod38fr/config-
235 model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
236 Skip a test
238 A test file can be skipped using "skip" global test parameter.
239 In this example, test is skipped when not running on a Debian system:
241 eval { require AptPkg::Config; };
243 my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
244 {
246 skip => $skip,
247 tests => [ ] ,
248 };
249 Internal tests or backend tests
251 Some tests require the creation of a configuration class dedicated for
252 test (typically to test corner cases on a backend).
253 This test class can be created directly in the test specification by
255 specifying tests classes in "config_classes" global test parameter in
256 an array ref. Each array element is a data structure that use
257 create_config_class parameters. See for instance the layer test
258 <https://github.com/dod38fr/config-
259 model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
260 shellvar backend <https://github.com/dod38fr/config-
261 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
262 In this case, no application exist for such classes so the model to
264 test must be specified in a global test parameter:
265 return {
267 config_classes => [ { name => "Foo", element => ... } , ... ],
268 model_to_test => "Foo",
269 tests => [ ... ]
270 };
271 Test specification with arbitrary file names
273 In some models, like "Multistrap", the config file is chosen by the
274 user. In this case, the file name must be specified for each tests
275 case:
276 {
278 tests => [ {
279 name => 'arm',
280 config_file => '/home/foo/my_arm.conf',
281 check => {},
282 }]
283 };
284 See the actual multistrap test <https://github.com/dod38fr/config-
286 model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
287 Backend argument
289 Some application like systemd requires a backend argument specified by
290 user (e.g. a service name for systemd). The parameter "backend_arg" can
291 be specified to emulate this behavior.
292 Re-use test data
294 When the input data for test is quite complex (several files), it may
295 be interesting to re-use these data for other test cases. Knowing that
296 test names must be unique, you can re-use test data with "data_from"
297 parameter. For instance:
298 tests => [
300 {
301 name => 'some-test',
302 # ...
303 },
304 {
305 name => 'some-other-test',
306 data_from => 'some-test', # re-use data from test above
307 # ...
308 },
309 ]
310 See plainfile backend test <https://github.com/dod38fr/config-
312 model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
313 real life example.
314 Test scenario
316 Each subtest follow a sequence explained below. Each step of this
317 sequence may be altered by adding test case parameters in
318 "<model-to-test>-test-conf.pl":
319 • Setup test in "wr_root/model_tests/<subtest name>/". If your
321 configuration file layout depend on the target system, you will
322 have to specify the path using "setup" parameter. See "Test file
323 layout depending on system".
324 • Create configuration instance, load config data and check its
326 validity. Use "load_check => 'no'" if your file is not valid.
327 • Check for config data warnings. You should pass the list of
329 expected warnings that are emitted through Log::Log4Perl. The array
330 ref is passed as is to the "expect" function of "expect" in
331 Test::Log::Lo4Perl. E.g:
332 log4perl_load_warnings => [
334 [ 'Tree.Node', (warn => qr/deprecated/) x 2 ] ,
335 [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
336 ]
337 The Log classes are specified in "cme/Logging".
339 Log levels below "warn" are ignored.
341 Note that log tests are disabled when "--log" option is used, hence
343 all warnings triggered by the tests are shown.
344 Config::Model is currently transitioning from traditional "warn" to
346 warn logs. To avoid breaking all tests based on this module, the
347 warnings are emitted through Log::Log4Perl only when
348 $::_use_log4perl_to_warn is set. This hack will be removed once all
349 warnings checks in tests are ported to log4perl checks.
350 • DEPRECATED. Check for config data warning. You should pass the list
352 of expected warnings. E.g.
353 load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
355 Use an empty array_ref to mask load warnings.
357 • Optionally run update command:
359 update => {
361 [ returns => 'foo' , ]
362 no_warnings => [ 0 | 1 ], # default 0
363 quiet => [ 0 | 1], # default 0, passed to update method
364 loag4perl_update_warnings => [ ... ] # Test::Log::Log4Perl::expect arguments
365 }
366 Where:
368 • "returns" is the expected return value (optional).
370 • "no_warnings" to suppress the warnings coming from
372 Config::Model::Value. Note that "no_warnings => 1" may be
373 useful for less verbose test.
374 • "quiet" to suppress progress messages during update.
376 • "log4perl_update_warnings" is used to check the warnings
378 produced during update. The argument is passed to "expect"
379 function of Test::Log::Log4Perl. See "load_warnings" parameter
380 above for more details.
381 • DEPRECATED. "update_warnings" is an array ref of quoted regexp
383 (See qr operator) to check the warnings produced during update.
384 use "update => []" to check that no warnings are issued during
385 update.
386 All other arguments are passed to "update" method.
388 • Optionally load configuration data. You should design this config
390 data to suppress any error or warning mentioned above. E.g:
391 load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
393 See Config::Model::Loader for the syntax of the string accepted by
395 "load" parameter.
396 • Optionally, run a check before running apply_fix (if any). This
398 step is useful to check warning messages:
399 check_before_fix => {
401 dump_errors => [ ... ] # optional, see below
402 load4perl_dump_warnings => [ ... ] # optional, see below
403 }
404 Use "dump_errors" if you expect issues:
406 check_before_fix => {
408 dump_errors => [
409 # the issues and a way to fix the issue using Config::Model::Node::load
410 qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
411 qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
412 ],
413 }
414 Likewise, specify any expected warnings:
416 check_before_fix => {
418 log4perl_dump_warnings => [ ... ],
419 }
420 "log4perl_dump_warnings" passes the array ref content to "expect"
422 function of Test::Log::Log4Perl.
423 Both "log4perl_dump_warnings" and "dump_errors" can be specified in
425 "check_before_fix" hash.
426 • Optionally, call apply_fixes:
428 apply_fix => 1,
430 • Call dump_tree to check the validity of the data after optional
432 "apply_fix". This step is not optional.
433 As with "check_before_fix", both "dump_errors" or "dump_warnings"
435 can be used.
436 • Run specific content check to verify that configuration data was
438 retrieved correctly:
439 check => {
441 'fs:/proc fs_spec' => "proc",
442 'fs:/proc fs_file' => "/proc",
443 'fs:/home fs_file' => "/home",
444 },
445 The keys of the hash points to the value to be checked using the
447 syntax described in "grab" in Config::Model::Role::Grab.
448 Multiple check on the same item can be applied with a array ref:
450 check => [
452 Synopsis => 'fix undefined path_max for st_size zero',
453 Description => [ qr/^The downstream/, qr/yada yada/ ]
454 ]
455 You can run check using different check modes (See "fetch" in
457 Config::Model::Value) by passing a hash ref instead of a scalar :
458 check => {
460 'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
461 'sections:base packages:0' => { mode => 'layered', value => "gcc-4.2-base' },
462 },
463 The whole hash content (except "value") is passed to grab and
465 fetch
466 A regexp can also be used to check value:
468 check => {
470 "License text" => qr/gnu/i,
471 }
472 And specification can nest hash or array style:
474 check => {
476 "License:0 text" => qr/gnu/i,
477 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
478 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
479 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
480 }
481 • Verify if a hash contains one or more keys (or keys matching a
483 regexp):
484 has_key => [
486 'sections' => 'debian', # sections must point to a hash element
487 'control' => [qw/source binary/],
488 'copyright Files' => qr/.c$/,
489 'copyright Files' => [qr/\.h$/], qr/\.c$/],
490 ],
491 • Verify that a hash does not have a key (or a key matching a
493 regexp):
494 has_not_key => [
496 'copyright Files' => qr/.virus$/ # silly, isn't ?
497 ],
498 • Verify annotation extracted from the configuration file comments:
500 verify_annotation => {
502 'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
503 'source Maintainer' => "what a fine\nteam this one is",
504 },
505 • Write back the config data in "wr_root/model_tests/<subtest
507 name>/". Note that write back is forced, so the tested
508 configuration files are written back even if the configuration
509 values were not changed during the test.
510 You can skip warning when writing back with the global :
512 no_warnings => 1,
514 • Check the content of the written files(s) with
516 Test::File::Contents. Tests can be grouped in an array ref:
517 file_contents => {
519 "/home/foo/my_arm.conf" => "really big string" ,
520 "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
521 }
522 file_contents_like => {
524 "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
525 }
526 file_contents_unlike => {
528 "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
529 }
530 • Check the mode of the written files:
532 file_mode => {
534 "~/.ssh/ssh_config" => oct(600), # better than 0600
535 "debian/stuff.postinst" => oct(755),
536 }
537 Only the last four octets of the mode are tested. I.e. the test is
539 done with " $file_mode & oct(7777) "
540 Note: this test is skipped on Windows
542 • Check added or removed configuration files. If you expect changes,
544 specify a subref to alter the file list:
545 file_check_sub => sub {
547 my $list_ref = shift ;
548 # file added during tests
549 push @$list_ref, "/debian/source/format" ;
550 },
551 Note that actual and expected file lists are sorted before check,
553 adding a file can be done with "push".
554 • Copy all config data from "wr_root/model_tests/<subtest name>/" to
556 "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
557 check that configuration written back has the same content as the
558 original configuration.
559 • Create a second configuration instance to read the conf file that
561 was just copied (configuration data is checked.)
562 • You can skip the load check if the written file still contain
564 errors (e.g. some errors were ignored and cannot be fixed) with
565 "load_check2 => 'no'"
566 • Optionally load configuration data in the second instance. You
568 should design this config data to suppress any error or warning
569 that occur in the step below. E.g:
570 load2 => 'binary:seaview',
572 See Config::Model::Loader for the syntax of the string accepted by
574 "load2" parameter.
575 • Compare data read from original data.
577 • Run specific content check on the written config file to verify
579 that configuration data was written and retrieved correctly:
580 wr_check => {
582 'fs:/proc fs_spec' => "proc" ,
583 'fs:/proc fs_file' => "/proc",
584 'fs:/home fs_file' => "/home",
585 },
586 Like the "check" item explained above, you can run "wr_check" using
588 different check modes.
589 Running the test
591 Run all tests with one of these commands:
592 prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
594 perl -Ilib t/model_test.t [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
595 By default, all tests are run on all models.
597 You can pass arguments to "t/model_test.t":
599 • Optional parameters: "--trace" to get test traces. "--error" to get
601 stack trace in case of errors, "--log" to have logs. E.g.
602 # run with log and error traces
604 prove -lv t/model_test.t :: --error --logl
605 • The model name to tests. E.g.:
607 # run only fstab tests
609 prove -lv t/model_test.t :: fstab
610 • A regexp to filter subtest E.g.:
612 # run only fstab tests foobar subtest
614 prove -lv t/model_test.t :: fstab foobar
615 # run only fstab tests foo subtest
617 prove -lv t/model_test.t :: fstab '^foo$'
618
620 Some of these examples may still use global variables (which is
621 deprecated). Such files may be considered as buggy after Aug 2019.
622 Please warn the author if you find one.
623 • LCDproc <http://lcdproc.org> has a single configuration file:
625 "/etc/LCDd.conf". Here's LCDproc test layout
626 <https://github.com/dod38fr/config-model-
627 lcdproc/tree/master/t/model_tests.d> and the test specification
628 <https://github.com/dod38fr/config-model-
629 lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
630 • Dpkg packages are constructed from several files. These files are
632 handled like configuration files by Config::Model::Dpkg. The test
633 layout <http://anonscm.debian.org/gitweb/?p=pkg-
634 perl/packages/libconfig-model-dpkg-
635 perl.git;a=tree;f=t/model_tests.d;hb=HEAD> features test with
636 multiple file in dpkg-examples
637 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
638 model-dpkg-perl.git;a=tree;f=t/model_tests.d/dpkg-
639 examples;hb=HEAD>. The test is specified in dpkg-test-conf.pl
640 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
641 model-dpkg-perl.git;a=blob_plain;f=t/model_tests.d/dpkg-test-
642 conf.pl;hb=HEAD>
643 • multistrap-test-conf.pl <https://github.com/dod38fr/config-
645 model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
646 multistrap-examples <https://github.com/dod38fr/config-
647 model/tree/master/t/model_tests.d/multistrap-examples> specify a
648 test where the configuration file name is not imposed by the
649 application. The file name must then be set in the test
650 specification.
651 • backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
653 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
654 a more complex example showing how to test a backend. The test is
655 done creating a dummy model within the test specification.
656
658 In alphabetical order:
659 • Cyrille Bollu
661 Many thanks for your help.
663
665 • Config::Model
666 • Test::More
668
670 Dominique Dumont
671
673 This software is Copyright (c) 2013-2020 by Dominique Dumont.
674 This is free software, licensed under:
676 The GNU Lesser General Public License, Version 2.1, February 1999
678
680 Websites
681 The following websites have more information about this module, and may
682 be of help to you. As always, in addition to those websites please use
683 your favorite search engine to discover more resources.
684 • CPANTS
686 The CPANTS is a website that analyzes the Kwalitee ( code metrics )
688 of a distribution.
689 <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
691 • CPAN Testers
693 The CPAN Testers is a network of smoke testers who run automated
695 tests on uploaded CPAN distributions.
696 <http://www.cpantesters.org/distro/C/Config-Model-Tester>
698 • CPAN Testers Matrix
700 The CPAN Testers Matrix is a website that provides a visual
702 overview of the test results for a distribution on various
703 Perls/platforms.
704 <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
706 • CPAN Testers Dependencies
708 The CPAN Testers Dependencies is a website that shows a chart of
710 the test results of all dependencies for a distribution.
711 <http://deps.cpantesters.org/?module=Config::Model::Tester>
713 Bugs / Feature Requests
715 Please report any bugs or feature requests by email to "ddumont at
716 cpan.org", or through the web interface at
717 <https://github.com/dod38fr/config-model-tester/issues>. You will be
718 automatically notified of any progress on the request by the system.
719 Source Code
721 The code is open to the world, and available for you to hack on. Please
722 feel free to browse it and play with it, or whatever. If you want to
723 contribute patches, please send me a diff or prod me to pull from your
724 repository :)
725 <http://github.com/dod38fr/config-model-tester.git>
727 git clone git://github.com/dod38fr/config-model-tester.git
729perl v5.32.1 2021-01-27 Config::Model::Tester(3)