1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
6 Config::Model::Tester - Test framework for Config::Model
7
9 version 4.005
10
12 In your test file (typically "t/model_test.t"):
13 use warnings;
15 use strict;
16 use Config::Model::Tester ;
18 use ExtUtils::testlib;
19 run_tests() ;
21 Run tests with:
23 perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25
27 This class provides a way to test configuration models with tests
28 files. This class was designed to tests several models and run several
29 tests cases per model.
30 A specific layout for test files must be followed.
32 Sub test specification
34 Each subtest is defined in a file like:
35 t/model_tests.d/<app-name>-test-conf.pl
37 This file specifies that "app-name" (which is defined in
39 "lib/Config/Model/*.d" directory) is used for the test cases defined in
40 the "*-test-conf.pl" file. The model to test is inferred from the
41 application name to test.
42 This file contains a list of test case (explained below) and expects a
44 set of files used as test data. The layout of these test data files is
45 explained in next section.
46 Simple test file layout
48 Each test case is represented by a configuration file (not a directory)
49 in the "*-examples" directory. This configuration file is used by the
50 model to test and is copied as "$confdir/$conf_file_name" using the
51 test data structure explained below.
52 In the example below, we have 1 app model to test: "lcdproc" and 2
54 tests cases. The app name matches the file specified in
55 "lib/Config/Model/*.d" directory. In this case, the app name matches
56 "lib/Config/Model/system.d/lcdproc"
57 t
59 |-- model_test.t
60 \-- model_tests.d # do not change directory name
61 |-- lcdproc-test-conf.pl # subtest specification for lcdproc app
62 \-- lcdproc-examples
63 |-- t0 # test case t0
64 \-- LCDD-0.5.5 # test case for older LCDproc
65 Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67 this module looks for files named like "<app-name>-test-conf.pl>").
68 Subtests data is provided in files in directory "lcdproc-examples" (
70 i.e. this modules looks for test data in directory
71 "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72 so that each file is used as a "/etc/LCDd.conf" file during each test
73 case.
74 "lcdproc-test-conf.pl" can contain specifications for more test cases.
76 Each test case requires a new file in "lcdproc-examples" directory.
77 See "Examples" for a link to the actual LCDproc model tests
79 Test file layout for multi-file configuration
81 When a configuration is spread over several files, each test case is
82 provided in a sub-directory. This sub-directory is copied in "conf_dir"
83 (a test parameter as explained below)
84 In the example below, the test specification is written in
86 "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87 "dpkg-test-conf.pl" contains instructions so that each directory under
88 "dpkg-examples" is used.
89 t/model_tests.d
91 \-- dpkg-test-conf.pl # subtest specification
92 \-- dpkg-examples
93 \-- libversion # example subdir, used as test case name
94 \-- debian # directory for used by test case
95 |-- changelog
96 |-- compat
97 |-- control
98 |-- copyright
99 |-- rules
100 |-- source
101 | \-- format
102 \-- watch
103 See "Examples" for a link to the (many) Dpkg model tests
105 More complex file layout
107 Each test case is a sub-directory on the "*-examples" directory and
108 contains several files. The destination of the test files may depend on
109 the system (e.g. the OS). For instance, system wide "ssh_config" is
110 stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111 These files are copied in a test directory using a "setup" parameter in
113 test case specification:
114 setup => {
116 test_file_in_example_dir => 'destination'
117 }
118 Let's consider this example of 2 tests cases for ssh:
120 t/model_tests.d/
122 |-- ssh-test-conf.pl
123 |-- ssh-examples
124 \-- basic
125 |-- system_ssh_config
126 \-- user_ssh_config
127 Unfortunately, "user_ssh_config" is a user file, so you need to specify
129 where is located the home directory of the test with another global
130 parameter:
131 home_for_test => '/home/joe' ;
133 For Linux only, the "setup" parameter is:
135 setup => {
137 system_ssh_config => '/etc/ssh/ssh_config',
138 user_ssh_config => "~/.ssh/config"
139 }
140 On the other hand, system wide config file is different on MacOS and
142 the test file must be copied in the correct location. When the value of
143 the "setup" hash is another hash, the key of this other hash is used as
144 to specify the target location for other OS (as returned by Perl $^O
145 variable:
146 setup => {
148 'system_ssh_config' => {
149 'darwin' => '/etc/ssh_config',
150 'default' => '/etc/ssh/ssh_config',
151 },
152 'user_ssh_config' => "~/.ssh/config"
153 }
154 "systemd" is another beast where configuration files can be symlinks to
156 "/dev/null" or other files. To emulate this situation, use an array as
157 setup target:
158 setup => {
160 # test data file => [ link (may be repeated), .. link(s) target contains test data ]
161 'ssh.service' => [ '/etc/systemd/system/sshd.service', '/lib/systemd/system/ssh.service' ]
162 }
163 This will result in a symlink like:
165 wr_root/model_tests/test-sshd-service/etc/systemd/system/sshd.service
167 -> /absolute_path_to/wr_root/model_tests/test-sshd-service/lib/systemd/system/ssh.service
168 See the actual Ssh and Sshd model tests
170 <https://github.com/dod38fr/config-model-
171 openssh/tree/master/t/model_tests.d>
172 Basic test specification
174 Each model subtest is specified in "<app>-test-conf.pl". This file must
175 return a data structure containing the test specifications. Each test
176 data structure contains global parameters (Applied to all tests cases)
177 and test cases parameters (parameters are applied to the test case)
178 use strict;
180 use warnings;
181 {
182 # global parameters
183 # config file name (used to copy test case into test wr_root/model_tests directory)
185 conf_file_name => "fstab",
186 # config dir where to copy the file (optional)
187 conf_dir => "etc",
188 # home directory for this test
189 home_for_test => '/home/joe'
190 tests => [
192 {
193 # test case 1
194 name => 'my_first_test',
195 # other test case parameters
196 },
197 {
198 # test case 2
199 name => 'my_second_test',
200 # other test case parameters
201 },
202 # ...
203 ],
204 };
205 # do not add 1; at the end of the file
207 In the example below, "t0" file is copied in
209 "wr_root/model_tests/test-t0/etc/fstab".
210 use strict;
212 use warnings;
213 {
214 # list of tests.
215 tests => [
216 {
217 # test name
218 name => 't0',
219 # add optional specification here for t0 test
220 },
221 {
222 name => 't1',
223 # add optional specification here for t1 test
224 },
225 ]
226 };
227 You can suppress warnings by specifying "no_warnings => 1" in each test
229 case. On the other hand, you may also want to check for warnings
230 specified to your model. In this case, you should avoid specifying
231 "no_warnings" here and specify warning tests or warning filters as
232 mentioned below.
233 See actual fstab test <https://github.com/dod38fr/config-
235 model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
236 Skip a test
238 A test file can be skipped using "skip" global test parameter.
239 In this example, test is skipped when not running on a Debian system:
241 eval { require AptPkg::Config; };
243 my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
244 {
246 skip => $skip,
247 tests => [ ] ,
248 };
249 Internal tests or backend tests
251 Some tests require the creation of a configuration class dedicated for
252 test (typically to test corner cases on a backend).
253 This test class can be created directly in the test specification by
255 specifying tests classes in "config_classes" global test parameter in
256 an array ref. Each array element is a data structure that use
257 create_config_class parameters. See for instance the layer test
258 <https://github.com/dod38fr/config-
259 model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
260 shellvar backend <https://github.com/dod38fr/config-
261 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
262 In this case, no application exist for such classes so the model to
264 test must be specified in a global test parameter:
265 return {
267 config_classes => [ { name => "Foo", element => ... } , ... ],
268 model_to_test => "Foo",
269 tests => [ ... ]
270 };
271 Test specification with arbitrary file names
273 In some models, like "Multistrap", the config file is chosen by the
274 user. In this case, the file name must be specified for each tests
275 case:
276 {
278 tests => [ {
279 name => 'arm',
280 config_file => '/home/foo/my_arm.conf',
281 check => {},
282 }]
283 };
284 See the actual multistrap test <https://github.com/dod38fr/config-
286 model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
287 Backend argument
289 Some application like systemd requires a backend argument specified by
290 user (e.g. a service name for systemd). The parameter "backend_arg" can
291 be specified to emulate this behavior.
292 Re-use test data
294 When the input data for test is quite complex (several files), it may
295 be interesting to re-use these data for other test cases. Knowing that
296 test names must be unique, you can re-use test data with "data_from"
297 parameter. For instance:
298 tests => [
300 {
301 name => 'some-test',
302 # ...
303 },
304 {
305 name => 'some-other-test',
306 data_from => 'some-test', # re-use data from test above
307 # ...
308 },
309 ]
310 See plainfile backend test <https://github.com/dod38fr/config-
312 model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
313 real life example.
314 Test scenario
316 Each subtest follow a sequence explained below. Each step of this
317 sequence may be altered by adding test case parameters in
318 "<model-to-test>-test-conf.pl":
319 · Setup test in "wr_root/model_tests/<subtest name>/". If your
321 configuration file layout depend on the target system, you will
322 have to specify the path using "setup" parameter. See "Test file
323 layout depending on system".
324 · Create configuration instance, load config data and check its
326 validity. Use "load_check => 'no'" if your file is not valid.
327 · Check for config data warnings. You should pass the list of
329 expected warnings that are emitted through Log::Log4Perl. The array
330 ref is passed as is to the "expect" function of "expect" in
331 Test::Log::Lo4Perl. E.g:
332 log4perl_load_warnings => [
334 [ 'Tree.Node', (warn => qr/deprecated/) x 2 ] ,
335 [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
336 ]
337 The Log classes are specified in "cme/Logging".
339 Log levels below "warn" are ignored.
341 Config::Model is currently transitioning from traditional "warn" to
343 warn logs. To avoid breaking all tests based on this module, the
344 warnings are emitted through Log::Log4Perl only when
345 c<$::_use_log4perl_to_warn> is set. This hack will be removed once
346 all warnings checks in tests are ported to log4perl checks.
347 · DEPRECATED. Check for config data warning. You should pass the list
349 of expected warnings. E.g.
350 load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
352 Use an empty array_ref to mask load warnings.
354 · Optionally run update command:
356 update => {
358 [ returns => 'foo' , ]
359 no_warnings => [ 0 | 1 ], # default 0
360 quiet => [ 0 | 1], # default 0, passed to update method
361 loag4perl_update_warnings => [ ... ] # Test::Log::Log4Perl::expect arguments
362 }
363 Where:
365 · "returns" is the expected return value (optional).
367 · "no_warnings" to suppress the warnings coming from
369 Config::Model::Value. Note that "no_warnings => 1" may be
370 useful for less verbose test.
371 · "quiet" to suppress progress messages during update.
373 · "log4perl_update_warnings" is used to check the warnings
375 produced during update. The argument is passed to "expect"
376 function of Test::Log::Log4Perl. See "load_warnings" parameter
377 above for more details.
378 · DEPRECATED. "update_warnings" is an array ref of quoted regexp
380 (See qr operator) to check the warnings produced during update.
381 use "update => []" to check that no warnings are issued during
382 update.
383 All other arguments are passed to "update" method.
385 · Optionally load configuration data. You should design this config
387 data to suppress any error or warning mentioned above. E.g:
388 load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
390 See Config::Model::Loader for the syntax of the string accepted by
392 "load" parameter.
393 · Optionally, run a check before running apply_fix (if any). This
395 step is useful to check warning messages:
396 check_before_fix => {
398 dump_errors => [ ... ] # optional, see below
399 load4perl_dump_warnings => [ ... ] # optional, see below
400 }
401 Use "dump_errors" if you expect issues:
403 check_before_fix => {
405 dump_errors => [
406 # the issues and a way to fix the issue using Config::Model::Node::load
407 qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
408 qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
409 ],
410 }
411 Likewise, specify any expected warnings:
413 check_before_fix => {
415 log4perl_dump_warnings => [ ... ],
416 }
417 "log4perl_dump_warnings" passes the array ref content to "expect"
419 function of Test::Log::Log4Perl.
420 Both "log4perl_dump_warnings" and "dump_errors" can be specified in
422 "check_before_fix" hash.
423 · Optionally, call apply_fixes:
425 apply_fix => 1,
427 · Call dump_tree to check the validity of the data after optional
429 "apply_fix". This step is not optional.
430 As with "check_before_fix", both "dump_errors" or "dump_warnings"
432 can be used.
433 · Run specific content check to verify that configuration data was
435 retrieved correctly:
436 check => {
438 'fs:/proc fs_spec' => "proc",
439 'fs:/proc fs_file' => "/proc",
440 'fs:/home fs_file' => "/home",
441 },
442 The keys of the hash points to the value to be checked using the
444 syntax described in "grab" in Config::Model::Role::Grab.
445 Multiple check on the same item can be applied with a array ref:
447 check => [
449 Synopsis => 'fix undefined path_max for st_size zero',
450 Description => [ qr/^The downstream/, qr/yada yada/ ]
451 ]
452 You can run check using different check modes (See "fetch" in
454 Config::Model::Value) by passing a hash ref instead of a scalar :
455 check => {
457 'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
458 'sections:base packages:0' => { mode => 'layered', value => "gcc-4.2-base' },
459 },
460 The whole hash content (except "value") is passed to grab and
462 fetch
463 A regexp can also be used to check value:
465 check => {
467 "License text" => qr/gnu/i,
468 }
469 And specification can nest hash or array style:
471 check => {
473 "License:0 text" => qr/gnu/i,
474 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
475 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
476 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
477 }
478 · Verify if a hash contains one or more keys (or keys matching a
480 regexp):
481 has_key => [
483 'sections' => 'debian', # sections must point to a hash element
484 'control' => [qw/source binary/],
485 'copyright Files' => qr/.c$/,
486 'copyright Files' => [qr/\.h$/], qr/\.c$/],
487 ],
488 · Verify that a hash does not have a key (or a key matching a
490 regexp):
491 has_not_key => [
493 'copyright Files' => qr/.virus$/ # silly, isn't ?
494 ],
495 · Verify annotation extracted from the configuration file comments:
497 verify_annotation => {
499 'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
500 'source Maintainer' => "what a fine\nteam this one is",
501 },
502 · Write back the config data in "wr_root/model_tests/<subtest
504 name>/". Note that write back is forced, so the tested
505 configuration files are written back even if the configuration
506 values were not changed during the test.
507 You can skip warning when writing back with the global :
509 no_warnings => 1,
511 · Check the content of the written files(s) with
513 Test::File::Contents. Tests can be grouped in an array ref:
514 file_contents => {
516 "/home/foo/my_arm.conf" => "really big string" ,
517 "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
518 }
519 file_contents_like => {
521 "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
522 }
523 file_contents_unlike => {
525 "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
526 }
527 · Check the mode of the written files:
529 file_mode => {
531 "~/.ssh/ssh_config" => oct(600), # better than 0600
532 "debian/stuff.postinst" => oct(755),
533 }
534 Only the last four octets of the mode are tested. I.e. the test is
536 done with " $file_mode & oct(7777) "
537 Note: this test is skipped on Windows
539 · Check added or removed configuration files. If you expect changes,
541 specify a subref to alter the file list:
542 file_check_sub => sub {
544 my $list_ref = shift ;
545 # file added during tests
546 push @$list_ref, "/debian/source/format" ;
547 },
548 Note that actual and expected file lists are sorted before check,
550 adding a file can be done with "push".
551 · Copy all config data from "wr_root/model_tests/<subtest name>/" to
553 "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
554 check that configuration written back has the same content as the
555 original configuration.
556 · Create a second configuration instance to read the conf file that
558 was just copied (configuration data is checked.)
559 · You can skip the load check if the written file still contain
561 errors (e.g. some errors were ignored and cannot be fixed) with
562 "load_check2 => 'no'"
563 · Optionally load configuration data in the second instance. You
565 should design this config data to suppress any error or warning
566 that occur in the step below. E.g:
567 load2 => 'binary:seaview',
569 See Config::Model::Loader for the syntax of the string accepted by
571 "load2" parameter.
572 · Compare data read from original data.
574 · Run specific content check on the written config file to verify
576 that configuration data was written and retrieved correctly:
577 wr_check => {
579 'fs:/proc fs_spec' => "proc" ,
580 'fs:/proc fs_file' => "/proc",
581 'fs:/home fs_file' => "/home",
582 },
583 Like the "check" item explained above, you can run "wr_check" using
585 different check modes.
586 Running the test
588 Run all tests with one of these commands:
589 prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
591 perl -Ilib t/model_test.t [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
592 By default, all tests are run on all models.
594 You can pass arguments to "t/model_test.t":
596 · Optional parameters: "--trace" to get test traces. "--error" to get
598 stack trace in case of errors, "--log" to have logs. E.g.
599 # run with log and error traces
601 prove -lv t/model_test.t :: --error --logl
602 · The model name to tests. E.g.:
604 # run only fstab tests
606 prove -lv t/model_test.t :: fstab
607 · A regexp to filter subtest E.g.:
609 # run only fstab tests foobar subtest
611 prove -lv t/model_test.t :: fstab foobar
612 # run only fstab tests foo subtest
614 prove -lv t/model_test.t :: fstab '^foo$'
615
617 Some of these examples may still use global variables (which is
618 deprecated). Such files may be considered as buggy after Aug 2019.
619 Please warn the author if you find one.
620 · LCDproc <http://lcdproc.org> has a single configuration file:
622 "/etc/LCDd.conf". Here's LCDproc test layout
623 <https://github.com/dod38fr/config-model-
624 lcdproc/tree/master/t/model_tests.d> and the test specification
625 <https://github.com/dod38fr/config-model-
626 lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
627 · Dpkg packages are constructed from several files. These files are
629 handled like configuration files by Config::Model::Dpkg. The test
630 layout <http://anonscm.debian.org/gitweb/?p=pkg-
631 perl/packages/libconfig-model-dpkg-
632 perl.git;a=tree;f=t/model_tests.d;hb=HEAD> features test with
633 multiple file in dpkg-examples
634 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
635 model-dpkg-perl.git;a=tree;f=t/model_tests.d/dpkg-
636 examples;hb=HEAD>. The test is specified in dpkg-test-conf.pl
637 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
638 model-dpkg-perl.git;a=blob_plain;f=t/model_tests.d/dpkg-test-
639 conf.pl;hb=HEAD>
640 · multistrap-test-conf.pl <https://github.com/dod38fr/config-
642 model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
643 multistrap-examples <https://github.com/dod38fr/config-
644 model/tree/master/t/model_tests.d/multistrap-examples> specify a
645 test where the configuration file name is not imposed by the
646 application. The file name must then be set in the test
647 specification.
648 · backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
650 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
651 a more complex example showing how to test a backend. The test is
652 done creating a dummy model within the test specification.
653
655 In alphabetical order:
656 · Cyrille Bollu
658 Many thanks for your help.
660
662 · Config::Model
663 · Test::More
665
667 Dominique Dumont
668
670 This software is Copyright (c) 2013-2019 by Dominique Dumont.
671 This is free software, licensed under:
673 The GNU Lesser General Public License, Version 2.1, February 1999
675
677 Websites
678 The following websites have more information about this module, and may
679 be of help to you. As always, in addition to those websites please use
680 your favorite search engine to discover more resources.
681 · Search CPAN
683 The default CPAN search engine, useful to view POD in HTML format.
685 <http://search.cpan.org/dist/Config-Model-Tester>
687 · AnnoCPAN
689 The AnnoCPAN is a website that allows community annotations of Perl
691 module documentation.
692 <http://annocpan.org/dist/Config-Model-Tester>
694 · CPAN Ratings
696 The CPAN Ratings is a website that allows community ratings and
698 reviews of Perl modules.
699 <http://cpanratings.perl.org/d/Config-Model-Tester>
701 · CPANTS
703 The CPANTS is a website that analyzes the Kwalitee ( code metrics )
705 of a distribution.
706 <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
708 · CPAN Testers
710 The CPAN Testers is a network of smoke testers who run automated
712 tests on uploaded CPAN distributions.
713 <http://www.cpantesters.org/distro/C/Config-Model-Tester>
715 · CPAN Testers Matrix
717 The CPAN Testers Matrix is a website that provides a visual
719 overview of the test results for a distribution on various
720 Perls/platforms.
721 <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
723 · CPAN Testers Dependencies
725 The CPAN Testers Dependencies is a website that shows a chart of
727 the test results of all dependencies for a distribution.
728 <http://deps.cpantesters.org/?module=Config::Model::Tester>
730 Bugs / Feature Requests
732 Please report any bugs or feature requests by email to "ddumont at
733 cpan.org", or through the web interface at
734 <https://github.com/dod38fr/config-model-tester/issues>. You will be
735 automatically notified of any progress on the request by the system.
736 Source Code
738 The code is open to the world, and available for you to hack on. Please
739 feel free to browse it and play with it, or whatever. If you want to
740 contribute patches, please send me a diff or prod me to pull from your
741 repository :)
742 <http://github.com/dod38fr/config-model-tester.git>
744 git clone git://github.com/dod38fr/config-model-tester.git
746perl v5.30.1 2020-01-29 Config::Model::Tester(3)