1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
6 Config::Model::Tester - Test framework for Config::Model
7
9 version 4.003
10
12 In your test file (typically "t/model_test.t"):
13 use warnings;
15 use strict;
16 use Config::Model::Tester ;
18 use ExtUtils::testlib;
19 run_tests() ;
21 Run tests with:
23 perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25
27 This class provides a way to test configuration models with tests
28 files. This class was designed to tests several models and run several
29 tests cases per model.
30 A specific layout for test files must be followed.
32 Sub test specification
34 Each subtest is defined in a file like:
35 t/model_tests.d/<app-name>-test-conf.pl
37 This file specifies that "app-name" (which is defined in
39 "lib/Config/Model/*.d" directory) is used for the test cases defined in
40 the "*-test-conf.pl" file. The model to test is inferred from the
41 application name to test.
42 This file contains a list of test case (explained below) and expects a
44 set of files used as test data. The layout of these test data files is
45 explained in next section.
46 Simple test file layout
48 Each test case is represented by a configuration file (not a directory)
49 in the "*-examples" directory. This configuration file is used by the
50 model to test and is copied as "$confdir/$conf_file_name" using the
51 test data structure explained below.
52 In the example below, we have 1 app model to test: "lcdproc" and 2
54 tests cases. The app name matches the file specified in
55 "lib/Config/Model/*.d" directory. In this case, the app name matches
56 "lib/Config/Model/system.d/lcdproc"
57 t
59 |-- model_test.t
60 \-- model_tests.d # do not change directory name
61 |-- lcdproc-test-conf.pl # subtest specification for lcdproc app
62 \-- lcdproc-examples
63 |-- t0 # test case t0
64 \-- LCDD-0.5.5 # test case for older LCDproc
65 Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67 this module looks for files named like "<app-name>-test-conf.pl>").
68 Subtests data is provided in files in directory "lcdproc-examples" (
70 i.e. this modules looks for test data in directory
71 "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72 so that each file is used as a "/etc/LCDd.conf" file during each test
73 case.
74 "lcdproc-test-conf.pl" can contain specifications for more test cases.
76 Each test case requires a new file in "lcdproc-examples" directory.
77 See "Examples" for a link to the actual LCDproc model tests
79 Test file layout for multi-file configuration
81 When a configuration is spread over several files, each test case is
82 provided in a sub-directory. This sub-directory is copied in "conf_dir"
83 (a test parameter as explained below)
84 In the example below, the test specification is written in
86 "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87 "dpkg-test-conf.pl" contains instructions so that each directory under
88 "dpkg-examples" is used.
89 t/model_tests.d
91 \-- dpkg-test-conf.pl # subtest specification
92 \-- dpkg-examples
93 \-- libversion # example subdir, used as test case name
94 \-- debian # directory for used by test case
95 |-- changelog
96 |-- compat
97 |-- control
98 |-- copyright
99 |-- rules
100 |-- source
101 | \-- format
102 \-- watch
103 See "Examples" for a link to the (many) Dpkg model tests
105 More complex file layout
107 Each test case is a sub-directory on the "*-examples" directory and
108 contains several files. The destination of the test files may depend on
109 the system (e.g. the OS). For instance, system wide "ssh_config" is
110 stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111 These files are copied in a test directory using a "setup" parameter in
113 test case specification:
114 setup => {
116 test_file_in_example_dir => 'destination'
117 }
118 Let's consider this example of 2 tests cases for ssh:
120 t/model_tests.d/
122 |-- ssh-test-conf.pl
123 |-- ssh-examples
124 \-- basic
125 |-- system_ssh_config
126 \-- user_ssh_config
127 Unfortunately, "user_ssh_config" is a user file, so you need to specify
129 where is located the home directory of the test with another global
130 parameter:
131 home_for_test => '/home/joe' ;
133 For Linux only, the "setup" parameter is:
135 setup => {
137 system_ssh_config => '/etc/ssh/ssh_config',
138 user_ssh_config => "~/.ssh/config"
139 }
140 On the other hand, system wide config file is different on MacOS and
142 the test file must be copied in the correct location. When the value of
143 the "setup" hash is another hash, the key of this other hash is used as
144 to specify the target location for other OS (as returned by Perl $^O
145 variable:
146 setup => {
148 'system_ssh_config' => {
149 'darwin' => '/etc/ssh_config',
150 'default' => '/etc/ssh/ssh_config',
151 },
152 'user_ssh_config' => "~/.ssh/config"
153 }
154 See the actual Ssh and Sshd model tests
156 <https://github.com/dod38fr/config-model-
157 openssh/tree/master/t/model_tests.d>
158 Basic test specification
160 Each model subtest is specified in "<app>-test-conf.pl". This file must
161 return a data structure containing the test specifications. Each test
162 data structure contains global parameters (Applied to all tests cases)
163 and test cases parameters (parameters are applied to the test case)
164 use strict;
166 use warnings;
167 {
168 # global parameters
169 # config file name (used to copy test case into test wr_root/model_tests directory)
171 conf_file_name => "fstab",
172 # config dir where to copy the file (optional)
173 conf_dir => "etc",
174 # home directory for this test
175 home_for_test => '/home/joe'
176 tests => [
178 {
179 # test case 1
180 name => 'my_first_test',
181 # other test case parameters
182 },
183 {
184 # test case 2
185 name => 'my_second_test',
186 # other test case parameters
187 },
188 # ...
189 ],
190 };
191 # do not add 1; at the end of the file
193 In the example below, "t0" file is copied in
195 "wr_root/model_tests/test-t0/etc/fstab".
196 use strict;
198 use warnings;
199 {
200 # list of tests.
201 tests => [
202 {
203 # test name
204 name => 't0',
205 # add optional specification here for t0 test
206 },
207 {
208 name => 't1',
209 # add optional specification here for t1 test
210 },
211 ]
212 };
213 You can suppress warnings by specifying "no_warnings => 1" in each test
215 case. On the other hand, you may also want to check for warnings
216 specified to your model. In this case, you should avoid specifying
217 "no_warnings" here and specify warning tests or warning filters as
218 mentioned below.
219 See actual fstab test <https://github.com/dod38fr/config-
221 model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
222 Skip a test
224 A test file can be skipped using "skip" global test parameter.
225 In this example, test is skipped when not running on a Debian system:
227 eval { require AptPkg::Config; };
229 my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
230 {
232 skip => $skip,
233 tests => [ ] ,
234 };
235 Internal tests or backend tests
237 Some tests require the creation of a configuration class dedicated for
238 test (typically to test corner cases on a backend).
239 This test class can be created directly in the test specification by
241 specifying tests classes in "config_classes" global test parameter in
242 an array ref. Each array element is a data structure that use
243 create_config_class parameters. See for instance the layer test
244 <https://github.com/dod38fr/config-
245 model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
246 shellvar backend <https://github.com/dod38fr/config-
247 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
248 In this case, no application exist for such classes so the model to
250 test must be specified in a global test parameter:
251 return {
253 config_classes => [ { name => "Foo", element => ... } , ... ],
254 model_to_test => "Foo",
255 tests => [ ... ]
256 };
257 Test specification with arbitrary file names
259 In some models, like "Multistrap", the config file is chosen by the
260 user. In this case, the file name must be specified for each tests
261 case:
262 {
264 tests => [ {
265 name => 'arm',
266 config_file => '/home/foo/my_arm.conf',
267 check => {},
268 }]
269 };
270 See the actual multistrap test <https://github.com/dod38fr/config-
272 model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
273 Backend argument
275 Some application like systemd requires a backend argument specified by
276 user (e.g. a service name for systemd). The parameter "backend_arg" can
277 be specified to emulate this behavior.
278 Re-use test data
280 When the input data for test is quite complex (several files), it may
281 be interesting to re-use these data for other test cases. Knowing that
282 test names must be unique, you can re-use test data with "data_from"
283 parameter. For instance:
284 tests => [
286 {
287 name => 'some-test',
288 # ...
289 },
290 {
291 name => 'some-other-test',
292 data_from => 'some-test', # re-use data from test above
293 # ...
294 },
295 ]
296 See plainfile backend test <https://github.com/dod38fr/config-
298 model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
299 real life example.
300 Test scenario
302 Each subtest follow a sequence explained below. Each step of this
303 sequence may be altered by adding test case parameters in
304 "<model-to-test>-test-conf.pl":
305 · Setup test in "wr_root/model_tests/<subtest name>/". If your
307 configuration file layout depend on the target system, you will
308 have to specify the path using "setup" parameter. See "Test file
309 layout depending on system".
310 · Create configuration instance, load config data and check its
312 validity. Use "load_check => 'no'" if your file is not valid.
313 · Check for config data warnings. You should pass the list of
315 expected warnings that are emitted through Log::Log4Perl. The array
316 ref is passed as is to the "expect" function of "expect" in
317 Test::Log::Lo4Perl. E.g:
318 log4perl_load_warnings => [
320 [ 'Tree.Node', (warn => qr/deprecated/) x 2 ] ,
321 [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
322 ]
323 The Log classes are specified in "cme/Logging".
325 Log levels below "warn" are ignored.
327 Config::Model is currently transitioning from traditional "warn" to
329 warn logs. To avoid breaking all tests based on this module, the
330 warnings are emitted through Log::Log4Perl only when
331 c<$::_use_log4perl_to_warn> is set. This hack will be removed once
332 all warnings checks in tests are ported to log4perl checks.
333 · DEPRECATED. Check for config data warning. You should pass the list
335 of expected warnings. E.g.
336 load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
338 Use an empty array_ref to mask load warnings.
340 · Optionally run update command:
342 update => {
344 [ returns => 'foo' , ]
345 no_warnings => [ 0 | 1 ], # default 0
346 quiet => [ 0 | 1], # default 0, passed to update method
347 loag4perl_update_warnings => [ ... ] # Test::Log::Log4Perl::expect arguments
348 }
349 Where:
351 · "returns" is the expected return value (optional).
353 · "no_warnings" to suppress the warnings coming from
355 Config::Model::Value. Note that "no_warnings => 1" may be
356 useful for less verbose test.
357 · "quiet" to suppress progress messages during update.
359 · "log4perl_update_warnings" is used to check the warnings
361 produced during update. The argument is passed to "expect"
362 function of Test::Log::Log4Perl. See "load_warnings" parameter
363 above for more details.
364 · DEPRECATED. "update_warnings" is an array ref of quoted regexp
366 (See qr operator) to check the warnings produced during update.
367 use "update => []" to check that no warnings are issued during
368 update.
369 All other arguments are passed to "update" method.
371 · Optionally load configuration data. You should design this config
373 data to suppress any error or warning mentioned above. E.g:
374 load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
376 See Config::Model::Loader for the syntax of the string accepted by
378 "load" parameter.
379 · Optionally, run a check before running apply_fix (if any). This
381 step is useful to check warning messages:
382 check_before_fix => {
384 dump_errors => [ ... ] # optional, see below
385 load4perl_dump_warnings => [ ... ] # optional, see below
386 }
387 Use "dump_errors" if you expect issues:
389 check_before_fix => {
391 dump_errors => [
392 # the issues and a way to fix the issue using Config::Model::Node::load
393 qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
394 qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
395 ],
396 }
397 Likewise, specify any expected warnings:
399 check_before_fix => {
401 log4perl_dump_warnings => [ ... ],
402 }
403 "log4perl_dump_warnings" passes the array ref content to "expect"
405 function of Test::Log::Log4Perl.
406 Both "log4perl_dump_warnings" and "dump_errors" can be specified in
408 "check_before_fix" hash.
409 · Optionally, call apply_fixes:
411 apply_fix => 1,
413 · Call dump_tree to check the validity of the data after optional
415 "apply_fix". This step is not optional.
416 As with "check_before_fix", both "dump_errors" or "dump_warnings"
418 can be used.
419 · Run specific content check to verify that configuration data was
421 retrieved correctly:
422 check => {
424 'fs:/proc fs_spec' => "proc",
425 'fs:/proc fs_file' => "/proc",
426 'fs:/home fs_file' => "/home",
427 },
428 The keys of the hash points to the value to be checked using the
430 syntax described in "grab" in Config::Model::Role::Grab.
431 Multiple check on the same item can be applied with a array ref:
433 check => [
435 Synopsis => 'fix undefined path_max for st_size zero',
436 Description => [ qr/^The downstream/, qr/yada yada/ ]
437 ]
438 You can run check using different check modes (See "fetch" in
440 Config::Model::Value) by passing a hash ref instead of a scalar :
441 check => {
443 'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
444 'sections:base packages:0' => { mode => 'layered', value => "gcc-4.2-base' },
445 },
446 The whole hash content (except "value") is passed to grab and
448 fetch
449 A regexp can also be used to check value:
451 check => {
453 "License text" => qr/gnu/i,
454 }
455 And specification can nest hash or array style:
457 check => {
459 "License:0 text" => qr/gnu/i,
460 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
461 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
462 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
463 }
464 · Verify if a hash contains one or more keys (or keys matching a
466 regexp):
467 has_key => [
469 'sections' => 'debian', # sections must point to a hash element
470 'control' => [qw/source binary/],
471 'copyright Files' => qr/.c$/,
472 'copyright Files' => [qr/\.h$/], qr/\.c$/],
473 ],
474 · Verify that a hash does not have a key (or a key matching a
476 regexp):
477 has_not_key => [
479 'copyright Files' => qr/.virus$/ # silly, isn't ?
480 ],
481 · Verify annotation extracted from the configuration file comments:
483 verify_annotation => {
485 'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
486 'source Maintainer' => "what a fine\nteam this one is",
487 },
488 · Write back the config data in "wr_root/model_tests/<subtest
490 name>/". Note that write back is forced, so the tested
491 configuration files are written back even if the configuration
492 values were not changed during the test.
493 You can skip warning when writing back with the global :
495 no_warnings => 1,
497 · Check the content of the written files(s) with
499 Test::File::Contents. Tests can be grouped in an array ref:
500 file_contents => {
502 "/home/foo/my_arm.conf" => "really big string" ,
503 "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
504 }
505 file_contents_like => {
507 "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
508 }
509 file_contents_unlike => {
511 "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
512 }
513 · Check the mode of the written files:
515 file_mode => {
517 "~/.ssh/ssh_config" => oct(600), # better than 0600
518 "debian/stuff.postinst" => oct(755),
519 }
520 Only the last four octets of the mode are tested. I.e. the test is
522 done with " $file_mode & oct(7777) "
523 Note: this test is skipped on Windows
525 · Check added or removed configuration files. If you expect changes,
527 specify a subref to alter the file list:
528 file_check_sub => sub {
530 my $list_ref = shift ;
531 # file added during tests
532 push @$list_ref, "/debian/source/format" ;
533 },
534 Note that actual and expected file lists are sorted before check,
536 adding a file can be done with "push".
537 · Copy all config data from "wr_root/model_tests/<subtest name>/" to
539 "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
540 check that configuration written back has the same content as the
541 original configuration.
542 · Create a second configuration instance to read the conf file that
544 was just copied (configuration data is checked.)
545 · You can skip the load check if the written file still contain
547 errors (e.g. some errors were ignored and cannot be fixed) with
548 "load_check2 => 'no'"
549 · Optionally load configuration data in the second instance. You
551 should design this config data to suppress any error or warning
552 that occur in the step below. E.g:
553 load2 => 'binary:seaview',
555 See Config::Model::Loader for the syntax of the string accepted by
557 "load2" parameter.
558 · Compare data read from original data.
560 · Run specific content check on the written config file to verify
562 that configuration data was written and retrieved correctly:
563 wr_check => {
565 'fs:/proc fs_spec' => "proc" ,
566 'fs:/proc fs_file' => "/proc",
567 'fs:/home fs_file' => "/home",
568 },
569 Like the "check" item explained above, you can run "wr_check" using
571 different check modes.
572 Running the test
574 Run all tests with one of these commands:
575 prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
577 perl -Ilib t/model_test.t [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
578 By default, all tests are run on all models.
580 You can pass arguments to "t/model_test.t":
582 · Optional parameters: "--trace" to get test traces. "--error" to get
584 stack trace in case of errors, "--log" to have logs. E.g.
585 # run with log and error traces
587 prove -lv t/model_test.t :: --error --logl
588 · The model name to tests. E.g.:
590 # run only fstab tests
592 prove -lv t/model_test.t :: fstab
593 · A regexp to filter subtest E.g.:
595 # run only fstab tests foobar subtest
597 prove -lv t/model_test.t :: fstab foobar
598 # run only fstab tests foo subtest
600 prove -lv t/model_test.t :: fstab '^foo$'
601
603 Some of these examples may still use global variables (which is
604 deprecated). Such files may be considered as buggy after Aug 2019.
605 Please warn the author if you find one.
606 · LCDproc <http://lcdproc.org> has a single configuration file:
608 "/etc/LCDd.conf". Here's LCDproc test layout
609 <https://github.com/dod38fr/config-model-
610 lcdproc/tree/master/t/model_tests.d> and the test specification
611 <https://github.com/dod38fr/config-model-
612 lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
613 · Dpkg packages are constructed from several files. These files are
615 handled like configuration files by Config::Model::Dpkg. The test
616 layout <http://anonscm.debian.org/gitweb/?p=pkg-
617 perl/packages/libconfig-model-dpkg-
618 perl.git;a=tree;f=t/model_tests.d;hb=HEAD> features test with
619 multiple file in dpkg-examples
620 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
621 model-dpkg-perl.git;a=tree;f=t/model_tests.d/dpkg-
622 examples;hb=HEAD>. The test is specified in dpkg-test-conf.pl
623 <http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libconfig-
624 model-dpkg-perl.git;a=blob_plain;f=t/model_tests.d/dpkg-test-
625 conf.pl;hb=HEAD>
626 · multistrap-test-conf.pl <https://github.com/dod38fr/config-
628 model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
629 multistrap-examples <https://github.com/dod38fr/config-
630 model/tree/master/t/model_tests.d/multistrap-examples> specify a
631 test where the configuration file name is not imposed by the
632 application. The file name must then be set in the test
633 specification.
634 · backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
636 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
637 a more complex example showing how to test a backend. The test is
638 done creating a dummy model within the test specification.
639
641 In alphabetical order:
642 · Cyrille Bollu
644 Many thanks for your help.
646
648 · Config::Model
649 · Test::More
651
653 Dominique Dumont
654
656 This software is Copyright (c) 2013-2019 by Dominique Dumont.
657 This is free software, licensed under:
659 The GNU Lesser General Public License, Version 2.1, February 1999
661
663 Websites
664 The following websites have more information about this module, and may
665 be of help to you. As always, in addition to those websites please use
666 your favorite search engine to discover more resources.
667 · Search CPAN
669 The default CPAN search engine, useful to view POD in HTML format.
671 <http://search.cpan.org/dist/Config-Model-Tester>
673 · AnnoCPAN
675 The AnnoCPAN is a website that allows community annotations of Perl
677 module documentation.
678 <http://annocpan.org/dist/Config-Model-Tester>
680 · CPAN Ratings
682 The CPAN Ratings is a website that allows community ratings and
684 reviews of Perl modules.
685 <http://cpanratings.perl.org/d/Config-Model-Tester>
687 · CPANTS
689 The CPANTS is a website that analyzes the Kwalitee ( code metrics )
691 of a distribution.
692 <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
694 · CPAN Testers
696 The CPAN Testers is a network of smoke testers who run automated
698 tests on uploaded CPAN distributions.
699 <http://www.cpantesters.org/distro/C/Config-Model-Tester>
701 · CPAN Testers Matrix
703 The CPAN Testers Matrix is a website that provides a visual
705 overview of the test results for a distribution on various
706 Perls/platforms.
707 <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
709 · CPAN Testers Dependencies
711 The CPAN Testers Dependencies is a website that shows a chart of
713 the test results of all dependencies for a distribution.
714 <http://deps.cpantesters.org/?module=Config::Model::Tester>
716 Bugs / Feature Requests
718 Please report any bugs or feature requests by email to "ddumont at
719 cpan.org", or through the web interface at
720 <https://github.com/dod38fr/config-model-tester/issues>. You will be
721 automatically notified of any progress on the request by the system.
722 Source Code
724 The code is open to the world, and available for you to hack on. Please
725 feel free to browse it and play with it, or whatever. If you want to
726 contribute patches, please send me a diff or prod me to pull from your
727 repository :)
728 <http://github.com/dod38fr/config-model-tester.git>
730 git clone git://github.com/dod38fr/config-model-tester.git
732perl v5.30.0 2019-07-26 Config::Model::Tester(3)