1Config::Model::Tester(3U)ser Contributed Perl DocumentatiCoonnfig::Model::Tester(3)
2
3
4
6 Config::Model::Tester - Test framework for Config::Model
7
9 version 4.007
10
12 In your test file (typically "t/model_test.t"):
13
14 use warnings;
15 use strict;
16
17 use Config::Model::Tester ;
18 use ExtUtils::testlib;
19
20 run_tests() ;
21
22 Run tests with:
23
24 perl t/model_test.t [ --log ] [--error] [--trace] [ subtest [ test_case ] ]
25
27 This class provides a way to test configuration models with tests
28 files. This class was designed to tests several models and run several
29 tests cases per model.
30
31 A specific layout for test files must be followed.
32
33 Sub test specification
34 Each subtest is defined in a file like:
35
36 t/model_tests.d/<app-name>-test-conf.pl
37
38 This file specifies that "app-name" (which is defined in
39 "lib/Config/Model/*.d" directory) is used for the test cases defined in
40 the "*-test-conf.pl" file. The model to test is inferred from the
41 application name to test.
42
43 This file contains a list of test case (explained below) and expects a
44 set of files used as test data. The layout of these test data files is
45 explained in next section.
46
47 Simple test file layout
48 Each test case is represented by a configuration file (not a directory)
49 in the "*-examples" directory. This configuration file is used by the
50 model to test and is copied as "$confdir/$conf_file_name" using the
51 test data structure explained below.
52
53 In the example below, we have 1 app model to test: "lcdproc" and 2
54 tests cases. The app name matches the file specified in
55 "lib/Config/Model/*.d" directory. In this case, the app name matches
56 "lib/Config/Model/system.d/lcdproc"
57
58 t
59 |-- model_test.t
60 \-- model_tests.d # do not change directory name
61 |-- lcdproc-test-conf.pl # subtest specification for lcdproc app
62 \-- lcdproc-examples
63 |-- t0 # test case t0
64 \-- LCDD-0.5.5 # test case for older LCDproc
65
66 Subtest specification is written in "lcdproc-test-conf.pl" file (i.e.
67 this module looks for files named like "<app-name>-test-conf.pl>").
68
69 Subtests data is provided in files in directory "lcdproc-examples" (
70 i.e. this modules looks for test data in directory
71 "<model-name>-examples>". "lcdproc-test-conf.pl" contains instructions
72 so that each file is used as a "/etc/LCDd.conf" file during each test
73 case.
74
75 "lcdproc-test-conf.pl" can contain specifications for more test cases.
76 Each test case requires a new file in "lcdproc-examples" directory.
77
78 See "Examples" for a link to the actual LCDproc model tests
79
80 Test file layout for multi-file configuration
81 When a configuration is spread over several files, each test case is
82 provided in a sub-directory. This sub-directory is copied in "conf_dir"
83 (a test parameter as explained below)
84
85 In the example below, the test specification is written in
86 "dpkg-test-conf.pl". Dpkg layout requires several files per test case.
87 "dpkg-test-conf.pl" contains instructions so that each directory under
88 "dpkg-examples" is used.
89
90 t/model_tests.d
91 \-- dpkg-test-conf.pl # subtest specification
92 \-- dpkg-examples
93 \-- libversion # example subdir, used as test case name
94 \-- debian # directory for used by test case
95 |-- changelog
96 |-- compat
97 |-- control
98 |-- copyright
99 |-- rules
100 |-- source
101 | \-- format
102 \-- watch
103
104 See "Examples" for a link to the (many) Dpkg model tests
105
106 More complex file layout
107 Each test case is a sub-directory on the "*-examples" directory and
108 contains several files. The destination of the test files may depend on
109 the system (e.g. the OS). For instance, system wide "ssh_config" is
110 stored in "/etc/ssh" on Linux, and directly in "/etc" on MacOS.
111
112 These files are copied in a test directory using a "setup" parameter in
113 test case specification.
114
115 Let's consider this example of 2 tests cases for ssh:
116
117 t/model_tests.d/
118 |-- ssh-test-conf.pl
119 |-- ssh-examples
120 \-- basic
121 |-- system_ssh_config
122 \-- user_ssh_config
123
124 Unfortunately, "user_ssh_config" is a user file, so you need to specify
125 where is located the home directory of the test with another global
126 parameter:
127
128 home_for_test => '/home/joe' ;
129
130 For Linux only, the "setup" parameter is:
131
132 setup => {
133 system_ssh_config => '/etc/ssh/ssh_config',
134 user_ssh_config => "~/.ssh/config"
135 }
136
137 On the other hand, system wide config file is different on MacOS and
138 the test file must be copied in the correct location. When the value of
139 the "setup" hash is another hash, the key of this other hash is used as
140 to specify the target location for other OS (as returned by Perl $^O
141 variable:
142
143 setup => {
144 'system_ssh_config' => {
145 'darwin' => '/etc/ssh_config',
146 'default' => '/etc/ssh/ssh_config',
147 },
148 'user_ssh_config' => "~/.ssh/config"
149 }
150
151 "systemd" is another beast where configuration files can be symlinks to
152 "/dev/null" or other files. To emulate this situation, use an array as
153 setup target:
154
155 setup => {
156 # test data file => [ link (may be repeated), .. link(s) target contains test data ]
157 'ssh.service' => [ '/etc/systemd/system/sshd.service', '/lib/systemd/system/ssh.service' ]
158 }
159
160 This will result in a symlink like:
161
162 wr_root/model_tests/test-sshd-service/etc/systemd/system/sshd.service
163 -> /absolute_path_to/wr_root/model_tests/test-sshd-service/lib/systemd/system/ssh.service
164
165 See the actual Ssh and Sshd model tests
166 <https://github.com/dod38fr/config-model-
167 openssh/tree/master/t/model_tests.d>
168
169 Basic test specification
170 Each model subtest is specified in "<app>-test-conf.pl". This file must
171 return a data structure containing the test specifications. Each test
172 data structure contains global parameters (Applied to all tests cases)
173 and test cases parameters (parameters are applied to the test case)
174
175 use strict;
176 use warnings;
177 {
178 # global parameters
179
180 # config file name (used to copy test case into test wr_root/model_tests directory)
181 conf_file_name => "fstab",
182 # config dir where to copy the file (optional)
183 conf_dir => "etc",
184 # home directory for this test
185 home_for_test => '/home/joe'
186
187 tests => [
188 {
189 # test case 1
190 name => 'my_first_test',
191 # other test case parameters
192 },
193 {
194 # test case 2
195 name => 'my_second_test',
196 # other test case parameters
197 },
198 # ...
199 ],
200 };
201
202 # do not add 1; at the end of the file
203
204 In the example below, "t0" file is copied in
205 "wr_root/model_tests/test-t0/etc/fstab".
206
207 use strict;
208 use warnings;
209 {
210 # list of tests.
211 tests => [
212 {
213 # test name
214 name => 't0',
215 # add optional specification here for t0 test
216 },
217 {
218 name => 't1',
219 # add optional specification here for t1 test
220 },
221 ]
222 };
223
224 You can suppress warnings by specifying "no_warnings => 1" in each test
225 case. On the other hand, you may also want to check for warnings
226 specified to your model. In this case, you should avoid specifying
227 "no_warnings" here and specify warning tests or warning filters as
228 mentioned below.
229
230 See actual fstab test <https://github.com/dod38fr/config-
231 model/blob/master/t/model_tests.d/fstab-test-conf.pl>.
232
233 Skip a test
234 A test file can be skipped using "skip" global test parameter.
235
236 In this example, test is skipped when not running on a Debian system:
237
238 eval { require AptPkg::Config; };
239 my $skip = ( $@ or not -r '/etc/debian_version' ) ? 1 : 0;
240
241 {
242 skip => $skip,
243 tests => [ ] ,
244 };
245
246 Internal tests or backend tests
247 Some tests require the creation of a configuration class dedicated for
248 test (typically to test corner cases on a backend).
249
250 This test class can be created directly in the test specification by
251 specifying tests classes in "config_classes" global test parameter in
252 an array ref. Each array element is a data structure that use
253 create_config_class parameters. See for instance the layer test
254 <https://github.com/dod38fr/config-
255 model/blob/master/t/model_tests.d/layer-test-conf.pl> or the test for
256 shellvar backend <https://github.com/dod38fr/config-
257 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl>.
258
259 In this case, no application exist for such classes so the model to
260 test must be specified in a global test parameter:
261
262 return {
263 config_classes => [ { name => "Foo", element => ... } , ... ],
264 model_to_test => "Foo",
265 tests => [ ... ]
266 };
267
268 Test specification with arbitrary file names
269 In some models, like "Multistrap", the config file is chosen by the
270 user. In this case, the file name must be specified for each tests
271 case:
272
273 {
274 tests => [ {
275 name => 'arm',
276 config_file => '/home/foo/my_arm.conf',
277 check => {},
278 }]
279 };
280
281 See the actual multistrap test <https://github.com/dod38fr/config-
282 model/blob/master/t/model_tests.d/multistrap-test-conf.pl>.
283
284 Backend argument
285 Some application like systemd requires a backend argument specified by
286 user (e.g. a service name for systemd). The parameter "backend_arg" can
287 be specified to emulate this behavior.
288
289 Re-use test data
290 When the input data for test is quite complex (several files), it may
291 be interesting to re-use these data for other test cases. Knowing that
292 test names must be unique, you can re-use test data with "data_from"
293 parameter. For instance:
294
295 tests => [
296 {
297 name => 'some-test',
298 # ...
299 },
300 {
301 name => 'some-other-test',
302 data_from => 'some-test', # re-use data from test above
303 # ...
304 },
305 ]
306
307 See plainfile backend test <https://github.com/dod38fr/config-
308 model/blob/master/t/model_tests.d/backend-plainfile-test-conf.pl> for a
309 real life example.
310
311 Likewise, it may be useful to re-use test data from another group of
312 test. Lets see this example from "systemd-service-test-conf.pl":
313
314 {
315 name => 'transmission',
316 data_from_group => 'systemd', # i.e from ../systemd-examples
317 }
318
319 "data_from" and "data_from_group" can be together.
320
321 Test scenario
322 Each subtest follow a sequence explained below. Each step of this
323 sequence may be altered by adding test case parameters in
324 "<model-to-test>-test-conf.pl":
325
326 • Setup test in "wr_root/model_tests/<subtest name>/". If your
327 configuration file layout depend on the target system, you will
328 have to specify the path using "setup" parameter. See "More
329 complex file layout".
330
331 • Create configuration instance, load config data and check its
332 validity. Use "load_check => 'no'" if your file is not valid.
333
334 • Check for config data warnings. You should pass the list of
335 expected warnings that are emitted through Log::Log4perl. The array
336 ref is passed as is to the "expect" function of "expect" in
337 Test::Log::Lo4Perl. E.g:
338
339 log4perl_load_warnings => [
340 [ 'Tree.Node', (warn => qr/deprecated/) x 2 ] ,
341 [ 'Tree.Element.Value' , ( warn => qr/skipping/) x 2 ]
342 ]
343
344 The Log classes are specified in "cme/Logging".
345
346 Log levels below "warn" are ignored.
347
348 Note that log tests are disabled when "--log" option is used, hence
349 all warnings triggered by the tests are shown.
350
351 Config::Model is currently transitioning from traditional "warn" to
352 warn logs. To avoid breaking all tests based on this module, the
353 warnings are emitted through Log::Log4perl only when
354 $::_use_log4perl_to_warn is set. This hack will be removed once all
355 warnings checks in tests are ported to log4perl checks.
356
357 • DEPRECATED. Check for config data warning. You should pass the list
358 of expected warnings. E.g.
359
360 load_warnings => [ qr/Missing/, (qr/deprecated/) x 3 , ],
361
362 Use an empty array_ref to mask load warnings.
363
364 • Optionally run update command:
365
366 update => {
367 returns => 'foo' , # optional
368 no_warnings => [ 0 | 1 ], # default 0
369 quiet => [ 0 | 1], # default 0, passed to update method
370 load4perl_update_warnings => [ ... ] # Test::Log::Log4perl::expect arguments
371 }
372
373 Where:
374
375 • "returns" is the expected return value (optional).
376
377 • "no_warnings" can be used to suppress the warnings coming from
378 Config::Model::Value. Note that "no_warnings => 1" may be
379 useful for less verbose test.
380
381 • "quiet" to suppress progress messages during update.
382
383 • "log4perl_update_warnings" is used to check the warnings
384 produced during update. The argument is passed to "expect"
385 function of Test::Log::Log4perl. See "load_warnings" parameter
386 above for more details.
387
388 • DEPRECATED. "update_warnings" is an array ref of quoted regexp
389 (See qr operator) to check the warnings produced during update.
390 Please use "log4perl_update_warnings" instead.
391
392 All other arguments are passed to "update" method.
393
394 • Optionally load configuration data. You should design this config
395 data to suppress any error or warning mentioned above. E.g:
396
397 load => 'binary:seaview Synopsis="multiplatform interface for sequence alignment"',
398
399 See Config::Model::Loader for the syntax of the string accepted by
400 "load" parameter.
401
402 • Optionally, run a check before running apply_fix (if any). This
403 step is useful to check warning messages:
404
405 check_before_fix => {
406 dump_errors => [ ... ] # optional, see below
407 log4perl_dump_warnings => [ ... ] # optional, see below
408 }
409
410 Use "dump_errors" if you expect issues:
411
412 check_before_fix => {
413 dump_errors => [
414 # the issues and a way to fix the issue using Config::Model::Node::load
415 qr/mandatory/ => 'Files:"*" Copyright:0="(c) foobar"',
416 qr/mandatory/ => ' License:FOO text="foo bar" ! Files:"*" License short_name="FOO" '
417 ],
418 }
419
420 Likewise, specify any expected warnings:
421
422 check_before_fix => {
423 log4perl_dump_warnings => [ ... ],
424 }
425
426 "log4perl_dump_warnings" passes the array ref content to "expect"
427 function of Test::Log::Log4perl.
428
429 Both "log4perl_dump_warnings" and "dump_errors" can be specified in
430 "check_before_fix" hash.
431
432 • Optionally, call apply_fixes:
433
434 apply_fix => 1,
435
436 • Call dump_tree to check the validity of the data after optional
437 "apply_fix". This step is not optional.
438
439 As with "check_before_fix", both "dump_errors" or
440 "log4perl_dump_warnings" can be specified in "full_dump" parameter:
441
442 full_dump => {
443 log4perl_dump_warnings => [ ... ], # optional
444 dump_errors => [ ... ], # optional
445 }
446
447 • Run specific content check to verify that configuration data was
448 retrieved correctly:
449
450 check => {
451 'fs:/proc fs_spec' => "proc",
452 'fs:/proc fs_file' => "/proc",
453 'fs:/home fs_file' => "/home",
454 },
455
456 The keys of the hash points to the value to be checked using the
457 syntax described in "grab" in Config::Model::Role::Grab.
458
459 Multiple check on the same item can be applied with a array ref:
460
461 check => [
462 Synopsis => 'fix undefined path_max for st_size zero',
463 Description => [ qr/^The downstream/, qr/yada yada/ ]
464 ]
465
466 You can run check using different check modes (See "fetch" in
467 Config::Model::Value) by passing a hash ref instead of a scalar :
468
469 check => {
470 'sections:debian packages:0' => { mode => 'layered', value => 'dpkg-dev' },
471 'sections:base packages:0' => { mode => 'layered', value => "gcc-4.2-base' },
472 },
473
474 The whole hash content (except "value") is passed to grab and
475 fetch
476
477 A regexp can also be used to check value:
478
479 check => {
480 "License text" => qr/gnu/i,
481 }
482
483 And specification can nest hash or array style:
484
485 check => {
486 "License:0 text" => qr/gnu/i,
487 "License:1 text" => [ qr/gnu/i, qr/Stallman/ ],
488 "License:2 text" => { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
489 "License:3 text" => [ qr/General/], { mode => 'custom', value => [ qr/gnu/i , qr/Stallman/ ] },
490 }
491
492 • Verify if a hash contains one or more keys (or keys matching a
493 regexp):
494
495 has_key => [
496 'sections' => 'debian', # sections must point to a hash element
497 'control' => [qw/source binary/],
498 'copyright Files' => qr/.c$/,
499 'copyright Files' => [qr/\.h$/], qr/\.c$/],
500 ],
501
502 • Verify that a hash does not have a key (or a key matching a
503 regexp):
504
505 has_not_key => [
506 'copyright Files' => qr/.virus$/ # silly, isn't ?
507 ],
508
509 • Verify annotation extracted from the configuration file comments:
510
511 verify_annotation => {
512 'source Build-Depends' => "do NOT add libgtk2-perl to build-deps (see bug #554704)",
513 'source Maintainer' => "what a fine\nteam this one is",
514 },
515
516 • Write back the config data in "wr_root/model_tests/<subtest
517 name>/". Note that write back is forced, so the tested
518 configuration files are written back even if the configuration
519 values were not changed during the test.
520
521 You can skip warning when writing back with the global :
522
523 no_warnings => 1,
524
525 • Check the content of the written files(s) with
526 Test::File::Contents. Tests can be grouped in an array ref:
527
528 file_contents => {
529 "/home/foo/my_arm.conf" => "really big string" ,
530 "/home/bar/my_arm.conf" => [ "really big string" , "another"], ,
531 }
532
533 file_contents_like => {
534 "/home/foo/my_arm.conf" => [ qr/should be there/, qr/as well/ ] ,
535 }
536
537 file_contents_unlike => {
538 "/home/foo/my_arm.conf" => qr/should NOT be there/ ,
539 }
540
541 • Check the mode of the written files:
542
543 file_mode => {
544 "~/.ssh/ssh_config" => oct(600), # better than 0600
545 "debian/stuff.postinst" => oct(755),
546 }
547
548 Only the last four octets of the mode are tested. I.e. the test is
549 done with " $file_mode & oct(7777) "
550
551 Note: this test is skipped on Windows
552
553 • Check added or removed configuration files. If you expect changes,
554 specify a subref to alter the file list:
555
556 file_check_sub => sub {
557 my $list_ref = shift ;
558 # file added during tests
559 push @$list_ref, "/debian/source/format" ;
560 },
561
562 Note that actual and expected file lists are sorted before check,
563 adding a file can be done with "push".
564
565 • Copy all config data from "wr_root/model_tests/<subtest name>/" to
566 "wr_root/model_tests/<subtest name>-w/". This steps is necessary to
567 check that configuration written back has the same content as the
568 original configuration.
569
570 • Create a second configuration instance to read the conf file that
571 was just copied (configuration data is checked.)
572
573 • You can skip the load check if the written file still contain
574 errors (e.g. some errors were ignored and cannot be fixed) with
575 "load_check2 => 'no'"
576
577 • Optionally load configuration data in the second instance. You
578 should design this config data to suppress any error or warning
579 that occur in the step below. E.g:
580
581 load2 => 'binary:seaview',
582
583 See Config::Model::Loader for the syntax of the string accepted by
584 "load2" parameter.
585
586 • Compare data read from original data.
587
588 • Run specific content check on the written config file to verify
589 that configuration data was written and retrieved correctly:
590
591 wr_check => {
592 'fs:/proc fs_spec' => "proc" ,
593 'fs:/proc fs_file' => "/proc",
594 'fs:/home fs_file' => "/home",
595 },
596
597 Like the "check" item explained above, you can run "wr_check" using
598 different check modes.
599
600 Running the test
601 Run all tests with one of these commands:
602
603 prove -l t/model_test.t :: [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
604 perl -Ilib t/model_test.t [ --trace ] [ --log ] [ --error ] [ <model_name> [ <regexp> ]]
605
606 By default, all tests are run on all models.
607
608 You can pass arguments to "t/model_test.t":
609
610 • Optional parameters: "--trace" to get test traces. "--error" to get
611 stack trace in case of errors, "--log" to have logs. E.g.
612
613 # run with log and error traces
614 prove -lv t/model_test.t :: --error --logl
615
616 • The model name to tests. E.g.:
617
618 # run only fstab tests
619 prove -lv t/model_test.t :: fstab
620
621 • A regexp to filter subtest E.g.:
622
623 # run only fstab tests foobar subtest
624 prove -lv t/model_test.t :: fstab foobar
625
626 # run only fstab tests foo subtest
627 prove -lv t/model_test.t :: fstab '^foo$'
628
630 Some of these examples may still use global variables (which is
631 deprecated). Such files may be considered as buggy after Aug 2019.
632 Please warn the author if you find one.
633
634 • LCDproc <http://lcdproc.org> has a single configuration file:
635 "/etc/LCDd.conf". Here's LCDproc test layout
636 <https://github.com/dod38fr/config-model-
637 lcdproc/tree/master/t/model_tests.d> and the test specification
638 <https://github.com/dod38fr/config-model-
639 lcdproc/blob/master/t/model_tests.d/lcdd-test-conf.pl>
640
641 • Dpkg packages are constructed from several files. These files are
642 handled like configuration files by Config::Model::Dpkg
643 <https://salsa.debian.org/perl-team/modules/packages/libconfig-
644 model-dpkg-perl>. The test layout <https://salsa.debian.org/perl-
645 team/modules/packages/libconfig-model-dpkg-
646 perl/-/tree/master/t/model_tests.d> features test with multiple
647 file in dpkg-examples <https://salsa.debian.org/perl-
648 team/modules/packages/libconfig-model-dpkg-
649 perl/-/tree/master/t/model_tests.d/dpkg-examples>. The test is
650 specified in
651 <https://salsa.debian.org/perl-team/modules/packages/libconfig-model-dpkg-perl/-/blob/master/t/model_tests.d/dpkg-test-conf.pl>
652
653 • multistrap-test-conf.pl <https://github.com/dod38fr/config-
654 model/blob/master/t/model_tests.d/multistrap-test-conf.pl> and
655 multistrap-examples <https://github.com/dod38fr/config-
656 model/tree/master/t/model_tests.d/multistrap-examples> specify a
657 test where the configuration file name is not imposed by the
658 application. The file name must then be set in the test
659 specification.
660
661 • backend-shellvar-test-conf.pl <https://github.com/dod38fr/config-
662 model/blob/master/t/model_tests.d/backend-shellvar-test-conf.pl> is
663 a more complex example showing how to test a backend. The test is
664 done creating a dummy model within the test specification.
665
667 In alphabetical order:
668
669 • Cyrille Bollu
670
671 Many thanks for your help.
672
674 • Config::Model
675
676 • Test::More
677
679 Dominique Dumont
680
682 This software is Copyright (c) 2013-2020 by Dominique Dumont.
683
684 This is free software, licensed under:
685
686 The GNU Lesser General Public License, Version 2.1, February 1999
687
689 Websites
690 The following websites have more information about this module, and may
691 be of help to you. As always, in addition to those websites please use
692 your favorite search engine to discover more resources.
693
694 • CPANTS
695
696 The CPANTS is a website that analyzes the Kwalitee ( code metrics )
697 of a distribution.
698
699 <http://cpants.cpanauthors.org/dist/Config-Model-Tester>
700
701 • CPAN Testers
702
703 The CPAN Testers is a network of smoke testers who run automated
704 tests on uploaded CPAN distributions.
705
706 <http://www.cpantesters.org/distro/C/Config-Model-Tester>
707
708 • CPAN Testers Matrix
709
710 The CPAN Testers Matrix is a website that provides a visual
711 overview of the test results for a distribution on various
712 Perls/platforms.
713
714 <http://matrix.cpantesters.org/?dist=Config-Model-Tester>
715
716 • CPAN Testers Dependencies
717
718 The CPAN Testers Dependencies is a website that shows a chart of
719 the test results of all dependencies for a distribution.
720
721 <http://deps.cpantesters.org/?module=Config::Model::Tester>
722
723 Bugs / Feature Requests
724 Please report any bugs or feature requests by email to "ddumont at
725 cpan.org", or through the web interface at
726 <https://github.com/dod38fr/config-model-tester/issues>. You will be
727 automatically notified of any progress on the request by the system.
728
729 Source Code
730 The code is open to the world, and available for you to hack on. Please
731 feel free to browse it and play with it, or whatever. If you want to
732 contribute patches, please send me a diff or prod me to pull from your
733 repository :)
734
735 <http://github.com/dod38fr/config-model-tester.git>
736
737 git clone git://github.com/dod38fr/config-model-tester.git
738
739
740
741perl v5.36.0 2023-01-20 Config::Model::Tester(3)