1CH-TEST(1) Charliecloud CH-TEST(1)
2
6 ch-test - Run some or all of the Charliecloud test suite
7
9 $ ch-test [PHASE] [--scope SCOPE] [ARGS]
10
12 Charliecloud comes with a comprehensive test suite that exercises the
13 container workflow itself as well as a few example applications.
14 ch-test coordinates running the test suite.
15 While the CLI has lots of options, the defaults are reasonable, and
17 bare ch-test will give useful results in a few minutes on single-node,
18 internet-connected systems with a few GB available in /var/tmp.
19 The test suite requires a few GB (standard scope) or tens of GB (full
21 scope) of storage for test fixtures:
22 • Builder storage (e.g., layer cache). This goes wherever the builder
24 puts it.
25 • Packed images directory: image tarballs or SquashFS files.
27 • Unpacked images directory. Images are unpacked into and then run from
29 here.
30 • Filesystem permissions directories. These are used to test that the
32 kernel is enforcing permissions correctly. Note that this exercises
33 the kernel, not Charliecloud, and can be omitted from routine Char‐
34 liecloud testing.
35 The first three are created when needed if they don’t exist, while the
37 filesystem permissions fixtures must be created manually, in order to
38 accommodate configurations where sudo is not available via the same lo‐
39 gin path used for running tests.
40 The packed and unpacked image directories specified for testing are
42 volatile. The contents of these directories are deleted before the
43 build and run phases, respectively.
44 In all four cases, when creating directories, only the final path com‐
46 ponent is created. Parent directories must already exist, i.e., ch-test
47 uses the behavior of mkdir rather than mkdir -p.
48 Some of the tests exercise parallel functionality. If ch-test is run on
50 a single node, multiple cores will be used; if in a Slurm allocation,
51 multiple nodes too.
52 The subset of tests to run mostly splits along two key dimensions. The
54 phase is which parts of the workflow to run. Different parts of the
55 workflow can be tested on different systems by copying the necessary
56 artifacts between them, e.g. by building images on one system and run‐
57 ning them on another. The scope allows trading off thoroughness versus
58 time.
59 PHASE must be one of the following:
61 build Image building and associated functionality, with the se‐
63 lected builder.
64 run Running containers and associated functionality. This re‐
66 quires a packed images directory produced by a successful
67 build phase, which can be copied from the build system if
68 it’s not also the run system.
69 examples
71 Example applications. Requires an unpacked images directory
72 produced by a successful run phase.
73 all Execute phases build, run, and examples, in that order.
75 mk-perm-dirs
77 Create the filesystem permissions directories. Requires
78 --perm-dirs.
79 clean Delete automatically-generated test files, and packed and un‐
81 packed image directories.
82 rm-perm-dirs
84 Remove the filesystem permissions directories. Requires
85 --perm-dirs.
86 -f, --file FILE[:TEST]
88 Run the tests in the given file only, which can be an arbi‐
89 trary .bats file, except for test.bats under examples, where
90 you must specify the corresponding Dockerfile or Build file
91 instead. This is somewhat brittle and typically used for de‐
92 velopment or debugging. For example, it does not check
93 whether the pre-requisites of whatever is in the file are
94 satisfied. Often running build and run first is sufficient,
95 but this varies.
96 If TEST is also given, then run only the test with that name,
98 skipping the others. The separator is a literal colon. Most
99 test names contain spaces, so you’ll usually need to quote
100 the argument to protect it from the shell.
101 Scope is specified with:
103 -s, --scope SCOPE
105 SCOPE must be one of the following; the default is standard.
106 • quick: Most important subset of workflow. Handy for devel‐
108 opment. Completion time: 1–2 minutes.
109 • standard: All tested workflow functionality and a selection
111 of more important examples. Completion time: 5–10 minutes.
112 • full: All available tests, including all examples. Comple‐
114 tion time, hot cache: 7–15 minutes; cold cache: 1–2 hours.
115 Additional arguments:
117 -b, --builder BUILDER
119 Image builder to use. See ch-build(1) for how the default is
120 selected.
121 --dry-run
123 Print summary of what would be tested and then exit.
124 -h, --help
126 Print usage and then exit.
127 --img-dir DIR
129 Set unpacked images directory to DIR. In a multi-node alloca‐
130 tion, this directory may not be shared between nodes. De‐
131 fault: $CH_TEST_IMGDIR if set; otherwise /var/tmp/img.
132 --pack-dir DIR
134 Set packed images directory to DIR. Default: $CH_TEST_TARDIR
135 if set; otherwise /var/tmp/pack.
136 --pedantic (yes|no)
138 Some tests require configurations that are very specific
139 (e.g., being a member of at least two groups) or unusual
140 (e.g., sudo to a non-root group). If yes, then fail if the
141 requirement is not met; if no, then skip. The default is yes
142 for CI environments or people listed in README.md, no other‐
143 wise.
144 If yes and sudo seems to be available, implies --sudo.
146 --perm-dir DIR
148 Add DIR to filesystem permission fixture directories; can be
149 specified multiple times. We recommend one such directory per
150 mounted filesystem type whose kernel module you do not trust;
151 e.g., you probably don’t need to test your tmpfses, but
152 out-of-tree filesystems very likely need this.
153 Implies --sudo. Default: CH_TEST_PERMDIRS if set; otherwise
155 skip the filesystem permissions tests.
156 --pack-fmt FMT
158 Use packed image format FMT (squash or tar).
159 --sudo Enable things that require sudo, such as certain privilege
161 escalation tests and creating/removing the filesystem permis‐
162 sions fixtures. Requires generic sudo capabilities. Note that
163 the Docker builder uses sudo docker even without this option.
164 --lustre DIR
166 Use DIR for run-phase Lustre tests. Default: CH_TEST_LUS‐
167 TREDIR if set; otherwise skip them.
168 The tests will create, populate, and delete a new subdirec‐
170 tory under DIR, leaving everything else in DIR untouched.
171
173 Zero if all tests passed; non-zero if any failed. For setup and tear‐
174 down phases, zero if everything was created or deleted correctly,
175 non-zero otherwise.
176
178 Bats will wait until all descendant processes finish before exiting, so
179 if you get into a failure mode where a test sequence doesn’t clean up
180 all its processes, ch-test will hang.
181
183 Many systems can simply use the defaults. To run the build, run, and
184 examples phases on a single system, without the filesystem permissions
185 tests:
186 $ ch-test
188 ch-test version 0.12
189 ch-run: 0.12 /usr/local/bin/ch-run
191 bats: 0.4.0 /usr/bin/bats
192 tests: /usr/local/libexec/charliecloud/test
193 phase: build run examples
195 scope: standard (default)
196 builder: docker (default)
197 use generic sudo: no (default)
198 unpacked images dir: /var/tmp/img (default)
199 packed images dir: /var/tmp/tar (default)
200 fs permissions dirs: skip (default)
201 checking namespaces ...
203 ok
204 checking builder ...
206 found: /usr/bin/docker 19.03.2
207 bats build.bats build_auto.bats build_post.bats
209 ✓ documentation seems sane
210 ✓ version number seems sane
211 [...]
212 All tests passed.
213 The next example is for a more complex setup like you might find in HPC
215 centers:
216 • Non-default fixture directories.
218 • Non-default scope.
220 • Different build and run systems.
222 • Run the filesystem permissions tests.
224 Output has been omitted.
226 (mybox)$ ssh hpc-admin
228 (hpc-admin)$ ch-test mk-perm-dirs --perm-dir /scratch/$USER/perms \
229 --perm-dir /home/$USER/perms
230 (hpc-admin)$ exit
231 (mybox)$ ch-test build --scope full
232 (mybox)$ scp -r /var/tmp/pack hpc:/scratch/$USER/pack
233 (mybox)$ ssh hpc
234 (hpc)$ salloc -N2
235 (cn001)$ export CH_TEST_TARDIR=/scratch/$USER/pack
236 (cn001)$ export CH_TEST_IMGDIR=/local/tmp
237 (cn001)$ export CH_TEST_PERMDIRS="/scratch/$USER/perms /home/$USER/perms"
238 (cn001)$ export CH_TEST_SCOPE=full
239 (cn001)$ ch-test run
240 (cn001)$ ch-test examples
241
243 If Charliecloud was obtained from your Linux distribution, use your
244 distribution’s bug reporting procedures.
245 Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
247
249 charliecloud(7)
250 Full documentation at: <https://hpc.github.io/charliecloud>
252
254 2014–2021, Triad National Security, LLC
2550.25 2021-09-20 00:00 UTC CH-TEST(1)