1CH-RUN(1) Charliecloud CH-RUN(1)
2
6 ch-run - Run a command in a Charliecloud container
7
9 $ ch-run [OPTION...] IMAGE -- CMD [ARG...]
10
12 Run command CMD in a fully unprivileged Charliecloud container using
13 the image specified by IMAGE, which can be: (1) a path to a directory,
14 (2) the name of an image in ch-image storage (e.g. exam‐
15 ple.com:5050/foo) or, if the proper support is enabled, a SquashFS ar‐
16 chive. ch-run does not use any setuid or setcap helpers, even for
17 mounting SquashFS images with FUSE.
18 -b, --bind=SRC[:DST]
20 Bind-mount SRC at guest DST. The default destination if not
21 specified is to use the same path as the host; i.e., the de‐
22 fault is --bind=SRC:SRC. Can be repeated.
23 If --write is given and DST does not exist, it will be cre‐
25 ated as an empty directory. However, DST must be entirely
26 within the image itself; DST cannot enter a previous bind
27 mount. For example, --bind /foo:/tmp/foo will fail because
28 /tmp is shared with the host via bind-mount (unless $TMPDIR
29 is set to something else or --private-tmp is given).
30 Most images do have ten directories /mnt/[0-9] already avail‐
32 able as mount points.
33 Symlinks in DST are followed, and absolute links can have
35 surprising behavior. Bind-mounting happens after namespace
36 setup but before pivoting into the container image, so abso‐
37 lute links use the host root. For example, suppose the image
38 has a symlink /foo -> /mnt. Then, --bind=/bar:/foo will
39 bind-mount on the host’s /mnt, which is inaccessible on the
40 host because namespaces are already set up and also inacces‐
41 sible in the container because of the subsequent pivot into
42 the image. Currently, this problem is only detected when DST
43 needs to be created: ch-run will refuse to follow absolute
44 symlinks in this case, to avoid directory creation surprises.
45 -c, --cd=DIR
47 Initial working directory in container.
48 --ch-ssh
50 Bind ch-ssh(1) into container at /usr/bin/ch-ssh.
51 --env-no-expand
53 Don’t expand variables when using --set-env.
54 -g, --gid=GID
56 Run as group GID within container.
57 --home Bind-mount your host home directory (i.e., $HOME) at guest
59 /home/$USER. This is accomplished by over-mounting a new
60 tmpfs at /home, which hides any image content under that
61 path. By default, neither of these things happens and the im‐
62 age’s /home is exposed unaltered.
63 -j, --join
65 Use the same container (namespaces) as peer ch-run invoca‐
66 tions.
67 --join-pid=PID
69 Join the namespaces of an existing process.
70 --join-ct=N
72 Number of ch-run peers (implies --join; default: see below).
73 --join-tag=TAG
75 Label for ch-run peer group (implies --join; default: see be‐
76 low).
77 -m, --mount=DIR
79 Use DIR for the SquashFS mount point, which must already ex‐
80 ist. If not specified, the default is /var/tmp/$USER.ch/mnt,
81 which will be created if needed.
82 --no-passwd
84 By default, temporary /etc/passwd and /etc/group files are
85 created according to the UID and GID maps for the container
86 and bind-mounted into it. If this is specified, no such tem‐
87 porary files are created and the image’s files are exposed.
88 -s, --storage DIR
90 Set the storage directory. Equivalent to the same option for
91 ch-image(1).
92 --seccomp
94 Using seccomp, intercept some system calls that would fail
95 due to lack of privilege, do nothing, and return fake success
96 to the calling program. This is intended for use by ch-im‐
97 age(1) when building images; see that man page for a detailed
98 discussion.
99 -t, --private-tmp
101 By default, the host’s /tmp (or $TMPDIR if set) is
102 bind-mounted at container /tmp. If this is specified, a new
103 tmpfs is mounted on the container’s /tmp instead.
104 --set-env, --set-env=FILE, --set-env=VAR=VALUE
106 Set environment variable(s). With:
107 • no argument: as listed in file /ch/environment within
109 the image. It is an error if the file does not exist or
110 cannot be read. (Note that with SquashFS images, it is
111 not currently possible to use other files within the
112 image.)
113 • FILE (i.e., no equals in argument): as specified in
115 file at host path FILE. Again, it is an error if the
116 file cannot be read.
117 • NAME=VALUE (i.e., equals sign in argument): set vari‐
119 able NAME to VALUE.
120 See below for details on how environment variables work in
122 ch-run.
123 -u, --uid=UID
125 Run as user UID within container.
126 --unsafe
128 Enable various unsafe behavior. For internal use only. Seri‐
129 ously, stay away from this option.
130 --unset-env=GLOB
132 Unset environment variables whose names match GLOB.
133 -v, --verbose
135 Be more verbose (can be repeated).
136 -w, --write
138 Mount image read-write (by default, the image is mounted
139 read-only).
140 -?, --help
142 Print help and exit.
143 --usage
145 Print a short usage message and exit.
146 -V, --version
148 Print version and exit.
149 Note: Because ch-run is fully unprivileged, it is not possible to
151 change UIDs and GIDs within the container (the relevant system calls
152 fail). In particular, setuid, setgid, and setcap executables do not
153 work. As a precaution, ch-run calls prctl(PR_SET_NO_NEW_PRIVS, 1) to
154 disable these executables within the container. This does not reduce
155 functionality but is a “belt and suspenders” precaution to reduce the
156 attack surface should bugs in these system calls or elsewhere arise.
157
159 ch-run supports two different image formats.
160 The first is a simple directory that contains a Linux filesystem tree.
162 This can be accomplished by:
163 • ch-convert directly from ch-image or another builder to a directory.
165 • Charliecloud’s tarball workflow: build or pull the image, ch-convert
167 it to a tarball, transfer the tarball to the target system, then
168 ch-convert the tarball to a directory.
169 • Manually mount a SquashFS image, e.g. with squashfuse(1) and then
171 un-mount it after run with fusermount -u.
172 • Any other workflow that produces an appropriate directory tree.
174 The second is a SquashFS image archive mounted internally by ch-run,
176 available if it’s linked with the optional libsquashfuse_ll shared li‐
177 brary. ch-run mounts the image filesystem, services all FUSE requests,
178 and unmounts it, all within ch-run. See --mount above to set the mount
179 point location.
180 Like other FUSE implementations, Charliecloud calls the fusermount3(1)
182 utility to mount the SquashFS filesystem. However, this executable does
183 not need to be installed setuid root, and in fact ch-run actively sup‐
184 presses its setuid bit if set (using prctl(2)).
185 Prior versions of Charliecloud provided wrappers for the squashfuse and
187 squashfuse_ll SquashFS mount commands and fusermount -u unmount com‐
188 mand. We removed these because we concluded they had minimal value-add
189 over the standard, unwrapped commands.
190 WARNING:
192 Currently, Charliecloud unmounts the SquashFS filesystem when user
193 command CMD’s process exits. It does not monitor any of its child
194 processes. Therefore, if the user command spawns child processes and
195 then exits before them (e.g., some daemons), those children will
196 have the image unmounted from underneath them. In this case, the
197 workaround is to mount/unmount using external tools. We expect to
198 remove this limitation in a future version.
199
201 In addition to any directories specified by the user with --bind,
202 ch-run has standard host files and directories that are bind-mounted in
203 as well.
204 The following host files and directories are bind-mounted at the same
206 location in the container. These give access to the host’s devices and
207 various kernel facilities. (Recall that Charliecloud provides minimal
208 isolation and containerized processes are mostly normal unprivileged
209 processes.) They cannot be disabled and are required; i.e., they must
210 exist both on host and within the image.
211 • /dev
213 • /proc
215 • /sys
217 Optional; bind-mounted only if path exists on both host and within the
219 image, without error or warning if not.
220 • /etc/hosts and /etc/resolv.conf. Because Charliecloud containers
222 share the host network namespace, they need the same hostname res‐
223 olution configuration.
224 • /etc/machine-id. Provides a unique ID for the OS installation;
226 matching the host works for most situations. Needed to support
227 D-Bus, some software licensing situations, and likely other use
228 cases. See also issue #1050.
229 • /var/lib/hugetlbfs at guest /var/opt/cray/hugetlbfs, and
231 /var/opt/cray/alps/spool. These support Cray MPI.
232 • $PREFIX/bin/ch-ssh at guest /usr/bin/ch-ssh. SSH wrapper that au‐
234 tomatically containerizes after connecting.
235 Additional bind mounts done by default but can be disabled; see the op‐
237 tions above.
238 • $HOME at /home/$USER (and image /home is hidden). Makes user data
240 and init files available.
241 • /tmp (or $TMPDIR if set) at guest /tmp. Provides a temporary di‐
243 rectory that persists between container runs and is shared with
244 non-containerized application components.
245 • temporary files at /etc/passwd and /etc/group. Usernames and group
247 names need to be customized for each container run.
248
250 By default, different ch-run invocations use different user and mount
251 namespaces (i.e., different containers). While this has no impact on
252 sharing most resources between invocations, there are a few important
253 exceptions. These include:
254 1. ptrace(2), used by debuggers and related tools. One can attach a de‐
256 bugger to processes in descendant namespaces, but not sibling name‐
257 spaces. The practical effect of this is that (without --join), you
258 can’t run a command with ch-run and then attach to it with a debug‐
259 ger also run with ch-run.
260 2. Cross-memory attach (CMA) is used by cooperating processes to commu‐
262 nicate by simply reading and writing one another’s memory. This is
263 also not permitted between sibling namespaces. This affects various
264 MPI implementations that use CMA to pass messages between ranks on
265 the same node, because it’s faster than traditional shared memory.
266 --join is designed to address this by placing related ch-run commands
268 (the “peer group”) in the same container. This is done by one of the
269 peers creating the namespaces with unshare(2) and the others joining
270 with setns(2).
271 To do so, we need to know the number of peers and a name for the group.
273 These are specified by additional arguments that can (hopefully) be
274 left at default values in most cases:
275 • --join-ct sets the number of peers. The default is the value of the
277 first of the following environment variables that is defined:
278 OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE,
279 SLURM_CPUS_ON_NODE.
280 • --join-tag sets the tag that names the peer group. The default is en‐
282 vironment variable SLURM_STEP_ID, if defined; otherwise, the PID of
283 ch-run’s parent. Tags can be re-used for peer groups that start at
284 different times, i.e., once all peer ch-run have replaced themselves
285 with the user command, the tag can be re-used.
286 Caveats:
288 • One cannot currently add peers after the fact, for example, if one
290 decides to start a debugger after the fact. (This is only required
291 for code with bugs and is thus an unusual use case.)
292 • ch-run instances race. The winner of this race sets up the name‐
294 spaces, and the other peers use the winner to find the namespaces to
295 join. Therefore, if the user command of the winner exits, any remain‐
296 ing peers will not be able to join the namespaces, even if they are
297 still active. There is currently no general way to specify which
298 ch-run should be the winner.
299 • If --join-ct is too high, the winning ch-run’s user command exits be‐
301 fore all peers join, or ch-run itself crashes, IPC resources such as
302 semaphores and shared memory segments will be leaked. These appear as
303 files in /dev/shm/ and can be removed with rm(1).
304 • Many of the arguments given to the race losers, such as the image
306 path and --bind, will be ignored in favor of what was given to the
307 winner.
308
310 ch-run leaves environment variables unchanged, i.e. the host environ‐
311 ment is passed through unaltered, except:
312 • limited tweaks to avoid significant guest breakage;
314 • user-set variables via --set-env;
316 • user-unset variables via --unset-env; and
318 • set CH_RUNNING.
320 This section describes these features.
322 The default tweaks happen first, then --set-env and --unset-env in the
324 order specified on the command line, and then CH_RUNNING. The two op‐
325 tions can be repeated arbitrarily many times, e.g. to add/remove multi‐
326 ple variable sets or add only some variables in a file.
327 Default behavior
329 By default, ch-run makes the following environment variable changes:
330 $CH_RUNNING
332 Set to Weird Al Yankovic. While a process can figure out that
333 it’s in an unprivileged container and what namespaces are active
334 without this hint, that can be messy, and there is no way to
335 tell that it’s a Charliecloud container specifically. This vari‐
336 able makes such a test simple and well-defined. (Note: This
337 variable is unaffected by --unset-env.)
338 $HOME If --home is specified, then your home directory is bind-mounted
340 into the guest at /home/$USER. If you also have a different home
341 directory path on the host, an inherited $HOME will be incorrect
342 inside the guest, which confuses lots of software, notably
343 Spack. Thus, with --home, $HOME is set to /home/$USER (by de‐
344 fault, it is unchanged.)
345 $PATH Newer Linux distributions replace some root-level directories,
347 such as /bin, with symlinks to their counterparts in /usr.
348 Some of these distributions (e.g., Fedora 24) have also dropped
350 /bin from the default $PATH. This is a problem when the guest OS
351 does not have a merged /usr (e.g., Debian 8 “Jessie”). Thus, we
352 add /bin to $PATH if it’s not already present.
353 Further reading:
355 • The case for the /usr Merge
357 • Fedora
359 • Debian
361 $TMPDIR
363 Unset, because this is almost certainly a host path, and that
364 host path is made available in the guest at /tmp unless --pri‐
365 vate-tmp is given.
366 Setting variables with --set-env
368 The purpose of --set-env is to set environment variables within the
369 container. Values given replace any already in the environment (i.e.,
370 inherited from the host shell) or set by earlier --set-env. This flag
371 takes an optional argument with two possible forms:
372 1. If the argument contains an equals sign (=, ASCII 61), that sets an
374 environment variable directly. For example, to set FOO to the string
375 value bar:
376 $ ch-run --set-env=FOO=bar ...
378 Single straight quotes around the value (', ASCII 39) are stripped,
380 though be aware that both single and double quotes are also inter‐
381 preted by the shell. For example, this example is similar to the
382 prior one; the double quotes are removed by the shell and the single
383 quotes are removed by ch-run:
384 $ ch-run --set-env="'BAZ=qux'" ...
386 2. If the argument does not contain an equals sign, it is a host path
388 to a file containing zero or more variables using the same syntax as
389 above (except with no prior shell processing). This file contains a
390 sequence of assignments separated by newlines. Empty lines are ig‐
391 nored, and no comments are interpreted. (This syntax is designed to
392 accept the output of printenv and be easily produced by other simple
393 mechanisms.) For example:
394 $ cat /tmp/env.txt
396 FOO=bar
397 BAZ='qux'
398 $ ch-run --set-env=/tmp/env.txt ...
399 For directory images only (because the file is read before con‐
401 tainerizing), guest paths can be given by prepending the image path.
402 3. If there is no argument, the file /ch/environment within the image
404 is used. This file is commonly populated by ENV instructions in the
405 Dockerfile. For example, equivalently to form 2:
406 $ cat Dockerfile
408 [...]
409 ENV FOO=bar
410 ENV BAZ=qux
411 [...]
412 $ ch-image build -t foo .
413 $ ch-convert foo /var/tmp/foo.sqfs
414 $ ch-run --set-env /var/tmp/foo.sqfs -- ...
415 (Note the image path is interpreted correctly, not as the --set-env
417 argument.)
418 At present, there is no way to use files other than /ch/environment
420 within SquashFS images.
421 Environment variables are expanded for values that look like search
423 paths, unless --env-no-expand is given prior to --set-env. In this
424 case, the value is a sequence of zero or more possibly-empty items sep‐
425 arated by colon (:, ASCII 58). If an item begins with dollar sign ($,
426 ASCII 36), then the rest of the item is the name of an environment
427 variable. If this variable is set to a non-empty value, that value is
428 substituted for the item; otherwise (i.e., the variable is unset or the
429 empty string), the item is deleted, including a delimiter colon. The
430 purpose of omitting empty expansions is to avoid surprising behavior
431 such as an empty element in $PATH meaning the current directory.
432 For example, to set HOSTPATH to the search path in the current shell
434 (this is expanded by ch-run, though letting the shell do it happens to
435 be equivalent):
436 $ ch-run --set-env='HOSTPATH=$PATH' ...
438 To prepend /opt/bin to this current search path:
440 $ ch-run --set-env='PATH=/opt/bin:$PATH' ...
442 To prepend /opt/bin to the search path set by the Dockerfile, as re‐
444 trieved from guest file /ch/environment (here we really cannot let the
445 shell expand $PATH):
446 $ ch-run --set-env --set-env='PATH=/opt/bin:$PATH' ...
448 Examples of valid assignment, assuming that environment variable BAR is
450 set to bar and UNSET is unset or set to the empty string:
451 ┌───────────────────┬───────┬─────────────────────┐
453 │Assignment │ Name │ Value │
454 ├───────────────────┼───────┼─────────────────────┤
455 │FOO=bar │ FOO │ bar │
456 ├───────────────────┼───────┼─────────────────────┤
457 │FOO=bar=baz │ FOO │ bar=baz │
458 ├───────────────────┼───────┼─────────────────────┤
459 │FLAGS=-march=foo │ FLAGS │ -march=foo │
460 │-mtune=bar │ │ -mtune=bar │
461 ├───────────────────┼───────┼─────────────────────┤
462 │FLAGS='-march=foo │ FLAGS │ -march=foo │
463 │-mtune=bar' │ │ -mtune=bar │
464 ├───────────────────┼───────┼─────────────────────┤
465 │FOO=$BAR │ FOO │ bar │
466 ├───────────────────┼───────┼─────────────────────┤
467 │FOO=$BAR:baz │ FOO │ bar:baz │
468 ├───────────────────┼───────┼─────────────────────┤
469 │FOO= │ FOO │ empty string │
470 ├───────────────────┼───────┼─────────────────────┤
471 │FOO=$UNSET │ FOO │ empty string │
472 ├───────────────────┼───────┼─────────────────────┤
473 │FOO=baz:$UNSET:qux │ FOO │ baz:qux (not │
474 │ │ │ baz::qux) │
475 ├───────────────────┼───────┼─────────────────────┤
476 │FOO=:bar:baz:: │ FOO │ :bar:baz:: │
477 └───────────────────┴───────┴─────────────────────┘
478 │FOO='' │ FOO │ empty string │
480 ├───────────────────┼───────┼─────────────────────┤
481 │FOO='''' │ FOO │ '' (two single │
482 │ │ │ quotes) │
483 └───────────────────┴───────┴─────────────────────┘
484 Example invalid assignments:
486 ┌───────────┬──────────────────────┐
488 │Assignment │ Problem │
489 ├───────────┼──────────────────────┤
490 │FOO bar │ no equals separator │
491 ├───────────┼──────────────────────┤
492 │=bar │ name cannot be empty │
493 └───────────┴──────────────────────┘
494 Example valid assignments that are probably not what you want:
496 ┌─────────────────┬───────┬───────────┬──────────────────┐
498 │Assignment │ Name │ Value │ Problem │
499 ├─────────────────┼───────┼───────────┼──────────────────┤
500 │FOO="bar" │ FOO │ "bar" │ double quotes │
501 │ │ │ │ aren’t stripped │
502 ├─────────────────┼───────┼───────────┼──────────────────┤
503 │FOO=bar # baz │ FOO │ bar # baz │ comments not │
504 │ │ │ │ supported │
505 ├─────────────────┼───────┼───────────┼──────────────────┤
506 │FOO=bartbaz │ FOO │ bartbaz │ backslashes are │
507 │ │ │ │ not special │
508 ├─────────────────┼───────┼───────────┼──────────────────┤
509 │ FOO=bar │ FOO │ bar │ leading space in │
510 │ │ │ │ key │
511 ├─────────────────┼───────┼───────────┼──────────────────┤
512 │FOO= bar │ FOO │ bar │ leading space in │
513 │ │ │ │ value │
514 ├─────────────────┼───────┼───────────┼──────────────────┤
515 │$FOO=bar │ $FOO │ bar │ variables not │
516 │ │ │ │ expanded in key │
517 ├─────────────────┼───────┼───────────┼──────────────────┤
518 │FOO=$BAR baz:qux │ FOO │ qux │ variable BAR baz │
519 │ │ │ │ not set │
520 └─────────────────┴───────┴───────────┴──────────────────┘
521 Removing variables with --unset-env
523 The purpose of --unset-env=GLOB is to remove unwanted environment vari‐
524 ables. The argument GLOB is a glob pattern (dialect fnmatch(3) with the
525 FNM_EXTMATCH flag where supported); all variables with matching names
526 are removed from the environment.
527 WARNING:
529 Because the shell also interprets glob patterns, if any wildcard
530 characters are in GLOB, it is important to put it in single quotes
531 to avoid surprises.
532 GLOB must be a non-empty string.
534 Example 1: Remove the single environment variable FOO:
536 $ export FOO=bar
538 $ env | fgrep FOO
539 FOO=bar
540 $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
541 $
542 Example 2: Hide from a container the fact that it’s running in a Slurm
544 allocation, by removing all variables beginning with SLURM. You might
545 want to do this to test an MPI program with one rank and no launcher:
546 $ salloc -N1
548 $ env | egrep '^SLURM' | wc
549 44 44 1092
550 $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
551 [... long error message ...]
552 $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
553 0: MPI version:
554 Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
555 0: init ok cn001.localdomain, 1 ranks, userns 4026532530
556 0: send/receive ok
557 0: finalize ok
558 Example 3: Clear the environment completely (remove all variables):
560 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
562 $
563 Example 4: Remove all environment variables except for those prefixed
565 with either WANTED_ or ALSO_WANTED_:
566 $ export WANTED_1=yes
568 $ export ALSO_WANTED_2=yes
569 $ export NOT_WANTED_1=no
570 $ ch-run --unset-env='!(WANTED_*|ALSO_WANTED_*)' $CH_TEST_IMGDIR/chtest -- env
571 WANTED_1=yes
572 ALSO_WANTED_2=yes
573 $
574 Note that some programs, such as shells, set some environment variables
576 even if started with no init files:
577 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian_9ch -- bash --noprofile --norc -c env
579 SHLVL=1
580 PWD=/
581 _=/usr/bin/env
582 $
583
585 Run the command echo hello inside a Charliecloud container using the
586 unpacked image at /data/foo:
587 $ ch-run /data/foo -- echo hello
589 hello
590 Run an MPI job that can use CMA to communicate:
592 $ srun ch-run --join /data/foo -- bar
594
596 By default, ch-run logs its command line to syslog. (This can be dis‐
597 abled by configuring with --disable-syslog.) This includes: (1) the in‐
598 voking real UID, (2) the number of command line arguments, and (3) the
599 arguments, separated by spaces. For example:
600 Dec 10 18:19:08 mybox ch-run: uid=1000 args=7: ch-run -v /var/tmp/00_tiny -- echo hello "wor l}\$d"
602 Logging is one of the first things done during program initialization,
604 even before command line parsing. That is, almost all command lines are
605 logged, even if erroneous, and there is no logging of program success
606 or failure.
607 Arguments are serialized with the following procedure. The purpose is
609 to provide a human-readable reconstruction of the command line while
610 also allowing each argument to be recovered byte-for-byte.
611 • If an argument contains only printable ASCII bytes that are not
613 whitespace, shell metacharacters, double quote (", ASCII 34 deci‐
614 mal), or backslash (, ASCII 92), then log it unchanged.
615 • Otherwise, (a) enclose the argument in double quotes and (b) back‐
617 slash-escape double quotes, backslashes, and characters inter‐
618 preted by Bash (including POSIX shells) within double quotes.
619 The verbatim command line typed in the shell cannot be recovered, be‐
621 cause not enough information is provided to UNIX programs. For example,
622 echo 'foo' is given to programs as a sequence of two arguments, echo
623 and foo; the two spaces and single quotes are removed by the shell. The
624 zero byte, ASCII NUL, cannot appear in arguments because it would ter‐
625 minate the string.
626
628 If there is an error during containerization, ch-run exits with status
629 non-zero. If the user command is started successfully, the exit status
630 is that of the user command, with one exception: if the image is an in‐
631 ternally mounted SquashFS filesystem and the user command is killed by
632 a signal, the exit status is 1 regardless of the signal value.
633
635 If Charliecloud was obtained from your Linux distribution, use your
636 distribution’s bug reporting procedures.
637 Otherwise, report bugs to: https://github.com/hpc/charliecloud/issues
639
641 charliecloud(7)
642 Full documentation at: <https://hpc.github.io/charliecloud>
644
646 2014–2022, Triad National Security, LLC and others
6470.32 2023-07-19 00:00 UTC CH-RUN(1)