1CH-RUN(1) Charliecloud CH-RUN(1)
2
6 ch-run - Run a command in a Charliecloud container
7
9 $ ch-run [OPTION...] NEWROOT CMD [ARG...]
10
12 Run command CMD in a Charliecloud container using the flattened and
13 unpacked image directory located at NEWROOT.
14 -b, --bind=SRC[:DST]
16 mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)
17 -c, --cd=DIR
19 initial working directory in container
20 --ch-ssh
22 bind ch-ssh(1) into container at /usr/bin/ch-ssh
23 -g, --gid=GID
25 run as group GID within container
26 -j, --join
28 use the same container (namespaces) as peer ch-run invoca‐
29 tions
30 --join-pid=PID
32 join the namespaces of an existing process
33 --join-ct=N
35 number of ch-run peers (implies --join; default: see below)
36 --join-tag=TAG
38 label for ch-run peer group (implies --join; default: see
39 below)
40 --no-home
42 do not bind-mount your home directory (by default, your home
43 directory is mounted at /home/$USER in the container)
44 -t, --private-tmp
46 use container-private /tmp (by default, /tmp is shared with
47 the host)
48 --set-env=FILE
50 set environment variables as specified in host path FILE
51 -u, --uid=UID
53 run as user UID within container
54 --unset-env=GLOB
56 unset environment variables whose names match GLOB
57 -v, --verbose
59 be more verbose (debug if repeated)
60 -w, --write
62 mount image read-write (by default, the image is mounted
63 read-only)
64 -?, --help
66 print help and exit
67 --usage
69 print a short usage message and exit
70 -V, --version
72 print version and exit
73
75 In addition to any directories specified by the user with --bind,
76 ch-run has standard host files and directories that are bind-mounted in
77 as well.
78 The following host files and directories are bind-mounted at the same
80 location in the container. These cannot be disabled.
81 · /dev
83 · /etc/passwd
85 · /etc/group
87 · /etc/hosts
89 · /etc/resolv.conf
91 · /proc
93 · /sys
95 Three additional bind mounts can be disabled by the user:
97 · Your home directory (i.e., $HOME) is mounted at guest /home/$USER
99 by default. This is accomplished by mounting a new tmpfs at /home,
100 which hides any image content under that path. If --no-home is
101 specified, neither of these things happens and the image’s /home
102 is exposed unaltered.
103 · /tmp is shared with the host by default. If --private-tmp is spec‐
105 ified, a new tmpfs is mounted on the guest’s /tmp instead.
106 · If file /usr/bin/ch-ssh is present in the image, it is
108 over-mounted with the ch-ssh binary in the same directory as
109 ch-run.
110
112 By default, different ch-run invocations use different user and mount
113 namespaces (i.e., different containers). While this has no impact on
114 sharing most resources between invocations, there are a few important
115 exceptions. These include:
116 1. ptrace(2), used by debuggers and related tools. One can attach a
118 debugger to processes in descendant namespaces, but not sibling
119 namespaces. The practical effect of this is that (without --join),
120 you can’t run a command with ch-run and then attach to it with a
121 debugger also run with ch-run.
122 2. Cross-memory attach (CMA) is used by cooperating processes to commu‐
124 nicate by simply reading and writing one another’s memory. This is
125 also not permitted between sibling namespaces. This affects various
126 MPI implementations that use CMA to pass messages between ranks on
127 the same node, because it’s faster than traditional shared memory.
128 --join is designed to address this by placing related ch-run commands
130 (the “peer group”) in the same container. This is done by one of the
131 peers creating the namespaces with unshare(2) and the others joining
132 with setns(2).
133 To do so, we need to know the number of peers and a name for the group.
135 These are specified by additional arguments that can (hopefully) be
136 left at default values in most cases:
137 · --join-ct sets the number of peers. The default is the value of the
139 first of the following environment variables that is defined:
140 OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE,
141 SLURM_CPUS_ON_NODE.
142 · --join-tag sets the tag that names the peer group. The default is
144 environment variable SLURM_STEP_ID, if defined; otherwise, the PID of
145 ch-run’s parent. Tags can be re-used for peer groups that start at
146 different times, i.e., once all peer ch-run have replaced themselves
147 with the user command, the tag can be re-used.
148 Caveats:
150 · One cannot currently add peers after the fact, for example, if one
152 decides to start a debugger after the fact. (This is only required
153 for code with bugs and is thus an unusual use case.)
154 · ch-run instances race. The winner of this race sets up the names‐
156 paces, and the other peers use the winner to find the namespaces to
157 join. Therefore, if the user command of the winner exits, any remain‐
158 ing peers will not be able to join the namespaces, even if they are
159 still active. There is currently no general way to specify which
160 ch-run should be the winner.
161 · If --join-ct is too high, the winning ch-run’s user command exits
163 before all peers join, or ch-run itself crashes, IPC resources such
164 as semaphores and shared memory segments will be leaked. These appear
165 as files in /dev/shm/ and can be removed with rm(1).
166 · Many of the arguments given to the race losers, such as the image
168 path and --bind, will be ignored in favor of what was given to the
169 winner.
170
172 ch-run leaves environment variables unchanged, i.e. the host environ‐
173 ment is passed through unaltered, except:
174 · limited tweaks to avoid significant guest breakage;
176 · user-set variables via --set-env;
178 · user-unset variables via --unset-env; and
180 · set CH_RUNNING.
182 This section describes these features.
184 The default tweaks happen first, and then --set-env and --unset-env in
186 the order specified on the command line. The latter two can be repeated
187 arbitrarily many times, e.g. to add/remove multiple variable sets or
188 add only some variables in a file.
189 Default behavior
191 By default, ch-run makes the following environment variable changes:
192 · $CH_RUNNING: Set to Weird Al Yankovic. While a process can figure out
194 that it’s in an unprivileged container and what namespaces are active
195 without this hint, the checks can be messy, and there is no way to
196 tell that it’s a Charliecloud container specifically. This variable
197 makes such a test simple and well-defined. (Note: This variable is
198 unaffected by --unset-env.)
199 · $HOME: If the path to your home directory is not /home/$USER on the
201 host, then an inherited $HOME will be incorrect inside the guest.
202 This confuses some software, such as Spack.
203 Thus, we change $HOME to /home/$USER, unless --no-home is specified,
205 in which case it is left unchanged.
206 · $PATH: Newer Linux distributions replace some root-level directories,
208 such as /bin, with symlinks to their counterparts in /usr.
209 Some of these distributions (e.g., Fedora 24) have also dropped /bin
211 from the default $PATH. This is a problem when the guest OS does not
212 have a merged /usr (e.g., Debian 8 “Jessie”). Thus, we add /bin to
213 $PATH if it’s not already present.
214 Further reading:
216 · The case for the /usr Merge
218 · Fedora
220 · Debian
222 Setting variables with --set-env
224 The purpose of --set-env=FILE is to set environment variables that can‐
225 not be inherited from the host shell, e.g. Dockerfile ENV directives or
226 other build-time configuration. FILE is a host path to provide the
227 greatest flexibility; guest paths can be specified by prepending the
228 image path.
229 ch-builder2tar(1) lists variables specified at build time in Docker‐
231 files in the image in file /ch/environment. To set these variables:
232 --set-env=$IMG/ch/environment.
233 Variable values in FILE replace any already set. If a variable is
235 repeated, the last value wins.
236 The syntax of FILE is key-value pairs separated by the first equals
238 character (=, ASCII 61), one per line, with optional single straight
239 quotes (', ASCII 39) around the value. Empty lines are ignored. New‐
240 lines (ASCII 10) are not permitted in either key or value. No variable
241 expansion, comments, etc. are provided. The value may be empty, but not
242 the key. (This syntax is designed to accept the output of printenv and
243 be easily produced by other simple mechanisms.) Examples of valid
244 lines:
245 ┌──────────────────┬───────┬─────────────────────┐
247 │Line │ Key │ Value │
248 ├──────────────────┼───────┼─────────────────────┤
249 │FOO=bar │ FOO │ bar │
250 ├──────────────────┼───────┼─────────────────────┤
251 │FOO=bar=baz │ FOO │ bar=baz │
252 ├──────────────────┼───────┼─────────────────────┤
253 │FLAGS=-march=foo │ FLAGS │ -march=foo │
254 │-mtune=bar │ │ -mtune=bar │
255 ├──────────────────┼───────┼─────────────────────┤
256 │FLAGS='-march=foo │ FLAGS │ -march=foo │
257 │-mtune=bar' │ │ -mtune=bar │
258 ├──────────────────┼───────┼─────────────────────┤
259 │FOO= │ FOO │ (empty string) │
260 ├──────────────────┼───────┼─────────────────────┤
261 │FOO='' │ FOO │ (empty string) │
262 ├──────────────────┼───────┼─────────────────────┤
263 │FOO='''' │ FOO │ '' (two single │
264 │ │ │ quotes) │
265 └──────────────────┴───────┴─────────────────────┘
266 Example invalid lines:
268 ┌────────┬─────────────────────┐
270 │Line │ Problem │
271 ├────────┼─────────────────────┤
272 │FOO bar │ no separator │
273 ├────────┼─────────────────────┤
274 │=bar │ key cannot be empty │
275 └────────┴─────────────────────┘
276 Example valid lines that are probably not what you want:
278 ┌────────────────────┬───────┬────────────────┬──────────────────┐
280 │Line │ Key │ Value │ Problem │
281 ├────────────────────┼───────┼────────────────┼──────────────────┤
282 │FOO="bar" │ FOO │ "bar" │ double quotes │
283 │ │ │ │ aren’t stripped │
284 ├────────────────────┼───────┼────────────────┼──────────────────┤
285 │FOO=bar # baz │ FOO │ bar # baz │ comments not │
286 │ │ │ │ supported │
287 ├────────────────────┼───────┼────────────────┼──────────────────┤
288 │PATH=$PATH:/opt/bin │ PATH │ $PATH:/opt/bin │ variables not │
289 │ │ │ │ expanded │
290 ├────────────────────┼───────┼────────────────┼──────────────────┤
291 │ FOO=bar │ FOO │ bar │ leading space in │
292 │ │ │ │ key │
293 ├────────────────────┼───────┼────────────────┼──────────────────┤
294 │FOO= bar │ FOO │ bar │ leading space in │
295 │ │ │ │ value │
296 └────────────────────┴───────┴────────────────┴──────────────────┘
297 Removing variables with --unset-env
299 The purpose of --unset-env=GLOB is to remove unwanted environment vari‐
300 ables. The argument GLOB is a glob pattern (dialect fnmatch(3) with no
301 flags); all variables with matching names are removed from the environ‐
302 ment.
303 WARNING:
305 Because the shell also interprets glob patterns, if any wildcard
306 characters are in GLOB, it is important to put it in single quotes
307 to avoid surprises.
308 GLOB must be a non-empty string.
310 Example 1: Remove the single environment variable FOO:
312 $ export FOO=bar
314 $ env | fgrep FOO
315 FOO=bar
316 $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
317 $
318 Example 2: Hide from a container the fact that it’s running in a Slurm
320 allocation, by removing all variables beginning with SLURM. You might
321 want to do this to test an MPI program with one rank and no launcher:
322 $ salloc -N1
324 $ env | egrep '^SLURM' | wc
325 44 44 1092
326 $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
327 [... long error message ...]
328 $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
329 0: MPI version:
330 Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
331 0: init ok cn001.localdomain, 1 ranks, userns 4026532530
332 0: send/receive ok
333 0: finalize ok
334 Example 3: Clear the environment completely (remove all variables):
336 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
338 $
339 Note that some programs, such as shells, set some environment variables
341 even if started with no init files:
342 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
344 SHLVL=1
345 PWD=/
346 _=/usr/bin/env
347 $
348
350 Run the command echo hello inside a Charliecloud container using the
351 unpacked image at /data/foo:
352 $ ch-run /data/foo -- echo hello
354 hello
355 Run an MPI job that can use CMA to communicate:
357 $ srun ch-run --join /data/foo -- bar
359
361 If Charliecloud was obtained from your Linux distribution, use your
362 distribution’s bug reporting procedures.
363 Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
365
367 charliecloud(7)
368 Full documentation at: <https://hpc.github.io/charliecloud>
370
372 2014–2020, Triad National Security, LLC
3730.22 2021-03-08 00:00 Coordinated Universal Time CH-RUN(1)