1CH-RUN(1) Charliecloud CH-RUN(1)
2
6 ch-run - Run a command in a Charliecloud container
7
9 $ ch-run [OPTION...] NEWROOT CMD [ARG...]
10
12 Run command CMD in a Charliecloud container using the flattened and
13 unpacked image directory located at NEWROOT.
14 -b, --bind=SRC[:DST]
16 mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)
17 -c, --cd=DIR
19 initial working directory in container
20 --ch-ssh
22 bind ch-ssh(1) into container at /usr/bin/ch-ssh
23 -g, --gid=GID
25 run as group GID within container
26 -j, --join
28 use the same container (namespaces) as peer ch-run invoca‐
29 tions
30 --join-pid=PID
32 join the namespaces of an existing process
33 --join-ct=N
35 number of ch-run peers (implies --join; default: see below)
36 --join-tag=TAG
38 label for ch-run peer group (implies --join; default: see
39 below)
40 --no-home
42 do not bind-mount your home directory (by default, your home
43 directory is mounted at /home/$USER in the container)
44 -t, --private-tmp
46 use container-private /tmp (by default, /tmp is shared with
47 the host)
48 --set-env=FILE
50 set environment variables as specified in host path FILE
51 -u, --uid=UID
53 run as user UID within container
54 --unset-env=GLOB
56 unset environment variables whose names match GLOB
57 -v, --verbose
59 be more verbose (debug if repeated)
60 -w, --write
62 mount image read-write (by default, the image is mounted
63 read-only)
64 -?, --help
66 print help and exit
67 --usage
69 print a short usage message and exit
70 -V, --version
72 print version and exit
73
75 In addition to any directories specified by the user with --bind,
76 ch-run has standard host files and directories that are bind-mounted in
77 as well.
78 The following host files and directories are bind-mounted at the same
80 location in the container. These cannot be disabled.
81 · /dev
83 · /etc/passwd
85 · /etc/group
87 · /etc/hosts
89 · /etc/resolv.conf
91 · /proc
93 · /sys
95 Three additional bind mounts can be disabled by the user:
97 · Your home directory (i.e., $HOME) is mounted at guest /home/$USER
99 by default. This is accomplished by mounting a new tmpfs at /home,
100 which hides any image content under that path. If --no-home is
101 specified, neither of these things happens and the image’s /home
102 is exposed unaltered.
103 · /tmp is shared with the host by default. If --private-tmp is spec‐
105 ified, a new tmpfs is mounted on the guest’s /tmp instead.
106 · If file /usr/bin/ch-ssh is present in the image, it is
108 over-mounted with the ch-ssh binary in the same directory as
109 ch-run.
110
112 By default, different ch-run invocations use different user and mount
113 namespaces (i.e., different containers). While this has no impact on
114 sharing most resources between invocations, there are a few important
115 exceptions. These include:
116 1. ptrace(2), used by debuggers and related tools. One can attach a
118 debugger to processes in descendant namespaces, but not sibling
119 namespaces. The practical effect of this is that (without --join),
120 you can’t run a command with ch-run and then attach to it with a
121 debugger also run with ch-run.
122 2. Cross-memory attach (CMA) is used by cooperating processes to commu‐
124 nicate by simply reading and writing one another’s memory. This is
125 also not permitted between sibling namespaces. This affects various
126 MPI implementations that use CMA to pass messages between ranks on
127 the same node, because it’s faster than traditional shared memory.
128 --join is designed to address this by placing related ch-run commands
130 (the “peer group”) in the same container. This is done by one of the
131 peers creating the namespaces with unshare(2) and the others joining
132 with setns(2).
133 To do so, we need to know the number of peers and a name for the group.
135 These are specified by additional arguments that can (hopefully) be
136 left at default values in most cases:
137 · --join-ct sets the number of peers. The default is the value of the
139 first of the following environment variables that is defined:
140 OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE,
141 SLURM_CPUS_ON_NODE.
142 · --join-tag sets the tag that names the peer group. The default is
144 environment variable SLURM_STEP_ID, if defined; otherwise, the PID of
145 ch-run’s parent. Tags can be re-used for peer groups that start at
146 different times, i.e., once all peer ch-run have replaced themselves
147 with the user command, the tag can be re-used.
148 Caveats:
150 · One cannot currently add peers after the fact, for example, if one
152 decides to start a debugger after the fact. (This is only required
153 for code with bugs and is thus an unusual use case.)
154 · ch-run instances race. The winner of this race sets up the names‐
156 paces, and the other peers use the winner to find the namespaces to
157 join. Therefore, if the user command of the winner exits, any remain‐
158 ing peers will not be able to join the namespaces, even if they are
159 still active. There is currently no general way to specify which
160 ch-run should be the winner.
161 · If --join-ct is too high, the winning ch-run’s user command exits
163 before all peers join, or ch-run itself crashes, IPC resources such
164 as semaphores and shared memory segments will be leaked. These appear
165 as files in /dev/shm/ and can be removed with rm(1).
166 · Many of the arguments given to the race losers, such as the image
168 path and --bind, will be ignored in favor of what was given to the
169 winner.
170
172 ch-run leaves environment variables unchanged, i.e. the host environ‐
173 ment is passed through unaltered, except:
174 · limited tweaks to avoid significant guest breakage;
176 · user-set variables via --set-env; and
178 · user-unset variables via --unset-env.
180 This section describes these features.
182 The default tweaks happen first, and then --set-env and --unset-env in
184 the order specified on the command line. The latter two can be repeated
185 arbitrarily many times, e.g. to add/remove multiple variable sets or
186 add only some variables in a file.
187 Default behavior
189 By default, ch-run makes the following environment variable changes:
190 · $HOME: If the path to your home directory is not /home/$USER on the
192 host, then an inherited $HOME will be incorrect inside the guest.
193 This confuses some software, such as Spack.
194 Thus, we change $HOME to /home/$USER, unless --no-home is specified,
196 in which case it is left unchanged.
197 · $PATH: Newer Linux distributions replace some root-level directories,
199 such as /bin, with symlinks to their counterparts in /usr.
200 Some of these distributions (e.g., Fedora 24) have also dropped /bin
202 from the default $PATH. This is a problem when the guest OS does not
203 have a merged /usr (e.g., Debian 8 “Jessie”). Thus, we add /bin to
204 $PATH if it’s not already present.
205 Further reading:
207 · The case for the /usr Merge
209 · Fedora
211 · Debian
213 Setting variables with --set-env
215 The purpose of --set-env=FILE is to set environment variables that can‐
216 not be inherited from the host shell, e.g. Dockerfile ENV directives or
217 other build-time configuration. FILE is a host path to provide the
218 greatest flexibility; guest paths can be specified by prepending the
219 image path.
220 ch-docker2tar(1) lists variables specified at build time in Dockerfiles
222 in the image in file /environment. To set these variables:
223 --set-env=$IMG/environment.
224 Variable values in FILE replace any already set. If a variable is
226 repeated, the last value wins.
227 The syntax of FILE is key-value pairs separated by the first equals
229 character (=, ASCII 61), one per line, with optional single straight
230 quotes (', ASCII 39) around the value. Empty lines are ignored. New‐
231 lines (ASCII 10) are not permitted in either key or value. No variable
232 expansion, comments, etc. are provided. The value may be empty, but not
233 the key. (This syntax is designed to accept the output of printenv and
234 be easily produced by other simple mechanisms.) Examples of valid
235 lines:
236 ┌──────────────────┬───────┬─────────────────────┐
238 │Line │ Key │ Value │
239 ├──────────────────┼───────┼─────────────────────┤
240 │FOO=bar │ FOO │ bar │
241 ├──────────────────┼───────┼─────────────────────┤
242 │FOO=bar=baz │ FOO │ bar=baz │
243 ├──────────────────┼───────┼─────────────────────┤
244 │FLAGS=-march=foo │ FLAGS │ -march=foo │
245 │-mtune=bar │ │ -mtune=bar │
246 ├──────────────────┼───────┼─────────────────────┤
247 │FLAGS='-march=foo │ FLAGS │ -march=foo │
248 │-mtune=bar' │ │ -mtune=bar │
249 ├──────────────────┼───────┼─────────────────────┤
250 │FOO= │ FOO │ (empty string) │
251 ├──────────────────┼───────┼─────────────────────┤
252 │FOO='' │ FOO │ (empty string) │
253 ├──────────────────┼───────┼─────────────────────┤
254 │FOO='''' │ FOO │ '' (two single │
255 │ │ │ quotes) │
256 └──────────────────┴───────┴─────────────────────┘
257 Example invalid lines:
259 ┌────────┬─────────────────────┐
261 │Line │ Problem │
262 ├────────┼─────────────────────┤
263 │FOO bar │ no separator │
264 ├────────┼─────────────────────┤
265 │=bar │ key cannot be empty │
266 └────────┴─────────────────────┘
267 Example valid lines that are probably not what you want:
269 ┌────────────────────┬───────┬────────────────┬──────────────────┐
271 │Line │ Key │ Value │ Problem │
272 ├────────────────────┼───────┼────────────────┼──────────────────┤
273 │FOO="bar" │ FOO │ "bar" │ double quotes │
274 │ │ │ │ aren’t stripped │
275 ├────────────────────┼───────┼────────────────┼──────────────────┤
276 │FOO=bar # baz │ FOO │ bar # baz │ comments not │
277 │ │ │ │ supported │
278 ├────────────────────┼───────┼────────────────┼──────────────────┤
279 │PATH=$PATH:/opt/bin │ PATH │ $PATH:/opt/bin │ variables not │
280 │ │ │ │ expanded │
281 ├────────────────────┼───────┼────────────────┼──────────────────┤
282 │ FOO=bar │ FOO │ bar │ leading space in │
283 │ │ │ │ key │
284 ├────────────────────┼───────┼────────────────┼──────────────────┤
285 │FOO= bar │ FOO │ bar │ leading space in │
286 │ │ │ │ value │
287 └────────────────────┴───────┴────────────────┴──────────────────┘
288 Removing variables with --unset-env
290 The purpose of --unset-env=GLOB is to remove unwanted environment vari‐
291 ables. The argument GLOB is a glob pattern (dialect fnmatch(3) with no
292 flags); all variables with matching names are removed from the environ‐
293 ment.
294 WARNING:
296 Because the shell also interprets glob patterns, if any wildcard
297 characters are in GLOB, it is important to put it in single quotes
298 to avoid surprises.
299 GLOB must be a non-empty string.
301 Example 1: Remove the single environment variable FOO:
303 $ export FOO=bar
305 $ env | fgrep FOO
306 FOO=bar
307 $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
308 $
309 Example 2: Hide from a container the fact that it’s running in a Slurm
311 allocation, by removing all variables beginning with SLURM. You might
312 want to do this to test an MPI program with one rank and no launcher:
313 $ salloc -N1
315 $ env | egrep '^SLURM' | wc
316 44 44 1092
317 $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
318 [... long error message ...]
319 $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
320 0: MPI version:
321 Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
322 0: init ok cn001.localdomain, 1 ranks, userns 4026532530
323 0: send/receive ok
324 0: finalize ok
325 Example 3: Clear the environment completely (remove all variables):
327 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
329 $
330 Note that some programs, such as shells, set some environment variables
332 even if started with no init files:
333 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
335 SHLVL=1
336 PWD=/
337 _=/usr/bin/env
338 $
339
341 Run the command echo hello inside a Charliecloud container using the
342 unpacked image at /data/foo:
343 $ ch-run /data/foo -- echo hello
345 hello
346 Run an MPI job that can use CMA to communicate:
348 $ srun ch-run --join /data/foo -- bar
350
352 If Charliecloud was obtained from your Linux distribution, use your
353 distribution’s bug reporting procedures.
354 Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
356
358 charliecloud(1)
359 Full documentation at: <https://hpc.github.io/charliecloud>
361
363 Reid Priedhorsky, Tim Randles, and others
364
366 2014–2018, Los Alamos National Security, LLC
367 2019-08-22 00:00 Coordinated Universal Time CH-RUN(1)