1CH-RUN(1) Charliecloud CH-RUN(1)
2
3
4
6 ch-run - Run a command in a Charliecloud container
7
9 $ ch-run [OPTION...] NEWROOT CMD [ARG...]
10
12 Run command CMD in a Charliecloud container using the flattened and
13 unpacked image directory located at NEWROOT.
14
15 -b, --bind=SRC[:DST]
16 mount SRC at guest DST (default /mnt/0, /mnt/1, etc.)
17
18 -c, --cd=DIR
19 initial working directory in container
20
21 --ch-ssh
22 bind ch-ssh(1) into container at /usr/bin/ch-ssh
23
24 -g, --gid=GID
25 run as group GID within container
26
27 -j, --join
28 use the same container (namespaces) as peer ch-run invoca‐
29 tions
30
31 --join-pid=PID
32 join the namespaces of an existing process
33
34 --join-ct=N
35 number of ch-run peers (implies --join; default: see below)
36
37 --join-tag=TAG
38 label for ch-run peer group (implies --join; default: see
39 below)
40
41 --no-home
42 do not bind-mount your home directory (by default, your home
43 directory is mounted at /home/$USER in the container)
44
45 -t, --private-tmp
46 use container-private /tmp (by default, /tmp is shared with
47 the host)
48
49 --set-env=FILE
50 set environment variables as specified in host path FILE
51
52 -u, --uid=UID
53 run as user UID within container
54
55 --unset-env=GLOB
56 unset environment variables whose names match GLOB
57
58 -v, --verbose
59 be more verbose (debug if repeated)
60
61 -w, --write
62 mount image read-write (by default, the image is mounted
63 read-only)
64
65 -?, --help
66 print help and exit
67
68 --usage
69 print a short usage message and exit
70
71 -V, --version
72 print version and exit
73
75 In addition to any directories specified by the user with --bind,
76 ch-run has standard host files and directories that are bind-mounted in
77 as well.
78
79 The following host files and directories are bind-mounted at the same
80 location in the container. These cannot be disabled.
81
82 · /dev
83
84 · /etc/passwd
85
86 · /etc/group
87
88 · /etc/hosts
89
90 · /etc/resolv.conf
91
92 · /proc
93
94 · /sys
95
96 Three additional bind mounts can be disabled by the user:
97
98 · Your home directory (i.e., $HOME) is mounted at guest /home/$USER
99 by default. This is accomplished by mounting a new tmpfs at /home,
100 which hides any image content under that path. If --no-home is
101 specified, neither of these things happens and the image’s /home
102 is exposed unaltered.
103
104 · /tmp is shared with the host by default. If --private-tmp is spec‐
105 ified, a new tmpfs is mounted on the guest’s /tmp instead.
106
107 · If file /usr/bin/ch-ssh is present in the image, it is
108 over-mounted with the ch-ssh binary in the same directory as
109 ch-run.
110
112 By default, different ch-run invocations use different user and mount
113 namespaces (i.e., different containers). While this has no impact on
114 sharing most resources between invocations, there are a few important
115 exceptions. These include:
116
117 1. ptrace(2), used by debuggers and related tools. One can attach a
118 debugger to processes in descendant namespaces, but not sibling
119 namespaces. The practical effect of this is that (without --join),
120 you can’t run a command with ch-run and then attach to it with a
121 debugger also run with ch-run.
122
123 2. Cross-memory attach (CMA) is used by cooperating processes to commu‐
124 nicate by simply reading and writing one another’s memory. This is
125 also not permitted between sibling namespaces. This affects various
126 MPI implementations that use CMA to pass messages between ranks on
127 the same node, because it’s faster than traditional shared memory.
128
129 --join is designed to address this by placing related ch-run commands
130 (the “peer group”) in the same container. This is done by one of the
131 peers creating the namespaces with unshare(2) and the others joining
132 with setns(2).
133
134 To do so, we need to know the number of peers and a name for the group.
135 These are specified by additional arguments that can (hopefully) be
136 left at default values in most cases:
137
138 · --join-ct sets the number of peers. The default is the value of the
139 first of the following environment variables that is defined:
140 OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE,
141 SLURM_CPUS_ON_NODE.
142
143 · --join-tag sets the tag that names the peer group. The default is
144 environment variable SLURM_STEP_ID, if defined; otherwise, the PID of
145 ch-run’s parent. Tags can be re-used for peer groups that start at
146 different times, i.e., once all peer ch-run have replaced themselves
147 with the user command, the tag can be re-used.
148
149 Caveats:
150
151 · One cannot currently add peers after the fact, for example, if one
152 decides to start a debugger after the fact. (This is only required
153 for code with bugs and is thus an unusual use case.)
154
155 · ch-run instances race. The winner of this race sets up the names‐
156 paces, and the other peers use the winner to find the namespaces to
157 join. Therefore, if the user command of the winner exits, any remain‐
158 ing peers will not be able to join the namespaces, even if they are
159 still active. There is currently no general way to specify which
160 ch-run should be the winner.
161
162 · If --join-ct is too high, the winning ch-run’s user command exits
163 before all peers join, or ch-run itself crashes, IPC resources such
164 as semaphores and shared memory segments will be leaked. These appear
165 as files in /dev/shm/ and can be removed with rm(1).
166
167 · Many of the arguments given to the race losers, such as the image
168 path and --bind, will be ignored in favor of what was given to the
169 winner.
170
172 ch-run leaves environment variables unchanged, i.e. the host environ‐
173 ment is passed through unaltered, except:
174
175 · limited tweaks to avoid significant guest breakage;
176
177 · user-set variables via --set-env; and
178
179 · user-unset variables via --unset-env.
180
181 This section describes these features.
182
183 The default tweaks happen first, and then --set-env and --unset-env in
184 the order specified on the command line. The latter two can be repeated
185 arbitrarily many times, e.g. to add/remove multiple variable sets or
186 add only some variables in a file.
187
188 Default behavior
189 By default, ch-run makes the following environment variable changes:
190
191 · $HOME: If the path to your home directory is not /home/$USER on the
192 host, then an inherited $HOME will be incorrect inside the guest.
193 This confuses some software, such as Spack.
194
195 Thus, we change $HOME to /home/$USER, unless --no-home is specified,
196 in which case it is left unchanged.
197
198 · $PATH: Newer Linux distributions replace some root-level directories,
199 such as /bin, with symlinks to their counterparts in /usr.
200
201 Some of these distributions (e.g., Fedora 24) have also dropped /bin
202 from the default $PATH. This is a problem when the guest OS does not
203 have a merged /usr (e.g., Debian 8 “Jessie”). Thus, we add /bin to
204 $PATH if it’s not already present.
205
206 Further reading:
207
208 · The case for the /usr Merge
209
210 · Fedora
211
212 · Debian
213
214 Setting variables with --set-env
215 The purpose of --set-env=FILE is to set environment variables that can‐
216 not be inherited from the host shell, e.g. Dockerfile ENV directives or
217 other build-time configuration. FILE is a host path to provide the
218 greatest flexibility; guest paths can be specified by prepending the
219 image path.
220
221 ch-docker2tar(1) lists variables specified at build time in Dockerfiles
222 in the image in file /environment. To set these variables:
223 --set-env=$IMG/environment.
224
225 Variable values in FILE replace any already set. If a variable is
226 repeated, the last value wins.
227
228 The syntax of FILE is key-value pairs separated by the first equals
229 character (=, ASCII 61), one per line, with optional single straight
230 quotes (', ASCII 39) around the value. Empty lines are ignored. New‐
231 lines (ASCII 10) are not permitted in either key or value. No variable
232 expansion, comments, etc. are provided. The value may be empty, but not
233 the key. (This syntax is designed to accept the output of printenv and
234 be easily produced by other simple mechanisms.) Examples of valid
235 lines:
236
237 ┌──────────────────┬───────┬─────────────────────┐
238 │Line │ Key │ Value │
239 ├──────────────────┼───────┼─────────────────────┤
240 │FOO=bar │ FOO │ bar │
241 ├──────────────────┼───────┼─────────────────────┤
242 │FOO=bar=baz │ FOO │ bar=baz │
243 ├──────────────────┼───────┼─────────────────────┤
244 │FLAGS=-march=foo │ FLAGS │ -march=foo │
245 │-mtune=bar │ │ -mtune=bar │
246 ├──────────────────┼───────┼─────────────────────┤
247 │FLAGS='-march=foo │ FLAGS │ -march=foo │
248 │-mtune=bar' │ │ -mtune=bar │
249 ├──────────────────┼───────┼─────────────────────┤
250 │FOO= │ FOO │ (empty string) │
251 ├──────────────────┼───────┼─────────────────────┤
252 │FOO='' │ FOO │ (empty string) │
253 ├──────────────────┼───────┼─────────────────────┤
254 │FOO='''' │ FOO │ '' (two single │
255 │ │ │ quotes) │
256 └──────────────────┴───────┴─────────────────────┘
257
258 Example invalid lines:
259
260 ┌────────┬─────────────────────┐
261 │Line │ Problem │
262 ├────────┼─────────────────────┤
263 │FOO bar │ no separator │
264 ├────────┼─────────────────────┤
265 │=bar │ key cannot be empty │
266 └────────┴─────────────────────┘
267
268 Example valid lines that are probably not what you want:
269
270 ┌────────────────────┬───────┬────────────────┬──────────────────┐
271 │Line │ Key │ Value │ Problem │
272 ├────────────────────┼───────┼────────────────┼──────────────────┤
273 │FOO="bar" │ FOO │ "bar" │ double quotes │
274 │ │ │ │ aren’t stripped │
275 ├────────────────────┼───────┼────────────────┼──────────────────┤
276 │FOO=bar # baz │ FOO │ bar # baz │ comments not │
277 │ │ │ │ supported │
278 ├────────────────────┼───────┼────────────────┼──────────────────┤
279 │PATH=$PATH:/opt/bin │ PATH │ $PATH:/opt/bin │ variables not │
280 │ │ │ │ expanded │
281 ├────────────────────┼───────┼────────────────┼──────────────────┤
282 │ FOO=bar │ FOO │ bar │ leading space in │
283 │ │ │ │ key │
284 ├────────────────────┼───────┼────────────────┼──────────────────┤
285 │FOO= bar │ FOO │ bar │ leading space in │
286 │ │ │ │ value │
287 └────────────────────┴───────┴────────────────┴──────────────────┘
288
289 Removing variables with --unset-env
290 The purpose of --unset-env=GLOB is to remove unwanted environment vari‐
291 ables. The argument GLOB is a glob pattern (dialect fnmatch(3) with no
292 flags); all variables with matching names are removed from the environ‐
293 ment.
294
295 WARNING:
296 Because the shell also interprets glob patterns, if any wildcard
297 characters are in GLOB, it is important to put it in single quotes
298 to avoid surprises.
299
300 GLOB must be a non-empty string.
301
302 Example 1: Remove the single environment variable FOO:
303
304 $ export FOO=bar
305 $ env | fgrep FOO
306 FOO=bar
307 $ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
308 $
309
310 Example 2: Hide from a container the fact that it’s running in a Slurm
311 allocation, by removing all variables beginning with SLURM. You might
312 want to do this to test an MPI program with one rank and no launcher:
313
314 $ salloc -N1
315 $ env | egrep '^SLURM' | wc
316 44 44 1092
317 $ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
318 [... long error message ...]
319 $ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
320 0: MPI version:
321 Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
322 0: init ok cn001.localdomain, 1 ranks, userns 4026532530
323 0: send/receive ok
324 0: finalize ok
325
326 Example 3: Clear the environment completely (remove all variables):
327
328 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
329 $
330
331 Note that some programs, such as shells, set some environment variables
332 even if started with no init files:
333
334 $ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian9 -- bash --noprofile --norc -c env
335 SHLVL=1
336 PWD=/
337 _=/usr/bin/env
338 $
339
341 Run the command echo hello inside a Charliecloud container using the
342 unpacked image at /data/foo:
343
344 $ ch-run /data/foo -- echo hello
345 hello
346
347 Run an MPI job that can use CMA to communicate:
348
349 $ srun ch-run --join /data/foo -- bar
350
352 If Charliecloud was obtained from your Linux distribution, use your
353 distribution’s bug reporting procedures.
354
355 Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues>
356
358 charliecloud(1)
359
360 Full documentation at: <https://hpc.github.io/charliecloud>
361
363 Reid Priedhorsky, Tim Randles, and others
364
366 2014–2018, Los Alamos National Security, LLC
367
368
369
370
371 2019-08-22 00:00 Coordinated Universal Time CH-RUN(1)