1MPIRUN(1) OPEN MPI COMMANDS MPIRUN(1)
2
6 orterun, mpirun, mpiexec - Execute serial and parallel jobs in Open
7 MPI.
8 Note: mpirun, mpiexec, and orterun are all exact synonyms for each
10 other. Using any of the names will result in exactly identical behav‐
11 ior.
12
14 Single Process Multiple Data (SPMD) Model:
15 mpirun [ options ] <program> [ <args> ]
17 Multiple Instruction Multiple Data (MIMD) Model:
19 mpirun [ global_options ]
21 [ local_options1 ] <program1> [ <args1> ] :
22 [ local_options2 ] <program2> [ <args2> ] :
23 ... :
24 [ local_optionsN ] <programN> [ <argsN> ]
25 Note that in both models, invoking mpirun via an absolute path name is
27 equivalent to specifying the --prefix option with a <dir> value equiva‐
28 lent to the directory where mpirun resides, minus its last subdirec‐
29 tory. For example:
30 shell$ /usr/local/bin/mpirun ...
32 is equivalent to
34 shell$ mpirun --prefix /usr/local
36
39 If you are simply looking for how to run an MPI application, you proba‐
40 bly want to use a command line of the following form:
41 shell$ mpirun -np X [ --hostfile <filename> ] <program>
43 This will run X copies of <program> in your current run-time environ‐
45 ment (if running under a supported resource manager, Open MPI's mpirun
46 will usually automatically use the corresponding resource manager
47 process starter, as opposed to, for example, rsh or ssh, which require
48 the use of a hostfile, or will default to running all X copies on the
49 localhost), scheduling (by default) in a round-robin fashion by CPU
50 slot. See the rest of this page for more details.
51
53 mpirun will send the name of the directory where it was invoked on the
54 local node to each of the remote nodes, and attempt to change to that
55 directory. See the "Current Working Directory" section below for fur‐
56 ther details.
57 <args> Pass these run-time arguments to every new process. These
59 must always be the last arguments to mpirun. If an app con‐
60 text file is used, <args> will be ignored.
61 <program> The program executable. This is identified as the first non-
63 recognized argument to mpirun.
64 -aborted, --aborted <#>
66 Set the maximum number of aborted processes to display.
67 --app <appfile>
69 Provide an appfile, ignoring all other command line options.
70 -bynode, --bynode
72 Allocate (map) the processes by node in a round-robin scheme.
73 -byslot, --byslot
75 Allocate (map) the processes by slot in a round-robin scheme.
76 This is the default.
77 -c <#> Synonym for -np.
79 -debug, --debug
81 Invoke the user-level debugger indicated by the
82 orte_base_user_debugger MCA parameter.
83 -debugger, --debugger
85 Sequence of debuggers to search for when --debug is used
86 (i.e. a synonym for orte_base_user_debugger MCA parameter).
87 -gmca, --gmca <key> <value>
89 Pass global MCA parameters that are applicable to all con‐
90 texts. <key> is the parameter name; <value> is the parameter
91 value.
92 -h, --help
94 Display help for this command
95 -H <host1,host2,...,hostN>
97 Synonym for -host.
98 -host, --host <host1,host2,...,hostN>
100 List of hosts on which to invoke processes.
101 -hostfile, --hostfile <hostfile>
103 Provide a hostfile to use.
104 -machinefile, --machinefile <machinefile>
106 Synonym for -hostfile.
107 -mca, --mca <key> <value>
109 Send arguments to various MCA modules. See the "MCA" sec‐
110 tion, below.
111 -n, --n <#>
113 Synonym for -np.
114 -np <#> Run this many copies of the program on the given nodes. This
116 option indicates that the specified file is an executable
117 program and not an application context.
118 -nw, --nw Launch the processes and do not wait for their completion.
120 mpirun will complete as soon as successful launch occurs.
121 -path, --path <path>
123 <path> that will be used when attempting to locate requested
124 executables.
125 --prefix <dir>
127 Prefix directory that will be used to set the PATH and
128 LD_LIBRARY_PATH on the remote node before invoking Open MPI
129 or the target process. See the "Remote Execution" section,
130 below.
131 --tmpdir <dir>
133 Set the root for the session directory tree for mpirun only.
134 -tv, --tv Launch processes under the TotalView debugger. Deprecated
136 backwards compatibility flag. Synonym for --debug.
137 --universe <username@hostname:universe_name>
139 For this application, set the universe name as:
140 username@hostname:universe_name
141 -v, --verbose
143 Be verbose
144 -wd <dir> Change to the directory <dir> before the user's program exe‐
146 cutes. See the "Current Working Directory" section for notes
147 on relative paths. Note: If the -wd option appears both on
148 the command line and in an application context, the context
149 will take precedence over the command line.
150 -x <env> Export the specified environment variables to the remote
152 nodes before executing the program. Existing environment
153 variables can be specified (see the Examples section, below),
154 or new variable names specified with corresponding values.
155 The parser for the -x option is not very sophisticated; it
156 does not even understand quoted values. Users are advised to
157 set variables in the environment, and then use -x to export
158 (not define) them.
159 The following options are useful for developers; they are not generally
161 useful to most ORTE and/or MPI users:
162 -d, --debug-devel
164 Enable debugging of the OpenRTE (the run-time layer in Open
165 MPI). This is not generally useful for most users.
166 --debug-daemons
168 Enable debugging of any OpenRTE daemons used by this applica‐
169 tion.
170 --debug-daemons-file
172 Enable debugging of any OpenRTE daemons used by this applica‐
173 tion, storing output in files.
174 --no-daemonize
176 Do not detach OpenRTE daemons used by this application.
177
179 One invocation of mpirun starts an MPI application running under Open
180 MPI. If the application is single process multiple data (SPMD), the
181 application can be specified on the mpirun command line.
182 If the application is multiple instruction multiple data (MIMD), com‐
184 prising of multiple programs, the set of programs and argument can be
185 specified in one of two ways: Extended Command Line Arguments, and
186 Application Context.
187 An application context describes the MIMD program set including all
189 arguments in a separate file. This file essentially contains multiple
190 mpirun command lines, less the command name itself. The ability to
191 specify different options for different instantiations of a program is
192 another reason to use an application context.
193 Extended command line arguments allow for the description of the appli‐
195 cation layout on the command line using colons (:) to separate the
196 specification of programs and arguments. Some options are globally set
197 across all specified programs (e.g. --hostfile), while others are spe‐
198 cific to a single program (e.g. -np).
199 Location Nomenclature
201 As described above, mpirun can specify arbitrary locations in the cur‐
202 rent Open MPI universe. Locations can be specified either by CPU or by
203 node.
204 Note: Open MPI does not bind processes to CPUs -- specifying a location
206 "by CPU" is really a convenience mechanism for SMPs that ultimately
207 maps down to a specific node.
208 Specifying locations by node will launch one copy of an executable per
210 specified node. Using the --bynode option tells Open MPI to use all
211 available nodes. Using the --byslot option tells Open MPI to use all
212 slots on an available node before allocating resources on the next
213 available node. For example:
214 mpirun --bynode -np 4 a.out
216 Runs one copy of the the executable a.out on all available nodes in
217 the Open MPI universe. MPI_COMM_WORLD rank 0 will be on node0,
218 rank 1 will be on node1, etc. Regardless of how many slots are
219 available on each of the nodes.
220 mpirun --byslot -np 4 a.out
222 Runs one copy of the the executable a.out on each slot on a given
223 node before running the executable on other available nodes.
224 Specifying Hosts
226 Hosts can be specified in a number of ways. The most common of which is
227 in a
228 shell$ cat my-hostfile
230 node00 slots=2
231 node01 slots=2
232 node02 slots=2
233 mpirun --hostfile my-hostfile -np 3 a.out
236 This will run one copy of the executable a.out on hosts
237 node00,node01, and node02.
238 Another method for specifying hosts is directly on the command line.
240 Here can can include and exclude hosts from the set of hosts to run on.
241 For example:
242 mpirun -np 3 --host a a.out
244 Runs three copies of the executable a.out on host a.
245 mpirun -np 3 --host a,b,c a.out
247 Runs one copy of the executable a.out on hosts a, b, and c.
248 mpirun -np 3 --hostfile my-hostfile --host node00 a.out
250 Runs three copies of the executable a.out on host node00.
251 mpirun -np 3 --hostfile my-hostfile --host node10 a.out
253 This will prompt an error since node10 is not in my-hostfile;
254 mpirun will abort.
255 shell$ mpirun -np 1 --host a hostname : -np 2 --host b,c uptime
257 Runs one copy of the executable hostname on host a. And runs one
258 copy of the executable uptime on hosts b and c.
259 Application Context or Executable Program?
261 To distinguish the two different forms, mpirun looks on the command
262 line for --app option. If it is specified, then the file named on the
263 command line is assumed to be an application context. If it is not
264 specified, then the file is assumed to be an executable program.
265 Locating Files
267 If no relative or absolute path is specified for a file, Open MPI will
268 look for files by searching the directories in the user's PATH environ‐
269 ment variable as defined on the source node(s).
270 If a relative directory is specified, it must be relative to the ini‐
272 tial working directory determined by the specific starter used. For
273 example when using the rsh or ssh starters, the initial directory is
274 $HOME by default. Other starters may set the initial directory to the
275 current working directory from the invocation of mpirun.
276 Current Working Directory
278 The -wd mpirun option allows the user to change to an arbitrary direc‐
279 tory before their program is invoked. It can also be used in applica‐
280 tion context files to specify working directories on specific nodes
281 and/or for specific applications.
282 If the -wd option appears both in a context file and on the command
284 line, the context file directory will override the command line value.
285 If the -wd option is specified, Open MPI will attempt to change to the
287 specified directory on all of the remote nodes. If this fails, mpirun
288 will abort.
289 If the -wd option is not specified, Open MPI will send the directory
291 name where mpirun was invoked to each of the remote nodes. The remote
292 nodes will try to change to that directory. If they are unable (e.g.,
293 if the directory does not exit on that node), then Open MPI will use
294 the default directory determined by the starter.
295 All directory changing occurs before the user's program is invoked; it
297 does not wait until MPI_INIT is called.
298 Standard I/O
300 Open MPI directs UNIX standard input to /dev/null on all processes
301 except the MPI_COMM_WORLD rank 0 process. The MPI_COMM_WORLD rank 0
302 process inherits standard input from mpirun. Note: The node that
303 invoked mpirun need not be the same as the node where the
304 MPI_COMM_WORLD rank 0 process resides. Open MPI handles the redirection
305 of mpirun's standard input to the rank 0 process.
306 Open MPI directs UNIX standard output and error from remote nodes to
308 the node that invoked mpirun and prints it on the standard output/error
309 of mpirun. Local processes inherit the standard output/error of mpirun
310 and transfer to it directly.
311 Thus it is possible to redirect standard I/O for Open MPI applications
313 by using the typical shell redirection procedure on mpirun.
314 shell$ mpirun -np 2 my_app < my_input > my_output
316 Note that in this example only the MPI_COMM_WORLD rank 0 process will
318 receive the stream from my_input on stdin. The stdin on all the other
319 nodes will be tied to /dev/null. However, the stdout from all nodes
320 will be collected into the my_output file.
321 Process Termination / Signal Handling
323 During the run of an MPI application, if any rank dies abnormally
324 (either exiting before invoking MPI_FINALIZE, or dying as the result of
325 a signal), mpirun will print out an error message and kill the rest of
326 the MPI application.
327 User signal handlers should probably avoid trying to cleanup MPI state
329 (Open MPI is, currently, neither thread-safe nor async-signal-safe).
330 For example, if a segmentation fault occurs in MPI_SEND (perhaps
331 because a bad buffer was passed in) and a user signal handler is
332 invoked, if this user handler attempts to invoke MPI_FINALIZE, Bad
333 Things could happen since Open MPI was already "in" MPI when the error
334 occurred. Since mpirun will notice that the process died due to a sig‐
335 nal, it is probably not necessary (and safest) for the user to only
336 clean up non-MPI state.
337 Process Environment
339 Processes in the MPI application inherit their environment from the
340 Open RTE daemon upon the node on which they are running. The environ‐
341 ment is typically inherited from the user's shell. On remote nodes,
342 the exact environment is determined by the boot MCA module used. The
343 rsh launch module, for example, uses either rsh/ssh to launch the Open
344 RTE daemon on remote nodes, and typically executes one or more of the
345 user's shell-setup files before launching the Open RTE daemon. When
346 running dynamically linked applications which require the
347 LD_LIBRARY_PATH environment variable to be set, care must be taken to
348 ensure that it is correctly set when booting Open MPI.
349 See the "Remote Execution" section for more details.
351 Remote Execution
353 Open MPI requires that the PATH environment variable be set to find
354 executables on remote nodes (this is typically only necessary in rsh-
355 or ssh-based environments -- batch/scheduled environments typically
356 copy the current environment to the execution of remote jobs, so if the
357 current environment has PATH and/or LD_LIBRARY_PATH set properly, the
358 remote nodes will also have it set properly). If Open MPI was compiled
359 with shared library support, it may also be necessary to have the
360 LD_LIBRARY_PATH environment variable set on remote nodes as well (espe‐
361 cially to find the shared libraries required to run user MPI applica‐
362 tions).
363 However, it is not always desirable or possible to edit shell startup
365 files to set PATH and/or LD_LIBRARY_PATH. The --prefix option is pro‐
366 vided for some simple configurations where this is not possible.
367 The --prefix option takes a single argument: the base directory on the
369 remote node where Open MPI is installed. Open MPI will use this direc‐
370 tory to set the remote PATH and LD_LIBRARY_PATH before executing any
371 Open MPI or user applications. This allows running Open MPI jobs with‐
372 out having pre-configued the PATH and LD_LIBRARY_PATH on the remote
373 nodes.
374 Open MPI adds the basename of the current node's "bindir" (the direc‐
376 tory where Open MPI's executables are installed) to the prefix and uses
377 that to set the PATH on the remote node. Similarly, Open MPI adds the
378 basename of the current node's "libdir" (the directory where Open MPI's
379 libraries are installed) to the prefix and uses that to set the
380 LD_LIBRARY_PATH on the remote node. For example:
381 Local bindir: /local/node/directory/bin
383 Local libdir: /local/node/directory/lib64
385 If the following command line is used:
387 shell$ mpirun --prefix /remote/node/directory
389 Open MPI will add "/remote/node/directory/bin" to the PATH and
391 "/remote/node/directory/lib64" to the D_LIBRARY_PATH on the remote node
392 before attempting to execute anything.
393 Note that --prefix can be set on a per-context basis, allowing for dif‐
395 ferent values for different nodes.
396 The --prefix option is not sufficient if the installation paths on the
398 remote node are different than the local node (e.g., if "/lib" is used
399 on the local node, but "/lib64" is used on the remote node), or if the
400 installation paths are something other than a subdirectory under a com‐
401 mon prefix.
402 Note that executing mpirun via an absolute pathname is equivalent to
404 specifying --prefix without the last subdirectory in the absolute path‐
405 name to mpirun. For example:
406 shell$ /usr/local/bin/mpirun ...
408 is equivalent to
410 shell$ mpirun --prefix /usr/local
412 Exported Environment Variables
414 All environment variables that are named in the form OMPI_* will auto‐
415 matically be exported to new processes on the local and remote nodes.
416 The -x option to mpirun can be used to export specific environment
417 variables to the new processes. While the syntax of the -x option
418 allows the definition of new variables, note that the parser for this
419 option is currently not very sophisticated - it does not even under‐
420 stand quoted values. Users are advised to set variables in the envi‐
421 ronment and use -x to export them; not to define them.
422 MCA (Modular Component Architecture)
424 The -mca switch allows the passing of parameters to various MCA mod‐
425 ules. MCA modules have direct impact on MPI programs because they
426 allow tunable parameters to be set at run time (such as which BTL com‐
427 munication device driver to use, what parameters to pass to that BTL,
428 etc.).
429 The -mca switch takes two arguments: <key> and <value>. The <key>
431 argument generally specifies which MCA module will receive the value.
432 For example, the <key> "btl" is used to select which BTL to be used for
433 transporting MPI messages. The <value> argument is the value that is
434 passed. For example:
435 mpirun -mca btl tcp,self -np 1 foo
437 Tells Open MPI to use the "tcp" and "self" BTLs, and to run a sin‐
438 gle copy of "foo" an allocated node.
439 mpirun -mca btl self -np 1 foo
441 Tells Open MPI to use the "self" BTL, and to run a single copy of
442 "foo" an allocated node.
443 The -mca switch can be used multiple times to specify different <key>
445 and/or <value> arguments. If the same <key> is specified more than
446 once, the <value>s are concatenated with a comma (",") separating them.
447 Note: The -mca switch is simply a shortcut for setting environment
449 variables. The same effect may be accomplished by setting correspond‐
450 ing environment variables before running mpirun. The form of the envi‐
451 ronment variables that Open MPI sets are:
452 OMPI_<key>=<value>
454 Note that the -mca switch overrides any previously set environment
456 variables. Also note that unknown <key> arguments are still set as
457 environment variable -- they are not checked (by mpirun) for correct‐
458 ness. Illegal or incorrect <value> arguments may or may not be
459 reported -- it depends on the specific MCA module.
460
462 Be sure to also see the examples in the "Location Nomenclature" sec‐
463 tion, above.
464 mpirun -np 1 prog1
466 Load and execute prog1 on one node. Search the user's $PATH for
467 the executable file on each node.
468 mpirun -np 8 --byslot prog1
470 Run 8 copies of prog1 wherever Open MPI wants to run them.
471 mpirun -np 4 -mca btl ib,tcp,self prog1
473 Run 4 copies of prog1 using the "ib", "tcp", and "self" BTL's for
474 the transport of MPI messages.
475
477 mpirun returns 0 if all ranks started by mpirun exit after calling
478 MPI_FINALIZE. A non-zero value is returned if an internal error
479 occurred in mpirun, or one or more ranks exited before calling
480 MPI_FINALIZE. If an internal error occurred in mpirun, the correspond‐
481 ing error code is returned. In the event that one or more ranks exit
482 before calling MPI_FINALIZE, the return value of the rank of the
483 process that mpirun first notices died before calling MPI_FINALIZE will
484 be returned. Note that, in general, this will be the first rank that
485 died but is not guaranteed to be so.
486 However, note that if the -nw switch is used, the return value from
488 mpirun does not indicate the exit status of the ranks.
489Open MPI March 2006 MPIRUN(1)