1prte-map(1) PRTE prte-map(1)
2
6 PRTE: Mapping, Ranking, and Binding
7
9 PRTE employs a three-phase procedure for assigning process locations
10 and ranks:
11 1. mapping: Assigns a default location to each process
13 2. ranking: Assigns a unique rank value to each process
15 3. binding: Constrains each process to run on specific processors
17 This document describes these three phases with examples. Unless oth‐
19 erwise noted, this behavior is shared by prun, prterun, and prte.
20
22 The two binaries that most influence process layout are prte and prun.
23 The prte process discovers the allocation, starts the daemons, and de‐
24 fines the default mapping/ranking/binding for all jobs. The prun
25 process defines the specific mapping/ranking/binding for a specific
26 job. Most of the command line controls are targeted to prun since each
27 job has its own unique requirements.
28 prterun is just a wrapper around prte for a single job PRTE DVM. It is
30 doing the job of both prte and prun, and, as such, accepts the sum all
31 of their command line arguments. Any example that uses prun can sub‐
32 stitute the use of prterun except where otherwise noted.
33 The prte process attempts to automatically discover the nodes in the
35 allocation by querying supported resource managers. If a support re‐
36 source manager is not present then prte relies on a hostfile provided
37 by the user. In the absence of such a hostfile it will run all pro‐
38 cesses on the localhost.
39 If running under a supported resource manager, the prte process will
41 start the daemon processes (prted) on the remote nodes using the corre‐
42 sponding resource manager process starter. If no such starter is
43 available then rsh or ssh is used.
44 PRTE automatically maps processes in a round-robin fashion by CPU slot
46 in one of two ways in the absence of any further directives:
47 Map by core:
49 when the number of total processes in the job is <= 2
50 Map by package:
52 when the number of total processes in the job is > 2
53 PRTE automatically binds processes. Three binding patterns are used in
55 the absence of any further directives:
56 Bind to core:
58 when the number of total processes in the job is <= 2
59 Bind to package:
61 when the number of total processes in the job is > 2
62 Bind to none:
64 when oversubscribed
65 If your application uses threads, then you probably want to ensure that
67 you are either not bound at all (by specifying --bind-to none), or
68 bound to multiple cores using an appropriate binding level or specific
69 number of processing elements per application process.
70 PRTE automatically ranks processes starting from 0. Two ranking pat‐
72 terns are used in the absence of any further directives:
73 Rank by slot:
75 when the number of total processes in the job is <= 2
76 Rank by package:
78 when the number of total processes in the job is > 2
79
81 Listed here are the subset of command line options that will be used in
82 the process mapping/ranking/binding discussion in this manual page.
83 Specifying Host Nodes
85 Use one of the following options to specify which hosts (nodes) within
86 the PRTE DVM environment to run on.
87 --host <host1,host2,...,hostN> or --host <host1:X,host2:Y,...,hostN:Z>
89 List of hosts on which to invoke processes. After each hostname
90 a colon (:) followed by a positive integer can be used to speci‐
91 fy the number of slots on that host (:X, :Y, and :Z). The de‐
92 fault is 1.
93 --hostfile <hostfile>
95 Provide a hostfile to use.
96 --machinefile <machinefile>
98 Synonym for -hostfile.
99 --default-hostfile <hostfile>
101 Provide a default hostfile to use.
102 Process Mapping / Ranking / Binding Options
104 The following options specify the number of processes to launch. Note
105 that none of the options imply a particular binding policy - e.g., re‐
106 questing N processes for each socket does not imply that the processes
107 will be bound to the socket.
108 -c, -n, --n, --np <#>
110 Run this many copies of the program on the given nodes. This
111 option indicates that the specified file is an executable pro‐
112 gram and not an application context. If no value is provided
113 for the number of copies to execute (i.e., neither the -np nor
114 its synonyms are provided on the command line), prun will auto‐
115 matically execute a copy of the program on each process slot
116 (see below for description of a “process slot”). This feature,
117 however, can only be used in the SPMD model and will return an
118 error (without beginning execution of the application) other‐
119 wise.
120 To map processes across sets of objects:
122 --map-by <object>
124 Map to the specified object. See defaults in Quick Summary.
125 Supported options include slot, hwthread, core, l1cache,
126 l2cache, l3cache, package, node, seq, dist, ppr, and rankfile.
127 Any object can include qualifier by adding a colon (:) and any combina‐
129 tion of one or more of the following to the --map-by option:
130 • PE=n bind n processing elements to each process
132 • SPAN load balance the processes across the allocation
134 • OVERSUBSCRIBE allow more processes on a node than processing elements
136 • NOOVERSUBSCRIBE means !OVERSUBSCRIBE
138 • NOLOCAL do not launch processes on the same node as prun
140 • HWTCPUS use hardware threads as CPU slots
142 • CORECPUS use cores as CPU slots (default)
144 • DEVICE=dev device specifier for the dist policy
146 • INHERIT
148 • NOINHERIT means !INHERIT
150 • PE-LIST=a,b comma-delimited ranges of cpus to use for this job pro‐
152 cessed as an unordered pool of CPUs
153 • FILE=%s (path to file containing sequential or rankfile entries).
155 ppr policy example: --map-by ppr:N:<object> will launch N times the
157 number of objects of the specified type on each node.
158 To order processes’ ranks:
160 --rank-by <object>
162 Rank in round-robin fashion according to the specified object.
163 See defaults in Quick Summary. Supported options include slot,
164 hwthread, core, l1cache, l2cache, l3cache, package, and node.
165 Any object can include qualifiers by adding a colon (:) and any combi‐
167 nation of one or more of the following to the --rank-by option:
168 • SPAN
170 • FILL
172 To bind processes to sets of objects:
174 --bind-to <object>
176 Bind processes to the specified object. See defaults in Quick
177 Summary. Supported options include none, hwthread, core,
178 l1cache, l2cache, l3cache, and package.
179 Any object can include qualifiers by adding a colon (:) and any combi‐
181 nation of one or more of the following to the --bind-to option:
182 • overload-allowed allows for binding more than one process in relation
184 to a CPU
185 • if-supported if that object is supported on this system
187 Diagnostics
189 --map-by :DISPLAY
190 Display a table showing the mapped location of each process pri‐
191 or to launch.
192 --map-by :DISPLAYALLOC
194 Display the detected allocation of resources (e.g., nodes,
195 slots)
196 --bind-to :REPORT
198 Report bindings for launched processes to stderr.
199
201 PRTE employs a three-phase procedure for assigning process locations
202 and ranks:
203 1. mapping: Assigns a default location to each process
205 2. ranking: Assigns a unique rank value to each process
207 3. binding: Constrains each process to run on specific processors
209 The first phase of mapping is used to assign a default location to each
211 process based on the mapper being employed. Mapping by slot, node, and
212 sequentially results in the assignment of the processes to the node
213 level. In contrast, mapping by object, allows the mapper to assign the
214 process to an actual object on each node.
215 Note: The location assigned to the process is independent of where it
217 will be bound - the assignment is used solely as input to the binding
218 algorithm.
219 The second phase focuses on the ranking of the process within the job’s
221 namespace. PRTE separates this from the mapping procedure to allow
222 more flexibility in the relative placement of processes.
223 The third phase of binding actually binds each process to a given set
225 of processors. This can improve performance if the operating system is
226 placing processes sub-optimally. For example, it might oversubscribe
227 some multi-core processor sockets, leaving other sockets idle; this can
228 lead processes to contend unnecessarily for common resources. Or, it
229 might spread processes out too widely; this can be suboptimal if appli‐
230 cation performance is sensitive to interprocess communication costs.
231 Binding can also keep the operating system from migrating processes ex‐
232 cessively, regardless of how optimally those processes were placed to
233 begin with.
234 PRTE’s support for process binding depends on the underlying operating
236 system. Therefore, certain process binding options may not be avail‐
237 able on every system.
238 Specifying Host Nodes
240 Host nodes can be identified on the command line with the --host option
241 or in a hostfile.
242 For example, assuming no other resource manager or scheduler is in‐
244 volved,
245 prte --host aa,aa,bb ./a.out
247 launches two processes on node aa and one on bb.
248 prun --host aa ./a.out
250 launches one process on node aa.
251 prun --host aa:5 ./a.out
253 launches five processes on node aa.
254 Or, consider the hostfile
256 $ cat myhostfile
258 aa slots=2
259 bb slots=2
260 cc slots=2
261 Here, we list both the host names (aa, bb, and cc) but also how many
263 “slots” there are for each. Slots indicate how many processes can po‐
264 tentially execute on a node. For best performance, the number of slots
265 may be chosen to be the number of cores on the node or the number of
266 processor sockets.
267 If the hostfile does not provide slots information, the PRTE DVM will
269 attempt to discover the number of cores (or hwthreads, if the :HWTCPUS
270 qualifier to the --map-by option is set) and set the number of slots to
271 that value.
272 Examples using the hostfile above with and without the --host option
274 prun --hostfile myhostfile ./a.out
276 will launch two processes on each of the three nodes.
277 prun --hostfile myhostfile --host aa ./a.out
279 will launch two processes, both on node aa.
280 prun --hostfile myhostfile --host dd ./a.out
282 will find no hosts to run on and abort with an error. That is,
283 the specified host dd is not in the specified hostfile.
284 When running under resource managers (e.g., SLURM, Torque, etc.), PRTE
286 will obtain both the hostnames and the number of slots directly from
287 the resource manger. The behavior of --host in that environment will
288 behave the same as if a hostfile was provided (since it is provided by
289 the resource manager).
290 Specifying Number of Processes
292 As we have just seen, the number of processes to run can be set using
293 the hostfile. Other mechanisms exist.
294 The number of processes launched can be specified as a multiple of the
296 number of nodes or processor sockets available. Consider the hostfile
297 below for the examples that follow.
298 $ cat myhostfile
300 aa
301 bb
302 For example,
304 prun --hostfile myhostfile --map-by ppr:2:package ./a.out
306 launches processes 0-3 on node aa and process 4-7 on node bb,
307 where aa and bb are both dual-package nodes. The --map-by
308 ppr:2:package option also turns on the --bind-to package option,
309 which is discussed in a later section.
310 prun --hostfile myhostfile --map-by ppr:2:node ./a.out
312 launches processes 0-1 on node aa and processes 2-3 on node bb.
313 prun --hostfile myhostfile --map-by ppr:1:node ./a.out
315 launches one process per host node.
316 Another alternative is to specify the number of processes with the --np
318 option. Consider now the hostfile
319 $ cat myhostfile
321 aa slots=4
322 bb slots=4
323 cc slots=4
324 Now,
326 prun --hostfile myhostfile --np 6 ./a.out
328 will launch processes 0-3 on node aa and processes 4-5 on node
329 bb. The remaining slots in the hostfile will not be used since
330 the -np option indicated that only 6 processes should be
331 launched.
332 Mapping Processes to Nodes: Using Policies
334 The examples above illustrate the default mapping of process processes
335 to nodes. This mapping can also be controlled with various
336 prun/prterun options that describe mapping policies.
337 $ cat myhostfile
339 aa slots=4
340 bb slots=4
341 cc slots=4
342 Consider the hostfile above, with --np 6:
344 node aa node bb node cc
346 prun 0 1 2 3 4 5
347 prun --map-by node 0 1 2 3 4 5
348 prun --map-by node:NOLOCAL 0 1 2 3 4 5
349 The --map-by node option will load balance the processes across the
351 available nodes, numbering each process in a round-robin fashion.
352 The :NOLOCAL qualifier to --map-by prevents any processes from being
354 mapped onto the local host (in this case node aa). While prun typical‐
355 ly consumes few system resources, the :NOLOCAL qualifier can be helpful
356 for launching very large jobs where prun may actually need to use no‐
357 ticeable amounts of memory and/or processing time.
358 Just as --np can specify fewer processes than there are slots, it can
360 also oversubscribe the slots. For example, with the same hostfile:
361 prun --hostfile myhostfile --np 14 ./a.out
363 will produce an error since the default :NOOVERSUBSCRIBE quali‐
364 fier to --map-by prevents oversubscription.
365 To oversubscribe the nodes you can use the :OVERSUBSCRIBE qualifier to
367 --map-by:
368 prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out
370 will launch processes 0-5 on node aa, 6-9 on bb, and 10-13 on
371 cc.
372 Limits to oversubscription can also be specified in the hostfile itself
374 with the max_slots field:
375 % cat myhostfile
377 aa slots=4 max_slots=4
378 bb max_slots=8
379 cc slots=4
380 The max_slots field specifies such a limit. When it does, the slots
382 value defaults to the limit. Now:
383 prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out
385 causes the first 12 processes to be launched as before, but the
386 remaining two processes will be forced onto node cc. The other
387 two nodes are protected by the hostfile against oversubscription
388 by this job.
389 Using the :NOOVERSUBSCRIBE qualifier to --map-by option can be helpful
391 since the PRTE DVM currently does not get “max_slots” values from the
392 resource manager.
393 Of course, --np can also be used with the --host option. For example,
395 prun --host aa,bb --np 8 ./a.out
397 will produce an error since the default :NOOVERSUBSCRIBE quali‐
398 fier to --map-by prevents oversubscription.
399 prun --host aa,bb --np 8 --map-by :OVERSUBSCRIBE ./a.out
401 launches 8 processes. Since only two hosts are specified, after
402 the first two processes are mapped, one to aa and one to bb, the
403 remaining processes oversubscribe the specified hosts evenly.
404 prun --host aa:2,bb:6 --np 8 ./a.out
406 launches 8 processes. Processes 0-1 on node aa since it has 2
407 slots and processes 2-7 on node bb since it has 6 slots.
408 And here is a MIMD example:
410 prun --host aa --np 1 hostname : --host bb,cc --np 2 uptime
412 will launch process 0 running hostname on node aa and processes
413 1 and 2 each running uptime on nodes bb and cc, respectively.
414 Mapping, Ranking, and Binding: Fundamentals
416 The mapping of process processes to nodes can be defined not just with
417 general policies but also, if necessary, using arbitrary mappings that
418 cannot be described by a simple policy. One can use the “sequential
419 mapper,” which reads the hostfile line by line, assigning processes to
420 nodes in whatever order the hostfile specifies. Use the --prtemca
421 rmaps seq option.
422 For example, using the hostfile below:
424 % cat myhostfile
426 aa slots=4
427 bb slots=4
428 cc slots=4
429 The command below will launch three processes, one on each of nodes aa,
431 bb, and cc, respectively. The slot counts don’t matter; one process is
432 launched per line on whatever node is listed on the line.
433 % prun --hostfile myhostfile --prtemca rmaps seq ./a.out
435 The ranking phase is best illustrated by considering the following
437 hostfile and test cases we used the --map-by ppr:2:package option:
438 % cat myhostfile
440 aa
441 bb
442 node aa node bb
444 --rank-by core 0 1 ! 2 3 4 5 ! 6 7
445 --rank-by package 0 2 ! 1 3 4 6 ! 5 7
446 --rank-by package:SPAN 0 4 ! 1 5 2 6 ! 3 7
447 Ranking by core and by slot provide the identical result - a simple
449 progression of ranks across each node. Ranking by package does a
450 round-robin ranking within each node until all processes have been as‐
451 signed a rank, and then progresses to the next node. Adding the :SPAN
452 qualifier to the ranking directive causes the ranking algorithm to
453 treat the entire allocation as a single entity - thus, the process
454 ranks are assigned across all sockets before circling back around to
455 the beginning.
456 The binding phase restricts the process to a subset of the CPU re‐
458 sources on the node.
459 The processors to be used for binding can be identified in terms of
461 topological groupings - e.g., binding to an l3cache will bind each
462 process to all processors within the scope of a single L3 cache within
463 their assigned location. Thus, if a process is assigned by the mapper
464 to a certain package, then a --bind-to l3cache directive will cause the
465 process to be bound to the processors that share a single L3 cache
466 within that package.
467 To help balance loads, the binding directive uses a round-robin method
469 when binding to levels lower than used in the mapper. For example,
470 consider the case where a job is mapped to the package level, and then
471 bound to core. Each package will have multiple cores, so if multiple
472 processes are mapped to a given package, the binding algorithm will as‐
473 sign each process located to a package to a unique core in a round-
474 robin manner.
475 Alternatively, processes mapped by l2cache and then bound to package
477 will simply be bound to all the processors in the package where they
478 are located. In this manner, users can exert detailed control over
479 relative process location and binding.
480 Process mapping/ranking/binding can also be set with MCA parameters.
482 Their usage is less convenient than that of the command line options.
483 On the other hand, MCA parameters can be set not only on the prun com‐
484 mand line, but alternatively in a system or user mca-params.conf file
485 or as environment variables, as described in the MCA section below.
486 Some examples include:
487 prun option MCA parameter key value
489 --map-by core rmaps_base_mapping_policy core
490 --map-by package rmaps_base_mapping_policy package
491 --rank-by core rmaps_base_ranking_policy core
492 --bind-to core hwloc_base_binding_policy core
493 --bind-to package hwloc_base_binding_policy package
494 --bind-to none hwloc_base_binding_policy none
495 Difference between overloading and oversubscription
497 This section explores the difference between these two options. Users
498 are often confused by the difference between these two scenarios. As
499 such this section provides a number of scenarios to help illustrate the
500 differences.
501 • --map-by :OVERSUBSCRIBE allow more processes on a node than process‐
503 ing elements
504 • --bind-to <object>:overload-allowed allows for binding more than one
506 process in relation to a CPU
507 The important thing to remember with oversubscribing is that it can be
509 defined separately from the actual number of CPUs on a node. This al‐
510 lows the mapper to place more or fewer processes per node than CPUs.
511 By default, PRTE uses cores to determine slots in the absence of such
512 information provided in the hostfile or by the resource manager (except
513 in the case of the --host as described in the “Specifying Host Nodes”
514 section).
515 The important thing to remember with overloading is that it is defined
517 as binding more processes than CPUs. By default, PRTE uses cores as a
518 means of counting the number of CPUs. However, the user can adjust
519 this. For example when using the :HWTCPUS qualifier to the --map-by
520 option PRTE will use hardware threads as a means of counting the number
521 of CPUs.
522 For the following examples consider a node with: - Two processor pack‐
524 ages, - Ten cores per package, and - Eight hardware threads per core.
525 Consider the node from above with the hostfile below:
527 $ cat myhostfile
529 node01 slots=32
530 node02 slots=32
531 The “slots” tells PRTE that it can place up to 32 processes before
533 oversubscribing the node.
534 If we run the following:
536 prun --np 34 --hostfile myhostfile --map-by core --bind-to core hostname
538 It will return an error at the binding time indicating an overloading
540 scenario.
541 The mapping mechanism assigns 32 processes to node01 matching the
543 “slots” specification in the hostfile. The binding mechanism will bind
544 the first 20 processes to unique cores leaving it with 12 processes
545 that it cannot bind without overloading one of the cores (putting more
546 than one process on the core).
547 Using the overload-allowed qualifier to the --bind-to core option tells
549 PRTE that it may assign more than one process to a core.
550 If we run the following:
552 prun --np 34 --hostfile myhostfile --map-by core --bind-to core:overload-allowed hostname
554 This will run correctly placing 32 processes on node01, and 2 processes
556 on node02. On node01 two processes are bound to cores 0-11 accounting
557 for the overloading of those cores.
558 Alternatively, we could use hardware threads to give binding a lower
560 level CPU to bind to without overloading.
561 If we run the following:
563 prun --np 34 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname
565 This will run correctly placing 32 processes on node01, and 2 processes
567 on node02. On node01 two processes are mapped to cores 0-11 but bound
568 to different hardware threads on those cores (the logical first and
569 second hardware thread) thus no hardware threads are overloaded at
570 binding time.
571 In both of the examples above the node is not oversubscribed at mapping
573 time because the hostfile set the oversubscription limit to “slots=32”
574 for each node. It is only after we exceed that limit that PRTE will
575 throw an oversubscription error.
576 Consider next if we ran the following:
578 prun --np 66 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname
580 This will return an error at mapping time indicating an oversubscrip‐
582 tion scenario. The mapping mechanism will assign all of the available
583 slots (64 across 2 nodes) and be left two processes to map. The only
584 way to map those processes is to exceed the number of available slots
585 putting the job into an oversubscription scenario.
586 You can force PRTE to oversubscribe the nodes by using the :OVERSUB‐
588 SCRIBE qualifier to the --map-by option as seen in the example below:
589 prun --np 66 --hostfile myhostfile --map-by core:HWTCPUS:OVERSUBSCRIBE --bind-to hwthread hostname
591 This will run correctly placing 34 processes on node01 and 32 on
593 node02. Each process is bound to a unique hardware thread.
594 Overloading vs Oversubscription: Package Example
596 Let’s extend these examples by considering the package level. Consider
597 the same node as before, but with the hostfile below:
598 $ cat myhostfile
600 node01 slots=22
601 node02 slots=22
602 The lowest level CPUs are `cores' and we have 20 total (10 per pack‐
604 age).
605 If we run:
607 prun --np 20 --hostfile myhostfile --map-by package --bind-to package:REPORT hostname
609 Then 10 processes are mapped to each package, and bound at the package
611 level. This is not overloading since we have 10 CPUs (cores) available
612 in the package at the hardware level.
613 However, if we run:
615 prun --np 21 --hostfile myhostfile --map-by package --bind-to package:REPORT hostname
617 Then 11 processes are mapped to the first package and 10 to the second
619 package. At binding time we have an overloading scenario because there
620 are only 10 CPUs (cores) available in the package at the hardware lev‐
621 el. So the first package is overloaded.
622 Overloading vs Oversubscription: Hardware Threads Example
624 Similarly, if we consider hardware threads.
625 Consider the same node as before, but with the hostfile below:
627 $ cat myhostfile
629 node01 slots=165
630 node02 slots=165
631 The lowest level CPUs are `hwthreads' (because we are going to use the
633 :HWTCPUS qualifier) and we have 160 total (80 per package).
634 If we re-run (from the package example) and add the :HWTCPUS qualifier:
636 prun --np 21 --hostfile myhostfile --map-by package:HWTCPUS --bind-to package:REPORT hostname
638 Without the :HWTCPUS qualifier this would be overloading (as we saw
640 previously). The mapper places 11 processes on the first package and
641 10 to the second package. The processes are still bound to the package
642 level. However, with the :HWTCPUS qualifier, it is not overloading
643 since we have 80 CPUs (hwthreads) available in the package at the hard‐
644 ware level.
645 Alternatively, if we run:
647 prun --np 161 --hostfile myhostfile --map-by package:HWTCPUS --bind-to package:REPORT hostname
649 Then 81 processes are mapped to the first package and 80 to the second
651 package. At binding time we have an overloading scenario because there
652 are only 80 CPUs (hwthreads) available in the package at the hardware
653 level. So the first package is overloaded.
654 Diagnostics
656 PRTE provides various diagnostic reports that aid the user in verifying
657 and tuning the mapping/ranking/binding for a specific job.
658 The :REPORT qualifier to --bind-to command line option can be used to
660 report process bindings.
661 As an example, consider a node with: - Two processor packages, - Four
663 cores per package, and - Eight hardware threads per core.
664 In each of the examples below the binding is reported in a human read‐
666 able format.
667 $ prun --np 4 --map-by core --bind-to core:REPORT ./a.out
669 [node01:103137] MCW rank 0 bound to package[0][core:0]
670 [node01:103137] MCW rank 1 bound to package[0][core:1]
671 [node01:103137] MCW rank 2 bound to package[0][core:2]
672 [node01:103137] MCW rank 3 bound to package[0][core:3]
673 The example above processes bind to successive cores on the first pack‐
675 age.
676 $ prun --np 4 --map-by package --bind-to package:REPORT ./a.out
678 [node01:103115] MCW rank 0 bound to package[0][core:0-9]
679 [node01:103115] MCW rank 1 bound to package[1][core:10-19]
680 [node01:103115] MCW rank 2 bound to package[0][core:0-9]
681 [node01:103115] MCW rank 3 bound to package[1][core:10-19]
682 The example above processes bind to all cores on successive packages.
684 The processes cycle though the packages in a round-robin fashion as
685 many times as are needed.
686 $ prun --np 4 --map-by package:PE=2 --bind-to core:REPORT ./a.out
688 [node01:103328] MCW rank 0 bound to package[0][core:0-1]
689 [node01:103328] MCW rank 1 bound to package[1][core:10-11]
690 [node01:103328] MCW rank 2 bound to package[0][core:2-3]
691 [node01:103328] MCW rank 3 bound to package[1][core:12-13]
692 The example above shows us that 2 cores have been bound per process.
694 The :PE=2 qualifier states that 2 processing elements underneath the
695 package (which would be cores in this case) are mapped to each process.
696 The processes cycle though the packages in a round-robin fashion as
697 many times as are needed.
698 $ prun --np 4 --map-by core:PE=2:HWTCPUS --bind-to :REPORT hostname
700 [node01:103506] MCW rank 0 bound to package[0][hwt:0-1]
701 [node01:103506] MCW rank 1 bound to package[0][hwt:8-9]
702 [node01:103506] MCW rank 2 bound to package[0][hwt:16-17]
703 [node01:103506] MCW rank 3 bound to package[0][hwt:24-25]
704 The example above shows us that 2 hardware threads have been bound per
706 process. In this case prun is mapping by hardware threads since we
707 used the :HWTCPUS qualifier. Without that qualifier this command would
708 return an error since by default prun will not map to resources smaller
709 than a core. The :PE=2 qualifier states that 2 processing elements un‐
710 derneath the core (which would be hardware threads in this case) are
711 mapped to each process. The processes cycle though the cores in a
712 round-robin fashion as many times as are needed.
713 $ prun --np 4 --bind-to none:REPORT hostname
715 [node01:107126] MCW rank 0 is not bound (or bound to all available processors)
716 [node01:107126] MCW rank 1 is not bound (or bound to all available processors)
717 [node01:107126] MCW rank 2 is not bound (or bound to all available processors)
718 [node01:107126] MCW rank 3 is not bound (or bound to all available processors)
719 The example above binding is turned off.
721 Rankfiles
723 Another way to specify arbitrary mappings is with a rankfile, which
724 gives you detailed control over process binding as well.
725 Rankfiles are text files that specify detailed information about how
727 individual processes should be mapped to nodes, and to which proces‐
728 sor(s) they should be bound. Each line of a rankfile specifies the lo‐
729 cation of one process. The general form of each line in the rankfile
730 is:
731 rank <N>=<hostname> slot=<slot list>
733 For example:
735 $ cat myrankfile
737 rank 0=c712f6n01 slot=10-12
738 rank 1=c712f6n02 slot=0,1,4
739 rank 2=c712f6n03 slot=1-2
740 $ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
741 Means that
743 Rank 0 runs on node aa, bound to logical cores 10-12.
745 Rank 1 runs on node bb, bound to logical cores 0, 1, and 4.
746 Rank 2 runs on node cc, bound to logical cores 1 and 2.
747 For example:
749 $ cat myrankfile
751 rank 0=aa slot=1:0-2
752 rank 1=bb slot=0:0,1,4
753 rank 2=cc slot=1-2
754 $ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
755 Means that
757 Rank 0 runs on node aa, bound to logical package 1, cores 10-12 (the 0th through 2nd cores on that package).
759 Rank 1 runs on node bb, bound to logical package 0, cores 0, 1, and 4.
760 Rank 2 runs on node cc, bound to logical cores 1 and 2.
761 The hostnames listed above are “absolute,” meaning that actual resolv‐
763 able hostnames are specified. However, hostnames can also be specified
764 as “relative,” meaning that they are specified in relation to an exter‐
765 nally-specified list of hostnames (e.g., by prun’s --host argument, a
766 hostfile, or a job scheduler).
767 The “relative” specification is of the form “+n<X>”, where X is an in‐
769 teger specifying the Xth hostname in the set of all available host‐
770 names, indexed from 0. For example:
771 $ cat myrankfile
773 rank 0=+n0 slot=10-12
774 rank 1=+n1 slot=0,1,4
775 rank 2=+n2 slot=1-2
776 $ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
777 All package/core slot locations are be specified as logical indexes.
779 You can use tools such as HWLOC’s “lstopo” to find the logical indexes
780 of packages and cores.
781 Deprecated Options
783 These deprecated options will be removed in a future release.
784 --bind-to-core
786 (Deprecated: Use --bind-to core) Bind processes to cores
787 -bind-to-socket, --bind-to-socket
789 (Deprecated: Use --bind-to package) Bind processes to processor
790 sockets
791 --bycore
793 (Deprecated: Use --map-by core) Map processes by core
794 -bynode, --bynode
796 (Deprecated: Use --map-by node) Launch processes one per node,
797 cycling by node in a round-robin fashion. This spreads process‐
798 es evenly among nodes and assigns ranks in a round-robin, “by
799 node” manner.
800 --byslot
802 (Deprecated: Use --map-by slot) Map and rank processes round-
803 robin by slot.
804 --cpus-per-proc <#perproc>
806 (Deprecated: Use --map-by <obj>:PE=<#perproc>) Bind each process
807 to the specified number of cpus.
808 --cpus-per-rank <#perrank>
810 (Deprecated: Use --map-by <obj>:PE=<#perrank>) Alias for --cpus-
811 per-proc.
812 --display-allocation
814 (Deprecated: Use --map-by :DISPLAYALLOC) Display the detected
815 resource allocation.
816 --display-devel-map
818 (Deprecated: Use --map-by :DISPLAYDEVEL) Display a detailed
819 process map (mostly intended for developers) just before launch.
820 --display-map
822 (Deprecated: Use --map-by :DISPLAY) Display a table showing the
823 mapped location of each process prior to launch.
824 --display-topo
826 (Deprecated: Use --map-by :DISPLAYTOPO) Display the topology as
827 part of the process map (mostly intended for developers) just
828 before launch.
829 --do-not-launch
831 (Deprecated: Use --map-by :DONOTLAUNCH) Perform all necessary
832 operations to prepare to launch the application, but do not ac‐
833 tually launch it (usually used to test mapping patterns).
834 --do-not-resolve
836 (Deprecated: Use --map-by :DONOTRESOLVE) Do not attempt to re‐
837 solve interfaces - usually used to determine proposed process
838 placement/binding prior to obtaining an allocation.
839 -N <num>
841 (Deprecated: Use --map-by prr:<num>:node) Launch num processes
842 per node on all allocated nodes.
843 --nolocal
845 (Deprecated: Use --map-by :NOLOCAL) Do not run any copies of the
846 launched application on the same node as prun is running. This
847 option will override listing the localhost with --host or any
848 other host-specifying mechanism.
849 --nooversubscribe
851 (Deprecated: Use --map-by :NOOVERSUBSCRIBE) Do not oversubscribe
852 any nodes; error (without starting any processes) if the re‐
853 quested number of processes would cause oversubscription. This
854 option implicitly sets “max_slots” equal to the “slots” value
855 for each node. (Enabled by default).
856 --npernode <#pernode>
858 (Deprecated: Use --map-by ppr:<#pernode>:node) On each node,
859 launch this many processes.
860 --npersocket <#persocket>
862 (Deprecated: Use --map-by ppr:<#perpackage>:package) On each
863 node, launch this many processes times the number of processor
864 sockets on the node. The --npersocket option also turns on the
865 --bind-to socket option. The term socket has been globally re‐
866 placed with package.
867 --oversubscribe
869 (Deprecated: Use --map-by :OVERSUBSCRIBE) Nodes are allowed to
870 be oversubscribed, even on a managed system, and overloading of
871 processing elements.
872 --pernode
874 (Deprecated: Use --map-by ppr:1:node) On each node, launch one
875 process.
876 --ppr (Deprecated: Use --map-by ppr:<list>) Comma-separated list of
878 number of processes on a given resource type [default: none].
879 --rankfile <FILENAME>
881 (Deprecated: Use --map-by rankfile:FILE=<FILENAME>) Use a rank‐
882 file for mapping/ranking/binding
883 --report-bindings
885 (Deprecated: Use --bind-to :REPORT) Report any bindings for
886 launched processes.
887 --tag-output
889 (Deprecated: Use --map-by :TAGOUTPUT) Tag all output with
890 [job,rank]
891 --timestamp-output
893 (Deprecated: Use --map-by :TIMESTAMPOUTPUT) Timestamp all appli‐
894 cation process output
895 --use-hwthread-cpus
897 (Deprecated: Use --map-by :HWTCPUS) Use hardware threads as in‐
898 dependent cpus.
899 --xml (Deprecated: Use --map-by :XMLOUTPUT) Provide all output in XML
901 format
9022021-06-29 prte-map(1)