1CPUSET(7) Linux Programmer's Manual CPUSET(7)
2
6 cpuset - confine processes to processor and memory node subsets
7
9 The cpuset file system is a pseudo-file-system interface to the kernel
10 cpuset mechanism, which is used to control the processor placement and
11 memory placement of processes. It is commonly mounted at /dev/cpuset.
12 On systems with kernels compiled with built in support for cpusets, all
14 processes are attached to a cpuset, and cpusets are always present. If
15 a system supports cpusets, then it will have the entry nodev cpuset in
16 the file /proc/filesystems. By mounting the cpuset file system (see
17 the EXAMPLE section below), the administrator can configure the cpusets
18 on a system to control the processor and memory placement of processes
19 on that system. By default, if the cpuset configuration on a system is
20 not modified or if the cpuset file system is not even mounted, then the
21 cpuset mechanism, though present, has no affect on the system's behav‐
22 ior.
23 A cpuset defines a list of CPUs and memory nodes.
25 The CPUs of a system include all the logical processing units on which
27 a process can execute, including, if present, multiple processor cores
28 within a package and Hyper-Threads within a processor core. Memory
29 nodes include all distinct banks of main memory; small and SMP systems
30 typically have just one memory node that contains all the system's main
31 memory, while NUMA (non-uniform memory access) systems have multiple
32 memory nodes.
33 Cpusets are represented as directories in a hierarchical pseudo-file
35 system, where the top directory in the hierarchy (/dev/cpuset) repre‐
36 sents the entire system (all online CPUs and memory nodes) and any
37 cpuset that is the child (descendant) of another parent cpuset contains
38 a subset of that parent's CPUs and memory nodes. The directories and
39 files representing cpusets have normal file-system permissions.
40 Every process in the system belongs to exactly one cpuset. A process
42 is confined to only run on the CPUs in the cpuset it belongs to, and to
43 allocate memory only on the memory nodes in that cpuset. When a
44 process fork(2)s, the child process is placed in the same cpuset as its
45 parent. With sufficient privilege, a process may be moved from one
46 cpuset to another and the allowed CPUs and memory nodes of an existing
47 cpuset may be changed.
48 When the system begins booting, a single cpuset is defined that
50 includes all CPUs and memory nodes on the system, and all processes are
51 in that cpuset. During the boot process, or later during normal system
52 operation, other cpusets may be created, as subdirectories of this top
53 cpuset, under the control of the system administrator, and processes
54 may be placed in these other cpusets.
55 Cpusets are integrated with the sched_setaffinity(2) scheduling affin‐
57 ity mechanism and the mbind(2) and set_mempolicy(2) memory-placement
58 mechanisms in the kernel. Neither of these mechanisms let a process
59 make use of a CPU or memory node that is not allowed by that process's
60 cpuset. If changes to a process's cpuset placement conflict with these
61 other mechanisms, then cpuset placement is enforced even if it means
62 overriding these other mechanisms. The kernel accomplishes this over‐
63 riding by silently restricting the CPUs and memory nodes requested by
64 these other mechanisms to those allowed by the invoking process's
65 cpuset. This can result in these other calls returning an error, if
66 for example, such a call ends up requesting an empty set of CPUs or
67 memory nodes, after that request is restricted to the invoking
68 process's cpuset.
69 Typically, a cpuset is used to manage the CPU and memory-node confine‐
71 ment for a set of cooperating processes such as a batch scheduler job,
72 and these other mechanisms are used to manage the placement of individ‐
73 ual processes or memory regions within that set or job.
74
76 Each directory below /dev/cpuset represents a cpuset and contains a
77 fixed set of pseudo-files describing the state of that cpuset.
78 New cpusets are created using the mkdir(2) system call or the mkdir(1)
80 command. The properties of a cpuset, such as its flags, allowed CPUs
81 and memory nodes, and attached processes, are queried and modified by
82 reading or writing to the appropriate file in that cpuset's directory,
83 as listed below.
84 The pseudo-files in each cpuset directory are automatically created
86 when the cpuset is created, as a result of the mkdir(2) invocation. It
87 is not possible to directly add or remove these pseudo-files.
88 A cpuset directory that contains no child cpuset directories, and has
90 no attached processes, can be removed using rmdir(2) or rmdir(1). It
91 is not necessary, or possible, to remove the pseudo-files inside the
92 directory before removing it.
93 The pseudo-files in each cpuset directory are small text files that may
95 be read and written using traditional shell utilities such as cat(1),
96 and echo(1), or from a program by using file I/O library functions or
97 system calls, such as open(2), read(2), write(2), and close(2).
98 The pseudo-files in a cpuset directory represent internal kernel state
100 and do not have any persistent image on disk. Each of these per-cpuset
101 files is listed and described below.
102 tasks List of the process IDs (PIDs) of the processes in that cpuset.
104 The list is formatted as a series of ASCII decimal numbers, each
105 followed by a newline. A process may be added to a cpuset
106 (automatically removing it from the cpuset that previously con‐
107 tained it) by writing its PID to that cpuset's tasks file (with
108 or without a trailing newline.)
109 Warning: only one PID may be written to the tasks file at a
111 time. If a string is written that contains more than one PID,
112 only the first one will be used.
113 notify_on_release
115 Flag (0 or 1). If set (1), that cpuset will receive special
116 handling after it is released, that is, after all processes
117 cease using it (i.e., terminate or are moved to a different
118 cpuset) and all child cpuset directories have been removed. See
119 the Notify On Release section, below.
120 cpus List of the physical numbers of the CPUs on which processes in
122 that cpuset are allowed to execute. See List Format below for a
123 description of the format of cpus.
124 The CPUs allowed to a cpuset may be changed by writing a new
126 list to its cpus file.
127 cpu_exclusive
129 Flag (0 or 1). If set (1), the cpuset has exclusive use of its
130 CPUs (no sibling or cousin cpuset may overlap CPUs). By default
131 this is off (0). Newly created cpusets also initially default
132 this to off (0).
133 Two cpusets are sibling cpusets if they share the same parent
135 cpuset in the /dev/cpuset hierarchy. Two cpusets are cousin
136 cpusets if neither is the ancestor of the other. Regardless of
137 the cpu_exclusive setting, if one cpuset is the ancestor of
138 another, and if both of these cpusets have non-empty cpus, then
139 their cpus must overlap, because the cpus of any cpuset are
140 always a subset of the cpus of its parent cpuset.
141 mems List of memory nodes on which processes in this cpuset are
143 allowed to allocate memory. See List Format below for a
144 description of the format of mems.
145 mem_exclusive
147 Flag (0 or 1). If set (1), the cpuset has exclusive use of its
148 memory nodes (no sibling or cousin may overlap). Also if set
149 (1), the cpuset is a Hardwall cpuset (see below.) By default
150 this is off (0). Newly created cpusets also initially default
151 this to off (0).
152 Regardless of the mem_exclusive setting, if one cpuset is the
154 ancestor of another, then their memory nodes must overlap,
155 because the memory nodes of any cpuset are always a subset of
156 that cpuset's parent cpuset.
157 mem_hardwall (since Linux 2.6.26)
159 Flag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see
160 below.) Unlike mem_exclusive, there is no constraint on whether
161 cpusets marked mem_hardwall may have overlapping memory nodes
162 with sibling or cousin cpusets. By default this is off (0).
163 Newly created cpusets also initially default this to off (0).
164 memory_migrate (since Linux 2.6.16)
166 Flag (0 or 1). If set (1), then memory migration is enabled.
167 By default this is off (0). See the Memory Migration section,
168 below.
169 memory_pressure (since Linux 2.6.16)
171 A measure of how much memory pressure the processes in this
172 cpuset are causing. See the Memory Pressure section, below.
173 Unless memory_pressure_enabled is enabled, always has value zero
174 (0). This file is read-only. See the WARNINGS section, below.
175 memory_pressure_enabled (since Linux 2.6.16)
177 Flag (0 or 1). This file is only present in the root cpuset,
178 normally /dev/cpuset. If set (1), the memory_pressure calcula‐
179 tions are enabled for all cpusets in the system. By default
180 this is off (0). See the Memory Pressure section, below.
181 memory_spread_page (since Linux 2.6.17)
183 Flag (0 or 1). If set (1), pages in the kernel page cache
184 (file-system buffers) are uniformly spread across the cpuset.
185 By default this is off (0) in the top cpuset, and inherited from
186 the parent cpuset in newly created cpusets. See the Memory
187 Spread section, below.
188 memory_spread_slab (since Linux 2.6.17)
190 Flag (0 or 1). If set (1), the kernel slab caches for file I/O
191 (directory and inode structures) are uniformly spread across the
192 cpuset. By default this is off (0) in the top cpuset, and
193 inherited from the parent cpuset in newly created cpusets. See
194 the Memory Spread section, below.
195 sched_load_balance (since Linux 2.6.24)
197 Flag (0 or 1). If set (1, the default) the kernel will automat‐
198 ically load balance processes in that cpuset over the allowed
199 CPUs in that cpuset. If cleared (0) the kernel will avoid load
200 balancing processes in this cpuset, unless some other cpuset
201 with overlapping CPUs has its sched_load_balance flag set. See
202 Scheduler Load Balancing, below, for further details.
203 sched_relax_domain_level (since Linux 2.6.26)
205 Integer, between -1 and a small positive value. The
206 sched_relax_domain_level controls the width of the range of CPUs
207 over which the kernel scheduler performs immediate rebalancing
208 of runnable tasks across CPUs. If sched_load_balance is dis‐
209 abled, then the setting of sched_relax_domain_level does not
210 matter, as no such load balancing is done. If sched_load_bal‐
211 ance is enabled, then the higher the value of the
212 sched_relax_domain_level, the wider the range of CPUs over which
213 immediate load balancing is attempted. See Scheduler Relax
214 Domain Level, below, for further details.
215 In addition to the above pseudo-files in each directory below
217 /dev/cpuset, each process has a pseudo-file, /proc/<pid>/cpuset, that
218 displays the path of the process's cpuset directory relative to the
219 root of the cpuset file system.
220 Also the /proc/<pid>/status file for each process has four added lines,
222 displaying the process's Cpus_allowed (on which CPUs it may be sched‐
223 uled) and Mems_allowed (on which memory nodes it may obtain memory), in
224 the two formats Mask Format and List Format (see below) as shown in the
225 following example:
226 Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
228 Cpus_allowed_list: 0-127
229 Mems_allowed: ffffffff,ffffffff
230 Mems_allowed_list: 0-63
231 The "allowed" fields were added in Linux 2.6.24; the "allowed_list"
233 fields were added in Linux 2.6.26.
234
236 In addition to controlling which cpus and mems a process is allowed to
237 use, cpusets provide the following extended capabilities.
238 Exclusive Cpusets
240 If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset,
241 other than a direct ancestor or descendant, may share any of the same
242 CPUs or memory nodes.
243 A cpuset that is mem_exclusive restricts kernel allocations for buffer
245 cache pages and other internal kernel data pages commonly shared by the
246 kernel across multiple users. All cpusets, whether mem_exclusive or
247 not, restrict allocations of memory for user space. This enables con‐
248 figuring a system so that several independent jobs can share common
249 kernel data, while isolating each job's user allocation in its own
250 cpuset. To do this, construct a large mem_exclusive cpuset to hold all
251 the jobs, and construct child, non-mem_exclusive cpusets for each indi‐
252 vidual job. Only a small amount of kernel memory, such as requests
253 from interrupt handlers, is allowed to be placed on memory nodes out‐
254 side even a mem_exclusive cpuset.
255 Hardwall
257 A cpuset that has mem_exclusive or mem_hardwall set is a hardwall
258 cpuset. A hardwall cpuset restricts kernel allocations for page, buf‐
259 fer, and other data commonly shared by the kernel across multiple
260 users. All cpusets, whether hardwall or not, restrict allocations of
261 memory for user space.
262 This enables configuring a system so that several independent jobs can
264 share common kernel data, such as file system pages, while isolating
265 each job's user allocation in its own cpuset. To do this, construct a
266 large hardwall cpuset to hold all the jobs, and construct child cpusets
267 for each individual job which are not hardwall cpusets.
268 Only a small amount of kernel memory, such as requests from interrupt
270 handlers, is allowed to be taken outside even a hardwall cpuset.
271 Notify On Release
273 If the notify_on_release flag is enabled (1) in a cpuset, then whenever
274 the last process in the cpuset leaves (exits or attaches to some other
275 cpuset) and the last child cpuset of that cpuset is removed, the kernel
276 will run the command /sbin/cpuset_release_agent, supplying the pathname
277 (relative to the mount point of the cpuset file system) of the aban‐
278 doned cpuset. This enables automatic removal of abandoned cpusets.
279 The default value of notify_on_release in the root cpuset at system
281 boot is disabled (0). The default value of other cpusets at creation
282 is the current value of their parent's notify_on_release setting.
283 The command /sbin/cpuset_release_agent is invoked, with the name
285 (/dev/cpuset relative path) of the to-be-released cpuset in argv[1].
286 The usual contents of the command /sbin/cpuset_release_agent is simply
288 the shell script:
289 #!/bin/sh
291 rmdir /dev/cpuset/$1
292 As with other flag values below, this flag can be changed by writing an
294 ASCII number 0 or 1 (with optional trailing newline) into the file, to
295 clear or set the flag, respectively.
296 Memory Pressure
298 The memory_pressure of a cpuset provides a simple per-cpuset running
299 average of the rate that the processes in a cpuset are attempting to
300 free up in-use memory on the nodes of the cpuset to satisfy additional
301 memory requests.
302 This enables batch managers that are monitoring jobs running in dedi‐
304 cated cpusets to efficiently detect what level of memory pressure that
305 job is causing.
306 This is useful both on tightly managed systems running a wide mix of
308 submitted jobs, which may choose to terminate or re-prioritize jobs
309 that are trying to use more memory than allowed on the nodes assigned
310 them, and with tightly coupled, long-running, massively parallel scien‐
311 tific computing jobs that will dramatically fail to meet required per‐
312 formance goals if they start to use more memory than allowed to them.
313 This mechanism provides a very economical way for the batch manager to
315 monitor a cpuset for signs of memory pressure. It's up to the batch
316 manager or other user code to decide what action to take if it detects
317 signs of memory pressure.
318 Unless memory pressure calculation is enabled by setting the pseudo-
320 file /dev/cpuset/memory_pressure_enabled, it is not computed for any
321 cpuset, and reads from any memory_pressure always return zero, as rep‐
322 resented by the ASCII string "0\n". See the WARNINGS section, below.
323 A per-cpuset, running average is employed for the following reasons:
325 * Because this meter is per-cpuset rather than per-process or per vir‐
327 tual memory region, the system load imposed by a batch scheduler
328 monitoring this metric is sharply reduced on large systems, because
329 a scan of the tasklist can be avoided on each set of queries.
330 * Because this meter is a running average rather than an accumulating
332 counter, a batch scheduler can detect memory pressure with a single
333 read, instead of having to read and accumulate results for a period
334 of time.
335 * Because this meter is per-cpuset rather than per-process, the batch
337 scheduler can obtain the key information — memory pressure in a
338 cpuset — with a single read, rather than having to query and accumu‐
339 late results over all the (dynamically changing) set of processes in
340 the cpuset.
341 The memory_pressure of a cpuset is calculated using a per-cpuset simple
343 digital filter that is kept within the kernel. For each cpuset, this
344 filter tracks the recent rate at which processes attached to that
345 cpuset enter the kernel direct reclaim code.
346 The kernel direct reclaim code is entered whenever a process has to
348 satisfy a memory page request by first finding some other page to
349 repurpose, due to lack of any readily available already free pages.
350 Dirty file system pages are repurposed by first writing them to disk.
351 Unmodified file system buffer pages are repurposed by simply dropping
352 them, though if that page is needed again, it will have to be re-read
353 from disk.
354 The memory_pressure file provides an integer number representing the
356 recent (half-life of 10 seconds) rate of entries to the direct reclaim
357 code caused by any process in the cpuset, in units of reclaims
358 attempted per second, times 1000.
359 Memory Spread
361 There are two Boolean flag files per cpuset that control where the ker‐
362 nel allocates pages for the file-system buffers and related in-kernel
363 data structures. They are called memory_spread_page and mem‐
364 ory_spread_slab.
365 If the per-cpuset Boolean flag file memory_spread_page is set, then the
367 kernel will spread the file-system buffers (page cache) evenly over all
368 the nodes that the faulting process is allowed to use, instead of pre‐
369 ferring to put those pages on the node where the process is running.
370 If the per-cpuset Boolean flag file memory_spread_slab is set, then the
372 kernel will spread some file-system-related slab caches, such as those
373 for inodes and directory entries, evenly over all the nodes that the
374 faulting process is allowed to use, instead of preferring to put those
375 pages on the node where the process is running.
376 The setting of these flags does not affect the data segment (see
378 brk(2)) or stack segment pages of a process.
379 By default, both kinds of memory spreading are off and the kernel
381 prefers to allocate memory pages on the node local to where the
382 requesting process is running. If that node is not allowed by the
383 process's NUMA memory policy or cpuset configuration or if there are
384 insufficient free memory pages on that node, then the kernel looks for
385 the nearest node that is allowed and has sufficient free memory.
386 When new cpusets are created, they inherit the memory spread settings
388 of their parent.
389 Setting memory spreading causes allocations for the affected page or
391 slab caches to ignore the process's NUMA memory policy and be spread
392 instead. However, the affect of these changes in memory placement
393 caused by cpuset-specified memory spreading is hidden from the mbind(2)
394 or set_mempolicy(2) calls. These two NUMA memory policy calls always
395 appear to behave as if no cpuset-specified memory spreading is in
396 affect, even if it is. If cpuset memory spreading is subsequently
397 turned off, the NUMA memory policy most recently specified by these
398 calls is automatically re-applied.
399 Both memory_spread_page and memory_spread_slab are Boolean flag files.
401 By default they contain "0", meaning that the feature is off for that
402 cpuset. If a "1" is written to that file, that turns the named feature
403 on.
404 Cpuset-specified memory spreading behaves similarly to what is known
406 (in other contexts) as round-robin or interleave memory placement.
407 Cpuset-specified memory spreading can provide substantial performance
409 improvements for jobs that:
410 a) need to place thread-local data on memory nodes close to the CPUs
412 which are running the threads that most frequently access that data;
413 but also
414 b) need to access large file-system data sets that must to be spread
416 across the several nodes in the job's cpuset in order to fit.
417 Without this policy, the memory allocation across the nodes in the
419 job's cpuset can become very uneven, especially for jobs that might
420 have just a single thread initializing or reading in the data set.
421 Memory Migration
423 Normally, under the default setting (disabled) of memory_migrate, once
424 a page is allocated (given a physical page of main memory) then that
425 page stays on whatever node it was allocated, so long as it remains
426 allocated, even if the cpuset's memory-placement policy mems subse‐
427 quently changes.
428 When memory migration is enabled in a cpuset, if the mems setting of
430 the cpuset is changed, then any memory page in use by any process in
431 the cpuset that is on a memory node that is no longer allowed will be
432 migrated to a memory node that is allowed.
433 Furthermore, if a process is moved into a cpuset with memory_migrate
435 enabled, any memory pages it uses that were on memory nodes allowed in
436 its previous cpuset, but which are not allowed in its new cpuset, will
437 be migrated to a memory node allowed in the new cpuset.
438 The relative placement of a migrated page within the cpuset is pre‐
440 served during these migration operations if possible. For example, if
441 the page was on the second valid node of the prior cpuset, then the
442 page will be placed on the second valid node of the new cpuset, if pos‐
443 sible.
444 Scheduler Load Balancing
446 The kernel scheduler automatically load balances processes. If one CPU
447 is underutilized, the kernel will look for processes on other more
448 overloaded CPUs and move those processes to the underutilized CPU,
449 within the constraints of such placement mechanisms as cpusets and
450 sched_setaffinity(2).
451 The algorithmic cost of load balancing and its impact on key shared
453 kernel data structures such as the process list increases more than
454 linearly with the number of CPUs being balanced. For example, it costs
455 more to load balance across one large set of CPUs than it does to bal‐
456 ance across two smaller sets of CPUs, each of half the size of the
457 larger set. (The precise relationship between the number of CPUs being
458 balanced and the cost of load balancing depends on implementation
459 details of the kernel process scheduler, which is subject to change
460 over time, as improved kernel scheduler algorithms are implemented.)
461 The per-cpuset flag sched_load_balance provides a mechanism to suppress
463 this automatic scheduler load balancing in cases where it is not needed
464 and suppressing it would have worthwhile performance benefits.
465 By default, load balancing is done across all CPUs, except those marked
467 isolated using the kernel boot time "isolcpus=" argument. (See Sched‐
468 uler Relax Domain Level, below, to change this default.)
469 This default load balancing across all CPUs is not well suited to the
471 following two situations:
472 * On large systems, load balancing across many CPUs is expensive. If
474 the system is managed using cpusets to place independent jobs on
475 separate sets of CPUs, full load balancing is unnecessary.
476 * Systems supporting real-time on some CPUs need to minimize system
478 overhead on those CPUs, including avoiding process load balancing if
479 that is not needed.
480 When the per-cpuset flag sched_load_balance is enabled (the default
482 setting), it requests load balancing across all the CPUs in that
483 cpuset's allowed CPUs, ensuring that load balancing can move a process
484 (not otherwise pinned, as by sched_setaffinity(2)) from any CPU in that
485 cpuset to any other.
486 When the per-cpuset flag sched_load_balance is disabled, then the
488 scheduler will avoid load balancing across the CPUs in that cpuset,
489 except in so far as is necessary because some overlapping cpuset has
490 sched_load_balance enabled.
491 So, for example, if the top cpuset has the flag sched_load_balance
493 enabled, then the scheduler will load balance across all CPUs, and the
494 setting of the sched_load_balance flag in other cpusets has no affect,
495 as we're already fully load balancing.
496 Therefore in the above two situations, the flag sched_load_balance
498 should be disabled in the top cpuset, and only some of the smaller,
499 child cpusets would have this flag enabled.
500 When doing this, you don't usually want to leave any unpinned processes
502 in the top cpuset that might use nontrivial amounts of CPU, as such
503 processes may be artificially constrained to some subset of CPUs,
504 depending on the particulars of this flag setting in descendant
505 cpusets. Even if such a process could use spare CPU cycles in some
506 other CPUs, the kernel scheduler might not consider the possibility of
507 load balancing that process to the underused CPU.
508 Of course, processes pinned to a particular CPU can be left in a cpuset
510 that disables sched_load_balance as those processes aren't going any‐
511 where else anyway.
512 Scheduler Relax Domain Level
514 The kernel scheduler performs immediate load balancing whenever a CPU
515 becomes free or another task becomes runnable. This load balancing
516 works to ensure that as many CPUs as possible are usefully employed
517 running tasks. The kernel also performs periodic load balancing off
518 the software clock described in time(7). The setting of
519 sched_relax_domain_level only applies to immediate load balancing.
520 Regardless of the sched_relax_domain_level setting, periodic load bal‐
521 ancing is attempted over all CPUs (unless disabled by turning off
522 sched_load_balance.) In any case, of course, tasks will only be sched‐
523 uled to run on CPUs allowed by their cpuset, as modified by
524 sched_setaffinity(2) system calls.
525 On small systems, such as those with just a few CPUs, immediate load
527 balancing is useful to improve system interactivity and to minimize
528 wasteful idle CPU cycles. But on large systems, attempting immediate
529 load balancing across a large number of CPUs can be more costly than it
530 is worth, depending on the particular performance characteristics of
531 the job mix and the hardware.
532 The exact meaning of the small integer values of
534 sched_relax_domain_level will depend on internal implementation details
535 of the kernel scheduler code and on the non-uniform architecture of the
536 hardware. Both of these will evolve over time and vary by system
537 architecture and kernel version.
538 As of this writing, when this capability was introduced in Linux
540 2.6.26, on certain popular architectures, the positive values of
541 sched_relax_domain_level have the following meanings.
542 [1m(1) Perform immediate load balancing across Hyper-Thread siblings on
544 the same core.
545 [1m(2) Perform immediate load balancing across other cores in the same
546 package.
547 [1m(3) Perform immediate load balancing across other CPUs on the same node
548 or blade.
549 [1m(4) Perform immediate load balancing across over several (implementa‐
550 tion detail) nodes [On NUMA systems].
551 [1m(5) Perform immediate load balancing across over all CPUs in system [On
552 NUMA systems].
553 The sched_relax_domain_level value of zero (0) always means don't per‐
555 form immediate load balancing, hence that load balancing is only done
556 periodically, not immediately when a CPU becomes available or another
557 task becomes runnable.
558 The sched_relax_domain_level value of minus one (-1) always means use
560 the system default value. The system default value can vary by archi‐
561 tecture and kernel version. This system default value can be changed
562 by kernel boot-time "relax_domain_level=" argument.
563 In the case of multiple overlapping cpusets which have conflicting
565 sched_relax_domain_level values, then the highest such value applies to
566 all CPUs in any of the overlapping cpusets. In such cases, the value
567 minus one (-1) is the lowest value, overridden by any other value, and
568 the value zero (0) is the next lowest value.
569
571 The following formats are used to represent sets of CPUs and memory
572 nodes.
573 Mask Format
575 The Mask Format is used to represent CPU and memory-node bitmasks in
576 the /proc/<pid>/status file.
577 This format displays each 32-bit word in hexadecimal (using ASCII char‐
579 acters "0" - "9" and "a" - "f"); words are filled with leading zeros,
580 if required. For masks longer than one word, a comma separator is used
581 between words. Words are displayed in big-endian order, which has the
582 most significant bit first. The hex digits within a word are also in
583 big-endian order.
584 The number of 32-bit words displayed is the minimum number needed to
586 display all bits of the bitmask, based on the size of the bitmask.
587 Examples of the Mask Format:
589 00000001 # just bit 0 set
591 40000000,00000000,00000000 # just bit 94 set
592 00000001,00000000,00000000 # just bit 64 set
593 000000ff,00000000 # bits 32-39 set
594 00000000,000E3862 # 1,5,6,11-13,17-19 set
595 A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:
597 00000001,00000001,00010117
599 The first "1" is for bit 64, the second for bit 32, the third for bit
601 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits
602 2, 1, and 0.
603 List Format
605 The List Format for cpus and mems is a comma-separated list of CPU or
606 memory-node numbers and ranges of numbers, in ASCII decimal.
607 Examples of the List Format:
609 0-4,9 # bits 0, 1, 2, 3, 4, and 9 set
611 0-2,7,12-14 # bits 0, 1, 2, 7, 12, 13, and 14 set
612
614 The following rules apply to each cpuset:
615 * Its CPUs and memory nodes must be a (possibly equal) subset of its
617 parent's.
618 * It can only be marked cpu_exclusive if its parent is.
620 * It can only be marked mem_exclusive if its parent is.
622 * If it is cpu_exclusive, its CPUs may not overlap any sibling.
624 * If it is memory_exclusive, its memory nodes may not overlap any sib‐
626 ling.
627
629 The permissions of a cpuset are determined by the permissions of the
630 directories and pseudo-files in the cpuset file system, normally
631 mounted at /dev/cpuset.
632 For instance, a process can put itself in some other cpuset (than its
634 current one) if it can write the tasks file for that cpuset. This
635 requires execute permission on the encompassing directories and write
636 permission on the tasks file.
637 An additional constraint is applied to requests to place some other
639 process in a cpuset. One process may not attach another to a cpuset
640 unless it would have permission to send that process a signal (see
641 kill(2)).
642 A process may create a child cpuset if it can access and write the par‐
644 ent cpuset directory. It can modify the CPUs or memory nodes in a
645 cpuset if it can access that cpuset's directory (execute permissions on
646 the each of the parent directories) and write the corresponding cpus or
647 mems file.
648 There is one minor difference between the manner in which these permis‐
650 sions are evaluated and the manner in which normal file-system opera‐
651 tion permissions are evaluated. The kernel interprets relative path‐
652 names starting at a process's current working directory. Even if one
653 is operating on a cpuset file, relative pathnames are interpreted rela‐
654 tive to the process's current working directory, not relative to the
655 process's current cpuset. The only ways that cpuset paths relative to
656 a process's current cpuset can be used are if either the process's cur‐
657 rent working directory is its cpuset (it first did a cd or chdir(2) to
658 its cpuset directory beneath /dev/cpuset, which is a bit unusual) or if
659 some user code converts the relative cpuset path to a full file-system
660 path.
661 In theory, this means that user code should specify cpusets using abso‐
663 lute pathnames, which requires knowing the mount point of the cpuset
664 file system (usually, but not necessarily, /dev/cpuset). In practice,
665 all user level code that this author is aware of simply assumes that if
666 the cpuset file system is mounted, then it is mounted at /dev/cpuset.
667 Furthermore, it is common practice for carefully written user code to
668 verify the presence of the pseudo-file /dev/cpuset/tasks in order to
669 verify that the cpuset pseudo-file system is currently mounted.
670
672 Enabling memory_pressure
673 By default, the per-cpuset file memory_pressure always contains zero
674 (0). Unless this feature is enabled by writing "1" to the pseudo-file
675 /dev/cpuset/memory_pressure_enabled, the kernel does not compute per-
676 cpuset memory_pressure.
677 Using the echo command
679 When using the echo command at the shell prompt to change the values of
680 cpuset files, beware that the built-in echo command in some shells does
681 not display an error message if the write(2) system call fails. For
682 example, if the command:
683 echo 19 > mems
685 failed because memory node 19 was not allowed (perhaps the current sys‐
687 tem does not have a memory node 19), then the echo command might not
688 display any error. It is better to use the /bin/echo external command
689 to change cpuset file settings, as this command will display write(2)
690 errors, as in the example:
691 /bin/echo 19 > mems
693 /bin/echo: write error: Invalid argument
694
696 Memory placement
697 Not all allocations of system memory are constrained by cpusets, for
698 the following reasons.
699 If hot-plug functionality is used to remove all the CPUs that are cur‐
701 rently assigned to a cpuset, then the kernel will automatically update
702 the cpus_allowed of all processes attached to CPUs in that cpuset to
703 allow all CPUs. When memory hot-plug functionality for removing memory
704 nodes is available, a similar exception is expected to apply there as
705 well. In general, the kernel prefers to violate cpuset placement,
706 rather than starving a process that has had all its allowed CPUs or
707 memory nodes taken offline. User code should reconfigure cpusets to
708 only refer to online CPUs and memory nodes when using hot-plug to add
709 or remove such resources.
710 A few kernel-critical, internal memory-allocation requests, marked
712 GFP_ATOMIC, must be satisfied immediately. The kernel may drop some
713 request or malfunction if one of these allocations fail. If such a
714 request cannot be satisfied within the current process's cpuset, then
715 we relax the cpuset, and look for memory anywhere we can find it. It's
716 better to violate the cpuset than stress the kernel.
717 Allocations of memory requested by kernel drivers while processing an
719 interrupt lack any relevant process context, and are not confined by
720 cpusets.
721 Renaming cpusets
723 You can use the rename(2) system call to rename cpusets. Only simple
724 renaming is supported; that is, changing the name of a cpuset directory
725 is permitted, but moving a directory into a different directory is not
726 permitted.
727
729 The Linux kernel implementation of cpusets sets errno to specify the
730 reason for a failed system call affecting cpusets.
731 The possible errno settings and their meaning when set on a failed
733 cpuset call are as listed below.
734 E2BIG Attempted a write(2) on a special cpuset file with a length
736 larger than some kernel-determined upper limit on the length of
737 such writes.
738 EACCES Attempted to write(2) the process ID (PID) of a process to a
740 cpuset tasks file when one lacks permission to move that
741 process.
742 EACCES Attempted to add, using write(2), a CPU or memory node to a
744 cpuset, when that CPU or memory node was not already in its par‐
745 ent.
746 EACCES Attempted to set, using write(2), cpu_exclusive or mem_exclusive
748 on a cpuset whose parent lacks the same setting.
749 EACCES Attempted to write(2) a memory_pressure file.
751 EACCES Attempted to create a file in a cpuset directory.
753 EBUSY Attempted to remove, using rmdir(2), a cpuset with attached pro‐
755 cesses.
756 EBUSY Attempted to remove, using rmdir(2), a cpuset with child
758 cpusets.
759 EBUSY Attempted to remove a CPU or memory node from a cpuset that is
761 also in a child of that cpuset.
762 EEXIST Attempted to create, using mkdir(2), a cpuset that already
764 exists.
765 EEXIST Attempted to rename(2) a cpuset to a name that already exists.
767 EFAULT Attempted to read(2) or write(2) a cpuset file using a buffer
769 that is outside the writing processes accessible address space.
770 EINVAL Attempted to change a cpuset, using write(2), in a way that
772 would violate a cpu_exclusive or mem_exclusive attribute of that
773 cpuset or any of its siblings.
774 EINVAL Attempted to write(2) an empty cpus or mems list to a cpuset
776 which has attached processes or child cpusets.
777 EINVAL Attempted to write(2) a cpus or mems list which included a range
779 with the second number smaller than the first number.
780 EINVAL Attempted to write(2) a cpus or mems list which included an
782 invalid character in the string.
783 EINVAL Attempted to write(2) a list to a cpus file that did not include
785 any online CPUs.
786 EINVAL Attempted to write(2) a list to a mems file that did not include
788 any online memory nodes.
789 EINVAL Attempted to write(2) a list to a mems file that included a node
791 that held no memory.
792 EIO Attempted to write(2) a string to a cpuset tasks file that does
794 not begin with an ASCII decimal integer.
795 EIO Attempted to rename(2) a cpuset into a different directory.
797 ENAMETOOLONG
799 Attempted to read(2) a /proc/<pid>/cpuset file for a cpuset path
800 that is longer than the kernel page size.
801 ENAMETOOLONG
803 Attempted to create, using mkdir(2), a cpuset whose base direc‐
804 tory name is longer than 255 characters.
805 ENAMETOOLONG
807 Attempted to create, using mkdir(2), a cpuset whose full path‐
808 name, including the mount point (typically "/dev/cpuset/") pre‐
809 fix, is longer than 4095 characters.
810 ENODEV The cpuset was removed by another process at the same time as a
812 write(2) was attempted on one of the pseudo-files in the cpuset
813 directory.
814 ENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset
816 that doesn't exist.
817 ENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset
819 directory.
820 ENOMEM Insufficient memory is available within the kernel; can occur on
822 a variety of system calls affecting cpusets, but only if the
823 system is extremely short of memory.
824 ENOSPC Attempted to write(2) the process ID (PID) of a process to a
826 cpuset tasks file when the cpuset had an empty cpus or empty
827 mems setting.
828 ENOSPC Attempted to write(2) an empty cpus or mems setting to a cpuset
830 that has tasks attached.
831 ENOTDIR
833 Attempted to rename(2) a nonexistent cpuset.
834 EPERM Attempted to remove a file from a cpuset directory.
836 ERANGE Specified a cpus or mems list to the kernel which included a
838 number too large for the kernel to set in its bitmasks.
839 ESRCH Attempted to write(2) the process ID (PID) of a nonexistent
841 process to a cpuset tasks file.
842
844 Cpusets appeared in version 2.6.12 of the Linux kernel.
845
847 Despite its name, the pid parameter is actually a thread ID, and each
848 thread in a threaded group can be attached to a different cpuset. The
849 value returned from a call to gettid(2) can be passed in the argument
850 pid.
851
853 memory_pressure cpuset files can be opened for writing, creation, or
854 truncation, but then the write(2) fails with errno set to EACCES, and
855 the creation and truncation options on open(2) have no affect.
856
858 The following examples demonstrate querying and setting cpuset options
859 using shell commands.
860 Creating and attaching to a cpuset.
862 To create a new cpuset and attach the current command shell to it, the
863 steps are:
864 1) mkdir /dev/cpuset (if not already done)
866 2) mount -t cpuset none /dev/cpuset (if not already done)
867 3) Create the new cpuset using mkdir(1).
868 4) Assign CPUs and memory nodes to the new cpuset.
869 5) Attach the shell to the new cpuset.
870 For example, the following sequence of commands will set up a cpuset
872 named "Charlie", containing just CPUs 2 and 3, and memory node 1, and
873 then attach the current shell to that cpuset.
874 $ mkdir /dev/cpuset
876 $ mount -t cpuset cpuset /dev/cpuset
877 $ cd /dev/cpuset
878 $ mkdir Charlie
879 $ cd Charlie
880 $ /bin/echo 2-3 > cpus
881 $ /bin/echo 1 > mems
882 $ /bin/echo $$ > tasks
883 # The current shell is now running in cpuset Charlie
884 # The next line should display '/Charlie'
885 $ cat /proc/self/cpuset
886 Migrating a job to different memory nodes.
888 To migrate a job (the set of processes attached to a cpuset) to differ‐
889 ent CPUs and memory nodes in the system, including moving the memory
890 pages currently allocated to that job, perform the following steps.
891 1) Let's say we want to move the job in cpuset alpha (CPUs 4-7 and
893 memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes
894 8-9).
895 2) First create the new cpuset beta.
896 3) Then allow CPUs 16-19 and memory nodes 8-9 in beta.
897 4) Then enable memory_migration in beta.
898 5) Then move each process from alpha to beta.
899 The following sequence of commands accomplishes this.
901 $ cd /dev/cpuset
903 $ mkdir beta
904 $ cd beta
905 $ /bin/echo 16-19 > cpus
906 $ /bin/echo 8-9 > mems
907 $ /bin/echo 1 > memory_migrate
908 $ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks
909 The above should move any processes in alpha to beta, and any memory
911 held by these processes on memory nodes 2-3 to memory nodes 8-9,
912 respectively.
913 Notice that the last step of the above sequence did not do:
915 $ cp ../alpha/tasks tasks
917 The while loop, rather than the seemingly easier use of the cp(1) com‐
919 mand, was necessary because only one process PID at a time may be writ‐
920 ten to the tasks file.
921 The same affect (writing one PID at a time) as the while loop can be
923 accomplished more efficiently, in fewer keystrokes and in syntax that
924 works on any shell, but alas more obscurely, by using the -u
925 (unbuffered) option of sed(1):
926 $ sed -un p < ../alpha/tasks > tasks
928
930 taskset(1), get_mempolicy(2), getcpu(2), mbind(2), sched_getaffin‐
931 ity(2), sched_setaffinity(2), sched_setscheduler(2), set_mempolicy(2),
932 CPU_SET(3), proc(5), numa(7), migratepages(8), numactl(8)
933 The kernel source file Documentation/cpusets.txt.
935
937 This page is part of release 3.22 of the Linux man-pages project. A
938 description of the project, and information about reporting bugs, can
939 be found at http://www.kernel.org/doc/man-pages/.
940Linux 2008-11-12 CPUSET(7)