1cgroup.conf(5) Slurm Configuration File cgroup.conf(5)
2
6 cgroup.conf - Slurm configuration file for the cgroup support
7
10 cgroup.conf is an ASCII file which defines parameters used by Slurm's
11 Linux cgroup related plugins. The file location can be modified at
12 system build time using the DEFAULT_SLURM_CONF parameter or at execu‐
13 tion time by setting the SLURM_CONF environment variable. The file will
14 always be located in the same directory as the slurm.conf file.
15 Parameter names are case insensitive. Any text following a "#" in the
17 configuration file is treated as a comment through the end of that
18 line. Changes to the configuration file take effect upon restart of
19 Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the
20 command "scontrol reconfigure" unless otherwise noted.
21 For general Slurm Cgroups information, see the Cgroups Guide at
24 <https://slurm.schedmd.com/cgroups.html>.
25 The following cgroup.conf parameters are defined to control the general
28 behavior of Slurm cgroup plugins.
29 CgroupAutomount=<yes|no>
32 Slurm cgroup plugins require valid and functional cgroup subsys‐
33 tem to be mounted under /sys/fs/cgroup/<subsystem_name>. When
34 launched, plugins check their subsystem availability. If not
35 available, the plugin launch fails unless CgroupAutomount is set
36 to yes. In that case, the plugin will first try to mount the
37 required subsystems.
38 CgroupMountpoint=PATH
41 Specify the PATH under which cgroups should be mounted. This
42 should be a writable directory which will contain cgroups
43 mounted one per subsystem. The default PATH is /sys/fs/cgroup.
44
47 The following cgroup.conf parameters are defined to control the behav‐
48 ior of this particular plugin:
49 AllowedKmemSpace=<number>
52 Constrain the job cgroup kernel memory to this amount of the
53 allocated memory, specified in bytes. The AllowedKmemSpace must
54 be between the upper and lower memory limits, specified by MaxK‐
55 memPercent and MinKmemSpace, respectivley. If AllowedKmemSpace
56 goes beyond the upper or lower limit, it will be reset to that
57 upper or lower limit, whichever has been exceeded.
58 AllowedRAMSpace=<number>
61 Constrain the job cgroup RAM to this percentage of the allocated
62 memory. The percentage supplied may be expressed as floating
63 point number, e.g. 98.5. If the AllowedRAMSpace limit is
64 exceeded, the job steps will be killed and a warning message
65 will be written to standard error. Also see ConstrainRAMSpace.
66 The default value is 100.
67 AllowedSwapSpace=<number>
70 Constrain the job cgroup swap space to this percentage of the
71 allocated memory. The default value is 0, which means that
72 RAM+Swap will be limited to AllowedRAMSpace. The supplied per‐
73 centage may be expressed as a floating point number, e.g. 50.5.
74 If the limit is exceeded, the job steps will be killed and a
75 warning message will be written to standard error. Also see
76 ConstrainSwapSpace. NOTE: Setting AllowedSwapSpace to 0 does
77 not restrict the Linux kernel from using swap space. To control
78 how the kernel uses swap space, see MemorySwappiness.
79 ConstrainCores=<yes|no>
82 If configured to "yes" then constrain allowed cores to the sub‐
83 set of allocated resources. This functionality makes use of the
84 cpuset subsystem. Due to a bug fixed in version 1.11.5 of
85 HWLOC, the task/affinity plugin may be required in addition to
86 task/cgroup for this to function properly. The default value is
87 "no".
88 ConstrainDevices=<yes|no>
91 If configured to "yes" then constrain the job's allowed devices
92 based on GRES allocated resources. It uses the devices subsystem
93 for that. The default value is "no".
94 ConstrainKmemSpace=<yes|no>
97 If configured to "yes" then constrain the job's Kmem RAM usage
98 in addition to RAM usage. Only takes effect if ConstrainRAMSpace
99 is set to "yes". The default value is "no". If set to yes, the
100 job's Kmem limit will be set to AllowedKmemSpace if set; other‐
101 wise, the job's Kmem limit will be set to its RAM limit. Also
102 see AllowedKmemSpace.
103 ConstrainRAMSpace=<yes|no>
106 If configured to "yes" then constrain the job's RAM usage by
107 setting the memory soft limit to the allocated memory and the
108 hard limit to the allocated memory * AllowedRAMSpace. The
109 default value is "no", in which case the job's RAM limit will be
110 set to its swap space limit if ConstrainSwapSpace is set to
111 "yes". Also see AllowedSwapSpace, AllowedRAMSpace and Con‐
112 strainSwapSpace. NOTE: When enabled, ConstrainRAMSpace can lead
113 to a noticiable decline in per-node job throughout. Sites with
114 high-throughput requirements should carefully weigh the tradeoff
115 between per-node throughput, versus potential problems that can
116 arise from unconstrained memory usage on the node. See
117 <https://slurm.schedmd.com/high_throughput.html> for further
118 discussion.
119 ConstrainSwapSpace=<yes|no>
122 If configured to "yes" then constrain the job's swap space
123 usage. The default value is "no". Note that when set to "yes"
124 and ConstrainRAMSpace is set to "no", AllowedRAMSpace is auto‐
125 matically set to 100% in order to limit the RAM+Swap amount to
126 100% of job's requirement plus the percent of allowed swap
127 space. This amount is thus set to both RAM and RAM+Swap limits.
128 This means that in that particular case, ConstrainRAMSpace is
129 automatically enabled with the same limit than the one used to
130 constrain swap space. Also see AllowedSwapSpace.
131 MaxRAMPercent=PERCENT
134 Set an upper bound in percent of total RAM on the RAM constraint
135 for a job. This will be the memory constraint applied to jobs
136 that are not explicitly allocated memory by Slurm (i.e. Slurm's
137 select plugin is not configured to manage memory allocations).
138 The PERCENT may be an arbitrary floating point number. The
139 default value is 100.
140 MaxSwapPercent=PERCENT
143 Set an upper bound (in percent of total RAM) on the amount of
144 RAM+Swap that may be used for a job. This will be the swap limit
145 applied to jobs on systems where memory is not being explicitly
146 allocated to job. The PERCENT may be an arbitrary floating point
147 number between 0 and 100. The default value is 100.
148 MaxKmemPercent=PERCENT
151 Set an upper bound in percent of total Kmem for a job. The PER‐
152 CENT may be an arbitrary floating point number. The default
153 value is 100.
154 MemorySwappiness=<number>
157 Configure the kernel's priority for swapping out anonymous pages
158 (such as program data) verses file cache pages for the job
159 cgroup. Valid values are between 0 and 100, inclusive. A value
160 of 0 prevents the kernel from swapping out program data. A value
161 of 100 gives equal priorioty to swapping out file cache or
162 anonymous pages. If not set, then the kernel's default swappi‐
163 ness value will be used. Either ConstrainRAMSpace or Constrain‐
164 SwapSpace must be set to yes in order for this parameter to be
165 applied.
166 MinKmemSpace=<number>
169 Set a lower bound (in MB) on the memory limits defined by
170 AllowedKmemSpace. The default limit is 30M.
171 MinRAMSpace=<number>
174 Set a lower bound (in MB) on the memory limits defined by
175 AllowedRAMSpace and AllowedSwapSpace. This prevents accidentally
176 creating a memory cgroup with such a low limit that slurmstepd
177 is immediately killed due to lack of RAM. The default limit is
178 30M.
179 TaskAffinity=<yes|no>
182 If configured to "yes" then set a default task affinity to bind
183 each step task to a subset of the allocated cores using
184 sched_setaffinity. The default value is "no". Note: This fea‐
185 ture requires the Portable Hardware Locality (hwloc) library to
186 be installed.
187
190 Debian and derivatives (e.g. Ubuntu) usually exclude the memory and
191 memsw (swap) cgroups by default. To include them, add the following
192 parameters to the kernel command line: cgroup_enable=memory swapac‐
193 count=1
194 This can usually be placed in /etc/default/grub inside the GRUB_CMD‐
196 LINE_LINUX variable. A command such as update-grub must be run after
197 updating the file.
198
201 ###
202 # Slurm cgroup support configuration file
203 ###
204 CgroupAutomount=yes
205 ConstrainCores=yes
206 #
207
210 Copyright (C) 2010-2012 Lawrence Livermore National Security. Produced
211 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
212 Copyright (C) 2010-2016 SchedMD LLC.
213 This file is part of Slurm, a resource management program. For
215 details, see <https://slurm.schedmd.com/>.
216 Slurm is free software; you can redistribute it and/or modify it under
218 the terms of the GNU General Public License as published by the Free
219 Software Foundation; either version 2 of the License, or (at your
220 option) any later version.
221 Slurm is distributed in the hope that it will be useful, but WITHOUT
223 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
224 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
225 for more details.
226
229 slurm.conf(5)
230December 2016 Slurm Configuration File cgroup.conf(5)