1cgroup.conf(5)             Slurm Configuration File             cgroup.conf(5)
2
3
4

NAME

6       cgroup.conf - Slurm configuration file for the cgroup support
7
8

DESCRIPTION

10       cgroup.conf  is  an ASCII file which defines parameters used by Slurm's
11       Linux cgroup related plugins.  The file location  can  be  modified  at
12       system  build  time using the DEFAULT_SLURM_CONF parameter or at execu‐
13       tion time by setting the SLURM_CONF environment variable. The file will
14       always be located in the same directory as the slurm.conf file.
15
16       Parameter  names are case insensitive.  Any text following a "#" in the
17       configuration file is treated as a comment  through  the  end  of  that
18       line.   Changes  to  the configuration file take effect upon restart of
19       Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the
20       command "scontrol reconfigure" unless otherwise noted.
21
22
23       For  general  Slurm  Cgroups  information,  see  the  Cgroups  Guide at
24       <https://slurm.schedmd.com/cgroups.html>.
25
26
27       The following cgroup.conf parameters are defined to control the general
28       behavior of Slurm cgroup plugins.
29
30
31       CgroupAutomount=<yes|no>
32              Slurm cgroup plugins require valid and functional cgroup subsys‐
33              tem to be mounted under  /sys/fs/cgroup/<subsystem_name>.   When
34              launched,  plugins  check  their  subsystem availability. If not
35              available, the plugin launch fails unless CgroupAutomount is set
36              to  yes.  In  that  case, the plugin will first try to mount the
37              required subsystems.
38
39
40       CgroupMountpoint=PATH
41              Specify the PATH under which cgroups  should  be  mounted.  This
42              should  be  a  writable  directory  which  will  contain cgroups
43              mounted one per subsystem. The default PATH is /sys/fs/cgroup.
44
45

TASK/CGROUP PLUGIN

47       The following cgroup.conf parameters are defined to control the  behav‐
48       ior of this particular plugin:
49
50
51       AllowedKmemSpace=<number>
52              Constrain  the  job  cgroup  kernel memory to this amount of the
53              allocated memory, specified in bytes. The AllowedKmemSpace  must
54              be between the upper and lower memory limits, specified by MaxK‐
55              memPercent and MinKmemSpace, respectivley.  If  AllowedKmemSpace
56              goes  beyond  the upper or lower limit, it will be reset to that
57              upper or lower limit, whichever has been exceeded.
58
59
60       AllowedRAMSpace=<number>
61              Constrain the job cgroup RAM to this percentage of the allocated
62              memory.   The  percentage  supplied may be expressed as floating
63              point  number,  e.g.  98.5.  If  the  AllowedRAMSpace  limit  is
64              exceeded,  the  job  steps  will be killed and a warning message
65              will be written to standard error.  Also see  ConstrainRAMSpace.
66              The default value is 100.
67
68
69       AllowedSwapSpace=<number>
70              Constrain  the  job  cgroup swap space to this percentage of the
71              allocated memory.  The default value  is  0,  which  means  that
72              RAM+Swap  will  be limited to AllowedRAMSpace. The supplied per‐
73              centage may be expressed as a floating point number, e.g.  50.5.
74              If  the  limit  is  exceeded, the job steps will be killed and a
75              warning message will be written to  standard  error.   Also  see
76              ConstrainSwapSpace.   NOTE:  Setting  AllowedSwapSpace to 0 does
77              not restrict the Linux kernel from using swap space. To  control
78              how the kernel uses swap space, see MemorySwappiness.
79
80
81       ConstrainCores=<yes|no>
82              If  configured to "yes" then constrain allowed cores to the sub‐
83              set of allocated resources. This functionality makes use of  the
84              cpuset  subsystem.   Due  to  a  bug  fixed in version 1.11.5 of
85              HWLOC, the task/affinity plugin may be required in  addition  to
86              task/cgroup for this to function properly.  The default value is
87              "no".
88
89
90       ConstrainDevices=<yes|no>
91              If configured to "yes" then constrain the job's allowed  devices
92              based on GRES allocated resources. It uses the devices subsystem
93              for that.  The default value is "no".
94
95
96       ConstrainKmemSpace=<yes|no>
97              If configured to "yes" then constrain the job's Kmem  RAM  usage
98              in addition to RAM usage. Only takes effect if ConstrainRAMSpace
99              is set to "yes". The default value is "no". If set to  yes,  the
100              job's  Kmem limit will be set to AllowedKmemSpace if set; other‐
101              wise, the job's Kmem limit will be set to its RAM  limit.   Also
102              see AllowedKmemSpace.
103
104
105       ConstrainRAMSpace=<yes|no>
106              If  configured  to  "yes"  then constrain the job's RAM usage by
107              setting the memory soft limit to the allocated  memory  and  the
108              hard  limit  to  the  allocated  memory  * AllowedRAMSpace.  The
109              default value is "no", in which case the job's RAM limit will be
110              set  to  its  swap  space  limit if ConstrainSwapSpace is set to
111              "yes".  Also  see  AllowedSwapSpace,  AllowedRAMSpace  and  Con‐
112              strainSwapSpace.  NOTE: When enabled, ConstrainRAMSpace can lead
113              to a noticiable decline in per-node job throughout.  Sites  with
114              high-throughput requirements should carefully weigh the tradeoff
115              between per-node throughput, versus potential problems that  can
116              arise   from   unconstrained  memory  usage  on  the  node.  See
117              <https://slurm.schedmd.com/high_throughput.html>   for   further
118              discussion.
119
120
121       ConstrainSwapSpace=<yes|no>
122              If  configured  to  "yes"  then  constrain  the job's swap space
123              usage.  The default value is "no". Note that when set  to  "yes"
124              and  ConstrainRAMSpace  is set to "no", AllowedRAMSpace is auto‐
125              matically set to 100% in order to limit the RAM+Swap  amount  to
126              100%  of  job's  requirement  plus  the  percent of allowed swap
127              space. This amount is thus set to both RAM and RAM+Swap  limits.
128              This  means  that  in that particular case, ConstrainRAMSpace is
129              automatically enabled with the same limit than the one  used  to
130              constrain swap space.  Also see AllowedSwapSpace.
131
132
133       MaxRAMPercent=PERCENT
134              Set an upper bound in percent of total RAM on the RAM constraint
135              for a job.  This will be the memory constraint applied  to  jobs
136              that  are not explicitly allocated memory by Slurm (i.e. Slurm's
137              select plugin is not configured to manage  memory  allocations).
138              The  PERCENT  may  be  an  arbitrary  floating point number. The
139              default value is 100.
140
141
142       MaxSwapPercent=PERCENT
143              Set an upper bound (in percent of total RAM) on  the  amount  of
144              RAM+Swap that may be used for a job. This will be the swap limit
145              applied to jobs on systems where memory is not being  explicitly
146              allocated to job. The PERCENT may be an arbitrary floating point
147              number between 0 and 100.  The default value is 100.
148
149
150       MaxKmemPercent=PERCENT
151              Set an upper bound in percent of total Kmem for a job. The  PER‐
152              CENT  may  be  an  arbitrary  floating point number. The default
153              value is 100.
154
155
156       MemorySwappiness=<number>
157              Configure the kernel's priority for swapping out anonymous pages
158              (such  as  program  data)  verses  file  cache pages for the job
159              cgroup. Valid values are between 0 and 100, inclusive.  A  value
160              of 0 prevents the kernel from swapping out program data. A value
161              of 100 gives equal priorioty  to  swapping  out  file  cache  or
162              anonymous  pages.  If not set, then the kernel's default swappi‐
163              ness value will be used. Either ConstrainRAMSpace or  Constrain‐
164              SwapSpace  must  be set to yes in order for this parameter to be
165              applied.
166
167
168       MinKmemSpace=<number>
169              Set a lower bound (in  MB)  on  the  memory  limits  defined  by
170              AllowedKmemSpace. The default limit is 30M.
171
172
173       MinRAMSpace=<number>
174              Set  a  lower  bound  (in  MB)  on  the memory limits defined by
175              AllowedRAMSpace and AllowedSwapSpace. This prevents accidentally
176              creating  a  memory cgroup with such a low limit that slurmstepd
177              is immediately killed due to lack of RAM. The default  limit  is
178              30M.
179
180
181       TaskAffinity=<yes|no>
182              If  configured to "yes" then set a default task affinity to bind
183              each step  task  to  a  subset  of  the  allocated  cores  using
184              sched_setaffinity.   The default value is "no".  Note: This fea‐
185              ture requires the Portable Hardware Locality (hwloc) library  to
186              be installed.
187
188

DISTRIBUTION-SPECIFIC NOTES

190       Debian  and  derivatives  (e.g.  Ubuntu) usually exclude the memory and
191       memsw (swap) cgroups by default. To include  them,  add  the  following
192       parameters  to  the  kernel  command line: cgroup_enable=memory swapac‐
193       count=1
194
195       This can usually be placed in /etc/default/grub  inside  the  GRUB_CMD‐
196       LINE_LINUX  variable.  A  command such as update-grub must be run after
197       updating the file.
198
199

EXAMPLE

201       ###
202       # Slurm cgroup support configuration file
203       ###
204       CgroupAutomount=yes
205       ConstrainCores=yes
206       #
207
208

COPYING

210       Copyright (C) 2010-2012 Lawrence Livermore National Security.  Produced
211       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
212       Copyright (C) 2010-2016 SchedMD LLC.
213
214       This  file  is  part  of  Slurm,  a  resource  management program.  For
215       details, see <https://slurm.schedmd.com/>.
216
217       Slurm is free software; you can redistribute it and/or modify it  under
218       the  terms  of  the GNU General Public License as published by the Free
219       Software Foundation; either version 2  of  the  License,  or  (at  your
220       option) any later version.
221
222       Slurm  is  distributed  in the hope that it will be useful, but WITHOUT
223       ANY WARRANTY; without even the implied warranty of  MERCHANTABILITY  or
224       FITNESS  FOR  A PARTICULAR PURPOSE.  See the GNU General Public License
225       for more details.
226
227

SEE ALSO

229       slurm.conf(5)
230
231
232
233December 2016              Slurm Configuration File             cgroup.conf(5)
Impressum