1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an Ad‐
17 ministrator. If an attempt to view or modify configuration information
18 is made by an unauthorized user, an error message will be printed and
19 the requested action will not occur. If no command is entered on the
20 execute line, scontrol will operate in an interactive mode and prompt
21 for input. It will continue prompting for input and executing commands
22 until explicitly terminated. If a command is entered on the execute
23 line, scontrol will execute that command and terminate. All commands
24 and options are case-insensitive, although node names, partition names,
25 and reservation names are case-sensitive (node names "LX" and "lx" are
26 distinct). All commands and options can be abbreviated to the extent
27 that the specification is unique. A modified Slurm configuration can be
28 written to a file using the scontrol write config command. The result‐
29 ing file will be named using the convention "slurm.conf.<datetime>" and
30 located in the same directory as the original "slurm.conf" file. The
31 directory containing the original slurm.conf must be writable for this
32 to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -M, --clusters=<string>
43 The cluster to issue commands to. Only one cluster name may be
44 specified. Note that the SlurmDBD must be up for this option to
45 work properly. This option implicitly sets the --local option.
46 -d, --details
48 Causes the show command to provide additional details where
49 available.
50 --federation
52 Report jobs from federation if a member of one.
53 -F, --future
55 Report nodes in FUTURE state.
56 -h, --help
58 Print a help message describing the usage of scontrol.
59 --hide Do not display information about hidden partitions, their jobs
61 and job steps. By default, neither partitions that are config‐
62 ured as hidden nor those partitions unavailable to user's group
63 will be displayed (i.e. this is the default behavior).
64 --local
66 Show only information local to this cluster. Ignore other clus‐
67 ters in the federated if a member of one. Overrides --federa‐
68 tion.
69 -o, --oneliner
71 Print information one line per record.
72 -Q, --quiet
74 Print no warning or informational messages, only fatal error
75 messages.
76 --sibling
78 Show all sibling jobs on a federated cluster. Implies --federa‐
79 tion.
80 -u, --uid=<uid>
82 Attempt to update a job as user <uid> instead of the invoking
83 user id.
84 -v, --verbose
86 Print detailed event logging. Multiple -v's will further in‐
87 crease the verbosity of logging. By default only errors will be
88 displayed.
89 -V , --version
91 Print version information and exit.
92
94 abort Instruct the Slurm controller to terminate immediately and gen‐
95 erate a core file. See "man slurmctld" for information about
96 where the core file will be written.
97 cancel_reboot <NodeList>
99 Cancel pending reboots on nodes. The node will be undrain'ed and
100 the reason cleared if the node was drained by an ASAP reboot.
101 cluster <CLUSTER_NAME>
103 The cluster to issue commands to. Only one cluster name may be
104 specified.
105 create <SPECIFICATION>
107 Create a new node, partition, or reservation. See the full list
108 of parameters below.
109 completing
111 Display all jobs in a COMPLETING state along with associated
112 nodes in either a COMPLETING or DOWN state.
113 delete <SPECIFICATION>
115 Delete the entry with the specified SPECIFICATION. The three
116 SPECIFICATION choices are NodeName=<nodelist>, Partition‐
117 Name=<name> and Reservation=<name>. Nodes can't be deleted if
118 they are in a reservation or have jobs on running. Reservations
119 and partitions should have no associated jobs at the time of
120 their deletion (modify the jobs first). If the specified parti‐
121 tion is in use, the request is denied.
122 errnumstr <ERRNO>
124 Given a Slurm error number, return a descriptive string.
125 fsdampeningfactor <FACTOR>
127 Set the FairShareDampeningFactor in slurmctld.
128 help Display a description of scontrol options and commands.
130 hold <job_list>
132 Prevent a pending job from being started (sets its priority to
133 0). Use the release command to permit the job to be scheduled.
134 The job_list argument is a comma separated list of job IDs OR
135 "jobname=" with the job's name, which will attempt to hold all
136 jobs having that name. Note that when a job is held by a system
137 administrator using the hold command, only a system administra‐
138 tor may release the job for execution (also see the uhold com‐
139 mand). When the job is held by its owner, it may also be re‐
140 leased by the job's owner. Additionally, attempting to hold a
141 running job will have not suspend or cancel it. But, it will set
142 the job priority to 0 and update the job reason field, which
143 would hold the job if it was requeued at a later time.
144 notify <job_id> <message>
146 Send a message to standard error of the salloc or srun command
147 or batch job associated with the specified job_id.
148 pidinfo <proc_id>
150 Print the Slurm job id and scheduled termination time corre‐
151 sponding to the supplied process id, proc_id, on the current
152 node. This will work only with processes on node on which scon‐
153 trol is run, and only for those processes spawned by Slurm and
154 their descendants.
155 listpids [<job_id>[.<step_id>]] [<NodeName>]
157 Print a listing of the process IDs in a job step (if JO‐
158 BID.STEPID is provided), or all of the job steps in a job (if
159 job_id is provided), or all of the job steps in all of the jobs
160 on the local node (if job_id is not provided or job_id is "*").
161 This will work only with processes on the node on which scontrol
162 is run, and only for those processes spawned by Slurm and their
163 descendants. Note that some Slurm configurations (ProctrackType
164 value of pgid) are unable to identify all processes associated
165 with a job or job step.
166 Note that the NodeName option is only really useful when you
168 have multiple slurmd daemons running on the same host machine.
169 Multiple slurmd daemons on one host are, in general, only used
170 by Slurm developers.
171 ping Ping the primary and secondary slurmctld daemon and report if
173 they are responding.
174 reboot [ASAP] [nextstate={RESUME|DOWN}] [reason=<reason>] {ALL|<Node‐
176 List>}
177 Reboot the nodes in the system when they become idle using the
178 RebootProgram as configured in Slurm's slurm.conf file. Each
179 node will have the "REBOOT" flag added to its node state. After
180 a node reboots and the slurmd daemon starts up again, the
181 HealthCheckProgram will run once. Then, the slurmd daemon will
182 register itself with the slurmctld daemon and the "REBOOT" flag
183 will be cleared. The node's "DRAIN" state flag will be cleared
184 if the reboot was "ASAP", nextstate=resume or down. The "ASAP"
185 option adds the "DRAIN" flag to each node's state, preventing
186 additional jobs from running on the node so it can be rebooted
187 and returned to service "As Soon As Possible" (i.e. ASAP).
188 "ASAP" will also set the node reason to "Reboot ASAP" if the
189 "reason" option isn't specified. If the "nextstate" option is
190 specified as "DOWN", then the node will remain in a down state
191 after rebooting. If "nextstate" is specified as "RESUME", then
192 the nodes will resume as normal and the node's reason and
193 "DRAIN" state will be cleared. Resuming nodes will be consid‐
194 ered as available in backfill future scheduling and won't be re‐
195 placed by idle nodes in a reservation. The "reason" option sets
196 each node's reason to a user-defined message. A default reason
197 of "reboot requested" is set if no other reason is set on the
198 node. The reason will be appended with: "reboot issued" when
199 the reboot is issued; "reboot complete" when the node registers
200 and has a "nextstate" of "DOWN"; or "reboot timed out" when the
201 node fails to register within ResumeTimeout. You must specify
202 either a list of nodes or that ALL nodes are to be rebooted.
203 NOTE: By default, this command does not prevent additional jobs
204 from being scheduled on any nodes before reboot. To do this,
205 you can either use the "ASAP" option or explicitly drain the
206 nodes beforehand. You can alternately create an advanced reser‐
207 vation to prevent additional jobs from being initiated on nodes
208 to be rebooted. Pending reboots can be cancelled by using
209 "scontrol cancel_reboot <node>" or setting the node state to
210 "CANCEL_REBOOT". A node will be marked "DOWN" if it doesn't re‐
211 boot within ResumeTimeout.
212 reconfigure
214 Instruct all Slurm daemons to re-read the configuration file.
215 This command does not restart the daemons. This mechanism can
216 be used to modify configuration parameters set in slurm.conf.
217 The Slurm controller (slurmctld) forwards the request to all
218 other daemons (slurmd daemon on each compute node). Running jobs
219 continue execution. Most configuration parameters can be
220 changed by just running this command; however, there are parame‐
221 ters that require a restart of the relevant Slurm daemons. Pa‐
222 rameters requiring a restart will be noted in the slurm.conf(5)
223 man page. The slurmctld daemon and all slurmd daemons must also
224 be restarted if nodes are added to or removed from the cluster.
225 release <job_list>
227 Release a previously held job to begin execution. The job_list
228 argument is a comma separated list of job IDs OR "jobname=" with
229 the job's name, which will attempt to hold all jobs having that
230 name. Also see hold.
231 requeue [<option>] <job_list>
233 Requeue a running, suspended or finished Slurm batch job into
234 pending state. The job_list argument is a comma separated list
235 of job IDs. The command accepts the following option:
236 Incomplete
238 Operate only on jobs (or tasks of a job array) which have
239 not completed. Specifically only jobs in the following
240 states will be requeued: CONFIGURING, RUNNING, STOPPED or
241 SUSPENDED.
242 requeuehold [<option>] <job_list>
244 Requeue a running, suspended or finished Slurm batch job into
245 pending state, moreover the job is put in held state (priority
246 zero). The job_list argument is a comma separated list of job
247 IDs. A held job can be released using scontrol to reset its
248 priority (e.g. "scontrol release <job_id>"). The command ac‐
249 cepts the following options:
250 Incomplete
252 Operate only on jobs (or tasks of a job array) which have
253 not completed. Specifically only jobs in the following
254 states will be requeued: CONFIGURING, RUNNING, STOPPED or
255 SUSPENDED.
256 State=SpecialExit
258 The "SpecialExit" keyword specifies that the job has to
259 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
260 trol show job" command will display the JobState as SPE‐
261 CIAL_EXIT, while the "squeue" command as SE.
262 resume <job_list>
264 Resume a previously suspended job. The job_list argument is a
265 comma separated list of job IDs. Also see suspend.
266 NOTE: A suspended job releases its CPUs for allocation to other
268 jobs. Resuming a previously suspended job may result in multi‐
269 ple jobs being allocated the same CPUs, which could trigger gang
270 scheduling with some configurations or severe degradation in
271 performance with other configurations. Use of the scancel com‐
272 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
273 out releasing its CPUs for allocation to other jobs and would be
274 a preferable mechanism in many cases. If performing system
275 maintenance you may want to use suspend/resume in the following
276 way. Before suspending set all nodes to draining or set all par‐
277 titions to down so that no new jobs can be scheduled. Then sus‐
278 pend jobs. Once maintenance is done resume jobs then resume
279 nodes and/or set all partitions back to up. Use with caution.
280 schedloglevel <LEVEL>
282 Enable or disable scheduler logging. LEVEL may be "0", "1",
283 "disable" or "enable". "0" has the same effect as "disable". "1"
284 has the same effect as "enable". This value is temporary and
285 will be overwritten when the slurmctld daemon reads the
286 slurm.conf configuration file (e.g. when the daemon is restarted
287 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
288 parameter is present.
289 setdebug <LEVEL>
291 Change the debug level of the slurmctld daemon for all active
292 logging channels not originally configured off (quiet). LEVEL
293 may be an integer value between zero and nine (using the same
294 values as SlurmctldDebug in the slurm.conf file) or the name of
295 the most detailed message type to be printed: "quiet", "fatal",
296 "error", "info", "verbose", "debug", "debug2", "debug3", "de‐
297 bug4", or "debug5". This value is temporary and will be over‐
298 written whenever the slurmctld daemon reads the slurm.conf con‐
299 figuration file (e.g. when the daemon is restarted or scontrol
300 reconfigure is executed).
301 setdebugflags [+|-]<FLAG>
303 Add or remove DebugFlags of the slurmctld daemon. See "man
304 slurm.conf" for a list of supported DebugFlags.
305 NOTE: Changing the value of some DebugFlags will have no effect
307 without restarting the slurmctld daemon, which would set De‐
308 bugFlags based upon the contents of the slurm.conf configuration
309 file.
310 show <ENTITY>[=<ID>] or <ENTITY> [<ID>]
312 Display the state of the specified entity with the specified
313 identification.
314 aliases
316 Returns all NodeName values associated with a given Node‐
317 Hostname (useful to get the list of virtual nodes associ‐
318 ated with a real node in a configuration where multiple
319 slurmd daemons execute on a single compute node).
320 assoc_mgr
322 Displays the current contents of the slurmctld's internal
323 cache for users, associations and/or qos. The output can
324 be filtered by different record types:
325 users=<user1>[...,<userN>]
327 Limit the User Records displayed to those with the
328 specified user name(s).
329 accounts=<acct1>[...,<acctN>]
331 Limit the Association Records displayed to those
332 with the specified account name(s).
333 qos=<qos1>[...,<qosN>]
335 Limit the QOS Records displayed to those with the
336 specified QOS name(s).
337 flags={users|assoc|qos}
339 Specify the desired record type to be displayed.
340 If no flags are specified, all record types are
341 displayed.
342 bbstat Displays output from Cray's burst buffer status tool. Op‐
344 tions following bbstat are passed directly to the dwstat
345 command by the slurmctld daemon and the response returned
346 to the user. Equivalent to dwstat.
347 burstbuffer
349 Displays the current status of the BurstBuffer plugin.
350 config Displays parameter names from the configuration files in
352 mixed case (e.g. SlurmdPort=7003) while derived parame‐
353 ters names are in upper case only (e.g. SLURM_VERSION).
354 daemons
356 Reports which daemons should be running on this node.
357 dwstat Displays output from Cray's burst buffer status tool. Op‐
359 tions following dwstat are passed directly to the dwstat
360 command by the slurmctld daemon and the response returned
361 to the user. Equivalent to bbstat.
362 federation
364 The federation name that the controller is part of and
365 the sibling clusters part of the federation will be
366 listed.
367 frontend
369 Shows configured frontend nodes.
370 hostlist
372 Takes a list of host names and prints the hostlist ex‐
373 pression for them (the inverse of hostnames). hostlist
374 can also take the absolute pathname of a file (beginning
375 with the character '/') containing a list of hostnames.
376 Multiple node names may be specified using simple node
377 range expressions (e.g. "lx[10-20]"). By default
378 hostlist does not sort the node list or make it unique
379 (e.g. tux2,tux1,tux2 = tux[2,1-2]). If you wanted a
380 sorted list use hostlistsorted (e.g. tux2,tux1,tux2 =
381 tux[1-2,2]).
382 hostlistsorted
384 Takes a list of host names and prints a sorted (but not
385 unique) hostlist expression for them. See hostlist.
386 hostnames
388 Takes an optional hostlist expression as input and writes
389 a list of individual host names to standard output (one
390 per line). If no hostlist expression is supplied, the
391 contents of the SLURM_JOB_NODELIST environment variable
392 is used. For example "tux[1-3]" is mapped to
393 "tux1","tux2" and "tux3" (one hostname per line).
394 job Displays statistics about all jobs by default. If an op‐
396 tional jobid is specified, details for just that job will
397 be displayed. If the job does not specify
398 socket-per-node, cores-per-socket or threads-per-core
399 then it will display '*' in the ReqS:C:T=*:*:* field.
400 licenses
402 Displays statistics about all configured licenses (local
403 and remote) by default. If an optional license name is
404 specified, details for just that license will be dis‐
405 played.
406 node Displays statistics about all nodes by default. If an op‐
409 tional nodename is specified, details for just that node
410 will be displayed.
411 partition
413 Displays statistics about all partitions by default. If
414 an optional partition name is specified, details for just
415 that partition will be displayed.
416 reservation
418 Displays statistics about all reservations by default. If
419 an optional reservation name is specified, details for
420 just that reservation will be displayed.
421 slurmd Displays statistics for the slurmd running on the current
423 node.
424 step Displays statistics about all job steps by default. If an
426 optional jobid is specified, details about steps for just
427 that job will be displayed. If a jobid.stepid is speci‐
428 fied, details for just that step will be displayed.
429 topology
431 Displays information about the defined topology layout.
432 If a switch is specified, information about that switch
433 will be shown. If one node name is specified, all
434 switches connected to that node (and their parent
435 switches) will be shown. If more than one node name is
436 specified, only switches that connect to all named nodes
437 will be shown.
438 shutdown <OPTION>
440 Instruct Slurm daemons to save current state and terminate. By
441 default, the Slurm controller (slurmctld) forwards the request
442 all other daemons (slurmd daemon on each compute node). An OP‐
443 TION of slurmctld or controller results in only the slurmctld
444 daemon being shutdown and the slurmd daemons remaining active.
445 suspend <job_list>
447 Suspend a running job. The job_list argument is a comma sepa‐
448 rated list of job IDs. Use the resume command to resume its ex‐
449 ecution. User processes must stop on receipt of SIGSTOP signal
450 and resume upon receipt of SIGCONT for this operation to be ef‐
451 fective. Not all architectures and configurations support job
452 suspension. If a suspended job is requeued, it will be placed
453 in a held state. The time a job is suspended will not count
454 against a job's time limit. Only an operator, administrator,
455 SlurmUser, or root can suspend jobs.
456 takeover [<INDEX>]
458 Instruct one of Slurm's backup controllers (slurmctld) to take
459 over system control. By default the first backup controller (IN‐
460 DEX=1) requests control from the primary and waits for its ter‐
461 mination. After that, it switches from backup mode to controller
462 mode. If primary controller can not be contacted, it directly
463 switches to controller mode. This can be used to speed up the
464 Slurm controller fail-over mechanism when the primary node is
465 down. This can be used to minimize disruption if the computer
466 executing the primary Slurm controller is scheduled down.
467 (Note: Slurm's primary controller will take the control back at
468 startup.)
469 top <job_list>
471 Move the specified job IDs to the top of the queue of jobs be‐
472 longing to the identical user ID, partition name, account, and
473 QOS. The job_list argument is a comma separated ordered list of
474 job IDs. Any job not matching all of those fields will not be
475 effected. Only jobs submitted to a single partition will be ef‐
476 fected. This operation changes the order of jobs by adjusting
477 job nice values. The net effect on that user's throughput will
478 be negligible to slightly negative. This operation is disabled
479 by default for non-privileged (non-operator, admin, SlurmUser,
480 or root) users. This operation may be enabled for non-privileged
481 users by the system administrator by including the option "en‐
482 able_user_top" in the SchedulerParameters configuration parame‐
483 ter.
484 token [lifespan=<lifespan>] [username=<username>]
486 Return an auth token which can be used to support JWT authenti‐
487 cation if AuthAltTypes=auth/jwt has been enabled on the system.
488 Supports two optional arguments. lifespan= may be used to spec‐
489 ify the token's lifespan in seconds. username (only available to
490 SlurmUser/root) may be used to request a token for a different
491 username.
492 uhold <job_list>
494 Prevent a pending job from being started (sets its priority to
495 0). The job_list argument is a space separated list of job IDs
496 or job names. Use the release command to permit the job to be
497 scheduled. This command is designed for a system administrator
498 to hold a job so that the job owner may release it rather than
499 requiring the intervention of a system administrator (also see
500 the hold command).
501 update <SPECIFICATION>
503 Update job, step, node, partition, or reservation configuration
504 per the supplied specification. SPECIFICATION is in the same
505 format as the Slurm configuration file and the output of the
506 show command described above. It may be desirable to execute the
507 show command (described above) on the specific entity you want
508 to update, then use cut-and-paste tools to enter updated config‐
509 uration values to the update. Note that while most configuration
510 values can be changed using this command, not all can be changed
511 using this mechanism. In particular, the hardware configuration
512 of a node or the physical addition or removal of nodes from the
513 cluster may only be accomplished through editing the Slurm con‐
514 figuration file and executing the reconfigure command (described
515 above).
516 version
518 Display the version number of scontrol being executed.
519 wait_job <job_id>
521 Wait until a job and all of its nodes are ready for use or the
522 job has entered some termination state. This option is particu‐
523 larly useful in the Slurm Prolog or in the batch script itself
524 if nodes are powered down and restarted automatically as needed.
525 NOTE: Don't use scontrol wait_job in PrologSlurmctld or Prolog
527 with PrologFlags=Alloc as this will result in a deadlock.
528 NOTE: When using wait_job for an array job, use the SLURM_JOB_ID
530 environment variable to reference the job rather than the
531 SLURM_ARRAY_JOB_ID variable.
532 write batch_script <job_id> [<optional_filename>]
535 Write the batch script for a given job_id to a file or to std‐
536 out. The file will default to slurm-<job_id>.sh if the optional
537 filename argument is not given. The script will be written to
538 stdout if - is given instead of a filename. The batch script
539 can only be retrieved by an admin or operator, or by the owner
540 of the job.
541 write config <optional_filename>
543 Write the current configuration to a file with the naming con‐
544 vention of "slurm.conf.<datetime>" in the same directory as the
545 original slurm.conf file. If a filename is given that file lo‐
546 cation with a .<datetime> suffix is created.
547
549 NOTE: All commands listed below can be used in the interactive mode,
550 but NOT on the initial command line.
551 all Show all partitions, their jobs and jobs steps. This causes in‐
554 formation to be displayed about partitions that are configured
555 as hidden and partitions that are unavailable to user's group.
556 details
558 Causes the show command to provide additional details where
559 available. Job information will include CPUs and NUMA memory
560 allocated on each node. Note that on computers with hyper‐
561 threading enabled and Slurm configured to allocate cores, each
562 listed CPU represents one physical core. Each hyperthread on
563 that core can be allocated a separate task, so a job's CPU count
564 and task count may differ. See the --cpu-bind and --mem-bind
565 option descriptions in srun man pages for more information. The
566 details option is currently only supported for the show job com‐
567 mand.
568 exit Terminate scontrol interactive session.
570 hide Do not display partition, job or jobs step information for par‐
572 titions that are configured as hidden or partitions that are un‐
573 available to the user's group. This is the default behavior.
574 oneliner
576 Print information one line per record.
577 quiet Print no warning or informational messages, only fatal error
579 messages.
580 quit Terminate the execution of scontrol.
582 verbose
584 Print detailed event logging. This includes time-stamps on data
585 structures, record counts, etc.
586 !! Repeat the last command executed.
588
590 Note that update requests done by either root, SlurmUser or Administra‐
591 tors are not subject to certain restrictions. For instance, if an Ad‐
592 ministrator changes the QOS on a pending job, certain limits such as
593 the TimeLimit will not be changed automatically as changes made by the
594 Administrators are allowed to violate these restrictions.
595 Account=<account>
598 Account name to be changed for this job's resource use. Value
599 may be cleared with blank data value, "Account=".
600 AdminComment=<spec>
602 Arbitrary descriptive string. Can only be set by a Slurm admin‐
603 istrator.
604 ArrayTaskThrottle=<count>
606 Specify the maximum number of tasks in a job array that can exe‐
607 cute at the same time. Set the count to zero in order to elimi‐
608 nate any limit. The task throttle count for a job array is re‐
609 ported as part of its ArrayTaskId field, preceded with a percent
610 sign. For example "ArrayTaskId=1-10%2" indicates the maximum
611 number of running tasks is limited to 2.
612 BurstBuffer=<spec>
614 Burst buffer specification to be changed for this job's resource
615 use. Value may be cleared with blank data value, "Burst‐
616 Buffer=". Format is burst buffer plugin specific.
617 Clusters=<spec>
619 Specifies the clusters that the federated job can run on.
620 ClusterFeatures=<spec>
622 Specifies features that a federated cluster must have to have a
623 sibling job submitted to it. Slurm will attempt to submit a sib‐
624 ling job to a cluster if it has at least one of the specified
625 features.
626 Comment=<spec>
628 Arbitrary descriptive string.
629 Contiguous={yes|no}
631 Set the job's requirement for contiguous (consecutive) nodes to
632 be allocated. Possible values are "YES" and "NO". Only the
633 Slurm administrator or root can change this parameter.
634 CoreSpec=<count>
636 Number of cores to reserve per node for system use. The job
637 will be charged for these cores, but be unable to use them.
638 Will be reported as "*" if not constrained.
639 CPUsPerTask=<count>
641 Change the CPUsPerTask job's value.
642 Deadline=<time_spec>
644 It accepts times of the form HH:MM:SS to specify a deadline to a
645 job at a specific time of day (seconds are optional). You may
646 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
647 you can have a time-of-day suffixed with AM or PM for a deadline
648 in the morning or the evening. You can specify a deadline for
649 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
650 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
651 give times like now + count time-units, where the time-units can
652 be seconds (default), minutes, hours, days, or weeks and you can
653 tell Slurm to put a deadline for tomorrow with the keyword to‐
654 morrow. The specified deadline must be later than the current
655 time. Only pending jobs can have the deadline updated. Only
656 the Slurm administrator or root can change this parameter.
657 DelayBoot=<time_spec>
659 Change the time to decide whether to reboot nodes in order to
660 satisfy job's feature specification if the job has been eligible
661 to run for less than this time period. See salloc/sbatch man
662 pages option --delay-boot.
663 Dependency=<dependency_list>
665 Defer job's initiation until specified job dependency specifica‐
666 tion is satisfied. Cancel dependency with an empty depen‐
667 dency_list (e.g. "Dependency="). <dependency_list> is of the
668 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
669 can share the same dependency and these jobs may even belong to
670 different users.
671 after:job_id[:jobid...]
673 This job can begin execution after the specified jobs
674 have begun execution or been canceled.
675 afterany:job_id[:jobid...]
677 This job can begin execution after the specified jobs
678 have terminated.
679 afternotok:job_id[:jobid...]
681 This job can begin execution after the specified jobs
682 have terminated in some failed state (non-zero exit code,
683 node failure, timed out, etc).
684 afterok:job_id[:jobid...]
686 This job can begin execution after the specified jobs
687 have successfully executed (ran to completion with an
688 exit code of zero).
689 singleton
691 This job can begin execution after any previously
692 launched jobs sharing the same job name and user have
693 terminated. In other words, only one job by that name
694 and owned by that user can be running or suspended at any
695 point in time.
696 EligibleTime=<time_spec>
698 See StartTime.
699 EndTime
701 The time the job is expected to terminate based on the job's
702 time limit. When the job ends sooner, this field will be up‐
703 dated with the actual end time.
704 ExcNodeList=<nodes>
706 Set the job's list of excluded node. Multiple node names may be
707 specified using simple node range expressions (e.g.
708 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
709 NodeList=".
710 Features=<features>
712 Set the job's required node features. The list of features may
713 include multiple feature names separated by ampersand (AND)
714 and/or vertical bar (OR) operators. For example: Fea‐
715 tures="opteron&video" or Features="fast|faster". In the first
716 example, only nodes having both the feature "opteron" AND the
717 feature "video" will be used. There is no mechanism to specify
718 that you want one node with feature "opteron" and another node
719 with feature "video" in case no node has both features. If only
720 one of a set of possible options should be used for all allo‐
721 cated nodes, then use the OR operator and enclose the options
722 within square brackets. For example: "Fea‐
723 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
724 all nodes must be allocated on a single rack of the cluster, but
725 any of those four racks can be used. A request can also specify
726 the number of nodes needed with some feature by appending an as‐
727 terisk and count after the feature name. For example "Fea‐
728 tures=graphics*4" indicates that at least four allocated nodes
729 must have the feature "graphics." Parenthesis are also sup‐
730 ported for features to be ANDed together. For example "Fea‐
731 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
732 cation should include 4 nodes with ALL of the features "knl",
733 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
734 straints with node counts may only be combined with AND opera‐
735 tors. Value may be cleared with blank data value, for example
736 "Features=".
737 Gres=<list>
739 Specifies a comma-delimited list of generic consumable re‐
740 sources. The format of each entry on the list is
741 "name[:count[*cpu]]". The name is that of the consumable re‐
742 source. The count is the number of those resources with a de‐
743 fault value of 1. The specified resources will be allocated to
744 the job on each node allocated unless "*cpu" is appended, in
745 which case the resources will be allocated on a per cpu basis.
746 The available generic consumable resources is configurable by
747 the system administrator. A list of available generic consum‐
748 able resources will be printed and the command will exit if the
749 option argument is "help". Examples of use include
750 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
751 JobId=<job_list>
753 Identify the job(s) to be updated. The job_list may be a comma
754 separated list of job IDs. Either JobId or JobName is required.
755 Licenses=<name>
757 Specification of licenses (or other resources available on all
758 nodes of the cluster) as described in salloc/sbatch/srun man
759 pages.
760 MailType=<types>
762 Set the mail event types. Valid type values are NONE, BEGIN,
763 END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, RE‐
764 QUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and
765 teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 per‐
766 cent of time limit), TIME_LIMIT_80 (reached 80 percent of time
767 limit), TIME_LIMIT_50 (reached 50 percent of time limit) and AR‐
768 RAY_TASKS (send emails for each array task). Multiple type val‐
769 ues may be specified in a comma separated list. Unless the AR‐
770 RAY_TASKS option is specified, mail notifications on job BEGIN,
771 END and FAIL apply to a job array as a whole rather than gener‐
772 ating individual email messages for each task in the job array.
773 MailUser=<name>
775 Set the user to receive email notification of state changes. A
776 blank string will set the mail user to the default which is the
777 submitting user.
778 MinCPUsNode=<count>
780 Set the job's minimum number of CPUs per node to the specified
781 value.
782 MinMemoryCPU=<megabytes>
784 Set the job's minimum real memory required per allocated CPU to
785 the specified value. Either MinMemoryCPU or MinMemoryNode may be
786 set, but not both.
787 MinMemoryNode=<megabytes>
789 Set the job's minimum real memory required per node to the spec‐
790 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
791 but not both.
792 MinTmpDiskNode=<megabytes>
794 Set the job's minimum temporary disk space required per node to
795 the specified value. Only the Slurm administrator or root can
796 change this parameter.
797 TimeMin=<timespec>
799 Change TimeMin value which specifies the minimum time limit min‐
800 utes of the job.
801 JobName=<name>
803 Identify the name of jobs to be modified or set the job's name
804 to the specified value. When used to identify jobs to be modi‐
805 fied, all jobs belonging to all users are modified unless the
806 UserID option is used to identify a specific user. Either JobId
807 or JobName is required.
808 Name[=<name>]
810 See JobName.
811 Nice[=<adjustment>]
813 Update the job with an adjusted scheduling priority within
814 Slurm. With no adjustment value the scheduling priority is de‐
815 creased by 100. A negative nice value increases the priority,
816 otherwise decreases it. The adjustment range is +/- 2147483645.
817 Only privileged users can specify a negative adjustment.
818 NodeList=<nodes>
820 Change the nodes allocated to a running job to shrink its size.
821 The specified list of nodes must be a subset of the nodes cur‐
822 rently allocated to the job. Multiple node names may be speci‐
823 fied using simple node range expressions (e.g. "lx[10-20]"). Af‐
824 ter a job's allocation is reduced, subsequent srun commands must
825 explicitly specify node and task counts which are valid for the
826 new allocation.
827 NumCPUs=<min_count>[-<max_count>]
829 Set the job's minimum and optionally maximum count of CPUs to be
830 allocated.
831 NumNodes=<min_count>[-<max_count>]
833 Set the job's minimum and optionally maximum count of nodes to
834 be allocated. If the job is already running, use this to spec‐
835 ify a node count less than currently allocated and resources
836 previously allocated to the job will be relinquished. After a
837 job's allocation is reduced, subsequent srun commands must ex‐
838 plicitly specify node and task counts which are valid for the
839 new allocation. Also see the NodeList parameter above. This is
840 the same as ReqNodes.
841 NumTasks=<count>
843 Set the job's count of requested tasks to the specified value.
844 The number of tasks started in a specific step inside the allo‐
845 cation may differ from this value, for instance when a different
846 number of tasks is requested on step creation. This is the same
847 as ReqProcs.
848 OverSubscribe={yes|no}
850 Set the job's ability to share compute resources (i.e. individ‐
851 ual CPUs) with other jobs. Possible values are "YES" and "NO".
852 This option can only be changed for pending jobs.
853 Partition=<name>
855 Set the job's partition to the specified value.
856 Prefer=<features>
858 Set the job's preferred node features. This list is only pre‐
859 ferred, not required like Features is. This list will override
860 what is requested in Features. See Features option above.
861 Priority=<number>
863 Set the job's priority to the specified value. Note that a job
864 priority of zero prevents the job from ever being scheduled. By
865 setting a job's priority to zero it is held. Set the priority
866 to a non-zero value to permit it to run. Explicitly setting a
867 job's priority clears any previously set nice value and removes
868 the priority/multifactor plugin's ability to manage a job's pri‐
869 ority. In order to restore the priority/multifactor plugin's
870 ability to manage a job's priority, hold and then release the
871 job. Only the Slurm administrator or root can increase job's
872 priority.
873 QOS=<name>
875 Set the job's QOS (Quality Of Service) to the specified value.
876 Value may be cleared with blank data value, "QOS=".
877 Reboot={yes|no}
879 Set the job's flag that specifies whether to force the allocated
880 nodes to reboot before starting the job. This is only supported
881 with some system configurations and therefore it could be
882 silently ignored.
883 ReqCores=<count>
885 Change the job's requested Cores count.
886 ReqNodeList=<nodes>
888 Set the job's list of required node. Multiple node names may be
889 specified using simple node range expressions (e.g.
890 "lx[10-20]"). Value may be cleared with blank data value, "Re‐
891 qNodeList=".
892 ReqNodes=<min_count>[-<max_count>]
894 See NumNodes.
895 ReqProcs=<count>
897 See NumTasks.
898 ReqSockets=<count>
900 Change the job's requested socket count.
901 ReqThreads=<count>
903 Change the job's requested threads count.
904 IRequeue={0|1}
906 Stipulates whether a job should be requeued after a node fail‐
907 ure: 0 for no, 1 for yes.
908 ReservationName=<name>
910 Set the job's reservation to the specified value. Value may be
911 cleared with blank data value, "ReservationName=".
912 ResetAccrueTime
914 Set the job's accrue time value to 'now' meaning it will lose
915 any time previously accrued for priority. Helpful if you have a
916 large queue of jobs already in the queue and want to start lim‐
917 iting how many jobs can accrue time without waiting for the
918 queue to flush out.
919 SiteFactor=<account>
921 Specify the job's admin priority factor in the range of
922 +/-2147483645. Only privileged users can modify the value.
923 StdOut=<filepath>
925 Set the batch job's stdout file path.
926 Shared={yes|no}
928 See OverSubscribe option above.
929 StartTime=<time_spec>
931 Set the job's earliest initiation time. It accepts times of the
932 form HH:MM:SS to run a job at a specific time of day (seconds
933 are optional). (If that time is already past, the next day is
934 assumed.) You may also specify midnight, noon, fika (3 PM) or
935 teatime (4 PM) and you can have a time-of-day suffixed with AM
936 or PM for running in the morning or the evening. You can also
937 say what day the job will be run, by specifying a date of the
938 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
939 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
940 count time-units, where the time-units can be seconds (default),
941 minutes, hours, days, or weeks and you can tell Slurm to run the
942 job today with the keyword today and to run the job tomorrow
943 with the keyword tomorrow.
944 Notes on date/time specifications:
946 - although the 'seconds' field of the HH:MM:SS time specifica‐
947 tion is allowed by the code, note that the poll time of the
948 Slurm scheduler is not precise enough to guarantee dispatch of
949 the job on the exact second. The job will be eligible to start
950 on the next poll following the specified time. The exact poll
951 interval depends on the Slurm scheduler (e.g., 60 seconds with
952 the default sched/builtin).
953 - if no time (HH:MM:SS) is specified, the default is
954 (00:00:00).
955 - if a date is specified without a year (e.g., MM/DD) then the
956 current year is assumed, unless the combination of MM/DD and
957 HH:MM:SS has already passed for that year, in which case the
958 next year is used.
959 Switches=<count>[@<max-time-to-wait>]
961 When a tree topology is used, this defines the maximum count of
962 switches desired for the job allocation. If Slurm finds an allo‐
963 cation containing more switches than the count specified, the
964 job remain pending until it either finds an allocation with de‐
965 sired switch count or the time limit expires. By default there
966 is no switch count limit and no time limit delay. Set the count
967 to zero in order to clean any previously set count (disabling
968 the limit). The job's maximum time delay may be limited by the
969 system administrator using the SchedulerParameters configuration
970 parameter with the max_switch_wait parameter option. Also see
971 wait-for-switch.
972 wait-for-switch=<seconds>
974 Change max time to wait for a switch <seconds> secs.
975 TasksPerNode=<count>
977 Change the job's requested TasksPerNode.
978 ThreadSpec=<count>
980 Number of threads to reserve per node for system use. The job
981 will be charged for these threads, but be unable to use them.
982 Will be reported as "*" if not constrained.
983 TimeLimit=<time>
985 The job's time limit. Output format is [days-]hours:min‐
986 utes:seconds or "UNLIMITED". Input format (for update command)
987 set is minutes, minutes:seconds, hours:minutes:seconds,
988 days-hours, days-hours:minutes or days-hours:minutes:seconds.
989 Time resolution is one minute and second values are rounded up
990 to the next minute. If changing the time limit of a job, either
991 specify a new time limit value or precede the time and equal
992 sign with a "+" or "-" to increment or decrement the current
993 time limit (e.g. "TimeLimit+=30"). In order to increment or
994 decrement the current time limit, the JobId specification must
995 precede the TimeLimit specification. Note that incrementing or
996 decrementing the time limit for a job array is only allowed be‐
997 fore the job array has been split into more than one job record.
998 Only the Slurm administrator or root can increase job's Time‐
999 Limit.
1000 UserID=<UID or name>
1002 Used with the JobName option to identify jobs to be modified.
1003 Either a user name or numeric ID (UID), may be specified.
1004 WCKey=<key>
1006 Set the job's workload characterization key to the specified
1007 value.
1008 WorkDir=<directory_name>
1010 Set the job's working directory to the specified value. Note
1011 that this may only be set for jobs in the PENDING state, and
1012 that jobs may fail to launch if they rely on relative paths to
1013 the originally submitted WorkDir.
1014
1016 The "show" command, when used with the "job" or "job <jobid>" entity
1017 displays detailed information about a job or jobs. Much of this infor‐
1018 mation may be modified using the "update job" command as described
1019 above. However, the following fields displayed by the show job command
1020 are read-only and cannot be modified:
1021 AllocNode:Sid
1024 Local node and system id making the resource allocation.
1025 BatchFlag
1027 Jobs submitted using the sbatch command have BatchFlag set to 1.
1028 Jobs submitted using other commands have BatchFlag set to 0.
1029 ExitCode=<exit>:<sig>
1031 Exit status reported for the job by the wait() function. The
1032 first number is the exit code, typically as set by the exit()
1033 function. The second number of the signal that caused the
1034 process to terminate if it was terminated by a signal.
1035 GroupId
1037 The group under which the job was submitted.
1038 JobState
1040 The current state of the job.
1041 NodeListIndices
1043 The NodeIndices expose the internal indices into the node table
1044 associated with the node(s) allocated to the job.
1045 NtasksPerN:B:S:C=<tasks_per_node>:<tasks_per_base‐
1047 board>:<tasks_per_socket>:<tasks_per_core>
1048 Specifies the number of tasks to be started per hardware compo‐
1049 nent (node, baseboard, socket and core). Unconstrained values
1050 may be shown as "0" or "*".
1051 PreemptEligibleTime
1053 Time the job becomes eligible for preemption. Modified by Pre‐
1054 emptExemptTime, either from the global option in slurm.conf or
1055 the job QOS. This is hidden if the job has not started or if
1056 PreemptMode=OFF.
1057 PreemptTime
1059 Time at which job was signaled that it was selected for preemp‐
1060 tion. This value is only meaningful for PreemptMode=CANCEL and
1061 PreemptMode=REQUEUE and for jobs in a partition or QOS that has
1062 a GraceTime value designated. This is hidden if the job has not
1063 started or if PreemptMode=OFF.
1064 PreSusTime
1066 Time the job ran prior to last suspend.
1067 Reason The reason job is not running: e.g., waiting "Resources".
1069 ReqB:S:C:T=<baseboard_count>:<socket_per_base‐
1071 board_count>:<core_per_socket_count>:<thread_per_core_count>
1072 Specifies the count of various hardware components requested by
1073 the job. Unconstrained values may be shown as "0" or "*".
1074 SecsPreSuspend=<seconds>
1076 If the job is suspended, this is the run time accumulated by the
1077 job (in seconds) prior to being suspended.
1078 Socks/Node=<count>
1080 Count of desired sockets per node
1081 SubmitTime
1083 The time and date stamp (in localtime) the job was submitted.
1084 The format of the output is identical to that of the EndTime
1085 field.
1086 NOTE: If a job is requeued, the submit time is reset. To obtain
1088 the original submit time it is necessary to use the "sacct -j
1089 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1090 cate option to display all duplicate entries for a job.
1091 SuspendTime
1093 Time the job was last suspended or resumed.
1094 NOTE on information displayed for various job states: When you
1096 submit a request for the "show job" function the scontrol
1097 process makes an RPC request call to slurmctld with a RE‐
1098 QUEST_JOB_INFO message type. If the state of the job is PEND‐
1099 ING, then it returns some detail information such as: min_nodes,
1100 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1101 ING the code assumes that it is in a further state such as RUN‐
1102 NING, COMPLETE, etc. In these cases the code explicitly returns
1103 zero for these values. These values are meaningless once the job
1104 resources have been allocated and the job has started.
1105
1107 StepId=<job_id>[.<step_id>]
1108 Identify the step to be updated. If the job_id is given, but no
1109 step_id is specified then all steps of the identified job will
1110 be modified. This specification is required.
1111 TimeLimit=<time>
1113 The job's time limit. Output format is [days-]hours:min‐
1114 utes:seconds or "UNLIMITED". Input format (for update command)
1115 set is minutes, minutes:seconds, hours:minutes:seconds,
1116 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1117 Time resolution is one minute and second values are rounded up
1118 to the next minute. If changing the time limit of a step, ei‐
1119 ther specify a new time limit value or precede the time with a
1120 "+" or "-" to increment or decrement the current time limit
1121 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1122 current time limit, the StepId specification must precede the
1123 TimeLimit specification.
1124
1126 Provide the same NodeName configuration as found in the slurm.conf. See
1127 slurm.conf man page for details. Only State=CLOUD and State=FUTURE
1128 nodes are allowed.
1129
1132 NodeName=<name>
1133 Identify the node(s) to be updated. Multiple node names may be
1134 specified using simple node range expressions (e.g.
1135 "lx[10-20]"). Nodesets can also be specified by themselves or
1136 mixed with node range expressions, using a comma as a list sepa‐
1137 rator. If the keyword "ALL" is specified alone, then the update
1138 will be attempted against all the nodes in the local cluster.
1139 This specification is required.
1140 ActiveFeatures=<features>
1142 Identify the feature(s) currently active on the specified node.
1143 Any previously active feature specification will be overwritten
1144 with the new value. Also see AvailableFeatures. Typically Ac‐
1145 tiveFeatures will be identical to AvailableFeatures; however Ac‐
1146 tiveFeatures may be configured as a subset of the AvailableFea‐
1147 tures. For example, a node may be booted in multiple configura‐
1148 tions. In that case, all possible configurations may be identi‐
1149 fied as AvailableFeatures, while ActiveFeatures would identify
1150 the current node configuration. When updating the ActiveFea‐
1151 tures with scontrol, the change is only made in slurmctld. When
1152 using a node_features plugin the state/features of the node must
1153 be updated on the node such that a new node start will report
1154 the updated state/features.
1155 AvailableFeatures=<features>
1157 Identify the feature(s) available on the specified node. Any
1158 previously defined available feature specification will be over‐
1159 written with the new value. AvailableFeatures assigned via
1160 scontrol will only persist across the restart of the slurmctld
1161 daemon with the -R option and state files preserved or slurm‐
1162 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1163 meant to be persistent across normal restarts of slurmctld or
1164 the execution of scontrol reconfig.
1165 Note: Available features being removed via scontrol must not be
1167 active (i.e. remove them from ActiveFeatures first).
1168 Comment=<comment>
1170 Arbitrary descriptive string. Use quotes to enclose a comment
1171 having more than one word
1172 CpuBind=<node>
1174 Specify the task binding mode to be used by default for this
1175 node. Supported options include: "none", "socket", "ldom"
1176 (NUMA), "core", "thread" and "off" (remove previous binding
1177 mode).
1178 Extra=<comment>
1180 Arbitrary string on the node. Use quotes to enclose a string
1181 having more than one word.
1182 Gres=<gres>
1184 Identify generic resources to be associated with the specified
1185 node. Any previously defined generic resources will be over‐
1186 written with the new value. Specifications for multiple generic
1187 resources should be comma separated. Each resource specifica‐
1188 tion consists of a name followed by an optional colon with a nu‐
1189 meric value (default value is one) (e.g. "Gres=band‐
1190 width:10000"). Modification of GRES count associated with spe‐
1191 cific files (e.g. GPUs) is not allowed other than to set their
1192 count on a node to zero. In order to change the GRES count to
1193 another value, modify your slurm.conf and gres.conf files and
1194 restart daemons. If GRES are associated with specific sockets,
1195 that information will be reported For example if all 4 GPUs on a
1196 node are all associated with socket zero, then
1197 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1198 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1199 associated with specific GPUs is not reported, but only avail‐
1200 able by parsing the gres.conf file. Generic resources assigned
1201 via scontrol will only persist across the restart of the slurm‐
1202 ctld daemon with the -R option and state files preserved or
1203 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1204 changes meant to be persistent across normal restarts of slurm‐
1205 ctld or the execution of scontrol reconfig.
1206 NodeAddr=<node address>
1208 Name that a node should be referred to in establishing a commu‐
1209 nications path. This name will be used as an argument to the
1210 getaddrinfo() function for identification. If a node range ex‐
1211 pression is used to designate multiple nodes, they must exactly
1212 match the entries in the NodeName (e.g. "NodeName=lx[0-7]
1213 NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses.
1214 NodeHostname=<node hostname>
1216 Typically this would be the string that "/bin/hostname -s" re‐
1217 turns. It may also be the fully qualified domain name as re‐
1218 turned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid
1219 domain name associated with the host through the host database
1220 (/etc/hosts) or DNS, depending on the resolver settings. Note
1221 that if the short form of the hostname is not used, it may pre‐
1222 vent use of hostlist expressions (the numeric portion in brack‐
1223 ets must be at the end of the string). A node range expression
1224 can be used to specify a set of nodes. If an expression is used,
1225 the number of nodes identified by NodeHostname must be identical
1226 to the number of nodes identified by NodeName.
1227 Reason=<reason>
1229 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1230 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1231 having more than one word.
1232 State=<state>
1234 Assign one of the following states/actions to the node(s) speci‐
1235 fied by the update command.
1236 CANCEL_REBOOT
1238 Cancels a pending reboot on the node (same as scontrol
1239 cancel_reboot <node>).
1240 DOWN Stop all running and suspended jobs and make the node un‐
1242 available for new jobs.
1243 DRAIN Indicates that no new jobs may be started on this node.
1245 Existing jobs are allowed to run to completion, leaving
1246 the node in a DRAINED state once all the jobs have com‐
1247 pleted.
1248 FAIL Similar to DRAIN except that some applications will seek
1250 to relinquish those nodes before the job completes.
1251 FUTURE Indicates the node is not fully configured, but is ex‐
1253 pected to be available at some point in the future.
1254 NoResp This will set the "Not Responding" flag for a node with‐
1256 out changing its underlying state.
1257 POWER_DOWN
1259 Will use the configured SuspendProgram program to explic‐
1260 itly place a node in power saving mode. If a node is al‐
1261 ready in the process of being powered down, the command
1262 will only change the state of the node but won't have any
1263 effect until the configured SuspendTimeout is reached.
1264 Use of this command can be useful in situations where a
1265 ResumeProgram, like capmc in Cray machines, is stalled
1266 and one wants to restore the node to "IDLE" manually. In
1267 this case rebooting the node and setting the state to
1268 "POWER_DOWN" will cancel the previous "POWER_UP" state
1269 and the node will become "IDLE".
1270 POWER_DOWN_ASAP
1272 Will drain the node and mark it for power down. Currently
1273 running jobs will complete first and no additional jobs
1274 will be allocated to the node.
1275 POWER_DOWN_FORCE
1277 Will cancel all jobs on the node, power it down, and re‐
1278 set its state to "IDLE".
1279 POWER_UP
1281 Will use the configured ResumeProgram program to explic‐
1282 itly move a node out of power saving mode. If a node is
1283 already in the process of being powered up, the command
1284 will only change the state of the node but won't have any
1285 effect until the configured ResumeTimeout is reached.
1286 RESUME Not an actual node state, but will change a node state
1288 from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp.
1289 slurmctld will then attempt to contact slurmd to request
1290 that the node register itself. Once registered, the node
1291 state will then remove the NoResp flag and will resume
1292 normal operations. It will also clear the POWERING_DOWN
1293 state of a node and make it eligible to be allocted.
1294 UNDRAIN
1296 Clears the node from being drained (like RESUME), but
1297 will not change the node's base state (e.g. DOWN).
1298 UNDRAIN requires a valid node registration before new
1299 jobs can be scheduled on the node. Setting a node DOWN
1300 will cause all running and suspended jobs on that node to
1301 be terminated.
1302 While all of the above states are valid, some of them are not
1304 valid new node states given their prior state.
1305 NOTE: The scontrol command should not be used to change node
1307 state on Cray systems. Use Cray tools such as xtprocadmin in‐
1308 stead.
1309 Weight=<weight>
1311 Identify weight to be associated with specified nodes. This al‐
1312 lows dynamic changes to weight associated with nodes, which will
1313 be used for the subsequent node allocation decisions. Weight
1314 assigned via scontrol will only persist across the restart of
1315 the slurmctld daemon with the -R option and state files pre‐
1316 served or slurmctld's receipt of a SIGHUP. Update slurm.conf
1317 with any changes meant to be persistent across normal restarts
1318 of slurmctld or the execution of scontrol reconfig.
1319
1321 NodeName=<nodes>
1322 Identify the node(s) to be deleted. Multiple node names may be
1323 specified using simple node range expressions (e.g.
1324 "lx[10-20]"). Nodesets can also be specified by themselves or
1325 mixed with node range expressions, using a comma as a list sepa‐
1326 rator. If the keyword "ALL" is specified alone, then the update
1327 will be attempted against all the nodes in the local cluster.
1328 This specification is required.
1329
1331 AllocMem
1332 The total memory, in MB, currently allocated by jobs on the
1333 node.
1334 CPULoad
1336 CPU load of a node as reported by the OS.
1337 CPUSpecList
1339 The list of Slurm abstract CPU IDs on this node reserved for ex‐
1340 clusive use by the Slurm compute node daemons (slurmd, slurm‐
1341 stepd).
1342 FreeMem
1344 The total memory, in MB, currently free on the node as reported
1345 by the OS.
1346 LastBusyTime
1348 The last time the node was busy (i.e. last time the node had
1349 jobs on it). This time is used in PowerSave to determine when to
1350 suspend nodes (e.g. now - LastBusy > SuspendTime).
1351 MemSpecLimit
1353 The combined memory limit, in megabytes, on this node for the
1354 Slurm compute node daemons (slurmd, slurmstepd).
1355 RealMemory
1357 The total memory, in MB, on the node.
1358 State Identify the state(s) assigned to the node with '+' delimited
1360 state flags.
1361 States:
1363 ALLOCATED
1365 Indicates that the node has all CPUs allocated to job(s)
1366 running on the node.
1367 DOWN The node does not have any running jobs and is unavail‐
1369 able for new work.
1370 ERROR The node is in an error state. Consult the logs for more
1372 information about what caused this state.
1373 FUTURE The node is currently not fully configured, but expected
1375 to be available at some point in the indefinite future
1376 for use.
1377 IDLE Indicates that the node is available for work but does
1379 not currently have any jobs assigned to it.
1380 MIXED Indicates that the node is in multiple states. For in‐
1382 stance if only part of the node is ALLOCATED and the rest
1383 of the node is IDLE the state will be MIXED.
1384 UNKNOWN
1386 The node has not yet registered with the controller and
1387 its state is not known.
1388 Flags:
1390 CLOUD Indicates that the node is configured as a cloud node, to
1392 be brought up on demand, but not currently running.
1393 COMPLETING
1395 Indicates that the only job on the node or that all jobs
1396 on the node are in the process of completing.
1397 DRAIN The node is not accepting any new jobs and any currently
1399 running jobs will complete.
1400 DYNAMIC
1402 Slurm allows you to define multiple types of nodes in a
1403 FUTURE state. When starting slurmd on a node you can
1404 specify the -F flag to have the node match and use an ex‐
1405 isting definition in your slurm.conf file. The DYNAMIC
1406 state indicates that the node was started as a Dynamic
1407 Future node.
1408 INVALID_REG
1410 The node did not register correctly with the controller.
1411 This happens when a node registers with less resources
1412 than configured in the slurm.conf file. The node will
1413 clear from this state with a valid registration (i.e. a
1414 slurmd restart is required).
1415 MAINTENANCE
1417 The node is currently in a reservation that includes the
1418 maintenance flag.
1419 NOT_RESPONDING
1421 Node is not responding.
1422 PERFCTRS
1424 Indicates that Network Performance Counters associated
1425 with this node are in use, rendering this node as not us‐
1426 able for any other jobs.
1427 POWER_DOWN
1429 Node is pending power down.
1430 POWERED_DOWN
1432 Node is currently powered down and not capable of running
1433 any jobs.
1434 POWERING_DOWN
1436 Node is in the process of powering down.
1437 POWERING_UP
1439 Node is in the process of powering up.
1440 PLANNED
1442 The node is earmarked for a job that will start in the
1443 future.
1444 REBOOT_ISSUED
1446 A reboot request has been sent to the agent configured to
1447 handle this request.
1448 REBOOT_REQUESTED
1450 A request to reboot this node has been made, but hasn't
1451 been handled yet.
1452 RESERVED
1454 Indicates the node is in an advanced reservation and not
1455 generally available.
1456 The meaning of the energy information is as follows:
1458 CurrentWatts
1460 The instantaneous power consumption of the node at the
1461 time of the last node energy accounting sample, in watts.
1462 LowestJoules
1464 The energy consumed by the node between the last time it
1465 was powered on and the last time it was registered by
1466 slurmd, in joules.
1467 ConsumedJoules
1469 The energy consumed by the node between the last time it
1470 was registered by the slurmd daemon and the last node en‐
1471 ergy accounting sample, in joules.
1472 If the reported value is "n/s" (not supported), the node does
1474 not support the configured AcctGatherEnergyType plugin. If the
1475 reported value is zero, energy accounting for nodes is disabled.
1476 The meaning of the external sensors information is as follows:
1479 ExtSensorsJoules
1481 The energy consumed by the node between the last time it
1482 was powered on and the last external sensors plugin node
1483 sample, in joules.
1484 ExtSensorsWatts
1486 The instantaneous power consumption of the node at the
1487 time of the last external sensors plugin node sample, in
1488 watts.
1489 ExtSensorsTemp
1491 The temperature of the node at the time of the last ex‐
1492 ternal sensors plugin node sample, in celsius.
1493 If the reported value is "n/s" (not supported), the node does
1495 not support the configured ExtSensorsType plugin.
1496
1498 FrontendName=<name>
1499 Identify the front end node to be updated. This specification is
1500 required.
1501 Reason=<reason>
1503 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1504 Use quotes to enclose a reason having more than one word.
1505 State=<state>
1507 Identify the state to be assigned to the front end node. Possi‐
1508 ble values are "DOWN", "DRAIN" or "RESUME". If you want to re‐
1509 move a front end node from service, you typically want to set
1510 its state to "DRAIN". "RESUME" is not an actual node state, but
1511 will return a "DRAINED", "DRAINING", or "DOWN" front end node to
1512 service, either "IDLE" or "ALLOCATED" state as appropriate.
1513 Setting a front end node "DOWN" will cause all running and sus‐
1514 pended jobs on that node to be terminated.
1515
1517 AllocNodes=<name>
1518 Comma separated list of nodes from which users can execute jobs
1519 in the partition. Node names may be specified using the node
1520 range expression syntax described above. The default value is
1521 "ALL".
1522 AllowAccounts=<name>
1524 Identify the Accounts which may use this partition. Multiple
1525 Accounts may be specified in a comma separated list. To permit
1526 all Accounts to use the partition specify "AllowAccounts=ALL".
1527 AllowGroups=<name>
1529 Identify the user groups which may use this partition. Multiple
1530 groups may be specified in a comma separated list. To permit
1531 all groups to use the partition specify "AllowGroups=ALL".
1532 AllowQOS=<name>
1534 Identify the QOS's which may use this partition. Multiple QOS's
1535 may be specified in a comma separated list. To permit all QOS's
1536 to use the partition specify "AllowQOS=ALL".
1537 Alternate=<partition name>
1539 Alternate partition to be used if the state of this partition is
1540 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1541 set alternate partition.
1542 CpuBind=<node>
1544 Specify the task binding mode to be used by default for this
1545 partition. Supported options include: "none", "socket", "ldom"
1546 (NUMA), "core", "thread" and "off" (remove previous binding
1547 mode).
1548 Default={yes|no}
1550 Specify if this partition is to be used by jobs which do not ex‐
1551 plicitly identify a partition to use. Possible output values
1552 are "YES" and "NO". In order to change the default partition of
1553 a running system, use the scontrol update command and set De‐
1554 fault=yes for the partition that you want to become the new de‐
1555 fault.
1556 DefaultTime=<time>
1558 Run time limit used for jobs that don't specify a value. If not
1559 set then MaxTime will be used. Format is the same as for Max‐
1560 Time.
1561 DefMemPerCPU=<MB>
1563 Set the default memory to be allocated per CPU for jobs in this
1564 partition. The memory size is specified in megabytes.
1565 DefMemPerNode=<MB>
1567 Set the default memory to be allocated per node for jobs in this
1568 partition. The memory size is specified in megabytes.
1569 DenyAccounts=<name>
1571 Identify the Accounts which should be denied access to this par‐
1572 tition. Multiple Accounts may be specified in a comma separated
1573 list.
1574 DenyQOS=<name>
1576 Identify the QOS's which should be denied access to this parti‐
1577 tion. Multiple QOS's may be specified in a comma separated
1578 list.
1579 DisableRootJobs={yes|no}
1581 Specify if jobs can be executed as user root. Possible values
1582 are "YES" and "NO".
1583 ExclusiveUser={yes|no}
1585 When enabled nodes will be exclusively allocated to users. Mul‐
1586 tiple jobs can be run simultaneously, but those jobs must be
1587 from a single user.
1588 GraceTime=<seconds>
1590 Specifies, in units of seconds, the preemption grace time to be
1591 extended to a job which has been selected for preemption. The
1592 default value is zero, no preemption grace time is allowed on
1593 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1594 Hidden={yes|no}
1596 Specify if the partition and its jobs should be hidden from
1597 view. Hidden partitions will by default not be reported by
1598 Slurm APIs or commands. Possible values are "YES" and "NO".
1599 JobDefaults=<specs>
1601 Specify job default values using a comma-delimited list of
1602 "key=value" pairs. Supported keys include
1603 DefCpuPerGPU Default number of CPUs per allocated GPU.
1605 DefMemPerGPU Default memory limit (in megabytes) per allocated
1607 GPU.
1608 MaxCPUsPerNode=<count>
1610 Set the maximum number of CPUs that can be allocated per node to
1611 all jobs in this partition.
1612 LLN={yes|no}
1614 Schedule jobs on the least loaded nodes (based on the number of
1615 idle CPUs).
1616 MaxMemPerCPU=<MB>
1618 Set the maximum memory to be allocated per CPU for jobs in this
1619 partition. The memory size is specified in megabytes.
1620 MaxMemPerNode=<MB>
1622 Set the maximum memory to be allocated per node for jobs in this
1623 partition. The memory size is specified in megabytes.
1624 MaxNodes=<count>
1626 Set the maximum number of nodes which will be allocated to any
1627 single job in the partition. Specify a number, "INFINITE" or
1628 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1629 upon jobs that have already begun execution.
1630 MaxTime=<time>
1632 The maximum run time for jobs. Output format is
1633 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1634 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1635 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1636 onds. Time resolution is one minute and second values are
1637 rounded up to the next minute. Changing the MaxTime of a parti‐
1638 tion has no effect upon jobs that have already begun execution.
1639 MinNodes=<count>
1641 Set the minimum number of nodes which will be allocated to any
1642 single job in the partition. Changing the MinNodes of a parti‐
1643 tion has no effect upon jobs that have already begun execution.
1644 Increasing this value may prevent pending jobs from starting,
1645 even if they were submitted without -N/--nodes specification.
1646 If you do get in that situation, updating the MinNodes value of
1647 a pending job using the scontrol command will allow that job to
1648 be scheduled.
1649 Nodes=<name>
1651 Identify the node(s) to be associated with this partition. Mul‐
1652 tiple node names may be specified using simple node range ex‐
1653 pressions (e.g. "lx[10-20]"). Note that jobs may only be asso‐
1654 ciated with one partition at any time. Specify a blank data
1655 value to remove all nodes from a partition: "Nodes=". Changing
1656 the Nodes in a partition has no effect upon jobs that have al‐
1657 ready begun execution.
1658 OverSubscribe={yes|no|exclusive|force}[:<job_count>]
1660 Specify if compute resources (i.e. individual CPUs) in this par‐
1661 tition can be shared by multiple jobs. Possible values are
1662 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1663 specifies how many jobs can be allocated to use each resource.
1664 OverTimeLimit=<count>
1666 Number of minutes by which a job can exceed its time limit be‐
1667 fore being canceled. The configured job time limit is treated
1668 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1669 vides a hard limit, at which point the job is canceled. This is
1670 particularly useful for backfill scheduling, which bases upon
1671 each job's soft time limit. A partition-specific OverTimeLimit
1672 will override any global OverTimeLimit value. If not specified,
1673 the global OverTimeLimit value will take precedence. May not
1674 exceed 65533 minutes. An input value of "UNLIMITED" will clear
1675 any previously configured partition-specific OverTimeLimit
1676 value.
1677 PartitionName=<name>
1679 Identify the partition to be updated. This specification is re‐
1680 quired.
1681 PreemptMode=<mode>
1683 Reset the mechanism used to preempt jobs in this partition if
1684 PreemptType is configured to preempt/partition_prio. The default
1685 preemption mechanism is specified by the cluster-wide Preempt‐
1686 Mode configuration parameter. Possible values are "OFF", "CAN‐
1687 CEL", "REQUEUE" and "SUSPEND".
1688 Priority=<count>
1690 Jobs submitted to a higher priority partition will be dispatched
1691 before pending jobs in lower priority partitions and if possible
1692 they will preempt running jobs from lower priority partitions.
1693 Note that a partition's priority takes precedence over a job's
1694 priority. The value may not exceed 65533.
1695 PriorityJobFactor=<count>
1697 Partition factor used by priority/multifactor plugin in calcu‐
1698 lating job priority. The value may not exceed 65533. Also see
1699 PriorityTier.
1700 PriorityTier=<count>
1702 Jobs submitted to a partition with a higher priority tier value
1703 will be dispatched before pending jobs in partition with lower
1704 priority tier value and, if possible, they will preempt
1705 running jobs from partitions with lower priority tier values.
1706 Note that a partition's priority tier takes precedence over a
1707 job's priority. The value may not exceed 65533. Also see Pri‐
1708 orityJobFactor.
1709 QOS=<QOSname|blank to remove>
1711 Set the partition QOS with a QOS name or to remove the Partition
1712 QOS leave the option blank.
1713 ReqResv={yes|no}
1715 Specify if only allocation requests designating a reservation
1716 will be satisfied. This is used to restrict partition usage to
1717 be allowed only within a reservation. Possible values are "YES"
1718 and "NO".
1719 RootOnly={yes|no}
1721 Specify if only allocation requests initiated by user root will
1722 be satisfied. This can be used to restrict control of the par‐
1723 tition to some meta-scheduler. Possible values are "YES" and
1724 "NO".
1725 Shared={yes|no|exclusive|force}[:<job_count>]
1727 Renamed to OverSubscribe, see option descriptions above.
1728 State={up|down|drain|inactive}
1730 Specify if jobs can be allocated nodes or queued in this parti‐
1731 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1732 UP Designates that new jobs may queued on the partition,
1734 and that jobs may be allocated nodes and run from the
1735 partition.
1736 DOWN Designates that new jobs may be queued on the parti‐
1738 tion, but queued jobs may not be allocated nodes and
1739 run from the partition. Jobs already running on the
1740 partition continue to run. The jobs must be explicitly
1741 canceled to force their termination.
1742 DRAIN Designates that no new jobs may be queued on the par‐
1744 tition (job submission requests will be denied with an
1745 error message), but jobs already queued on the parti‐
1746 tion may be allocated nodes and run. See also the
1747 "Alternate" partition specification.
1748 INACTIVE Designates that no new jobs may be queued on the par‐
1750 tition, and jobs already queued may not be allocated
1751 nodes and run. See also the "Alternate" partition
1752 specification.
1753 TRESBillingWeights=<TRES Billing Weights>
1755 TRESBillingWeights is used to define the billing weights of each
1756 TRES type that will be used in calculating the usage of a job.
1757 The calculated usage is used when calculating fairshare and when
1758 enforcing the TRES billing limit on jobs. Updates affect new
1759 jobs and not existing jobs. See the slurm.conf man page for
1760 more information.
1761
1763 Reservation=<name>
1764 Identify the name of the reservation to be created, updated, or
1765 deleted. This parameter is required for update and is the only
1766 parameter for delete. For create, if you do not want to give a
1767 reservation name, use "scontrol create reservation ..." and a
1768 name will be created automatically.
1769 Accounts=<account list>
1771 List of accounts permitted to use the reserved nodes, for exam‐
1772 ple "Accounts=physcode1,physcode2". A user in any of the ac‐
1773 counts may use the reserved nodes. A new reservation must spec‐
1774 ify Users or Groups and/or Accounts. If both Users/Groups and
1775 Accounts are specified, a job must match both in order to use
1776 the reservation. Accounts can also be denied access to reserva‐
1777 tions by preceding all of the account names with '-'. Alter‐
1778 nately precede the equal sign with '-'. For example, "Ac‐
1779 counts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2"
1780 will permit any account except physcode1 and physcode2 to use
1781 the reservation. You can add or remove individual accounts from
1782 an existing reservation by using the update command and adding a
1783 '+' or '-' sign before the '=' sign. If accounts are denied ac‐
1784 cess to a reservation (account name preceded by a '-'), then all
1785 other accounts are implicitly allowed to use the reservation and
1786 it is not possible to also explicitly specify allowed accounts.
1787 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1789 Specification of burst buffer resources which are to be re‐
1790 served. "buffer_spec" consists of four elements:
1791 [plugin:][type:]#[units] "plugin" is the burst buffer plugin
1792 name, currently either "datawarp" or "generic". If no plugin is
1793 specified, the reservation applies to all configured burst buf‐
1794 fer plugins. "type" specifies a Cray generic burst buffer re‐
1795 source, for example "nodes". if "type" is not specified, the
1796 number is a measure of storage space. The "units" may be "N"
1797 (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers
1798 of 1024) and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).
1799 The default units are bytes for reservations of storage space.
1800 For example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage
1801 plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (re‐
1802 serve 100 GB of storage from all configured burst buffer plug‐
1803 ins). Jobs using this reservation are not restricted to these
1804 burst buffer resources, but may use these reserved resources
1805 plus any which are generally available. NOTE: Usually Slurm in‐
1806 terprets KB, MB, GB, TB, PB, TB units as powers of 1024, but for
1807 Burst Buffers size specifications Slurm supports both IEC/SI
1808 formats. This is because the CRAY API for managing DataWarps
1809 supports both formats.
1810 CoreCnt=<num>
1812 This option is only supported when SelectType=select/cons_res or
1813 select/cons_tres. Identify number of cores to be reserved. If
1814 NodeCnt is used without the FIRST_CORES flag, this is the total
1815 number of cores to reserve where cores per node is
1816 CoreCnt/NodeCnt. If a nodelist is used, or if NodeCnt is used
1817 with the FIRST_CORES flag, this should be an array of core num‐
1818 bers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or
1819 flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
1820 Licenses=<license>
1822 Specification of licenses (or other resources available on all
1823 nodes of the cluster) which are to be reserved. License names
1824 can be followed by a colon and count (the default count is one).
1825 Multiple license names should be comma separated (e.g. "Li‐
1826 censes=foo:4,bar"). A new reservation must specify one or more
1827 resource to be included: NodeCnt, Nodes and/or Licenses. If a
1828 reservation includes Licenses, but no NodeCnt or Nodes, then the
1829 option Flags=LICENSE_ONLY must also be specified. Jobs using
1830 this reservation are not restricted to these licenses, but may
1831 use these reserved licenses plus any which are generally avail‐
1832 able.
1833 MaxStartDelay[=<timespec>]
1835 Change MaxStartDelay value which specifies the maximum time an
1836 eligible job not requesting this reservation can delay a job re‐
1837 questing it. Default is none. Valid formats are minutes, min‐
1838 utes:seconds, hours:minutes:seconds, days-hours, days-hours:min‐
1839 utes, days-hours:minutes:seconds. Time resolution is one minute
1840 and second values are rounded up to the next minute. Output for‐
1841 mat is always [days-]hours:minutes:seconds.
1842 NodeCnt=<num>[,<num>,...]
1844 Identify number of nodes to be reserved. The number can include
1845 a suffix of "k" or "K", in which case the number specified is
1846 multiplied by 1024. A new reservation must specify one or more
1847 resource to be included: NodeCnt, Nodes and/or Licenses.
1848 Nodes=<name>
1850 Identify the node(s) to be reserved. Multiple node names may be
1851 specified using simple node range expressions (e.g.
1852 "Nodes=lx[10-20]"). When using Nodes to specify more or fewer
1853 nodes, NodeCnt will be updated to honor the new number of nodes.
1854 However, when setting an empty list ("Nodes="), the nodelist
1855 will be filled with random nodes to fulfill the previous nodecnt
1856 and the SPEC_NODES flag will be removed. A new reservation must
1857 specify one or more resource to be included: NodeCnt, Nodes
1858 and/or Licenses. A specification of "ALL" will reserve all
1859 nodes. Set Flags=PART_NODES and PartitionName= in order for
1860 changes in the nodes associated with a partition to also be re‐
1861 flected in the nodes associated with a reservation.
1862 NOTE: When updating a reservation, if Nodes and Nodecnt are set
1864 simultaneously, nodecnt will always be honored. The reservation
1865 will get a subset of nodes if nodes > nodecnt, or it will add
1866 extra nodes to the list when nodes < nodecnt.
1867 StartTime=<time_spec>
1869 The start time for the reservation. A new reservation must
1870 specify a start time. It accepts times of the form HH:MM:SS for
1871 a specific time of day (seconds are optional). (If that time is
1872 already past, the next day is assumed.) You may also specify
1873 midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
1874 time-of-day suffixed with AM or PM for running in the morning or
1875 the evening. You can also say what day the job will be run, by
1876 specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or
1877 a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also give
1878 times like now + count time-units, where the time-units can be
1879 seconds (default), minutes, hours, days, or weeks and you can
1880 tell Slurm to run the job today with the keyword today and to
1881 run the job tomorrow with the keyword tomorrow. You cannot up‐
1882 date the StartTime of a reservation in ACTIVE state.
1883 EndTime=<time_spec>
1885 The end time for the reservation. A new reservation must spec‐
1886 ify an end time or a duration. Valid formats are the same as
1887 for StartTime.
1888 Duration=<time>
1890 The length of a reservation. A new reservation must specify an
1891 end time or a duration. Valid formats are minutes, minutes:sec‐
1892 onds, hours:minutes:seconds, days-hours, days-hours:minutes,
1893 days-hours:minutes:seconds, or UNLIMITED. Time resolution is
1894 one minute and second values are rounded up to the next minute.
1895 Output format is always [days-]hours:minutes:seconds.
1896 PartitionName=<name>
1898 Partition used to reserve nodes from. This will attempt to allo‐
1899 cate all nodes in the specified partition unless you request
1900 fewer resources than are available with CoreCnt, NodeCnt or
1901 TRES. Jobs will be allowed to use this reservation even if run‐
1902 ning in a different partition. There only needs to be overlap‐
1903 ping nodes from that different partition and the nodes used in
1904 the reservation.
1905 Flags=<flags>
1907 Flags associated with the reservation. You can add or remove
1908 individual flags from an existing reservation by adding a '+' or
1909 '-' sign before the '=' sign. For example: Flags-=DAILY (NOTE:
1910 this shortcut is not supported for all flags). Currently sup‐
1911 ported flags include:
1912 ANY_NODES This is a reservation for burst buffers and/or li‐
1914 censes only and not compute nodes. If this flag
1915 is set, a job using this reservation may use the
1916 associated burst buffers and/or licenses plus any
1917 compute nodes. If this flag is not set, a job us‐
1918 ing this reservation may use only the nodes and
1919 licenses associated with the reservation.
1920 DAILY Repeat the reservation at the same time every day.
1922 FLEX Permit jobs requesting the reservation to begin
1924 prior to the reservation's start time, end after
1925 the reservation's end time, and use any resources
1926 inside and/or outside of the reservation regard‐
1927 less of any constraints possibly set in the reser‐
1928 vation. A typical use case is to prevent jobs not
1929 explicitly requesting the reservation from using
1930 those reserved resources rather than forcing jobs
1931 requesting the reservation to use those resources
1932 in the time frame reserved. Another use case could
1933 be to always have a particular number of nodes
1934 with a specific feature reserved for a specific
1935 account so users in this account may use this
1936 nodes plus possibly other nodes without this fea‐
1937 ture.
1938 FIRST_CORES Use the lowest numbered cores on a node only. Flag
1940 removal with '-=' is not supported.
1941 IGNORE_JOBS Ignore currently running jobs when creating the
1943 reservation. This can be especially useful when
1944 reserving all nodes in the system for maintenance.
1945 HOURLY Repeat the reservation at the same time every
1947 hour.
1948 LICENSE_ONLY See ANY_NODES.
1950 MAINT Maintenance mode, receives special accounting
1952 treatment. This reservation is permitted to use
1953 resources that are already in another reservation.
1954 MAGNETIC This flag allows jobs to be considered for this
1956 reservation even if they didn't request it.
1957 NO_HOLD_JOBS_AFTER
1959 By default, when a reservation ends the reserva‐
1960 tion request will be removed from any pending jobs
1961 submitted to the reservation and will be put into
1962 a held state. Use this flag to let jobs run out‐
1963 side of the reservation after the reservation is
1964 gone. Flag removal with '-=' is not supported.
1965 OVERLAP This reservation can be allocated resources that
1967 are already in another reservation. Flag removal
1968 with '-=' is not supported.
1969 PART_NODES This flag can be used to reserve all nodes within
1971 the specified partition. PartitionName and
1972 Nodes=ALL must be specified with this flag.
1973 PURGE_COMP[=<timespec>]
1975 Purge the reservation if it is ever idle for time‐
1976 spec (no jobs associated with it). If timespec
1977 isn't given then 5 minutes is the default. Valid
1978 timespec formats are minutes, minutes:seconds,
1979 hours:minutes:seconds, days-hours, days-hours:min‐
1980 utes, days-hours:minutes:seconds. Time resolution
1981 is one minute and second values are rounded up to
1982 the next minute. Output format is always
1983 [days-]hours:minutes:seconds.
1984 REPLACE Nodes which are DOWN, DRAINED, or allocated to
1986 jobs are automatically replenished using idle re‐
1987 sources. This option can be used to maintain a
1988 constant number of idle resources available for
1989 pending jobs (subject to availability of idle re‐
1990 sources). This should be used with the NodeCnt
1991 reservation option; do not identify specific nodes
1992 to be included in the reservation. Flag removal
1993 with '-=' is not supported.
1994 NOTE: Removing a node from the cluster while in a
1996 reservation with the REPLACE flag will not cause
1997 it to be replaced.
1998 REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically
2000 replenished using idle resources. This option can
2001 be used to maintain a constant sized pool of re‐
2002 sources available for pending jobs (subject to
2003 availability of idle resources). This should be
2004 used with the NodeCnt reservation option; do not
2005 identify specific nodes to be included in the
2006 reservation. Flag removal with '-=' is not sup‐
2007 ported.
2008 NOTE: Removing a node from the cluster while in a
2010 reservation with the REPLACE_DOWN flag will not
2011 cause it to be replaced.
2012 SPEC_NODES Reservation is for specific nodes (output only).
2014 STATIC_ALLOC Make it so after the nodes are selected for a
2016 reservation they don't change. Without this op‐
2017 tion when nodes are selected for a reservation and
2018 one goes down the reservation will select a new
2019 node to fill the spot.
2020 TIME_FLOAT The reservation start time is relative to the cur‐
2022 rent time and moves forward through time (e.g. a
2023 StartTime=now+10minutes will always be 10 minutes
2024 in the future). Repeating (e.g. DAILY) floating
2025 reservations are not supported. Flag cannot be
2026 added to or removed from an existing reservation.
2027 WEEKDAY Repeat the reservation at the same time on every
2029 weekday (Monday, Tuesday, Wednesday, Thursday and
2030 Friday).
2031 WEEKEND Repeat the reservation at the same time on every
2033 weekend day (Saturday and Sunday).
2034 WEEKLY Repeat the reservation at the same time every
2036 week.
2037 Features=<features>
2039 Set the reservation's required node features. Multiple values
2040 may be "&" separated if all features are required (AND opera‐
2041 tion) or separated by "|" if any of the specified features are
2042 required (OR operation). Parenthesis are also supported for
2043 features to be ANDed together with counts of nodes having the
2044 specified features. For example "Fea‐
2045 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
2046 reservation should include 4 nodes with ALL of the features
2047 "knl", "a2a", and "flat" plus 2 nodes with the feature
2048 "haswell".
2049 Value may be cleared with blank data value, "Features=".
2051 Groups=<group list>
2053 List of groups permitted to use the reserved nodes, for example
2054 "Group=bio,chem". A new reservation must specify Users or
2055 Groups and/or Accounts. If both Users/Groups and Accounts are
2056 specified, a job must match both in order to use the reserva‐
2057 tion. Unlike users groups do not allow denied access to reser‐
2058 vations. You can add or remove individual groups from an exist‐
2059 ing reservation by using the update command and adding a '+' or
2060 '-' sign before the '=' sign. NOTE: Groups and Users are mutu‐
2061 ally exclusive in reservations, if you want to switch between
2062 the 2 you must update the reservation with a group='' or user=''
2063 and fill in the opposite with the appropriate setting.
2064 Skip Used on a reoccurring reservation, skip to the next reservation
2066 iteration. NOTE: Only available for update.
2067 Users=<user list>
2069 List of users permitted to use the reserved nodes, for example
2070 "User=jones1,smith2". A new reservation must specify Users or
2071 Groups and/or Accounts. If both Users/Groups and Accounts are
2072 specified, a job must match both in order to use the reserva‐
2073 tion. Users can also be denied access to reservations by pre‐
2074 ceding all of the user names with '-'. Alternately precede the
2075 equal sign with '-'. For example, "User=-jones1,-smith2" or
2076 "User-=jones1,smith2" will permit any user except jones1 and
2077 smith2 to use the reservation. You can add or remove individual
2078 users from an existing reservation by using the update command
2079 and adding a '+' or '-' sign before the '=' sign. If users are
2080 denied access to a reservation (user name preceded by a '-'),
2081 then all other users are implicitly allowed to use the reserva‐
2082 tion and it is not possible to also explicitly specify allowed
2083 users. NOTE: Groups and Users are mutually exclusive in reser‐
2084 vations, if you want to switch between the 2 you must update the
2085 reservation with a group='' or user='' and fill in the opposite
2086 with the appropriate setting.
2087 TRES=<tres_spec>
2089 Comma-separated list of TRES required for the reservation. Cur‐
2090 rent supported TRES types with reservations are: CPU, Node, Li‐
2091 cense and BB. CPU and Node follow the same format as CoreCnt and
2092 NodeCnt parameters respectively. License names can be followed
2093 by an equal '=' and a count:
2094 License/<name1>=<count1>[,License/<name2>=<count2>,...]
2096 BurstBuffer can be specified in a similar way as BurstBuffer pa‐
2098 rameter. The only difference is that colon symbol ':' should be
2099 replaced by an equal '=' in order to follow the TRES format.
2100 Some examples of TRES valid specifications:
2102 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
2104 TRES=node=5k,license/iop1=2
2106 As specified in CoreCnt, if a nodelist is specified, cpu can be
2108 an array of core numbers by node: nodes=compute[1-3]
2109 TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
2110 Please note that CPU, Node, License and BB can override CoreCnt,
2112 NodeCnt, Licenses and BurstBuffer parameters respectively. Also
2113 CPU represents CoreCnt, in a reservation and will be adjusted if
2114 you have threads per core on your nodes.
2115 Note that a reservation that contains nodes or cores is associ‐
2117 ated with one partition, and can't span resources over multiple
2118 partitions. The only exception from this is when the reserva‐
2119 tion is created with explicitly requested nodes.
2120
2122 Executing scontrol sends a remote procedure call to slurmctld. If
2123 enough calls from scontrol or other Slurm client commands that send re‐
2124 mote procedure calls to the slurmctld daemon come in at once, it can
2125 result in a degradation of performance of the slurmctld daemon, possi‐
2126 bly resulting in a denial of service.
2127 Do not run scontrol or other Slurm client commands that send remote
2129 procedure calls to slurmctld from loops in shell scripts or other pro‐
2130 grams. Ensure that programs limit calls to scontrol to the minimum nec‐
2131 essary for the information you are trying to gather.
2132
2135 Some scontrol options may be set via environment variables. These envi‐
2136 ronment variables, along with their corresponding options, are listed
2137 below. (Note: Command line options will always override these set‐
2138 tings.)
2139 SCONTROL_ALL -a, --all
2142 SCONTROL_FEDERATION --federation
2144 SCONTROL_FUTURE -F, --future
2146 SCONTROL_LOCAL --local
2148 SCONTROL_SIBLING --sibling
2150 SLURM_BITSTR_LEN Specifies the string length to be used for holding
2152 a job array's task ID expression. The default
2153 value is 64 bytes. A value of 0 will print the
2154 full expression with any length required. Larger
2155 values may adversely impact the application perfor‐
2156 mance.
2157 SLURM_CLUSTERS Same as --clusters
2159 SLURM_CONF The location of the Slurm configuration file.
2161 SLURM_CONF_OUT When running 'write config', the location of the
2163 Slurm configuration file to be written.
2164 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
2166 value of standard, the default value, generates
2167 output in the form
2168 "year-month-dateThour:minute:second". A value of
2169 relative returns only "hour:minute:second" if the
2170 current day. For other dates in the current year
2171 it prints the "hour:minute" preceded by "Tomorr"
2172 (tomorrow), "Ystday" (yesterday), the name of the
2173 day for the coming week (e.g. "Mon", "Tue", etc.),
2174 otherwise the date (e.g. "25 Apr"). For other
2175 years it returns a date month and year without a
2176 time (e.g. "6 Jun 2012"). All of the time stamps
2177 use a 24 hour format.
2178 A valid strftime() format can also be specified.
2180 For example, a value of "%a %T" will report the day
2181 of the week and a time stamp (e.g. "Mon 12:34:56").
2182 SLURM_TOPO_LEN Specify the maximum size of the line when printing
2184 Topology. If not set, the default value is unlim‐
2185 ited.
2186
2188 When using SlurmDBD, users who have an AdminLevel defined (Operator or
2189 Admin) and users who are account coordinators are given the authority
2190 to view and modify jobs, reservations, nodes, etc., as defined in the
2191 following table - regardless of whether a PrivateData restriction has
2192 been defined in the slurm.conf file.
2193 scontrol show job(s): Admin, Operator, Coordinator
2195 scontrol update job: Admin, Operator, Coordinator
2196 scontrol requeue: Admin, Operator, Coordinator
2197 scontrol show step(s): Admin, Operator, Coordinator
2198 scontrol update step: Admin, Operator, Coordinator
2199 scontrol show node: Admin, Operator
2201 scontrol update node: Admin
2202 scontrol create partition: Admin
2204 scontrol show partition: Admin, Operator
2205 scontrol update partition: Admin
2206 scontrol delete partition: Admin
2207 scontrol create reservation: Admin, Operator
2209 scontrol show reservation: Admin, Operator
2210 scontrol update reservation: Admin, Operator
2211 scontrol delete reservation: Admin, Operator
2212 scontrol reconfig: Admin
2214 scontrol shutdown: Admin
2215 scontrol takeover: Admin
2216
2219 $ scontrol
2220 scontrol: show part debug
2221 PartitionName=debug
2222 AllocNodes=ALL AllowGroups=ALL Default=YES
2223 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2224 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2225 Nodes=snowflake[0-48]
2226 Priority=1 RootOnly=NO OverSubscribe=YES:4
2227 State=UP TotalCPUs=694 TotalNodes=49
2228 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2229 scontrol: show job 71701
2230 JobId=71701 Name=hostname
2231 UserId=da(1000) GroupId=da(1000)
2232 Priority=66264 Account=none QOS=normal WCKey=*123
2233 JobState=COMPLETED Reason=None Dependency=(null)
2234 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
2235 SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
2236 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2237 SuspendTime=None SecsPreSuspend=0
2238 Partition=debug AllocNode:Sid=snowflake:4702
2239 ReqNodeList=(null) ExcNodeList=(null)
2240 NodeList=snowflake0
2241 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2242 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2243 Features=(null) Reservation=(null)
2244 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2245 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2246 scontrol: show hostnames tux[1-3]
2247 tux1
2248 tux2
2249 tux3
2250 scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
2251 Reservation created: dbremer_1
2252 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2253 scontrol: delete Reservation=dbremer_1
2254 scontrol: quit
2255
2258 Copyright (C) 2002-2007 The Regents of the University of California.
2259 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2260 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2261 Copyright (C) 2010-2022 SchedMD LLC.
2262 This file is part of Slurm, a resource management program. For de‐
2264 tails, see <https://slurm.schedmd.com/>.
2265 Slurm is free software; you can redistribute it and/or modify it under
2267 the terms of the GNU General Public License as published by the Free
2268 Software Foundation; either version 2 of the License, or (at your op‐
2269 tion) any later version.
2270 Slurm is distributed in the hope that it will be useful, but WITHOUT
2272 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2273 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2274 for more details.
2275
2277 /etc/slurm.conf
2278
2280 scancel(1), sinfo(1), squeue(1), slurm_create_partition [22m(3),
2281 slurm_delete_partition (3), slurm_load_ctl_conf [22m(3), slurm_load_jobs
2282 (3), slurm_load_node (3), slurm_load_partitions (3), slurm_reconfigure
2283 (3), slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2284 slurm_suspend [22m(3), slurm_takeover (3), slurm_update_job (3), slurm_up‐
2285 date_node (3), slurm_update_partition (3), slurm.conf(5), slurmctld(8)
2286January 2023 Slurm Commands scontrol(1)