1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an Ad‐
17 ministrator. If an attempt to view or modify configuration information
18 is made by an unauthorized user, an error message will be printed and
19 the requested action will not occur. If no command is entered on the
20 execute line, scontrol will operate in an interactive mode and prompt
21 for input. It will continue prompting for input and executing commands
22 until explicitly terminated. If a command is entered on the execute
23 line, scontrol will execute that command and terminate. All commands
24 and options are case-insensitive, although node names, partition names,
25 and reservation names are case-sensitive (node names "LX" and "lx" are
26 distinct). All commands and options can be abbreviated to the extent
27 that the specification is unique. A modified Slurm configuration can be
28 written to a file using the scontrol write config command. The result‐
29 ing file will be named using the convention "slurm.conf.<datetime>" and
30 located in the same directory as the original "slurm.conf" file. The
31 directory containing the original slurm.conf must be writable for this
32 to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -M, --clusters=<string>
43 The cluster to issue commands to. Only one cluster name may be
44 specified. Note that the SlurmDBD must be up for this option to
45 work properly. This option implicitly sets the --local option.
46 -d, --details
48 Causes the show command to provide additional details where
49 available.
50 --federation
52 Report jobs from federation if a member of one.
53 -F, --future
55 Report nodes in FUTURE state.
56 -h, --help
58 Print a help message describing the usage of scontrol.
59 --hide Do not display information about hidden partitions, their jobs
61 and job steps. By default, neither partitions that are config‐
62 ured as hidden nor those partitions unavailable to user's group
63 will be displayed (i.e. this is the default behavior).
64 --local
66 Show only information local to this cluster. Ignore other clus‐
67 ters in the federated if a member of one. Overrides --federa‐
68 tion.
69 -o, --oneliner
71 Print information one line per record.
72 -Q, --quiet
74 Print no warning or informational messages, only fatal error
75 messages.
76 --sibling
78 Show all sibling jobs on a federated cluster. Implies --federa‐
79 tion.
80 -u, --uid=<uid>
82 Attempt to update a job as user <uid> instead of the invoking
83 user id.
84 -v, --verbose
86 Print detailed event logging. Multiple -v's will further in‐
87 crease the verbosity of logging. By default only errors will be
88 displayed.
89 -V , --version
91 Print version information and exit.
92
94 abort Instruct the Slurm controller to terminate immediately and gen‐
95 erate a core file. See "man slurmctld" for information about
96 where the core file will be written.
97 cancel_reboot <NodeList>
99 Cancel pending reboots on nodes. The node will be undrain'ed and
100 the reason cleared if the node was drained by an ASAP reboot.
101 cluster <CLUSTER_NAME>
103 The cluster to issue commands to. Only one cluster name may be
104 specified.
105 create <SPECIFICATION>
107 Create a new partition or reservation. See the full list of pa‐
108 rameters below. Include the tag "res" to create a reservation
109 without specifying a reservation name.
110 completing
112 Display all jobs in a COMPLETING state along with associated
113 nodes in either a COMPLETING or DOWN state.
114 delete <SPECIFICATION>
116 Delete the entry with the specified SPECIFICATION. The two
117 SPECIFICATION choices are PartitionName=<name> and Reserva‐
118 tion=<name>. Reservations and partitions should have no associ‐
119 ated jobs at the time of their deletion (modify the jobs first).
120 If the specified partition is in use, the request is denied.
121 errnumstr <ERRNO>
123 Given a Slurm error number, return a descriptive string.
124 fsdampeningfactor <FACTOR>
126 Set the FairShareDampeningFactor in slurmctld.
127 help Display a description of scontrol options and commands.
129 hold <job_list>
131 Prevent a pending job from being started (sets its priority to
132 0). Use the release command to permit the job to be scheduled.
133 The job_list argument is a comma separated list of job IDs OR
134 "jobname=" with the job's name, which will attempt to hold all
135 jobs having that name. Note that when a job is held by a system
136 administrator using the hold command, only a system administra‐
137 tor may release the job for execution (also see the uhold com‐
138 mand). When the job is held by its owner, it may also be re‐
139 leased by the job's owner. Additionally, attempting to hold a
140 running job will have not suspend or cancel it. But, it will set
141 the job priority to 0 and update the job reason field, which
142 would hold the job if it was requeued at a later time.
143 notify <job_id> <message>
145 Send a message to standard error of the salloc or srun command
146 or batch job associated with the specified job_id.
147 pidinfo <proc_id>
149 Print the Slurm job id and scheduled termination time corre‐
150 sponding to the supplied process id, proc_id, on the current
151 node. This will work only with processes on node on which scon‐
152 trol is run, and only for those processes spawned by Slurm and
153 their descendants.
154 listpids [<job_id>[.<step_id>]] [<NodeName>]
156 Print a listing of the process IDs in a job step (if JO‐
157 BID.STEPID is provided), or all of the job steps in a job (if
158 job_id is provided), or all of the job steps in all of the jobs
159 on the local node (if job_id is not provided or job_id is "*").
160 This will work only with processes on the node on which scontrol
161 is run, and only for those processes spawned by Slurm and their
162 descendants. Note that some Slurm configurations (ProctrackType
163 value of pgid) are unable to identify all processes associated
164 with a job or job step.
165 Note that the NodeName option is only really useful when you
167 have multiple slurmd daemons running on the same host machine.
168 Multiple slurmd daemons on one host are, in general, only used
169 by Slurm developers.
170 ping Ping the primary and secondary slurmctld daemon and report if
172 they are responding.
173 reboot [ASAP] [nextstate={RESUME|DOWN}] [reason=<reason>] {ALL|<Node‐
175 List>}
176 Reboot the nodes in the system when they become idle using the
177 RebootProgram as configured in Slurm's slurm.conf file. Each
178 node will have the "REBOOT" flag added to its node state. After
179 a node reboots and the slurmd daemon starts up again, the
180 HealthCheckProgram will run once. Then, the slurmd daemon will
181 register itself with the slurmctld daemon and the "REBOOT" flag
182 will be cleared. The node's "DRAIN" state flag will be cleared
183 if the reboot was "ASAP", nextstate=resume or down. The "ASAP"
184 option adds the "DRAIN" flag to each node's state, preventing
185 additional jobs from running on the node so it can be rebooted
186 and returned to service "As Soon As Possible" (i.e. ASAP).
187 "ASAP" will also set the node reason to "Reboot ASAP" if the
188 "reason" option isn't specified. If the "nextstate" option is
189 specified as "DOWN", then the node will remain in a down state
190 after rebooting. If "nextstate" is specified as "RESUME", then
191 the nodes will resume as normal and the node's reason and
192 "DRAIN" state will be cleared. Resuming nodes will be consid‐
193 ered as available in backfill future scheduling and won't be re‐
194 placed by idle nodes in a reservation. The "reason" option sets
195 each node's reason to a user-defined message. A default reason
196 of "reboot requested" is set if no other reason is set on the
197 node. The reason will be appended with: "reboot issued" when
198 the reboot is issued; "reboot complete" when the node registers
199 and has a "nextstate" of "DOWN"; or "reboot timed out" when the
200 node fails to register within ResumeTimeout. You must specify
201 either a list of nodes or that ALL nodes are to be rebooted.
202 NOTE: By default, this command does not prevent additional jobs
203 from being scheduled on any nodes before reboot. To do this,
204 you can either use the "ASAP" option or explicitly drain the
205 nodes beforehand. You can alternately create an advanced reser‐
206 vation to prevent additional jobs from being initiated on nodes
207 to be rebooted. Pending reboots can be cancelled by using
208 "scontrol cancel_reboot <node>" or setting the node state to
209 "CANCEL_REBOOT". A node will be marked "DOWN" if it doesn't re‐
210 boot within ResumeTimeout.
211 reconfigure
213 Instruct all Slurm daemons to re-read the configuration file.
214 This command does not restart the daemons. This mechanism can
215 be used to modify configuration parameters set in slurm.conf.
216 The Slurm controller (slurmctld) forwards the request to all
217 other daemons (slurmd daemon on each compute node). Running jobs
218 continue execution. Most configuration parameters can be
219 changed by just running this command; however, there are parame‐
220 ters that require a restart of the relevant Slurm daemons. Pa‐
221 rameters requiring a restart will be noted in the slurm.conf(5)
222 man page. The slurmctld daemon and all slurmd daemons must also
223 be restarted if nodes are added to or removed from the cluster.
224 release <job_list>
226 Release a previously held job to begin execution. The job_list
227 argument is a comma separated list of job IDs OR "jobname=" with
228 the job's name, which will attempt to hold all jobs having that
229 name. Also see hold.
230 requeue [<option>] <job_list>
232 Requeue a running, suspended or finished Slurm batch job into
233 pending state. The job_list argument is a comma separated list
234 of job IDs. The command accepts the following option:
235 Incomplete
237 Operate only on jobs (or tasks of a job array) which have
238 not completed. Specifically only jobs in the following
239 states will be requeued: CONFIGURING, RUNNING, STOPPED or
240 SUSPENDED.
241 requeuehold [<option>] <job_list>
243 Requeue a running, suspended or finished Slurm batch job into
244 pending state, moreover the job is put in held state (priority
245 zero). The job_list argument is a comma separated list of job
246 IDs. A held job can be released using scontrol to reset its
247 priority (e.g. "scontrol release <job_id>"). The command ac‐
248 cepts the following options:
249 Incomplete
251 Operate only on jobs (or tasks of a job array) which have
252 not completed. Specifically only jobs in the following
253 states will be requeued: CONFIGURING, RUNNING, STOPPED or
254 SUSPENDED.
255 State=SpecialExit
257 The "SpecialExit" keyword specifies that the job has to
258 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
259 trol show job" command will display the JobState as SPE‐
260 CIAL_EXIT, while the "squeue" command as SE.
261 resume <job_list>
263 Resume a previously suspended job. The job_list argument is a
264 comma separated list of job IDs. Also see suspend.
265 NOTE: A suspended job releases its CPUs for allocation to other
267 jobs. Resuming a previously suspended job may result in multi‐
268 ple jobs being allocated the same CPUs, which could trigger gang
269 scheduling with some configurations or severe degradation in
270 performance with other configurations. Use of the scancel com‐
271 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
272 out releasing its CPUs for allocation to other jobs and would be
273 a preferable mechanism in many cases. If performing system
274 maintenance you may want to use suspend/resume in the following
275 way. Before suspending set all nodes to draining or set all par‐
276 titions to down so that no new jobs can be scheduled. Then sus‐
277 pend jobs. Once maintenance is done resume jobs then resume
278 nodes and/or set all partitions back to up. Use with caution.
279 schedloglevel <LEVEL>
281 Enable or disable scheduler logging. LEVEL may be "0", "1",
282 "disable" or "enable". "0" has the same effect as "disable". "1"
283 has the same effect as "enable". This value is temporary and
284 will be overwritten when the slurmctld daemon reads the
285 slurm.conf configuration file (e.g. when the daemon is restarted
286 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
287 parameter is present.
288 setdebug <LEVEL>
290 Change the debug level of the slurmctld daemon for all active
291 logging channels not originally configured off (quiet). LEVEL
292 may be an integer value between zero and nine (using the same
293 values as SlurmctldDebug in the slurm.conf file) or the name of
294 the most detailed message type to be printed: "quiet", "fatal",
295 "error", "info", "verbose", "debug", "debug2", "debug3", "de‐
296 bug4", or "debug5". This value is temporary and will be over‐
297 written whenever the slurmctld daemon reads the slurm.conf con‐
298 figuration file (e.g. when the daemon is restarted or scontrol
299 reconfigure is executed).
300 setdebugflags [+|-]<FLAG>
302 Add or remove DebugFlags of the slurmctld daemon. See "man
303 slurm.conf" for a list of supported DebugFlags. NOTE: Changing
304 the value of some DebugFlags will have no effect without
305 restarting the slurmctld daemon, which would set DebugFlags
306 based upon the contents of the slurm.conf configuration file.
307 show <ENTITY>[=<ID>] or <ENTITY> [<ID>]
309 Display the state of the specified entity with the specified
310 identification.
311 aliases
313 Returns all NodeName values associated with a given Node‐
314 Hostname (useful to get the list of virtual nodes associ‐
315 ated with a real node in a configuration where multiple
316 slurmd daemons execute on a single compute node).
317 assoc_mgr
319 Displays the current contents of the slurmctld's internal
320 cache for users, associations and/or qos. The output can
321 be filtered by different record types:
322 users=<user1>[...,<userN>]
324 Limit the User Records displayed to those with the
325 specified user name(s).
326 accounts=<acct1>[...,<acctN>]
328 Limit the Association Records displayed to those
329 with the specified account name(s).
330 qos=<qos1>[...,<qosN>]
332 Limit the QOS Records displayed to those with the
333 specified QOS name(s).
334 flags={users|assoc|qos}
336 Specify the desired record type to be displayed.
337 If no flags are specified, all record types are
338 displayed.
339 bbstat Displays output from Cray's burst buffer status tool. Op‐
341 tions following bbstat are passed directly to the dwstat
342 command by the slurmctld daemon and the response returned
343 to the user. Equivalent to dwstat.
344 burstbuffer
346 Displays the current status of the BurstBuffer plugin.
347 config Displays parameter names from the configuration files in
349 mixed case (e.g. SlurmdPort=7003) while derived parame‐
350 ters names are in upper case only (e.g. SLURM_VERSION).
351 daemons
353 Reports which daemons should be running on this node.
354 dwstat Displays output from Cray's burst buffer status tool. Op‐
356 tions following dwstat are passed directly to the dwstat
357 command by the slurmctld daemon and the response returned
358 to the user. Equivalent to bbstat.
359 federation
361 The federation name that the controller is part of and
362 the sibling clusters part of the federation will be
363 listed.
364 frontend
366 Shows configured frontend nodes.
367 hostlist
369 Takes a list of host names and prints the hostlist ex‐
370 pression for them (the inverse of hostnames). hostlist
371 can also take the absolute pathname of a file (beginning
372 with the character '/') containing a list of hostnames.
373 Multiple node names may be specified using simple node
374 range expressions (e.g. "lx[10-20]"). All other ID values
375 must identify a single element. The job step ID is of the
376 form "job_id.step_id", (e.g. "1234.1"). slurmd reports
377 the current status of the slurmd daemon executing on the
378 same node from which the scontrol command is executed
379 (the local host). It can be useful to diagnose problems.
380 By default hostlist does not sort the node list or make
381 it unique (e.g. tux2,tux1,tux2 = tux[2,1-2]). If you
382 wanted a sorted list use hostlistsorted (e.g.
383 tux2,tux1,tux2 = tux[1-2,2]). By default, all elements
384 of the entity type specified are printed.
385 hostlistsorted
387 Takes a list of host names and prints a sorted, unique
388 hostlist expression for them. See hostlist.
389 hostnames
391 Takes an optional hostlist expression as input and writes
392 a list of individual host names to standard output (one
393 per line). If no hostlist expression is supplied, the
394 contents of the SLURM_JOB_NODELIST environment variable
395 is used. For example "tux[1-3]" is mapped to
396 "tux1","tux2" and "tux3" (one hostname per line).
397 job Displays statistics about all jobs by default. If an op‐
399 tional jobid is specified, details for just that job will
400 be displayed. If the job does not specify
401 socket-per-node, cores-per-socket or threads-per-core
402 then it will display '*' in the ReqS:C:T=*:*:* field.
403 licenses
405 Displays statistics about all configured licenses (local
406 and remote) by default. If an optional license name is
407 specified, details for just that license will be dis‐
408 played.
409 node Displays statistics about all nodes by default. If an op‐
412 tional nodename is specified, details for just that node
413 will be displayed.
414 partition
416 Displays statistics about all partitions by default. If
417 an optional partition name is specified, details for just
418 that partition will be displayed.
419 reservation
421 Displays statistics about all reservations by default. If
422 an optional reservation name is specified, details for
423 just that reservation will be displayed.
424 slurmd Displays statistics for the slurmd running on the current
426 node.
427 step Displays statistics about all job steps by default. If an
429 optional jobid is specified, details about steps for just
430 that job will be displayed. If a jobid.stepid is speci‐
431 fied, details for just that step will be displayed.
432 topology
434 Displays information about the defined topology layout.
435 If a switch is specified, information about that switch
436 will be shown. If one node name is specified, all
437 switches connected to that node (and their parent
438 switches) will be shown. If more than one node name is
439 specified, only switches that connect to all named nodes
440 will be shown.
441 shutdown <OPTION>
443 Instruct Slurm daemons to save current state and terminate. By
444 default, the Slurm controller (slurmctld) forwards the request
445 all other daemons (slurmd daemon on each compute node). An OP‐
446 TION of slurmctld or controller results in only the slurmctld
447 daemon being shutdown and the slurmd daemons remaining active.
448 suspend <job_list>
450 Suspend a running job. The job_list argument is a comma sepa‐
451 rated list of job IDs. Use the resume command to resume its ex‐
452 ecution. User processes must stop on receipt of SIGSTOP signal
453 and resume upon receipt of SIGCONT for this operation to be ef‐
454 fective. Not all architectures and configurations support job
455 suspension. If a suspended job is requeued, it will be placed
456 in a held state. The time a job is suspended will not count
457 against a job's time limit. Only an operator, administrator,
458 SlurmUser, or root can suspend jobs.
459 takeover [<INDEX>]
461 Instruct one of Slurm's backup controllers (slurmctld) to take
462 over system control. By default the first backup controller (IN‐
463 DEX=1) requests control from the primary and waits for its ter‐
464 mination. After that, it switches from backup mode to controller
465 mode. If primary controller can not be contacted, it directly
466 switches to controller mode. This can be used to speed up the
467 Slurm controller fail-over mechanism when the primary node is
468 down. This can be used to minimize disruption if the computer
469 executing the primary Slurm controller is scheduled down.
470 (Note: Slurm's primary controller will take the control back at
471 startup.)
472 top <job_list>
474 Move the specified job IDs to the top of the queue of jobs be‐
475 longing to the identical user ID, partition name, account, and
476 QOS. The job_list argument is a comma separated ordered list of
477 job IDs. Any job not matching all of those fields will not be
478 effected. Only jobs submitted to a single partition will be ef‐
479 fected. This operation changes the order of jobs by adjusting
480 job nice values. The net effect on that user's throughput will
481 be negligible to slightly negative. This operation is disabled
482 by default for non-privileged (non-operator, admin, SlurmUser,
483 or root) users. This operation may be enabled for non-privileged
484 users by the system administrator by including the option "en‐
485 able_user_top" in the SchedulerParameters configuration parame‐
486 ter.
487 token [lifespan=<lifespan>] [username=<username>]
489 Return an auth token which can be used to support JWT authenti‐
490 cation if AuthAltTypes=auth/jwt has been enabled on the system.
491 Supports two optional arguments. lifespan= may be used to spec‐
492 ify the token's lifespan in seconds. username (only available to
493 SlurmUser/root) may be used to request a token for a different
494 username.
495 uhold <job_list>
497 Prevent a pending job from being started (sets its priority to
498 0). The job_list argument is a space separated list of job IDs
499 or job names. Use the release command to permit the job to be
500 scheduled. This command is designed for a system administrator
501 to hold a job so that the job owner may release it rather than
502 requiring the intervention of a system administrator (also see
503 the hold command).
504 update <SPECIFICATION>
506 Update job, step, node, partition, or reservation configuration
507 per the supplied specification. SPECIFICATION is in the same
508 format as the Slurm configuration file and the output of the
509 show command described above. It may be desirable to execute the
510 show command (described above) on the specific entity you want
511 to update, then use cut-and-paste tools to enter updated config‐
512 uration values to the update. Note that while most configuration
513 values can be changed using this command, not all can be changed
514 using this mechanism. In particular, the hardware configuration
515 of a node or the physical addition or removal of nodes from the
516 cluster may only be accomplished through editing the Slurm con‐
517 figuration file and executing the reconfigure command (described
518 above).
519 version
521 Display the version number of scontrol being executed.
522 wait_job <job_id>
524 Wait until a job and all of its nodes are ready for use or the
525 job has entered some termination state. This option is particu‐
526 larly useful in the Slurm Prolog or in the batch script itself
527 if nodes are powered down and restarted automatically as needed.
528 NOTE: Don't use scontrol wait_job in PrologSlurmctld or Prolog
530 with PrologFlags=Alloc as this will result in a deadlock.
531 NOTE: When using wait_job for an array job, use the SLURM_JOB_ID
533 environment variable to reference the job rather than the
534 SLURM_ARRAY_JOB_ID variable.
535 write batch_script <job_id> [<optional_filename>]
538 Write the batch script for a given job_id to a file or to std‐
539 out. The file will default to slurm-<job_id>.sh if the optional
540 filename argument is not given. The script will be written to
541 stdout if - is given instead of a filename. The batch script
542 can only be retrieved by an admin or operator, or by the owner
543 of the job.
544 write config <optional_filename>
546 Write the current configuration to a file with the naming con‐
547 vention of "slurm.conf.<datetime>" in the same directory as the
548 original slurm.conf file. If a filename is given that file lo‐
549 cation with a .<datetime> suffix is created.
550
552 NOTE: All commands listed below can be used in the interactive mode,
553 but NOT on the initial command line.
554 all Show all partitions, their jobs and jobs steps. This causes in‐
557 formation to be displayed about partitions that are configured
558 as hidden and partitions that are unavailable to user's group.
559 details
561 Causes the show command to provide additional details where
562 available. Job information will include CPUs and NUMA memory
563 allocated on each node. Note that on computers with hyper‐
564 threading enabled and Slurm configured to allocate cores, each
565 listed CPU represents one physical core. Each hyperthread on
566 that core can be allocated a separate task, so a job's CPU count
567 and task count may differ. See the --cpu-bind and --mem-bind
568 option descriptions in srun man pages for more information. The
569 details option is currently only supported for the show job com‐
570 mand.
571 exit Terminate scontrol interactive session.
573 hide Do not display partition, job or jobs step information for par‐
575 titions that are configured as hidden or partitions that are un‐
576 available to the user's group. This is the default behavior.
577 oneliner
579 Print information one line per record.
580 quiet Print no warning or informational messages, only fatal error
582 messages.
583 quit Terminate the execution of scontrol.
585 verbose
587 Print detailed event logging. This includes time-stamps on data
588 structures, record counts, etc.
589 !! Repeat the last command executed.
591
593 Note that update requests done by either root, SlurmUser or Administra‐
594 tors are not subject to certain restrictions. For instance, if an Ad‐
595 ministrator changes the QOS on a pending job, certain limits such as
596 the TimeLimit will not be changed automatically as changes made by the
597 Administrators are allowed to violate these restrictions.
598 Account=<account>
601 Account name to be changed for this job's resource use. Value
602 may be cleared with blank data value, "Account=".
603 AdminComment=<spec>
605 Arbitrary descriptive string. Can only be set by a Slurm admin‐
606 istrator.
607 ArrayTaskThrottle=<count>
609 Specify the maximum number of tasks in a job array that can exe‐
610 cute at the same time. Set the count to zero in order to elimi‐
611 nate any limit. The task throttle count for a job array is re‐
612 ported as part of its ArrayTaskId field, preceded with a percent
613 sign. For example "ArrayTaskId=1-10%2" indicates the maximum
614 number of running tasks is limited to 2.
615 BurstBuffer=<spec>
617 Burst buffer specification to be changed for this job's resource
618 use. Value may be cleared with blank data value, "Burst‐
619 Buffer=". Format is burst buffer plugin specific.
620 Clusters=<spec>
622 Specifies the clusters that the federated job can run on.
623 ClusterFeatures=<spec>
625 Specifies features that a federated cluster must have to have a
626 sibling job submitted to it. Slurm will attempt to submit a sib‐
627 ling job to a cluster if it has at least one of the specified
628 features.
629 Comment=<spec>
631 Arbitrary descriptive string.
632 Contiguous={yes|no}
634 Set the job's requirement for contiguous (consecutive) nodes to
635 be allocated. Possible values are "YES" and "NO". Only the
636 Slurm administrator or root can change this parameter.
637 CoreSpec=<count>
639 Number of cores to reserve per node for system use. The job
640 will be charged for these cores, but be unable to use them.
641 Will be reported as "*" if not constrained.
642 CPUsPerTask=<count>
644 Change the CPUsPerTask job's value.
645 Deadline=<time_spec>
647 It accepts times of the form HH:MM:SS to specify a deadline to a
648 job at a specific time of day (seconds are optional). You may
649 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
650 you can have a time-of-day suffixed with AM or PM for a deadline
651 in the morning or the evening. You can specify a deadline for
652 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
653 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
654 give times like now + count time-units, where the time-units can
655 be seconds (default), minutes, hours, days, or weeks and you can
656 tell Slurm to put a deadline for tomorrow with the keyword to‐
657 morrow. The specified deadline must be later than the current
658 time. Only pending jobs can have the deadline updated. Only
659 the Slurm administrator or root can change this parameter.
660 DelayBoot=<time_spec>
662 Change the time to decide whether to reboot nodes in order to
663 satisfy job's feature specification if the job has been eligible
664 to run for less than this time period. See salloc/sbatch man
665 pages option --delay-boot.
666 Dependency=<dependency_list>
668 Defer job's initiation until specified job dependency specifica‐
669 tion is satisfied. Cancel dependency with an empty depen‐
670 dency_list (e.g. "Dependency="). <dependency_list> is of the
671 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
672 can share the same dependency and these jobs may even belong to
673 different users.
674 after:job_id[:jobid...]
676 This job can begin execution after the specified jobs
677 have begun execution or been canceled.
678 afterany:job_id[:jobid...]
680 This job can begin execution after the specified jobs
681 have terminated.
682 afternotok:job_id[:jobid...]
684 This job can begin execution after the specified jobs
685 have terminated in some failed state (non-zero exit code,
686 node failure, timed out, etc).
687 afterok:job_id[:jobid...]
689 This job can begin execution after the specified jobs
690 have successfully executed (ran to completion with an
691 exit code of zero).
692 singleton
694 This job can begin execution after any previously
695 launched jobs sharing the same job name and user have
696 terminated. In other words, only one job by that name
697 and owned by that user can be running or suspended at any
698 point in time.
699 EligibleTime=<time_spec>
701 See StartTime.
702 EndTime
704 The time the job is expected to terminate based on the job's
705 time limit. When the job ends sooner, this field will be up‐
706 dated with the actual end time.
707 ExcNodeList=<nodes>
709 Set the job's list of excluded node. Multiple node names may be
710 specified using simple node range expressions (e.g.
711 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
712 NodeList=".
713 Features=<features>
715 Set the job's required node features. The list of features may
716 include multiple feature names separated by ampersand (AND)
717 and/or vertical bar (OR) operators. For example: Fea‐
718 tures="opteron&video" or Features="fast|faster". In the first
719 example, only nodes having both the feature "opteron" AND the
720 feature "video" will be used. There is no mechanism to specify
721 that you want one node with feature "opteron" and another node
722 with feature "video" in case no node has both features. If only
723 one of a set of possible options should be used for all allo‐
724 cated nodes, then use the OR operator and enclose the options
725 within square brackets. For example: "Fea‐
726 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
727 all nodes must be allocated on a single rack of the cluster, but
728 any of those four racks can be used. A request can also specify
729 the number of nodes needed with some feature by appending an as‐
730 terisk and count after the feature name. For example "Fea‐
731 tures=graphics*4" indicates that at least four allocated nodes
732 must have the feature "graphics." Parenthesis are also sup‐
733 ported for features to be ANDed together. For example "Fea‐
734 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
735 cation should include 4 nodes with ALL of the features "knl",
736 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
737 straints with node counts may only be combined with AND opera‐
738 tors. Value may be cleared with blank data value, for example
739 "Features=".
740 Gres=<list>
742 Specifies a comma-delimited list of generic consumable re‐
743 sources. The format of each entry on the list is
744 "name[:count[*cpu]]". The name is that of the consumable re‐
745 source. The count is the number of those resources with a de‐
746 fault value of 1. The specified resources will be allocated to
747 the job on each node allocated unless "*cpu" is appended, in
748 which case the resources will be allocated on a per cpu basis.
749 The available generic consumable resources is configurable by
750 the system administrator. A list of available generic consum‐
751 able resources will be printed and the command will exit if the
752 option argument is "help". Examples of use include
753 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
754 JobId=<job_list>
756 Identify the job(s) to be updated. The job_list may be a comma
757 separated list of job IDs. Either JobId or JobName is required.
758 Licenses=<name>
760 Specification of licenses (or other resources available on all
761 nodes of the cluster) as described in salloc/sbatch/srun man
762 pages.
763 MailType=<types>
765 Set the mail event types. Valid type values are NONE, BEGIN,
766 END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, RE‐
767 QUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and
768 teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 per‐
769 cent of time limit), TIME_LIMIT_80 (reached 80 percent of time
770 limit), TIME_LIMIT_50 (reached 50 percent of time limit) and AR‐
771 RAY_TASKS (send emails for each array task). Multiple type val‐
772 ues may be specified in a comma separated list. Unless the AR‐
773 RAY_TASKS option is specified, mail notifications on job BEGIN,
774 END and FAIL apply to a job array as a whole rather than gener‐
775 ating individual email messages for each task in the job array.
776 MailUser=<name>
778 Set the user to receive email notification of state changes. A
779 blank string will set the mail user to the default which is the
780 submitting user.
781 MinCPUsNode=<count>
783 Set the job's minimum number of CPUs per node to the specified
784 value.
785 MinMemoryCPU=<megabytes>
787 Set the job's minimum real memory required per allocated CPU to
788 the specified value. Either MinMemoryCPU or MinMemoryNode may be
789 set, but not both.
790 MinMemoryNode=<megabytes>
792 Set the job's minimum real memory required per node to the spec‐
793 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
794 but not both.
795 MinTmpDiskNode=<megabytes>
797 Set the job's minimum temporary disk space required per node to
798 the specified value. Only the Slurm administrator or root can
799 change this parameter.
800 TimeMin=<timespec>
802 Change TimeMin value which specifies the minimum time limit min‐
803 utes of the job.
804 JobName=<name>
806 Identify the name of jobs to be modified or set the job's name
807 to the specified value. When used to identify jobs to be modi‐
808 fied, all jobs belonging to all users are modified unless the
809 UserID option is used to identify a specific user. Either JobId
810 or JobName is required.
811 Name[=<name>]
813 See JobName.
814 Nice[=<adjustment>]
816 Update the job with an adjusted scheduling priority within
817 Slurm. With no adjustment value the scheduling priority is de‐
818 creased by 100. A negative nice value increases the priority,
819 otherwise decreases it. The adjustment range is +/- 2147483645.
820 Only privileged users can specify a negative adjustment.
821 NodeList=<nodes>
823 Change the nodes allocated to a running job to shrink its size.
824 The specified list of nodes must be a subset of the nodes cur‐
825 rently allocated to the job. Multiple node names may be speci‐
826 fied using simple node range expressions (e.g. "lx[10-20]"). Af‐
827 ter a job's allocation is reduced, subsequent srun commands must
828 explicitly specify node and task counts which are valid for the
829 new allocation.
830 NumCPUs=<min_count>[-<max_count>]
832 Set the job's minimum and optionally maximum count of CPUs to be
833 allocated.
834 NumNodes=<min_count>[-<max_count>]
836 Set the job's minimum and optionally maximum count of nodes to
837 be allocated. If the job is already running, use this to spec‐
838 ify a node count less than currently allocated and resources
839 previously allocated to the job will be relinquished. After a
840 job's allocation is reduced, subsequent srun commands must ex‐
841 plicitly specify node and task counts which are valid for the
842 new allocation. Also see the NodeList parameter above. This is
843 the same as ReqNodes.
844 NumTasks=<count>
846 Set the job's count of required tasks to the specified value.
847 This is the same as ReqProcs.
848 OverSubscribe={yes|no}
850 Set the job's ability to share compute resources (i.e. individ‐
851 ual CPUs) with other jobs. Possible values are "YES" and "NO".
852 This option can only be changed for pending jobs.
853 Partition=<name>
855 Set the job's partition to the specified value.
856 Priority=<number>
858 Set the job's priority to the specified value. Note that a job
859 priority of zero prevents the job from ever being scheduled. By
860 setting a job's priority to zero it is held. Set the priority
861 to a non-zero value to permit it to run. Explicitly setting a
862 job's priority clears any previously set nice value and removes
863 the priority/multifactor plugin's ability to manage a job's pri‐
864 ority. In order to restore the priority/multifactor plugin's
865 ability to manage a job's priority, hold and then release the
866 job. Only the Slurm administrator or root can increase job's
867 priority.
868 QOS=<name>
870 Set the job's QOS (Quality Of Service) to the specified value.
871 Value may be cleared with blank data value, "QOS=".
872 Reboot={yes|no}
874 Set the job's flag that specifies whether to force the allocated
875 nodes to reboot before starting the job. This is only supported
876 with some system configurations and therefore it could be
877 silently ignored.
878 ReqCores=<count>
880 Change the job's requested Cores count.
881 ReqNodeList=<nodes>
883 Set the job's list of required node. Multiple node names may be
884 specified using simple node range expressions (e.g.
885 "lx[10-20]"). Value may be cleared with blank data value, "Re‐
886 qNodeList=".
887 ReqNodes=<min_count>[-<max_count>]
889 See NumNodes.
890 ReqProcs=<count>
892 See NumTasks.
893 ReqSockets=<count>
895 Change the job's requested socket count.
896 ReqThreads=<count>
898 Change the job's requested threads count.
899 IRequeue={0|1}
901 Stipulates whether a job should be requeued after a node fail‐
902 ure: 0 for no, 1 for yes.
903 ReservationName=<name>
905 Set the job's reservation to the specified value. Value may be
906 cleared with blank data value, "ReservationName=".
907 ResetAccrueTime
909 Set the job's accrue time value to 'now' meaning it will lose
910 any time previously accrued for priority. Helpful if you have a
911 large queue of jobs already in the queue and want to start lim‐
912 iting how many jobs can accrue time without waiting for the
913 queue to flush out.
914 SiteFactor=<account>
916 Specify the job's admin priority factor in the range of
917 +/-2147483645. Only privileged users can modify the value.
918 StdOut=<filepath>
920 Set the batch job's stdout file path.
921 Shared={yes|no}
923 See OverSubscribe option above.
924 StartTime=<time_spec>
926 Set the job's earliest initiation time. It accepts times of the
927 form HH:MM:SS to run a job at a specific time of day (seconds
928 are optional). (If that time is already past, the next day is
929 assumed.) You may also specify midnight, noon, fika (3 PM) or
930 teatime (4 PM) and you can have a time-of-day suffixed with AM
931 or PM for running in the morning or the evening. You can also
932 say what day the job will be run, by specifying a date of the
933 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
934 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
935 count time-units, where the time-units can be seconds (default),
936 minutes, hours, days, or weeks and you can tell Slurm to run the
937 job today with the keyword today and to run the job tomorrow
938 with the keyword tomorrow.
939 Notes on date/time specifications:
941 - although the 'seconds' field of the HH:MM:SS time specifica‐
942 tion is allowed by the code, note that the poll time of the
943 Slurm scheduler is not precise enough to guarantee dispatch of
944 the job on the exact second. The job will be eligible to start
945 on the next poll following the specified time. The exact poll
946 interval depends on the Slurm scheduler (e.g., 60 seconds with
947 the default sched/builtin).
948 - if no time (HH:MM:SS) is specified, the default is
949 (00:00:00).
950 - if a date is specified without a year (e.g., MM/DD) then the
951 current year is assumed, unless the combination of MM/DD and
952 HH:MM:SS has already passed for that year, in which case the
953 next year is used.
954 Switches=<count>[@<max-time-to-wait>]
956 When a tree topology is used, this defines the maximum count of
957 switches desired for the job allocation. If Slurm finds an allo‐
958 cation containing more switches than the count specified, the
959 job remain pending until it either finds an allocation with de‐
960 sired switch count or the time limit expires. By default there
961 is no switch count limit and no time limit delay. Set the count
962 to zero in order to clean any previously set count (disabling
963 the limit). The job's maximum time delay may be limited by the
964 system administrator using the SchedulerParameters configuration
965 parameter with the max_switch_wait parameter option. Also see
966 wait-for-switch.
967 wait-for-switch=<seconds>
969 Change max time to wait for a switch <seconds> secs.
970 TasksPerNode=<count>
972 Change the job's requested TasksPerNode.
973 ThreadSpec=<count>
975 Number of threads to reserve per node for system use. The job
976 will be charged for these threads, but be unable to use them.
977 Will be reported as "*" if not constrained.
978 TimeLimit=<time>
980 The job's time limit. Output format is [days-]hours:min‐
981 utes:seconds or "UNLIMITED". Input format (for update command)
982 set is minutes, minutes:seconds, hours:minutes:seconds,
983 days-hours, days-hours:minutes or days-hours:minutes:seconds.
984 Time resolution is one minute and second values are rounded up
985 to the next minute. If changing the time limit of a job, either
986 specify a new time limit value or precede the time and equal
987 sign with a "+" or "-" to increment or decrement the current
988 time limit (e.g. "TimeLimit+=30"). In order to increment or
989 decrement the current time limit, the JobId specification must
990 precede the TimeLimit specification. Note that incrementing or
991 decrementing the time limit for a job array is only allowed be‐
992 fore the job array has been split into more than one job record.
993 Only the Slurm administrator or root can increase job's Time‐
994 Limit.
995 UserID=<UID or name>
997 Used with the JobName option to identify jobs to be modified.
998 Either a user name or numeric ID (UID), may be specified.
999 WCKey=<key>
1001 Set the job's workload characterization key to the specified
1002 value.
1003 WorkDir=<directory_name>
1005 Set the job's working directory to the specified value. Note
1006 that this may only be set for jobs in the PENDING state, and
1007 that jobs may fail to launch if they rely on relative paths to
1008 the originally submitted WorkDir.
1009
1011 The "show" command, when used with the "job" or "job <jobid>" entity
1012 displays detailed information about a job or jobs. Much of this infor‐
1013 mation may be modified using the "update job" command as described
1014 above. However, the following fields displayed by the show job command
1015 are read-only and cannot be modified:
1016 AllocNode:Sid
1019 Local node and system id making the resource allocation.
1020 BatchFlag
1022 Jobs submitted using the sbatch command have BatchFlag set to 1.
1023 Jobs submitted using other commands have BatchFlag set to 0.
1024 ExitCode=<exit>:<sig>
1026 Exit status reported for the job by the wait() function. The
1027 first number is the exit code, typically as set by the exit()
1028 function. The second number of the signal that caused the
1029 process to terminate if it was terminated by a signal.
1030 GroupId
1032 The group under which the job was submitted.
1033 JobState
1035 The current state of the job.
1036 NodeListIndices
1038 The NodeIndices expose the internal indices into the node table
1039 associated with the node(s) allocated to the job.
1040 NtasksPerN:B:S:C=<tasks_per_node>:<tasks_per_base‐
1042 board>:<tasks_per_socket>:<tasks_per_core>
1043 Specifies the number of tasks to be started per hardware compo‐
1044 nent (node, baseboard, socket and core). Unconstrained values
1045 may be shown as "0" or "*".
1046 PreemptEligibleTime
1048 Time the job becomes eligible for preemption. Modified by Pre‐
1049 emptExemptTime, either from the global option in slurm.conf or
1050 the job QOS. This is hidden if the job has not started or if
1051 PreemptMode=OFF.
1052 PreemptTime
1054 Time at which job was signaled that it was selected for preemp‐
1055 tion. This value is only meaningful for PreemptMode=CANCEL and
1056 PreemptMode=REQUEUE and for jobs in a partition or QOS that has
1057 a GraceTime value designated. This is hidden if the job has not
1058 started or if PreemptMode=OFF.
1059 PreSusTime
1061 Time the job ran prior to last suspend.
1062 Reason The reason job is not running: e.g., waiting "Resources".
1064 ReqB:S:C:T=<baseboard_count>:<socket_per_base‐
1066 board_count>:<core_per_socket_count>:<thread_per_core_count>
1067 Specifies the count of various hardware components requested by
1068 the job. Unconstrained values may be shown as "0" or "*".
1069 SecsPreSuspend=<seconds>
1071 If the job is suspended, this is the run time accumulated by the
1072 job (in seconds) prior to being suspended.
1073 Socks/Node=<count>
1075 Count of desired sockets per node
1076 SubmitTime
1078 The time and date stamp (in localtime) the job was submitted.
1079 The format of the output is identical to that of the EndTime
1080 field.
1081 NOTE: If a job is requeued, the submit time is reset. To obtain
1083 the original submit time it is necessary to use the "sacct -j
1084 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1085 cate option to display all duplicate entries for a job.
1086 SuspendTime
1088 Time the job was last suspended or resumed.
1089 NOTE on information displayed for various job states: When you
1091 submit a request for the "show job" function the scontrol
1092 process makes an RPC request call to slurmctld with a RE‐
1093 QUEST_JOB_INFO message type. If the state of the job is PEND‐
1094 ING, then it returns some detail information such as: min_nodes,
1095 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1096 ING the code assumes that it is in a further state such as RUN‐
1097 NING, COMPLETE, etc. In these cases the code explicitly returns
1098 zero for these values. These values are meaningless once the job
1099 resources have been allocated and the job has started.
1100
1102 StepId=<job_id>[.<step_id>]
1103 Identify the step to be updated. If the job_id is given, but no
1104 step_id is specified then all steps of the identified job will
1105 be modified. This specification is required.
1106 TimeLimit=<time>
1108 The job's time limit. Output format is [days-]hours:min‐
1109 utes:seconds or "UNLIMITED". Input format (for update command)
1110 set is minutes, minutes:seconds, hours:minutes:seconds,
1111 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1112 Time resolution is one minute and second values are rounded up
1113 to the next minute. If changing the time limit of a step, ei‐
1114 ther specify a new time limit value or precede the time with a
1115 "+" or "-" to increment or decrement the current time limit
1116 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1117 current time limit, the StepId specification must precede the
1118 TimeLimit specification.
1119
1121 NodeName=<name>
1122 Identify the node(s) to be updated. Multiple node names may be
1123 specified using simple node range expressions (e.g.
1124 "lx[10-20]"). This specification is required.
1125 ActiveFeatures=<features>
1127 Identify the feature(s) currently active on the specified node.
1128 Any previously active feature specification will be overwritten
1129 with the new value. Also see AvailableFeatures. Typically Ac‐
1130 tiveFeatures will be identical to AvailableFeatures; however Ac‐
1131 tiveFeatures may be configured as a subset of the AvailableFea‐
1132 tures. For example, a node may be booted in multiple configura‐
1133 tions. In that case, all possible configurations may be identi‐
1134 fied as AvailableFeatures, while ActiveFeatures would identify
1135 the current node configuration. When updating the ActiveFea‐
1136 tures with scontrol, the change is only made in slurmctld. When
1137 using a node_features plugin the state/features of the node must
1138 be updated on the node such that a new node start will report
1139 the updated state/features.
1140 AvailableFeatures=<features>
1142 Identify the feature(s) available on the specified node. Any
1143 previously defined available feature specification will be over‐
1144 written with the new value. AvailableFeatures assigned via
1145 scontrol will only persist across the restart of the slurmctld
1146 daemon with the -R option and state files preserved or slurm‐
1147 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1148 meant to be persistent across normal restarts of slurmctld or
1149 the execution of scontrol reconfig.
1150 Note: Available features being removed via scontrol must not be
1152 active (i.e. remove them from ActiveFeatures first).
1153 Comment=<comment>
1155 Arbitrary descriptive string. Use quotes to enclose a comment
1156 having more than one word
1157 CpuBind=<node>
1159 Specify the task binding mode to be used by default for this
1160 node. Supported options include: "none", "socket", "ldom"
1161 (NUMA), "core", "thread" and "off" (remove previous binding
1162 mode).
1163 Extra=<comment>
1165 Arbitrary string on the node. Use quotes to enclose a string
1166 having more than one word.
1167 Gres=<gres>
1169 Identify generic resources to be associated with the specified
1170 node. Any previously defined generic resources will be over‐
1171 written with the new value. Specifications for multiple generic
1172 resources should be comma separated. Each resource specifica‐
1173 tion consists of a name followed by an optional colon with a nu‐
1174 meric value (default value is one) (e.g. "Gres=band‐
1175 width:10000"). Modification of GRES count associated with spe‐
1176 cific files (e.g. GPUs) is not allowed other than to set their
1177 count on a node to zero. In order to change the GRES count to
1178 another value, modify your slurm.conf and gres.conf files and
1179 restart daemons. If GRES are associated with specific sockets,
1180 that information will be reported For example if all 4 GPUs on a
1181 node are all associated with socket zero, then
1182 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1183 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1184 associated with specific GPUs is not reported, but only avail‐
1185 able by parsing the gres.conf file. Generic resources assigned
1186 via scontrol will only persist across the restart of the slurm‐
1187 ctld daemon with the -R option and state files preserved or
1188 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1189 changes meant to be persistent across normal restarts of slurm‐
1190 ctld or the execution of scontrol reconfig.
1191 NodeAddr=<node address>
1193 Name that a node should be referred to in establishing a commu‐
1194 nications path. This name will be used as an argument to the
1195 getaddrinfo() function for identification. If a node range ex‐
1196 pression is used to designate multiple nodes, they must exactly
1197 match the entries in the NodeName (e.g. "NodeName=lx[0-7]
1198 NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses.
1199 NodeHostname=<node hostname>
1201 Typically this would be the string that "/bin/hostname -s" re‐
1202 turns. It may also be the fully qualified domain name as re‐
1203 turned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid
1204 domain name associated with the host through the host database
1205 (/etc/hosts) or DNS, depending on the resolver settings. Note
1206 that if the short form of the hostname is not used, it may pre‐
1207 vent use of hostlist expressions (the numeric portion in brack‐
1208 ets must be at the end of the string). A node range expression
1209 can be used to specify a set of nodes. If an expression is used,
1210 the number of nodes identified by NodeHostname must be identical
1211 to the number of nodes identified by NodeName.
1212 Reason=<reason>
1214 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1215 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1216 having more than one word.
1217 State=<state>
1219 Assign one of the following states/actions to the node(s) speci‐
1220 fied by the update command.
1221 CANCEL_REBOOT
1223 Cancels a pending reboot on the node (same as scontrol
1224 cancel_reboot <node>).
1225 DOWN Stop all running and suspended jobs and make the node un‐
1227 available for new jobs.
1228 DRAIN Indicates that no new jobs may be started on this node.
1230 Existing jobs are allowed to run to completion, leaving
1231 the node in a DRAINED state once all the jobs have com‐
1232 pleted.
1233 FAIL Similar to DRAIN except that some applications will seek
1235 to relinquish those nodes before the job completes.
1236 FUTURE Indicates the node is not fully configured, but is ex‐
1238 pected to be available at some point in the future.
1239 NoResp This will set the "Not Responding" flag for a node with‐
1241 out changing its underlying state.
1242 POWER_DOWN
1244 Will use the configured SuspendProgram program to explic‐
1245 itly place a node in power saving mode. If a node is al‐
1246 ready in the process of being powered down, the command
1247 will only change the state of the node but won't have any
1248 effect until the configured SuspendTimeout is reached.
1249 Use of this command can be useful in situations where a
1250 ResumeProgram, like capmc in Cray machines, is stalled
1251 and one wants to restore the node to "IDLE" manually. In
1252 this case rebooting the node and setting the state to
1253 "POWER_DOWN" will cancel the previous "POWER_UP" state
1254 and the node will become "IDLE".
1255 POWER_DOWN_ASAP
1257 Will drain the node and mark it for power down. Currently
1258 running jobs will complete first and no additional jobs
1259 will be allocated to the node.
1260 POWER_DOWN_FORCE
1262 Will cancel all jobs on the node, power it down, and re‐
1263 set its state to "IDLE".
1264 POWER_UP
1266 Will use the configured ResumeProgram program to explic‐
1267 itly move a node out of power saving mode. If a node is
1268 already in the process of being powered up, the command
1269 will only change the state of the node but won't have any
1270 effect until the configured ResumeTimeout is reached.
1271 RESUME Not an actual node state, but will change a node state
1273 from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp.
1274 slurmctld will then attempt to contact slurmd to request
1275 that the node register itself. Once registered, the node
1276 state will then remove the NoResp flag and will resume
1277 normal operations. It will also clear the POWERING_DOWN
1278 state of a node and make it eligible to be allocted.
1279 UNDRAIN
1281 Clears the node from being drained (like RESUME), but
1282 will not change the node's base state (e.g. DOWN).
1283 UNDRAIN requires a valid node registration before new
1284 jobs can be scheduled on the node. Setting a node DOWN
1285 will cause all running and suspended jobs on that node to
1286 be terminated.
1287 While all of the above states are valid, some of them are not
1289 valid new node states given their prior state.
1290 NOTE: The scontrol command should not be used to change node
1292 state on Cray systems. Use Cray tools such as xtprocadmin in‐
1293 stead.
1294 Weight=<weight>
1296 Identify weight to be associated with specified nodes. This al‐
1297 lows dynamic changes to weight associated with nodes, which will
1298 be used for the subsequent node allocation decisions. Weight
1299 assigned via scontrol will only persist across the restart of
1300 the slurmctld daemon with the -R option and state files pre‐
1301 served or slurmctld's receipt of a SIGHUP. Update slurm.conf
1302 with any changes meant to be persistent across normal restarts
1303 of slurmctld or the execution of scontrol reconfig.
1304
1306 AllocMem
1307 The total memory, in MB, currently allocated by jobs on the
1308 node.
1309 CPUSpecList
1311 The list of Slurm abstract CPU IDs on this node reserved for ex‐
1312 clusive use by the Slurm compute node daemons (slurmd, slurm‐
1313 stepd).
1314 FreeMem
1316 The total memory, in MB, currently free on the node as reported
1317 by the OS.
1318 LastBusyTime
1320 The last time the node was busy (i.e. last time the node had
1321 jobs on it). This time is used in PowerSave to determine when to
1322 suspend nodes (e.g. now - LastBusy > SuspendTime).
1323 MemSpecLimit
1325 The combined memory limit, in megabytes, on this node for the
1326 Slurm compute node daemons (slurmd, slurmstepd).
1327 RealMemory
1329 The total memory, in MB, on the node.
1330 State Identify the state(s) assigned to the node with '+' delimited
1332 state flags.
1333 States:
1335 ALLOCATED
1337 Indicates that the node has all CPUs allocated to job(s)
1338 running on the node.
1339 DOWN The node does not have any running jobs and is unavail‐
1341 able for new work.
1342 ERROR The node is in an error state. Consult the logs for more
1344 information about what caused this state.
1345 FUTURE The node is currently not fully configured, but expected
1347 to be available at some point in the indefinite future
1348 for use.
1349 IDLE Indicates that the node is available for work but does
1351 not currently have any jobs assigned to it.
1352 MIXED Indicates that the node is in multiple states. For in‐
1354 stance if only part of the node is ALLOCATED and the rest
1355 of the node is IDLE the state will be MIXED.
1356 UNKNOWN
1358 The node has not yet registered with the controller and
1359 its state is not known.
1360 Flags:
1362 CLOUD Indicates that the node is configured as a cloud node, to
1364 be brought up on demand, but not currently running.
1365 COMPLETING
1367 Indicates that the only job on the node or that all jobs
1368 on the node are in the process of completing.
1369 DRAIN The node is not accepting any new jobs and any currently
1371 running jobs will complete.
1372 DYNAMIC
1374 Slurm allows you to define multiple types of nodes in a
1375 FUTURE state. When starting slurmd on a node you can
1376 specify the -F flag to have the node match and use an ex‐
1377 isting definition in your slurm.conf file. The DYNAMIC
1378 state indicates that the node was started as a Dynamic
1379 Future node.
1380 INVALID_REG
1382 The node did not register correctly with the controller.
1383 This happens when a node registers with less resources
1384 than configured in the slurm.conf file. The node will
1385 clear from this state with a valid registration (i.e. a
1386 slurmd restart is required).
1387 MAINTENANCE
1389 The node is currently in a reservation that includes the
1390 maintenance flag.
1391 NOT_RESPONDING
1393 Node is not responding.
1394 PERFCTRS
1396 Indicates that Network Performance Counters associated
1397 with this node are in use, rendering this node as not us‐
1398 able for any other jobs.
1399 POWER_DOWN
1401 Node is pending power down.
1402 POWERED_DOWN
1404 Node is currently powered down and not capable of running
1405 any jobs.
1406 POWERING_DOWN
1408 Node is in the process of powering down.
1409 POWERING_UP
1411 Node is in the process of powering up.
1412 PLANNED
1414 The node is earmarked for a job that will start in the
1415 future.
1416 REBOOT_ISSUED
1418 A reboot request has been sent to the agent configured to
1419 handle this request.
1420 REBOOT_REQUESTED
1422 A request to reboot this node has been made, but hasn't
1423 been handled yet.
1424 RESERVED
1426 Indicates the node is in an advanced reservation and not
1427 generally available.
1428 The meaning of the energy information is as follows:
1430 CurrentWatts
1432 The instantaneous power consumption of the node at the
1433 time of the last node energy accounting sample, in watts.
1434 LowestJoules
1436 The energy consumed by the node between the last time it
1437 was powered on and the last time it was registered by
1438 slurmd, in joules.
1439 ConsumedJoules
1441 The energy consumed by the node between the last time it
1442 was registered by the slurmd daemon and the last node en‐
1443 ergy accounting sample, in joules.
1444 If the reported value is "n/s" (not supported), the node does
1446 not support the configured AcctGatherEnergyType plugin. If the
1447 reported value is zero, energy accounting for nodes is disabled.
1448 The meaning of the external sensors information is as follows:
1451 ExtSensorsJoules
1453 The energy consumed by the node between the last time it
1454 was powered on and the last external sensors plugin node
1455 sample, in joules.
1456 ExtSensorsWatts
1458 The instantaneous power consumption of the node at the
1459 time of the last external sensors plugin node sample, in
1460 watts.
1461 ExtSensorsTemp
1463 The temperature of the node at the time of the last ex‐
1464 ternal sensors plugin node sample, in celsius.
1465 If the reported value is "n/s" (not supported), the node does
1467 not support the configured ExtSensorsType plugin.
1468
1470 FrontendName=<name>
1471 Identify the front end node to be updated. This specification is
1472 required.
1473 Reason=<reason>
1475 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1476 Use quotes to enclose a reason having more than one word.
1477 State=<state>
1479 Identify the state to be assigned to the front end node. Possi‐
1480 ble values are "DOWN", "DRAIN" or "RESUME". If you want to re‐
1481 move a front end node from service, you typically want to set
1482 its state to "DRAIN". "RESUME" is not an actual node state, but
1483 will return a "DRAINED", "DRAINING", or "DOWN" front end node to
1484 service, either "IDLE" or "ALLOCATED" state as appropriate.
1485 Setting a front end node "DOWN" will cause all running and sus‐
1486 pended jobs on that node to be terminated.
1487
1489 AllowGroups=<name>
1490 Identify the user groups which may use this partition. Multiple
1491 groups may be specified in a comma separated list. To permit
1492 all groups to use the partition specify "AllowGroups=ALL".
1493 AllocNodes=<name>
1495 Comma separated list of nodes from which users can execute jobs
1496 in the partition. Node names may be specified using the node
1497 range expression syntax described above. The default value is
1498 "ALL".
1499 Alternate=<partition name>
1501 Alternate partition to be used if the state of this partition is
1502 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1503 set alternate partition.
1504 CpuBind=<node>
1506 Specify the task binding mode to be used by default for this
1507 partition. Supported options include: "none", "socket", "ldom"
1508 (NUMA), "core", "thread" and "off" (remove previous binding
1509 mode).
1510 Default={yes|no}
1512 Specify if this partition is to be used by jobs which do not ex‐
1513 plicitly identify a partition to use. Possible output values
1514 are "YES" and "NO". In order to change the default partition of
1515 a running system, use the scontrol update command and set De‐
1516 fault=yes for the partition that you want to become the new de‐
1517 fault.
1518 DefaultTime=<time>
1520 Run time limit used for jobs that don't specify a value. If not
1521 set then MaxTime will be used. Format is the same as for Max‐
1522 Time.
1523 DefMemPerCPU=<MB>
1525 Set the default memory to be allocated per CPU for jobs in this
1526 partition. The memory size is specified in megabytes.
1527 DefMemPerNode=<MB>
1529 Set the default memory to be allocated per node for jobs in this
1530 partition. The memory size is specified in megabytes.
1531 DisableRootJobs={yes|no}
1533 Specify if jobs can be executed as user root. Possible values
1534 are "YES" and "NO".
1535 GraceTime=<seconds>
1537 Specifies, in units of seconds, the preemption grace time to be
1538 extended to a job which has been selected for preemption. The
1539 default value is zero, no preemption grace time is allowed on
1540 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1541 Hidden={yes|no}
1543 Specify if the partition and its jobs should be hidden from
1544 view. Hidden partitions will by default not be reported by
1545 Slurm APIs or commands. Possible values are "YES" and "NO".
1546 JobDefaults=<specs>
1548 Specify job default values using a comma-delimited list of
1549 "key=value" pairs. Supported keys include
1550 DefCpuPerGPU Default number of CPUs per allocated GPU.
1552 DefMemPerGPU Default memory limit (in megabytes) per allocated
1554 GPU.
1555 MaxMemPerCPU=<MB>
1557 Set the maximum memory to be allocated per CPU for jobs in this
1558 partition. The memory size is specified in megabytes.
1559 MaxMemPerCNode=<MB>
1561 Set the maximum memory to be allocated per node for jobs in this
1562 partition. The memory size is specified in megabytes.
1563 MaxNodes=<count>
1565 Set the maximum number of nodes which will be allocated to any
1566 single job in the partition. Specify a number, "INFINITE" or
1567 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1568 upon jobs that have already begun execution.
1569 MaxTime=<time>
1571 The maximum run time for jobs. Output format is
1572 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1573 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1574 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1575 onds. Time resolution is one minute and second values are
1576 rounded up to the next minute. Changing the MaxTime of a parti‐
1577 tion has no effect upon jobs that have already begun execution.
1578 MinNodes=<count>
1580 Set the minimum number of nodes which will be allocated to any
1581 single job in the partition. Changing the MinNodes of a parti‐
1582 tion has no effect upon jobs that have already begun execution.
1583 Increasing this value may prevent pending jobs from starting,
1584 even if they were submitted without -N/--nodes specification.
1585 If you do get in that situation, updating the MinNodes value of
1586 a pending job using the scontrol command will allow that job to
1587 be scheduled.
1588 Nodes=<name>
1590 Identify the node(s) to be associated with this partition. Mul‐
1591 tiple node names may be specified using simple node range ex‐
1592 pressions (e.g. "lx[10-20]"). Note that jobs may only be asso‐
1593 ciated with one partition at any time. Specify a blank data
1594 value to remove all nodes from a partition: "Nodes=". Changing
1595 the Nodes in a partition has no effect upon jobs that have al‐
1596 ready begun execution.
1597 OverTimeLimit=<count>
1599 Number of minutes by which a job can exceed its time limit be‐
1600 fore being canceled. The configured job time limit is treated
1601 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1602 vides a hard limit, at which point the job is canceled. This is
1603 particularly useful for backfill scheduling, which bases upon
1604 each job's soft time limit. A partition-specific OverTimeLimit
1605 will override any global OverTimeLimit value. If not specified,
1606 the global OverTimeLimit value will take precedence. May not
1607 exceed 65533 minutes. An input value of "UNLIMITED" will clear
1608 any previously configured partition-specific OverTimeLimit
1609 value.
1610 OverSubscribe={yes|no|exclusive|force}[:<job_count>]
1612 Specify if compute resources (i.e. individual CPUs) in this par‐
1613 tition can be shared by multiple jobs. Possible values are
1614 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1615 specifies how many jobs can be allocated to use each resource.
1616 PartitionName=<name>
1618 Identify the partition to be updated. This specification is re‐
1619 quired.
1620 PreemptMode=<mode>
1622 Reset the mechanism used to preempt jobs in this partition if
1623 PreemptType is configured to preempt/partition_prio. The default
1624 preemption mechanism is specified by the cluster-wide Preempt‐
1625 Mode configuration parameter. Possible values are "OFF", "CAN‐
1626 CEL", "REQUEUE" and "SUSPEND".
1627 Priority=<count>
1629 Jobs submitted to a higher priority partition will be dispatched
1630 before pending jobs in lower priority partitions and if possible
1631 they will preempt running jobs from lower priority partitions.
1632 Note that a partition's priority takes precedence over a job's
1633 priority. The value may not exceed 65533.
1634 PriorityJobFactor=<count>
1636 Partition factor used by priority/multifactor plugin in calcu‐
1637 lating job priority. The value may not exceed 65533. Also see
1638 PriorityTier.
1639 PriorityTier=<count>
1641 Jobs submitted to a partition with a higher priority tier value
1642 will be dispatched before pending jobs in partition with lower
1643 priority tier value and, if possible, they will preempt
1644 running jobs from partitions with lower priority tier values.
1645 Note that a partition's priority tier takes precedence over a
1646 job's priority. The value may not exceed 65533. Also see Pri‐
1647 orityJobFactor.
1648 QOS=<QOSname|blank to remove>
1650 Set the partition QOS with a QOS name or to remove the Partition
1651 QOS leave the option blank.
1652 RootOnly={yes|no}
1654 Specify if only allocation requests initiated by user root will
1655 be satisfied. This can be used to restrict control of the par‐
1656 tition to some meta-scheduler. Possible values are "YES" and
1657 "NO".
1658 ReqResv={yes|no}
1660 Specify if only allocation requests designating a reservation
1661 will be satisfied. This is used to restrict partition usage to
1662 be allowed only within a reservation. Possible values are "YES"
1663 and "NO".
1664 Shared={yes|no|exclusive|force}[:<job_count>]
1666 Renamed to OverSubscribe, see option descriptions above.
1667 State={up|down|drain|inactive}
1669 Specify if jobs can be allocated nodes or queued in this parti‐
1670 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1671 UP Designates that new jobs may queued on the partition,
1673 and that jobs may be allocated nodes and run from the
1674 partition.
1675 DOWN Designates that new jobs may be queued on the parti‐
1677 tion, but queued jobs may not be allocated nodes and
1678 run from the partition. Jobs already running on the
1679 partition continue to run. The jobs must be explicitly
1680 canceled to force their termination.
1681 DRAIN Designates that no new jobs may be queued on the par‐
1683 tition (job submission requests will be denied with an
1684 error message), but jobs already queued on the parti‐
1685 tion may be allocated nodes and run. See also the
1686 "Alternate" partition specification.
1687 INACTIVE Designates that no new jobs may be queued on the par‐
1689 tition, and jobs already queued may not be allocated
1690 nodes and run. See also the "Alternate" partition
1691 specification.
1692 TRESBillingWeights=<TRES Billing Weights>
1694 TRESBillingWeights is used to define the billing weights of each
1695 TRES type that will be used in calculating the usage of a job.
1696 The calculated usage is used when calculating fairshare and when
1697 enforcing the TRES billing limit on jobs. Updates affect new
1698 jobs and not existing jobs. See the slurm.conf man page for
1699 more information.
1700
1702 Reservation=<name>
1703 Identify the name of the reservation to be created, updated, or
1704 deleted. This parameter is required for update and is the only
1705 parameter for delete. For create, if you do not want to give a
1706 reservation name, use "scontrol create res ..." and a name will
1707 be created automatically.
1708 Accounts=<account list>
1710 List of accounts permitted to use the reserved nodes, for exam‐
1711 ple "Accounts=physcode1,physcode2". A user in any of the ac‐
1712 counts may use the reserved nodes. A new reservation must spec‐
1713 ify Users or Groups and/or Accounts. If both Users/Groups and
1714 Accounts are specified, a job must match both in order to use
1715 the reservation. Accounts can also be denied access to reserva‐
1716 tions by preceding all of the account names with '-'. Alter‐
1717 nately precede the equal sign with '-'. For example, "Ac‐
1718 counts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2"
1719 will permit any account except physcode1 and physcode2 to use
1720 the reservation. You can add or remove individual accounts from
1721 an existing reservation by using the update command and adding a
1722 '+' or '-' sign before the '=' sign. If accounts are denied ac‐
1723 cess to a reservation (account name preceded by a '-'), then all
1724 other accounts are implicitly allowed to use the reservation and
1725 it is not possible to also explicitly specify allowed accounts.
1726 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1728 Specification of burst buffer resources which are to be re‐
1729 served. "buffer_spec" consists of four elements:
1730 [plugin:][type:]#[units] "plugin" is the burst buffer plugin
1731 name, currently either "datawarp" or "generic". If no plugin is
1732 specified, the reservation applies to all configured burst buf‐
1733 fer plugins. "type" specifies a Cray generic burst buffer re‐
1734 source, for example "nodes". if "type" is not specified, the
1735 number is a measure of storage space. The "units" may be "N"
1736 (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers
1737 of 1024) and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).
1738 The default units are bytes for reservations of storage space.
1739 For example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage
1740 plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (re‐
1741 serve 100 GB of storage from all configured burst buffer plug‐
1742 ins). Jobs using this reservation are not restricted to these
1743 burst buffer resources, but may use these reserved resources
1744 plus any which are generally available. NOTE: Usually Slurm in‐
1745 terprets KB, MB, GB, TB, PB, TB units as powers of 1024, but for
1746 Burst Buffers size specifications Slurm supports both IEC/SI
1747 formats. This is because the CRAY API for managing DataWarps
1748 supports both formats.
1749 CoreCnt=<num>
1751 This option is only supported when SelectType=select/cons_res or
1752 select/cons_tres. Identify number of cores to be reserved. If
1753 NodeCnt is used without the FIRST_CORES flag, this is the total
1754 number of cores to reserve where cores per node is
1755 CoreCnt/NodeCnt. If a nodelist is used, or if NodeCnt is used
1756 with the FIRST_CORES flag, this should be an array of core num‐
1757 bers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or
1758 flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
1759 Licenses=<license>
1761 Specification of licenses (or other resources available on all
1762 nodes of the cluster) which are to be reserved. License names
1763 can be followed by a colon and count (the default count is one).
1764 Multiple license names should be comma separated (e.g. "Li‐
1765 censes=foo:4,bar"). A new reservation must specify one or more
1766 resource to be included: NodeCnt, Nodes and/or Licenses. If a
1767 reservation includes Licenses, but no NodeCnt or Nodes, then the
1768 option Flags=LICENSE_ONLY must also be specified. Jobs using
1769 this reservation are not restricted to these licenses, but may
1770 use these reserved licenses plus any which are generally avail‐
1771 able.
1772 MaxStartDelay[=<timespec>]
1774 Change MaxStartDelay value which specifies the maximum time an
1775 eligible job not requesting this reservation can delay a job re‐
1776 questing it. Default is none. Valid formats are minutes, min‐
1777 utes:seconds, hours:minutes:seconds, days-hours, days-hours:min‐
1778 utes, days-hours:minutes:seconds. Time resolution is one minute
1779 and second values are rounded up to the next minute. Output for‐
1780 mat is always [days-]hours:minutes:seconds.
1781 NodeCnt=<num>[,<num>,...]
1783 Identify number of nodes to be reserved. The number can include
1784 a suffix of "k" or "K", in which case the number specified is
1785 multiplied by 1024. A new reservation must specify one or more
1786 resource to be included: NodeCnt, Nodes and/or Licenses.
1787 Nodes=<name>
1789 Identify the node(s) to be reserved. Multiple node names may be
1790 specified using simple node range expressions (e.g.
1791 "Nodes=lx[10-20]"). Specify a blank data value to remove all
1792 nodes from a reservation: "Nodes=". A new reservation must
1793 specify one or more resource to be included: NodeCnt, Nodes
1794 and/or Licenses. A specification of "ALL" will reserve all
1795 nodes. Set Flags=PART_NODES and PartitionName= in order for
1796 changes in the nodes associated with a partition to also be re‐
1797 flected in the nodes associated with a reservation.
1798 StartTime=<time_spec>
1800 The start time for the reservation. A new reservation must
1801 specify a start time. It accepts times of the form HH:MM:SS for
1802 a specific time of day (seconds are optional). (If that time is
1803 already past, the next day is assumed.) You may also specify
1804 midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
1805 time-of-day suffixed with AM or PM for running in the morning or
1806 the evening. You can also say what day the job will be run, by
1807 specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or
1808 a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also give
1809 times like now + count time-units, where the time-units can be
1810 seconds (default), minutes, hours, days, or weeks and you can
1811 tell Slurm to run the job today with the keyword today and to
1812 run the job tomorrow with the keyword tomorrow. You cannot up‐
1813 date the StartTime of a reservation in ACTIVE state.
1814 EndTime=<time_spec>
1816 The end time for the reservation. A new reservation must spec‐
1817 ify an end time or a duration. Valid formats are the same as
1818 for StartTime.
1819 Duration=<time>
1821 The length of a reservation. A new reservation must specify an
1822 end time or a duration. Valid formats are minutes, minutes:sec‐
1823 onds, hours:minutes:seconds, days-hours, days-hours:minutes,
1824 days-hours:minutes:seconds, or UNLIMITED. Time resolution is
1825 one minute and second values are rounded up to the next minute.
1826 Output format is always [days-]hours:minutes:seconds.
1827 PartitionName=<name>
1829 Partition used to reserve nodes from. This will attempt to allo‐
1830 cate all nodes in the specified partition unless you request
1831 fewer resources than are available with CoreCnt, NodeCnt or
1832 TRES. Jobs will be allowed to use this reservation even if run‐
1833 ning in a different partition. There only needs to be overlap‐
1834 ping nodes from that different partition and the nodes used in
1835 the reservation.
1836 Flags=<flags>
1838 Flags associated with the reservation. You can add or remove
1839 individual flags from an existing reservation by adding a '+' or
1840 '-' sign before the '=' sign. For example: Flags-=DAILY (NOTE:
1841 this shortcut is not supported for all flags). Currently sup‐
1842 ported flags include:
1843 ANY_NODES This is a reservation for burst buffers and/or li‐
1845 censes only and not compute nodes. If this flag
1846 is set, a job using this reservation may use the
1847 associated burst buffers and/or licenses plus any
1848 compute nodes. If this flag is not set, a job us‐
1849 ing this reservation may use only the nodes and
1850 licenses associated with the reservation.
1851 DAILY Repeat the reservation at the same time every day.
1853 FLEX Permit jobs requesting the reservation to begin
1855 prior to the reservation's start time, end after
1856 the reservation's end time, and use any resources
1857 inside and/or outside of the reservation regard‐
1858 less of any constraints possibly set in the reser‐
1859 vation. A typical use case is to prevent jobs not
1860 explicitly requesting the reservation from using
1861 those reserved resources rather than forcing jobs
1862 requesting the reservation to use those resources
1863 in the time frame reserved. Another use case could
1864 be to always have a particular number of nodes
1865 with a specific feature reserved for a specific
1866 account so users in this account may use this
1867 nodes plus possibly other nodes without this fea‐
1868 ture.
1869 FIRST_CORES Use the lowest numbered cores on a node only. Flag
1871 removal with '-=' is not supported.
1872 IGNORE_JOBS Ignore currently running jobs when creating the
1874 reservation. This can be especially useful when
1875 reserving all nodes in the system for maintenance.
1876 LICENSE_ONLY See ANY_NODES.
1878 MAINT Maintenance mode, receives special accounting
1880 treatment. This reservation is permitted to use
1881 resources that are already in another reservation.
1882 MAGNETIC This flag allows jobs to be considered for this
1884 reservation even if they didn't request it.
1885 NO_HOLD_JOBS_AFTER
1887 By default, when a reservation ends the reserva‐
1888 tion request will be removed from any pending jobs
1889 submitted to the reservation and will be put into
1890 a held state. Use this flag to let jobs run out‐
1891 side of the reservation after the reservation is
1892 gone. Flag removal with '-=' is not supported.
1893 OVERLAP This reservation can be allocated resources that
1895 are already in another reservation. Flag removal
1896 with '-=' is not supported.
1897 PART_NODES This flag can be used to reserve all nodes within
1899 the specified partition. PartitionName and
1900 Nodes=ALL must be specified with this flag.
1901 PURGE_COMP[=<timespec>]
1903 Purge the reservation if it is ever idle for time‐
1904 spec (no jobs associated with it). If timespec
1905 isn't given then 5 minutes is the default. Valid
1906 timespec formats are minutes, minutes:seconds,
1907 hours:minutes:seconds, days-hours, days-hours:min‐
1908 utes, days-hours:minutes:seconds. Time resolution
1909 is one minute and second values are rounded up to
1910 the next minute. Output format is always
1911 [days-]hours:minutes:seconds.
1912 REPLACE Nodes which are DOWN, DRAINED, or allocated to
1914 jobs are automatically replenished using idle re‐
1915 sources. This option can be used to maintain a
1916 constant number of idle resources available for
1917 pending jobs (subject to availability of idle re‐
1918 sources). This should be used with the NodeCnt
1919 reservation option; do not identify specific nodes
1920 to be included in the reservation. Flag removal
1921 with '-=' is not supported.
1922 NOTE: Removing a node from the cluster while in a
1924 reservation with the REPLACE flag will not cause
1925 it to be replaced.
1926 REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically
1928 replenished using idle resources. This option can
1929 be used to maintain a constant sized pool of re‐
1930 sources available for pending jobs (subject to
1931 availability of idle resources). This should be
1932 used with the NodeCnt reservation option; do not
1933 identify specific nodes to be included in the
1934 reservation. Flag removal with '-=' is not sup‐
1935 ported.
1936 NOTE: Removing a node from the cluster while in a
1938 reservation with the REPLACE_DOWN flag will not
1939 cause it to be replaced.
1940 SPEC_NODES Reservation is for specific nodes (output only).
1942 STATIC_ALLOC Make it so after the nodes are selected for a
1944 reservation they don't change. Without this op‐
1945 tion when nodes are selected for a reservation and
1946 one goes down the reservation will select a new
1947 node to fill the spot.
1948 TIME_FLOAT The reservation start time is relative to the cur‐
1950 rent time and moves forward through time (e.g. a
1951 StartTime=now+10minutes will always be 10 minutes
1952 in the future). Repeating (e.g. DAILY) floating
1953 reservations are not supported. Flag cannot be
1954 added to or removed from an existing reservation.
1955 WEEKDAY Repeat the reservation at the same time on every
1957 weekday (Monday, Tuesday, Wednesday, Thursday and
1958 Friday).
1959 WEEKEND Repeat the reservation at the same time on every
1961 weekend day (Saturday and Sunday).
1962 WEEKLY Repeat the reservation at the same time every
1964 week.
1965 Features=<features>
1967 Set the reservation's required node features. Multiple values
1968 may be "&" separated if all features are required (AND opera‐
1969 tion) or separated by "|" if any of the specified features are
1970 required (OR operation). Parenthesis are also supported for
1971 features to be ANDed together with counts of nodes having the
1972 specified features. For example "Fea‐
1973 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
1974 reservation should include 4 nodes with ALL of the features
1975 "knl", "a2a", and "flat" plus 2 nodes with the feature
1976 "haswell".
1977 Value may be cleared with blank data value, "Features=".
1979 Groups=<group list>
1981 List of groups permitted to use the reserved nodes, for example
1982 "Group=bio,chem". A new reservation must specify Users or
1983 Groups and/or Accounts. If both Users/Groups and Accounts are
1984 specified, a job must match both in order to use the reserva‐
1985 tion. Unlike users groups do not allow denied access to reser‐
1986 vations. You can add or remove individual groups from an exist‐
1987 ing reservation by using the update command and adding a '+' or
1988 '-' sign before the '=' sign. NOTE: Groups and Users are mutu‐
1989 ally exclusive in reservations, if you want to switch between
1990 the 2 you must update the reservation with a group='' or user=''
1991 and fill in the opposite with the appropriate setting.
1992 Skip Used on a reoccurring reservation, skip to the next reservation
1994 iteration. NOTE: Only available for update.
1995 Users=<user list>
1997 List of users permitted to use the reserved nodes, for example
1998 "User=jones1,smith2". A new reservation must specify Users or
1999 Groups and/or Accounts. If both Users/Groups and Accounts are
2000 specified, a job must match both in order to use the reserva‐
2001 tion. Users can also be denied access to reservations by pre‐
2002 ceding all of the user names with '-'. Alternately precede the
2003 equal sign with '-'. For example, "User=-jones1,-smith2" or
2004 "User-=jones1,smith2" will permit any user except jones1 and
2005 smith2 to use the reservation. You can add or remove individual
2006 users from an existing reservation by using the update command
2007 and adding a '+' or '-' sign before the '=' sign. If users are
2008 denied access to a reservation (user name preceded by a '-'),
2009 then all other users are implicitly allowed to use the reserva‐
2010 tion and it is not possible to also explicitly specify allowed
2011 users. NOTE: Groups and Users are mutually exclusive in reser‐
2012 vations, if you want to switch between the 2 you must update the
2013 reservation with a group='' or user='' and fill in the opposite
2014 with the appropriate setting.
2015 TRES=<tres_spec>
2017 Comma-separated list of TRES required for the reservation. Cur‐
2018 rent supported TRES types with reservations are: CPU, Node, Li‐
2019 cense and BB. CPU and Node follow the same format as CoreCnt and
2020 NodeCnt parameters respectively. License names can be followed
2021 by an equal '=' and a count:
2022 License/<name1>=<count1>[,License/<name2>=<count2>,...]
2024 BurstBuffer can be specified in a similar way as BurstBuffer pa‐
2026 rameter. The only difference is that colon symbol ':' should be
2027 replaced by an equal '=' in order to follow the TRES format.
2028 Some examples of TRES valid specifications:
2030 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
2032 TRES=node=5k,license/iop1=2
2034 As specified in CoreCnt, if a nodelist is specified, cpu can be
2036 an array of core numbers by node: nodes=compute[1-3]
2037 TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
2038 Please note that CPU, Node, License and BB can override CoreCnt,
2040 NodeCnt, Licenses and BurstBuffer parameters respectively. Also
2041 CPU represents CoreCnt, in a reservation and will be adjusted if
2042 you have threads per core on your nodes.
2043 Note that a reservation that contains nodes or cores is associ‐
2045 ated with one partition, and can't span resources over multiple
2046 partitions. The only exception from this is when the reserva‐
2047 tion is created with explicitly requested nodes.
2048
2050 Executing scontrol sends a remote procedure call to slurmctld. If
2051 enough calls from scontrol or other Slurm client commands that send re‐
2052 mote procedure calls to the slurmctld daemon come in at once, it can
2053 result in a degradation of performance of the slurmctld daemon, possi‐
2054 bly resulting in a denial of service.
2055 Do not run scontrol or other Slurm client commands that send remote
2057 procedure calls to slurmctld from loops in shell scripts or other pro‐
2058 grams. Ensure that programs limit calls to scontrol to the minimum nec‐
2059 essary for the information you are trying to gather.
2060
2063 Some scontrol options may be set via environment variables. These envi‐
2064 ronment variables, along with their corresponding options, are listed
2065 below. (Note: Command line options will always override these set‐
2066 tings.)
2067 SCONTROL_ALL -a, --all
2070 SCONTROL_FEDERATION --federation
2072 SCONTROL_FUTURE -F, --future
2074 SCONTROL_LOCAL --local
2076 SCONTROL_SIBLING --sibling
2078 SLURM_BITSTR_LEN Specifies the string length to be used for holding
2080 a job array's task ID expression. The default
2081 value is 64 bytes. A value of 0 will print the
2082 full expression with any length required. Larger
2083 values may adversely impact the application perfor‐
2084 mance.
2085 SLURM_CLUSTERS Same as --clusters
2087 SLURM_CONF The location of the Slurm configuration file.
2089 SLURM_CONF_OUT When running 'write config', the location of the
2091 Slurm configuration file to be written.
2092 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
2094 value of standard, the default value, generates
2095 output in the form
2096 "year-month-dateThour:minute:second". A value of
2097 relative returns only "hour:minute:second" if the
2098 current day. For other dates in the current year
2099 it prints the "hour:minute" preceded by "Tomorr"
2100 (tomorrow), "Ystday" (yesterday), the name of the
2101 day for the coming week (e.g. "Mon", "Tue", etc.),
2102 otherwise the date (e.g. "25 Apr"). For other
2103 years it returns a date month and year without a
2104 time (e.g. "6 Jun 2012"). All of the time stamps
2105 use a 24 hour format.
2106 A valid strftime() format can also be specified.
2108 For example, a value of "%a %T" will report the day
2109 of the week and a time stamp (e.g. "Mon 12:34:56").
2110 SLURM_TOPO_LEN Specify the maximum size of the line when printing
2112 Topology. If not set, the default value is unlim‐
2113 ited.
2114
2116 When using SlurmDBD, users who have an AdminLevel defined (Operator or
2117 Admin) and users who are account coordinators are given the authority
2118 to view and modify jobs, reservations, nodes, etc., as defined in the
2119 following table - regardless of whether a PrivateData restriction has
2120 been defined in the slurm.conf file.
2121 scontrol show job(s): Admin, Operator, Coordinator
2123 scontrol update job: Admin, Operator, Coordinator
2124 scontrol requeue: Admin, Operator, Coordinator
2125 scontrol show step(s): Admin, Operator, Coordinator
2126 scontrol update step: Admin, Operator, Coordinator
2127 scontrol show node: Admin, Operator
2129 scontrol update node: Admin
2130 scontrol create partition: Admin
2132 scontrol show partition: Admin, Operator
2133 scontrol update partition: Admin
2134 scontrol delete partition: Admin
2135 scontrol create reservation: Admin, Operator
2137 scontrol show reservation: Admin, Operator
2138 scontrol update reservation: Admin, Operator
2139 scontrol delete reservation: Admin, Operator
2140 scontrol reconfig: Admin
2142 scontrol shutdown: Admin
2143 scontrol takeover: Admin
2144
2147 $ scontrol
2148 scontrol: show part debug
2149 PartitionName=debug
2150 AllocNodes=ALL AllowGroups=ALL Default=YES
2151 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2152 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2153 Nodes=snowflake[0-48]
2154 Priority=1 RootOnly=NO OverSubscribe=YES:4
2155 State=UP TotalCPUs=694 TotalNodes=49
2156 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2157 scontrol: show job 71701
2158 JobId=71701 Name=hostname
2159 UserId=da(1000) GroupId=da(1000)
2160 Priority=66264 Account=none QOS=normal WCKey=*123
2161 JobState=COMPLETED Reason=None Dependency=(null)
2162 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
2163 SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
2164 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2165 SuspendTime=None SecsPreSuspend=0
2166 Partition=debug AllocNode:Sid=snowflake:4702
2167 ReqNodeList=(null) ExcNodeList=(null)
2168 NodeList=snowflake0
2169 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2170 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2171 Features=(null) Reservation=(null)
2172 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2173 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2174 scontrol: show hostnames tux[1-3]
2175 tux1
2176 tux2
2177 tux3
2178 scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
2179 Reservation created: dbremer_1
2180 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2181 scontrol: delete Reservation=dbremer_1
2182 scontrol: quit
2183
2186 Copyright (C) 2002-2007 The Regents of the University of California.
2187 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2188 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2189 Copyright (C) 2010-2022 SchedMD LLC.
2190 This file is part of Slurm, a resource management program. For de‐
2192 tails, see <https://slurm.schedmd.com/>.
2193 Slurm is free software; you can redistribute it and/or modify it under
2195 the terms of the GNU General Public License as published by the Free
2196 Software Foundation; either version 2 of the License, or (at your op‐
2197 tion) any later version.
2198 Slurm is distributed in the hope that it will be useful, but WITHOUT
2200 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2201 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2202 for more details.
2203
2205 /etc/slurm.conf
2206
2208 scancel(1), sinfo(1), squeue(1), slurm_create_partition [22m(3),
2209 slurm_delete_partition [22m(3), slurm_load_ctl_conf [22m(3), slurm_load_jobs
2210 (3), slurm_load_node (3), slurm_load_partitions (3), slurm_reconfigure
2211 (3), slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2212 slurm_suspend (3), slurm_takeover (3), slurm_update_job (3), slurm_up‐
2213 date_node (3), slurm_update_partition (3), slurm.conf(5), slurmctld(8)
2214March 2022 Slurm Commands scontrol(1)