1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an Ad‐
17 ministrator. If an attempt to view or modify configuration information
18 is made by an unauthorized user, an error message will be printed and
19 the requested action will not occur. If no command is entered on the
20 execute line, scontrol will operate in an interactive mode and prompt
21 for input. It will continue prompting for input and executing commands
22 until explicitly terminated. If a command is entered on the execute
23 line, scontrol will execute that command and terminate. All commands
24 and options are case-insensitive, although node names, partition names,
25 and reservation names are case-sensitive (node names "LX" and "lx" are
26 distinct). All commands and options can be abbreviated to the extent
27 that the specification is unique. A modified Slurm configuration can be
28 written to a file using the scontrol write config command. The result‐
29 ing file will be named using the convention "slurm.conf.<datetime>" and
30 located in the same directory as the original "slurm.conf" file. The
31 directory containing the original slurm.conf must be writable for this
32 to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -M, --clusters=<string>
43 The cluster to issue commands to. Only one cluster name may be
44 specified. Note that the SlurmDBD must be up for this option to
45 work properly. This option implicitly sets the --local option.
46 -d, --details
48 Causes the show command to provide additional details where
49 available.
50 --federation
52 Report jobs from federation if a member of one.
53 -F, --future
55 Report nodes in FUTURE state.
56 -h, --help
58 Print a help message describing the usage of scontrol.
59 --hide Do not display information about hidden partitions, their jobs
61 and job steps. By default, neither partitions that are config‐
62 ured as hidden nor those partitions unavailable to user's group
63 will be displayed (i.e. this is the default behavior).
64 --local
66 Show only information local to this cluster. Ignore other clus‐
67 ters in the federated if a member of one. Overrides --federa‐
68 tion.
69 -o, --oneliner
71 Print information one line per record.
72 -Q, --quiet
74 Print no warning or informational messages, only fatal error
75 messages.
76 --sibling
78 Show all sibling jobs on a federated cluster. Implies --federa‐
79 tion.
80 -u, --uid=<uid>
82 Attempt to update a job as user <uid> instead of the invoking
83 user id.
84 -v, --verbose
86 Print detailed event logging. Multiple -v's will further in‐
87 crease the verbosity of logging. By default only errors will be
88 displayed.
89 -V , --version
92 Print version information and exit.
93 COMMANDS
96 abort Instruct the Slurm controller to terminate immediately and gen‐
97 erate a core file. See "man slurmctld" for information about
98 where the core file will be written.
99 cancel_reboot <NodeList>
102 Cancel pending reboots on nodes. The node will be undrain'ed and
103 the reason cleared if the node was drained by an ASAP reboot.
104 cluster CLUSTER_NAME
107 The cluster to issue commands to. Only one cluster name may be
108 specified.
109 create SPECIFICATION
112 Create a new partition or reservation. See the full list of pa‐
113 rameters below. Include the tag "res" to create a reservation
114 without specifying a reservation name.
115 completing
118 Display all jobs in a COMPLETING state along with associated
119 nodes in either a COMPLETING or DOWN state.
120 delete SPECIFICATION
123 Delete the entry with the specified SPECIFICATION. The two
124 SPECIFICATION choices are PartitionName=<name> and Reserva‐
125 tion=<name>. Reservations and partitions should have no associ‐
126 ated jobs at the time of their deletion (modify the jobs first).
127 If the specified partition is in use, the request is denied.
128 errnumstr ERRNO
131 Given a Slurm error number, return a descriptive string.
132 fsdampeningfactor FACTOR
135 Set the FairShareDampeningFactor in slurmctld.
136 help Display a description of scontrol options and commands.
139 hold job_list
142 Prevent a pending job from being started (sets its priority to
143 0). Use the release command to permit the job to be scheduled.
144 The job_list argument is a comma separated list of job IDs OR
145 "jobname=" with the job's name, which will attempt to hold all
146 jobs having that name. Note that when a job is held by a system
147 administrator using the hold command, only a system administra‐
148 tor may release the job for execution (also see the uhold com‐
149 mand). When the job is held by its owner, it may also be re‐
150 leased by the job's owner. Additionally, attempting to hold a
151 running job will have not suspend or cancel it. But, it will set
152 the job priority to 0 and update the job reason field, which
153 would hold the job if it was requeued at a later time.
154 notify job_id message
157 Send a message to standard error of the salloc or srun command
158 or batch job associated with the specified job_id.
159 pidinfo proc_id
162 Print the Slurm job id and scheduled termination time corre‐
163 sponding to the supplied process id, proc_id, on the current
164 node. This will work only with processes on node on which scon‐
165 trol is run, and only for those processes spawned by Slurm and
166 their descendants.
167 listpids [job_id[.step_id]] [NodeName]
170 Print a listing of the process IDs in a job step (if JO‐
171 BID.STEPID is provided), or all of the job steps in a job (if
172 job_id is provided), or all of the job steps in all of the jobs
173 on the local node (if job_id is not provided or job_id is "*").
174 This will work only with processes on the node on which scontrol
175 is run, and only for those processes spawned by Slurm and their
176 descendants. Note that some Slurm configurations (ProctrackType
177 value of pgid) are unable to identify all processes associated
178 with a job or job step.
179 Note that the NodeName option is only really useful when you
181 have multiple slurmd daemons running on the same host machine.
182 Multiple slurmd daemons on one host are, in general, only used
183 by Slurm developers.
184 ping Ping the primary and secondary slurmctld daemon and report if
187 they are responding.
188 reboot [ASAP] [nextstate=<RESUME|DOWN>] [reason=<reason>] <ALL|Node‐
191 List>
192 Reboot the nodes in the system when they become idle using the
193 RebootProgram as configured in Slurm's slurm.conf file. Each
194 node will have the "REBOOT" flag added to its node state. After
195 a node reboots and the slurmd daemon starts up again, the
196 HealthCheckProgram will run once. Then, the slurmd daemon will
197 register itself with the slurmctld daemon and the "REBOOT" flag
198 will be cleared. The node's "DRAIN" state flag will be cleared
199 if the reboot was "ASAP", nextstate=resume or down. The "ASAP"
200 option adds the "DRAIN" flag to each node's state, preventing
201 additional jobs from running on the node so it can be rebooted
202 and returned to service "As Soon As Possible" (i.e. ASAP).
203 "ASAP" will also set the node reason to "Reboot ASAP" if the
204 "reason" option isn't specified. If the "nextstate" option is
205 specified as "DOWN", then the node will remain in a down state
206 after rebooting. If "nextstate" is specified as "RESUME", then
207 the nodes will resume as normal and the node's reason and
208 "DRAIN" state will be cleared. Resuming nodes will be consid‐
209 ered as available in backfill future scheduling and won't be re‐
210 placed by idle nodes in a reservation. The "reason" option sets
211 each node's reason to a user-defined message. A default reason
212 of "reboot requested" is set if no other reason is set on the
213 node. The reason will be appended with "reboot issued" when the
214 reboot is issued and "reboot complete" when the node registers
215 and has a "nextstate" of "DOWN". You must specify either a list
216 of nodes or that ALL nodes are to be rebooted. NOTE: By de‐
217 fault, this command does not prevent additional jobs from being
218 scheduled on any nodes before reboot. To do this, you can ei‐
219 ther use the "ASAP" option or explicitly drain the nodes before‐
220 hand. You can alternately create an advanced reservation to
221 prevent additional jobs from being initiated on nodes to be re‐
222 booted. Pending reboots can be cancelled by using "scontrol
223 cancel_reboot <node>" or setting the node state to "CANCEL_RE‐
224 BOOT". A node will be marked "DOWN" if it doesn't reboot within
225 ResumeTimeout.
226 reconfigure
229 Instruct all Slurm daemons to re-read the configuration file.
230 This command does not restart the daemons. This mechanism would
231 be used to modify configuration parameters (Epilog, Prolog,
232 SlurmctldLogFile, SlurmdLogFile, etc.). The Slurm controller
233 (slurmctld) forwards the request to all other daemons (slurmd
234 daemon on each compute node). Running jobs continue execution.
235 Most configuration parameters can be changed by just running
236 this command, however, Slurm daemons should be shutdown and
237 restarted if any of these parameters are to be changed: Au‐
238 thType, ControlMach, PluginDir, StateSaveLocation, Slurmctld‐
239 Host, SlurmctldPort, or SlurmdPort. The slurmctld daemon and all
240 slurmd daemons must be restarted if nodes are added to or re‐
241 moved from the cluster.
242 release job_list
245 Release a previously held job to begin execution. The job_list
246 argument is a comma separated list of job IDs OR "jobname=" with
247 the job's name, which will attempt to hold all jobs having that
248 name. Also see hold.
249 requeue [option] job_list
252 Requeue a running, suspended or finished Slurm batch job into
253 pending state. The job_list argument is a comma separated list
254 of job IDs. The command accepts the following option:
255 Incomplete
257 Operate only on jobs (or tasks of a job array) which have
258 not completed. Specifically only jobs in the following
259 states will be requeued: CONFIGURING, RUNNING, STOPPED or
260 SUSPENDED.
261 requeuehold [option] job_list
264 Requeue a running, suspended or finished Slurm batch job into
265 pending state, moreover the job is put in held state (priority
266 zero). The job_list argument is a comma separated list of job
267 IDs. A held job can be released using scontrol to reset its
268 priority (e.g. "scontrol release <job_id>"). The command ac‐
269 cepts the following options:
270 Incomplete
272 Operate only on jobs (or tasks of a job array) which have
273 not completed. Specifically only jobs in the following
274 states will be requeued: CONFIGURING, RUNNING, STOPPED or
275 SUSPENDED.
276 State=SpecialExit
278 The "SpecialExit" keyword specifies that the job has to
279 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
280 trol show job" command will display the JobState as SPE‐
281 CIAL_EXIT, while the "squeue" command as SE.
282 resume job_list
285 Resume a previously suspended job. The job_list argument is a
286 comma separated list of job IDs. Also see suspend.
287 NOTE: A suspended job releases its CPUs for allocation to other
289 jobs. Resuming a previously suspended job may result in multi‐
290 ple jobs being allocated the same CPUs, which could trigger gang
291 scheduling with some configurations or severe degradation in
292 performance with other configurations. Use of the scancel com‐
293 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
294 out releasing its CPUs for allocation to other jobs and would be
295 a preferable mechanism in many cases. If performing system
296 maintenance you may want to use suspend/resume in the following
297 way. Before suspending set all nodes to draining or set all par‐
298 titions to down so that no new jobs can be scheduled. Then sus‐
299 pend jobs. Once maintenance is done resume jobs then resume
300 nodes and/or set all partitions back to up. Use with caution.
301 schedloglevel LEVEL
304 Enable or disable scheduler logging. LEVEL may be "0", "1",
305 "disable" or "enable". "0" has the same effect as "disable". "1"
306 has the same effect as "enable". This value is temporary and
307 will be overwritten when the slurmctld daemon reads the
308 slurm.conf configuration file (e.g. when the daemon is restarted
309 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
310 parameter is present.
311 setdebug LEVEL
314 Change the debug level of the slurmctld daemon for all active
315 logging channels not originally configured off (quiet). LEVEL
316 may be an integer value between zero and nine (using the same
317 values as SlurmctldDebug in the slurm.conf file) or the name of
318 the most detailed message type to be printed: "quiet", "fatal",
319 "error", "info", "verbose", "debug", "debug2", "debug3", "de‐
320 bug4", or "debug5". This value is temporary and will be over‐
321 written whenever the slurmctld daemon reads the slurm.conf con‐
322 figuration file (e.g. when the daemon is restarted or scontrol
323 reconfigure is executed).
324 setdebugflags [+|-]FLAG
327 Add or remove DebugFlags of the slurmctld daemon. See "man
328 slurm.conf" for a list of supported DebugFlags. NOTE: Changing
329 the value of some DebugFlags will have no effect without
330 restarting the slurmctld daemon, which would set DebugFlags
331 based upon the contents of the slurm.conf configuration file.
332 show ENTITY[=ID] or ENTITY [ID]
335 Display the state of the specified entity with the specified
336 identification.
337 aliases
339 Returns all NodeName values associated with a given Node‐
340 Hostname (useful to get the list of virtual nodes associ‐
341 ated with a real node in a configuration where multiple
342 slurmd daemons execute on a single compute node).
343 assoc_mgr
345 Displays the current contents of the slurmctld's internal
346 cache for users, associations and/or qos. The output can
347 be filtered by different record types:
348 users=<user1>[...,<userN>]
350 Limit the User Records displayed to those with the
351 specified user name(s).
352 accounts=<acct1>[...,<acctN>]
354 Limit the Association Records displayed to those
355 with the specified account name(s).
356 qos=<qos1>[...,<qosN>]
358 Limit the QOS Records displayed to those with the
359 specified QOS name(s).
360 flags=users|assoc|qos
362 Specify the desired record type to be displayed.
363 If no flags are specified, all record types are
364 displayed.
365 bbstat Displays output from Cray's burst buffer status tool. Op‐
367 tions following bbstat are passed directly to the dwstat
368 command by the slurmctld daemon and the response returned
369 to the user. Equivalent to dwstat.
370 burstbuffer
372 Displays the current status of the BurstBuffer plugin.
373 config Displays parameter names from the configuration files in
375 mixed case (e.g. SlurmdPort=7003) while derived parame‐
376 ters names are in upper case only (e.g. SLURM_VERSION).
377 daemons
379 Reports which daemons should be running on this node.
380 dwstat Displays output from Cray's burst buffer status tool. Op‐
382 tions following dwstat are passed directly to the dwstat
383 command by the slurmctld daemon and the response returned
384 to the user. Equivalent to bbstat.
385 federation
387 The federation name that the controller is part of and
388 the sibling clusters part of the federation will be
389 listed.
390 frontend
392 Shows configured frontend nodes.
393 hostlist
395 Takes a list of host names and prints the hostlist ex‐
396 pression for them (the inverse of hostnames). hostlist
397 can also take the absolute pathname of a file (beginning
398 with the character '/') containing a list of hostnames.
399 Multiple node names may be specified using simple node
400 range expressions (e.g. "lx[10-20]"). All other ID values
401 must identify a single element. The job step ID is of the
402 form "job_id.step_id", (e.g. "1234.1"). slurmd reports
403 the current status of the slurmd daemon executing on the
404 same node from which the scontrol command is executed
405 (the local host). It can be useful to diagnose problems.
406 By default hostlist does not sort the node list or make
407 it unique (e.g. tux2,tux1,tux2 = tux[2,1-2]). If you
408 wanted a sorted list use hostlistsorted (e.g.
409 tux2,tux1,tux2 = tux[1-2,2]). By default, all elements
410 of the entity type specified are printed.
411 hostlistsorted
413 Takes a list of host names and prints a sorted, unique
414 hostlist expression for them. See hostlist.
415 hostnames
417 Takes an optional hostlist expression as input and writes
418 a list of individual host names to standard output (one
419 per line). If no hostlist expression is supplied, the
420 contents of the SLURM_JOB_NODELIST environment variable
421 is used. For example "tux[1-3]" is mapped to
422 "tux1","tux2" and "tux3" (one hostname per line).
423 job Displays statistics about all jobs by default. If an op‐
425 tional jobid is specified, details for just that job will
426 be displayed. If the job does not specify
427 socket-per-node, cores-per-socket or threads-per-core
428 then it will display '*' in the ReqS:C:T=*:*:* field.
429 node Displays statistics about all nodes by default. If an op‐
431 tional nodename is specified, details for just that node
432 will be displayed.
433 partition
435 Displays statistics about all partitions by default. If
436 an optional partition name is specified, details for just
437 that partition will be displayed.
438 reservation
440 Displays statistics about all reservations by default. If
441 an optional reservation name is specified, details for
442 just that reservation will be displayed.
443 slurmd Displays statistics for the slurmd running on the current
445 node.
446 step Displays statistics about all job steps by default. If an
448 optional jobid is specified, details about steps for just
449 that job will be displayed. If a jobid.stepid is speci‐
450 fied, details for just that step will be displayed.
451 topology
453 Displays information about the defined topology layout.
454 If a switch is specified, information about that switch
455 will be shown. If one node name is specified, all
456 switches connected to that node (and their parent
457 switches) will be shown. If more than one node name is
458 specified, only switches that connect to all named nodes
459 will be shown.
460 shutdown OPTION
463 Instruct Slurm daemons to save current state and terminate. By
464 default, the Slurm controller (slurmctld) forwards the request
465 all other daemons (slurmd daemon on each compute node). An OP‐
466 TION of slurmctld or controller results in only the slurmctld
467 daemon being shutdown and the slurmd daemons remaining active.
468 suspend job_list
471 Suspend a running job. The job_list argument is a comma sepa‐
472 rated list of job IDs. Use the resume command to resume its ex‐
473 ecution. User processes must stop on receipt of SIGSTOP signal
474 and resume upon receipt of SIGCONT for this operation to be ef‐
475 fective. Not all architectures and configurations support job
476 suspension. If a suspended job is requeued, it will be placed
477 in a held state. The time a job is suspended will not count
478 against a job's time limit. Only an operator, administrator,
479 SlurmUser, or root can suspend jobs.
480 takeover [INDEX]
483 Instruct one of Slurm's backup controllers (slurmctld) to take
484 over system control. By default the first backup controller (IN‐
485 DEX=1) requests control from the primary and waits for its ter‐
486 mination. After that, it switches from backup mode to controller
487 mode. If primary controller can not be contacted, it directly
488 switches to controller mode. This can be used to speed up the
489 Slurm controller fail-over mechanism when the primary node is
490 down. This can be used to minimize disruption if the computer
491 executing the primary Slurm controller is scheduled down.
492 (Note: Slurm's primary controller will take the control back at
493 startup.)
494 top job_list
497 Move the specified job IDs to the top of the queue of jobs be‐
498 longing to the identical user ID, partition name, account, and
499 QOS. The job_list argument is a comma separated ordered list of
500 job IDs. Any job not matching all of those fields will not be
501 effected. Only jobs submitted to a single partition will be ef‐
502 fected. This operation changes the order of jobs by adjusting
503 job nice values. The net effect on that user's throughput will
504 be negligible to slightly negative. This operation is disabled
505 by default for non-privileged (non-operator, admin, SlurmUser,
506 or root) users. This operation may be enabled for non-privileged
507 users by the system administrator by including the option "en‐
508 able_user_top" in the SchedulerParameters configuration parame‐
509 ter.
510 token [lifespan=<lifespan>] [username=<username>]
513 Return an auth token which can be used to support JWT authenti‐
514 cation if AuthAltTypes=auth/jwt has been enabled on the system.
515 Supports two optional arguments. lifespan= may be used to spec‐
516 ify the token's lifespan in seconds. username (only available to
517 SlurmUser/root) may be used to request a token for a different
518 username.
519 uhold job_list
522 Prevent a pending job from being started (sets its priority to
523 0). The job_list argument is a space separated list of job IDs
524 or job names. Use the release command to permit the job to be
525 scheduled. This command is designed for a system administrator
526 to hold a job so that the job owner may release it rather than
527 requiring the intervention of a system administrator (also see
528 the hold command).
529 update SPECIFICATION
532 Update job, step, node, partition, or reservation configuration
533 per the supplied specification. SPECIFICATION is in the same
534 format as the Slurm configuration file and the output of the
535 show command described above. It may be desirable to execute the
536 show command (described above) on the specific entity you want
537 to update, then use cut-and-paste tools to enter updated config‐
538 uration values to the update. Note that while most configuration
539 values can be changed using this command, not all can be changed
540 using this mechanism. In particular, the hardware configuration
541 of a node or the physical addition or removal of nodes from the
542 cluster may only be accomplished through editing the Slurm con‐
543 figuration file and executing the reconfigure command (described
544 above).
545 version
548 Display the version number of scontrol being executed.
549 wait_job job_id
552 Wait until a job and all of its nodes are ready for use or the
553 job has entered some termination state. This option is particu‐
554 larly useful in the Slurm Prolog or in the batch script itself
555 if nodes are powered down and restarted automatically as needed.
556 NOTE: Don't use scontrol wait_job in PrologSlurmctld or Prolog
558 with PrologFlags=Alloc as this will result in a deadlock.
559 write batch_script job_id optional_filename
562 Write the batch script for a given job_id to a file or to std‐
563 out. The file will default to slurm-<job_id>.sh if the optional
564 filename argument is not given. The script will be written to
565 stdout if - is given instead of a filename. The batch script
566 can only be retrieved by an admin or operator, or by the owner
567 of the job.
568 write config optional_filename
571 Write the current configuration to a file with the naming con‐
572 vention of "slurm.conf.<datetime>" in the same directory as the
573 original slurm.conf file. If a filename is given that file lo‐
574 cation with a .<datetime> suffix is created.
575 INTERACTIVE COMMANDS
578 NOTE: All commands listed below can be used in the interactive mode,
579 but NOT on the initial command line.
580 all Show all partitions, their jobs and jobs steps. This causes in‐
583 formation to be displayed about partitions that are configured
584 as hidden and partitions that are unavailable to user's group.
585 details
588 Causes the show command to provide additional details where
589 available. Job information will include CPUs and NUMA memory
590 allocated on each node. Note that on computers with hyper‐
591 threading enabled and Slurm configured to allocate cores, each
592 listed CPU represents one physical core. Each hyperthread on
593 that core can be allocated a separate task, so a job's CPU count
594 and task count may differ. See the --cpu-bind and --mem-bind
595 option descriptions in srun man pages for more information. The
596 details option is currently only supported for the show job com‐
597 mand.
598 exit Terminate scontrol interactive session.
601 hide Do not display partition, job or jobs step information for par‐
604 titions that are configured as hidden or partitions that are un‐
605 available to the user's group. This is the default behavior.
606 oneliner
609 Print information one line per record.
610 quiet Print no warning or informational messages, only fatal error
613 messages.
614 quit Terminate the execution of scontrol.
617 verbose
620 Print detailed event logging. This includes time-stamps on data
621 structures, record counts, etc.
622 !! Repeat the last command executed.
625 JOBS - SPECIFICATIONS FOR UPDATE COMMAND
628 Note that update requests done by either root, SlurmUser or Administra‐
629 tors are not subject to certain restrictions. For instance, if an Ad‐
630 ministrator changes the QOS on a pending job, certain limits such as
631 the TimeLimit will not be changed automatically as changes made by the
632 Administrators are allowed to violate these restrictions.
633 Account=<account>
636 Account name to be changed for this job's resource use. Value
637 may be cleared with blank data value, "Account=".
638 AdminComment=<spec>
640 Arbitrary descriptive string. Can only be set by a Slurm admin‐
641 istrator.
642 ArrayTaskThrottle=<count>
644 Specify the maximum number of tasks in a job array that can exe‐
645 cute at the same time. Set the count to zero in order to elimi‐
646 nate any limit. The task throttle count for a job array is re‐
647 ported as part of its ArrayTaskId field, preceded with a percent
648 sign. For example "ArrayTaskId=1-10%2" indicates the maximum
649 number of running tasks is limited to 2.
650 BurstBuffer=<spec>
652 Burst buffer specification to be changed for this job's resource
653 use. Value may be cleared with blank data value, "Burst‐
654 Buffer=". Format is burst buffer plugin specific.
655 Clusters=<spec>
657 Specifies the clusters that the federated job can run on.
658 ClusterFeatures=<spec>
660 Specifies features that a federated cluster must have to have a
661 sibling job submitted to it. Slurm will attempt to submit a sib‐
662 ling job to a cluster if it has at least one of the specified
663 features.
664 Comment=<spec>
666 Arbitrary descriptive string.
667 Contiguous=<yes|no>
669 Set the job's requirement for contiguous (consecutive) nodes to
670 be allocated. Possible values are "YES" and "NO". Only the
671 Slurm administrator or root can change this parameter.
672 CoreSpec=<count>
674 Number of cores to reserve per node for system use. The job
675 will be charged for these cores, but be unable to use them.
676 Will be reported as "*" if not constrained.
677 CPUsPerTask=<count>
679 Change the CPUsPerTask job's value.
680 Deadline=<time_spec>
682 It accepts times of the form HH:MM:SS to specify a deadline to a
683 job at a specific time of day (seconds are optional). You may
684 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
685 you can have a time-of-day suffixed with AM or PM for a deadline
686 in the morning or the evening. You can specify a deadline for
687 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
688 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
689 give times like now + count time-units, where the time-units can
690 be seconds (default), minutes, hours, days, or weeks and you can
691 tell Slurm to put a deadline for tomorrow with the keyword to‐
692 morrow. The specified deadline must be later than the current
693 time. Only pending jobs can have the deadline updated. Only
694 the Slurm administrator or root can change this parameter.
695 DelayBoot=<time_spec>
697 Change the time to decide whether to reboot nodes in order to
698 satisfy job's feature specification if the job has been eligible
699 to run for less than this time period. See salloc/sbatch man
700 pages option --delay-boot.
701 Dependency=<dependency_list>
703 Defer job's initiation until specified job dependency specifica‐
704 tion is satisfied. Cancel dependency with an empty depen‐
705 dency_list (e.g. "Dependency="). <dependency_list> is of the
706 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
707 can share the same dependency and these jobs may even belong to
708 different users.
709 after:job_id[:jobid...]
711 This job can begin execution after the specified jobs
712 have begun execution or been canceled.
713 afterany:job_id[:jobid...]
715 This job can begin execution after the specified jobs
716 have terminated.
717 afternotok:job_id[:jobid...]
719 This job can begin execution after the specified jobs
720 have terminated in some failed state (non-zero exit code,
721 node failure, timed out, etc).
722 afterok:job_id[:jobid...]
724 This job can begin execution after the specified jobs
725 have successfully executed (ran to completion with an
726 exit code of zero).
727 singleton
729 This job can begin execution after any previously
730 launched jobs sharing the same job name and user have
731 terminated. In other words, only one job by that name
732 and owned by that user can be running or suspended at any
733 point in time.
734 EligibleTime=<time_spec>
736 See StartTime.
737 EndTime
739 The time the job is expected to terminate based on the job's
740 time limit. When the job ends sooner, this field will be up‐
741 dated with the actual end time.
742 ExcNodeList=<nodes>
744 Set the job's list of excluded node. Multiple node names may be
745 specified using simple node range expressions (e.g.
746 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
747 NodeList=".
748 Features=<features>
750 Set the job's required node features. The list of features may
751 include multiple feature names separated by ampersand (AND)
752 and/or vertical bar (OR) operators. For example: Fea‐
753 tures="opteron&video" or Features="fast|faster". In the first
754 example, only nodes having both the feature "opteron" AND the
755 feature "video" will be used. There is no mechanism to specify
756 that you want one node with feature "opteron" and another node
757 with feature "video" in case no node has both features. If only
758 one of a set of possible options should be used for all allo‐
759 cated nodes, then use the OR operator and enclose the options
760 within square brackets. For example: "Fea‐
761 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
762 all nodes must be allocated on a single rack of the cluster, but
763 any of those four racks can be used. A request can also specify
764 the number of nodes needed with some feature by appending an as‐
765 terisk and count after the feature name. For example "Fea‐
766 tures=graphics*4" indicates that at least four allocated nodes
767 must have the feature "graphics." Parenthesis are also sup‐
768 ported for features to be ANDed together. For example "Fea‐
769 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
770 cation should include 4 nodes with ALL of the features "knl",
771 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
772 straints with node counts may only be combined with AND opera‐
773 tors. Value may be cleared with blank data value, for example
774 "Features=".
775 Gres=<list>
778 Specifies a comma-delimited list of generic consumable re‐
779 sources. The format of each entry on the list is
780 "name[:count[*cpu]]". The name is that of the consumable re‐
781 source. The count is the number of those resources with a de‐
782 fault value of 1. The specified resources will be allocated to
783 the job on each node allocated unless "*cpu" is appended, in
784 which case the resources will be allocated on a per cpu basis.
785 The available generic consumable resources is configurable by
786 the system administrator. A list of available generic consum‐
787 able resources will be printed and the command will exit if the
788 option argument is "help". Examples of use include
789 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
790 JobId=<job_list>
793 Identify the job(s) to be updated. The job_list may be a comma
794 separated list of job IDs. Either JobId or JobName is required.
795 Licenses=<name>
797 Specification of licenses (or other resources available on all
798 nodes of the cluster) as described in salloc/sbatch/srun man
799 pages.
800 MailType=<types>
802 Set the mail event types. Valid type values are NONE, BEGIN,
803 END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, RE‐
804 QUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and
805 teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 per‐
806 cent of time limit), TIME_LIMIT_80 (reached 80 percent of time
807 limit), TIME_LIMIT_50 (reached 50 percent of time limit) and AR‐
808 RAY_TASKS (send emails for each array task). Multiple type val‐
809 ues may be specified in a comma separated list. Unless the AR‐
810 RAY_TASKS option is specified, mail notifications on job BEGIN,
811 END and FAIL apply to a job array as a whole rather than gener‐
812 ating individual email messages for each task in the job array.
813 MailUser=<name>
815 Set the user to receive email notification of state changes. A
816 blank string will set the mail user to the default which is the
817 submitting user.
818 MinCPUsNode=<count>
820 Set the job's minimum number of CPUs per node to the specified
821 value.
822 MinMemoryCPU=<megabytes>
824 Set the job's minimum real memory required per allocated CPU to
825 the specified value. Either MinMemoryCPU or MinMemoryNode may be
826 set, but not both.
827 MinMemoryNode=<megabytes>
829 Set the job's minimum real memory required per node to the spec‐
830 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
831 but not both.
832 MinTmpDiskNode=<megabytes>
834 Set the job's minimum temporary disk space required per node to
835 the specified value. Only the Slurm administrator or root can
836 change this parameter.
837 TimeMin=<timespec>
839 Change TimeMin value which specifies the minimum time limit min‐
840 utes of the job.
841 JobName=<name>
843 Identify the name of jobs to be modified or set the job's name
844 to the specified value. When used to identify jobs to be modi‐
845 fied, all jobs belonging to all users are modified unless the
846 UserID option is used to identify a specific user. Either JobId
847 or JobName is required.
848 Name[=<name>]
850 See JobName.
851 Nice[=<adjustment>]
853 Update the job with an adjusted scheduling priority within
854 Slurm. With no adjustment value the scheduling priority is de‐
855 creased by 100. A negative nice value increases the priority,
856 otherwise decreases it. The adjustment range is +/- 2147483645.
857 Only privileged users can specify a negative adjustment.
858 NodeList=<nodes>
860 Change the nodes allocated to a running job to shrink its size.
861 The specified list of nodes must be a subset of the nodes cur‐
862 rently allocated to the job. Multiple node names may be speci‐
863 fied using simple node range expressions (e.g. "lx[10-20]"). Af‐
864 ter a job's allocation is reduced, subsequent srun commands must
865 explicitly specify node and task counts which are valid for the
866 new allocation.
867 NumCPUs=<min_count>[-<max_count>]
869 Set the job's minimum and optionally maximum count of CPUs to be
870 allocated.
871 NumNodes=<min_count>[-<max_count>]
873 Set the job's minimum and optionally maximum count of nodes to
874 be allocated. If the job is already running, use this to spec‐
875 ify a node count less than currently allocated and resources
876 previously allocated to the job will be relinquished. After a
877 job's allocation is reduced, subsequent srun commands must ex‐
878 plicitly specify node and task counts which are valid for the
879 new allocation. Also see the NodeList parameter above. This is
880 the same as ReqNodes.
881 NumTasks=<count>
883 Set the job's count of required tasks to the specified value.
884 This is the same as ReqProcs.
885 OverSubscribe=<yes|no>
887 Set the job's ability to share compute resources (i.e. individ‐
888 ual CPUs) with other jobs. Possible values are "YES" and "NO".
889 This option can only be changed for pending jobs.
890 Partition=<name>
892 Set the job's partition to the specified value.
893 Priority=<number>
895 Set the job's priority to the specified value. Note that a job
896 priority of zero prevents the job from ever being scheduled. By
897 setting a job's priority to zero it is held. Set the priority
898 to a non-zero value to permit it to run. Explicitly setting a
899 job's priority clears any previously set nice value and removes
900 the priority/multifactor plugin's ability to manage a job's pri‐
901 ority. In order to restore the priority/multifactor plugin's
902 ability to manage a job's priority, hold and then release the
903 job. Only the Slurm administrator or root can increase job's
904 priority.
905 QOS=<name>
907 Set the job's QOS (Quality Of Service) to the specified value.
908 Value may be cleared with blank data value, "QOS=".
909 Reboot=<yes|no>
911 Set the job's flag that specifies whether to force the allocated
912 nodes to reboot before starting the job. This is only supported
913 with some system configurations and therefore it could be
914 silently ignored.
915 ReqCores=<count>
917 Change the job's requested Cores count.
918 ReqNodeList=<nodes>
920 Set the job's list of required node. Multiple node names may be
921 specified using simple node range expressions (e.g.
922 "lx[10-20]"). Value may be cleared with blank data value, "Re‐
923 qNodeList=".
924 ReqNodes=<min_count>[-<max_count>]
926 See NumNodes.
927 ReqProcs=<count>
929 See NumTasks.
930 ReqSockets=<count>
932 Change the job's requested socket count.
933 ReqThreads=<count>
935 Change the job's requested threads count.
936 Requeue=<0|1>
938 Stipulates whether a job should be requeued after a node fail‐
939 ure: 0 for no, 1 for yes.
940 ReservationName=<name>
942 Set the job's reservation to the specified value. Value may be
943 cleared with blank data value, "ReservationName=".
944 ResetAccrueTime
946 Reset the job's accrue time value to 0 meaning it will lose any
947 time previously accrued for priority. Helpful if you have a
948 large queue of jobs already in the queue and want to start lim‐
949 iting how many jobs can accrue time without waiting for the
950 queue to flush out.
951 SiteFactor=<account>
953 Specify the job's admin priority factor in the range of
954 +/-2147483645. Only privileged users can modify the value.
955 StdOut=<filepath>
957 Set the batch job's stdout file path.
958 Shared=<yes|no>
960 See OverSubscribe option above.
961 StartTime=<time_spec>
963 Set the job's earliest initiation time. It accepts times of the
964 form HH:MM:SS to run a job at a specific time of day (seconds
965 are optional). (If that time is already past, the next day is
966 assumed.) You may also specify midnight, noon, fika (3 PM) or
967 teatime (4 PM) and you can have a time-of-day suffixed with AM
968 or PM for running in the morning or the evening. You can also
969 say what day the job will be run, by specifying a date of the
970 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
971 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
972 count time-units, where the time-units can be seconds (default),
973 minutes, hours, days, or weeks and you can tell Slurm to run the
974 job today with the keyword today and to run the job tomorrow
975 with the keyword tomorrow.
976 Notes on date/time specifications:
978 - although the 'seconds' field of the HH:MM:SS time specifica‐
979 tion is allowed by the code, note that the poll time of the
980 Slurm scheduler is not precise enough to guarantee dispatch of
981 the job on the exact second. The job will be eligible to start
982 on the next poll following the specified time. The exact poll
983 interval depends on the Slurm scheduler (e.g., 60 seconds with
984 the default sched/builtin).
985 - if no time (HH:MM:SS) is specified, the default is
986 (00:00:00).
987 - if a date is specified without a year (e.g., MM/DD) then the
988 current year is assumed, unless the combination of MM/DD and
989 HH:MM:SS has already passed for that year, in which case the
990 next year is used.
991 Switches=<count>[@<max-time-to-wait>]
993 When a tree topology is used, this defines the maximum count of
994 switches desired for the job allocation. If Slurm finds an allo‐
995 cation containing more switches than the count specified, the
996 job remain pending until it either finds an allocation with de‐
997 sired switch count or the time limit expires. By default there
998 is no switch count limit and no time limit delay. Set the count
999 to zero in order to clean any previously set count (disabling
1000 the limit). The job's maximum time delay may be limited by the
1001 system administrator using the SchedulerParameters configuration
1002 parameter with the max_switch_wait parameter option. Also see
1003 wait-for-switch.
1004 wait-for-switch=<seconds>
1007 Change max time to wait for a switch <seconds> secs.
1008 TasksPerNode=<count>
1011 Change the job's requested TasksPerNode.
1012 ThreadSpec=<count>
1015 Number of threads to reserve per node for system use. The job
1016 will be charged for these threads, but be unable to use them.
1017 Will be reported as "*" if not constrained.
1018 TimeLimit=<time>
1021 The job's time limit. Output format is [days-]hours:min‐
1022 utes:seconds or "UNLIMITED". Input format (for update command)
1023 set is minutes, minutes:seconds, hours:minutes:seconds,
1024 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1025 Time resolution is one minute and second values are rounded up
1026 to the next minute. If changing the time limit of a job, either
1027 specify a new time limit value or precede the time and equal
1028 sign with a "+" or "-" to increment or decrement the current
1029 time limit (e.g. "TimeLimit+=30"). In order to increment or
1030 decrement the current time limit, the JobId specification must
1031 precede the TimeLimit specification. Note that incrementing or
1032 decrementing the time limit for a job array is only allowed be‐
1033 fore the job array has been split into more than one job record.
1034 Only the Slurm administrator or root can increase job's Time‐
1035 Limit.
1036 UserID=<UID or name>
1039 Used with the JobName option to identify jobs to be modified.
1040 Either a user name or numeric ID (UID), may be specified.
1041 WCKey=<key>
1044 Set the job's workload characterization key to the specified
1045 value.
1046 WorkDir=<directory_name>
1049 Set the job's working directory to the specified value. Note
1050 that this may only be set for jobs in the PENDING state, and
1051 that jobs may fail to launch if they rely on relative paths to
1052 the originally submitted WorkDir.
1053 JOBS - SPECIFICATIONS FOR SHOW COMMAND
1056 The "show" command, when used with the "job" or "job <jobid>" entity
1057 displays detailed information about a job or jobs. Much of this infor‐
1058 mation may be modified using the "update job" command as described
1059 above. However, the following fields displayed by the show job command
1060 are read-only and cannot be modified:
1061 AllocNode:Sid
1064 Local node and system id making the resource allocation.
1065 BatchFlag
1067 Jobs submitted using the sbatch command have BatchFlag set to 1.
1068 Jobs submitted using other commands have BatchFlag set to 0.
1069 ExitCode=<exit>:<sig>
1071 Exit status reported for the job by the wait() function. The
1072 first number is the exit code, typically as set by the exit()
1073 function. The second number of the signal that caused the
1074 process to terminate if it was terminated by a signal.
1075 GroupId
1077 The group under which the job was submitted.
1078 JobState
1080 The current state of the job.
1081 NodeListIndices
1083 The NodeIndices expose the internal indices into the node table
1084 associated with the node(s) allocated to the job.
1085 NtasksPerN:B:S:C=
1087 <tasks_per_node>:<tasks_per_base‐
1088 board>:<tasks_per_socket>:<tasks_per_core> Specifies the number
1089 of tasks to be started per hardware component (node, baseboard,
1090 socket and core). Unconstrained values may be shown as "0" or
1091 "*".
1092 PreemptEligibleTime
1094 Time the job becomes eligible for preemption. Modified by Pre‐
1095 emptExemptTime, either from the global option in slurm.conf or
1096 the job QOS. This is hidden if the job has not started or if
1097 PreemptMode=OFF.
1098 PreemptTime
1100 Time at which job was signaled that it was selected for preemp‐
1101 tion. (Meaningful only for PreemptMode=CANCEL and the partition
1102 or QOS with which the job is associated has a GraceTime value
1103 designated.) This is hidden if the job has not started or if
1104 PreemptMode=OFF.
1105 PreSusTime
1107 Time the job ran prior to last suspend.
1108 Reason The reason job is not running: e.g., waiting "Resources".
1110 ReqB:S:C:T=
1112 <baseboard_count>:<socket_per_base‐
1113 board_count>:<core_per_socket_count>:<thread_per_core_count>
1114 Specifies the count of various hardware components requested by
1115 the job. Unconstrained values may be shown as "0" or "*".
1116 SecsPreSuspend=<seconds>
1118 If the job is suspended, this is the run time accumulated by the
1119 job (in seconds) prior to being suspended.
1120 Socks/Node=<count>
1122 Count of desired sockets per node
1123 SubmitTime
1125 The time and date stamp (in localtime) the job was submitted.
1126 The format of the output is identical to that of the EndTime
1127 field.
1128 NOTE: If a job is requeued, the submit time is reset. To obtain
1130 the original submit time it is necessary to use the "sacct -j
1131 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1132 cate option to display all duplicate entries for a job.
1133 SuspendTime
1135 Time the job was last suspended or resumed.
1136 NOTE on information displayed for various job states: When you
1138 submit a request for the "show job" function the scontrol
1139 process makes an RPC request call to slurmctld with a RE‐
1140 QUEST_JOB_INFO message type. If the state of the job is PEND‐
1141 ING, then it returns some detail information such as: min_nodes,
1142 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1143 ING the code assumes that it is in a further state such as RUN‐
1144 NING, COMPLETE, etc. In these cases the code explicitly returns
1145 zero for these values. These values are meaningless once the job
1146 resources have been allocated and the job has started.
1147 STEPS - SPECIFICATIONS FOR UPDATE COMMAND
1150 StepId=<job_id>[.<step_id>]
1151 Identify the step to be updated. If the job_id is given, but no
1152 step_id is specified then all steps of the identified job will
1153 be modified. This specification is required.
1154 TimeLimit=<time>
1156 The job's time limit. Output format is [days-]hours:min‐
1157 utes:seconds or "UNLIMITED". Input format (for update command)
1158 set is minutes, minutes:seconds, hours:minutes:seconds,
1159 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1160 Time resolution is one minute and second values are rounded up
1161 to the next minute. If changing the time limit of a step, ei‐
1162 ther specify a new time limit value or precede the time with a
1163 "+" or "-" to increment or decrement the current time limit
1164 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1165 current time limit, the StepId specification must precede the
1166 TimeLimit specification.
1167 NODES - SPECIFICATIONS FOR UPDATE COMMAND
1170 NodeName=<name>
1171 Identify the node(s) to be updated. Multiple node names may be
1172 specified using simple node range expressions (e.g.
1173 "lx[10-20]"). This specification is required.
1174 ActiveFeatures=<features>
1177 Identify the feature(s) currently active on the specified node.
1178 Any previously active feature specification will be overwritten
1179 with the new value. Also see AvailableFeatures. Typically Ac‐
1180 tiveFeatures will be identical to AvailableFeatures; however Ac‐
1181 tiveFeatures may be configured as a subset of the AvailableFea‐
1182 tures. For example, a node may be booted in multiple configura‐
1183 tions. In that case, all possible configurations may be identi‐
1184 fied as AvailableFeatures, while ActiveFeatures would identify
1185 the current node configuration.
1186 AvailableFeatures=<features>
1189 Identify the feature(s) available on the specified node. Any
1190 previously defined available feature specification will be over‐
1191 written with the new value. AvailableFeatures assigned via
1192 scontrol will only persist across the restart of the slurmctld
1193 daemon with the -R option and state files preserved or slurm‐
1194 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1195 meant to be persistent across normal restarts of slurmctld or
1196 the execution of scontrol reconfig.
1197 Note: Available features being removed via scontrol must not be
1199 active (i.e. remove them from ActiveFeatures first).
1200 Comment=<comment>
1203 Arbitrary descriptive string. Use quotes to enclose a comment
1204 having more than one word
1205 CpuBind=<node>
1208 Specify the task binding mode to be used by default for this
1209 node. Supported options include: "none", "board", "socket",
1210 "ldom" (NUMA), "core", "thread" and "off" (remove previous bind‐
1211 ing mode).
1212 Extra=<comment>
1215 Arbitrary string on the node. Use quotes to enclose a string
1216 having more than one word.
1217 Gres=<gres>
1220 Identify generic resources to be associated with the specified
1221 node. Any previously defined generic resources will be over‐
1222 written with the new value. Specifications for multiple generic
1223 resources should be comma separated. Each resource specifica‐
1224 tion consists of a name followed by an optional colon with a nu‐
1225 meric value (default value is one) (e.g. "Gres=band‐
1226 width:10000"). Modification of GRES count associated with spe‐
1227 cific files (e.g. GPUs) is not allowed other than to set their
1228 count on a node to zero. In order to change the GRES count to
1229 another value, modify your slurm.conf and gres.conf files and
1230 restart daemons. If GRES are associated with specific sockets,
1231 that information will be reported For example if all 4 GPUs on a
1232 node are all associated with socket zero, then
1233 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1234 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1235 associated with specific GPUs is not reported, but only avail‐
1236 able by parsing the gres.conf file. Generic resources assigned
1237 via scontrol will only persist across the restart of the slurm‐
1238 ctld daemon with the -R option and state files preserved or
1239 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1240 changes meant to be persistent across normal restarts of slurm‐
1241 ctld or the execution of scontrol reconfig.
1242 NodeAddr=<node address>
1245 Name that a node should be referred to in establishing a commu‐
1246 nications path. This name will be used as an argument to the
1247 getaddrinfo() function for identification. If a node range ex‐
1248 pression is used to designate multiple nodes, they must exactly
1249 match the entries in the NodeName (e.g. "NodeName=lx[0-7]
1250 NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses.
1251 NodeHostname=<node hostname>
1254 Typically this would be the string that "/bin/hostname -s" re‐
1255 turns. It may also be the fully qualified domain name as re‐
1256 turned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid
1257 domain name associated with the host through the host database
1258 (/etc/hosts) or DNS, depending on the resolver settings. Note
1259 that if the short form of the hostname is not used, it may pre‐
1260 vent use of hostlist expressions (the numeric portion in brack‐
1261 ets must be at the end of the string). A node range expression
1262 can be used to specify a set of nodes. If an expression is used,
1263 the number of nodes identified by NodeHostname must be identical
1264 to the number of nodes identified by NodeName.
1265 Reason=<reason>
1268 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1269 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1270 having more than one word.
1271 State=<state>
1274 Assign one of the following states/actions to the node(s) speci‐
1275 fied by the update command.
1276 CANCEL_REBOOT
1278 Cancels a pending reboot on the node (same as scontrol
1279 cancel_reboot <node>).
1280 DOWN Stop all running and suspended jobs and make the node un‐
1282 available for new jobs.
1283 DRAIN Indicates that no new jobs may be started on this node.
1285 Existing jobs are allowed to run to completion, leaving
1286 the node in a DRAINED state once all the jobs have com‐
1287 pleted.
1288 FAIL Similar to DRAIN except that some applications will seek
1290 to relinquish those nodes before the job completes.
1291 FUTURE Indicates the node is not fully configured, but is ex‐
1293 pected to be available at some point in the future.
1294 NoResp This will set the "Not Responding" flag for a node with‐
1296 out changing its underlying state.
1297 POWER_DOWN
1299 Will use the configured SuspendProgram program to explic‐
1300 itly place a node in power saving mode. If a node is al‐
1301 ready in the process of being powered down, the command
1302 will only change the state of the node but won't have any
1303 effect until the configured SuspendTimeout is reached.
1304 Use of this command can be useful in situations where a
1305 ResumeProgram, like capmc in Cray machines, is stalled
1306 and one wants to restore the node to "IDLE" manually. In
1307 this case rebooting the node and setting the state to
1308 "POWER_DOWN" will cancel the previous "POWER_UP" state
1309 and the node will become "IDLE".
1310 POWER_DOWN_ASAP
1312 Will drain the node and mark it for power down. Currently
1313 running jobs will complete first and no additional jobs
1314 will be allocated to the node.
1315 POWER_DOWN_FORCE
1317 Will cancel all jobs on the node, power it down, and re‐
1318 set its state to "IDLE".
1319 POWER_UP
1321 Will use the configured ResumeProgram program to explic‐
1322 itly move a node out of power saving mode. If a node is
1323 already in the process of being powered up, the command
1324 will only change the state of the node but won't have any
1325 effect until the configured ResumeTimeout is reached.
1326 RESUME Not an actual node state, but will change a node state
1328 from DRAIN, DRAINING, DOWN or REBOOT to IDLE and NoResp.
1329 slurmctld will then attempt to contact slurmd to request
1330 that the node register itself. Once registered, the node
1331 state will then remove the NoResp flag and will resume
1332 normal operations. It will also clear the POWERING_DOWN
1333 state of a node and make it eligible to be allocted.
1334 UNDRAIN
1336 Clears the node from being drained (like RESUME), but
1337 will not change the node's base state (e.g. DOWN).
1338 UNDRAIN requires a valid node registration before new
1339 jobs can be scheduled on the node. Setting a node DOWN
1340 will cause all running and suspended jobs on that node to
1341 be terminated.
1342 While all of the above states are valid, some of them are not
1344 valid new node states given their prior state.
1345 NOTE: The scontrol command should not be used to change node
1347 state on Cray systems. Use Cray tools such as xtprocadmin in‐
1348 stead.
1349 Weight=<weight>
1352 Identify weight to be associated with specified nodes. This al‐
1353 lows dynamic changes to weight associated with nodes, which will
1354 be used for the subsequent node allocation decisions. Weight
1355 assigned via scontrol will only persist across the restart of
1356 the slurmctld daemon with the -R option and state files pre‐
1357 served or slurmctld's receipt of a SIGHUP. Update slurm.conf
1358 with any changes meant to be persistent across normal restarts
1359 of slurmctld or the execution of scontrol reconfig.
1360 NODES - SPECIFICATIONS FOR SHOW COMMAND
1363 AllocMem
1364 The total memory, in MB, currently allocated by jobs on the
1365 node.
1366 CPUSpecList
1369 The list of Slurm abstract CPU IDs on this node reserved for ex‐
1370 clusive use by the Slurm compute node daemons (slurmd, slurm‐
1371 stepd).
1372 FreeMem
1375 The total memory, in MB, currently free on the node as reported
1376 by the OS.
1377 LastBusyTime
1380 The last time the node was busy (i.e. last time the node had
1381 jobs on it). This time is used in PowerSave to determine when to
1382 suspend nodes (e.g. now - LastBusy > SuspendTime).
1383 MemSpecLimit
1386 The combined memory limit, in megabytes, on this node for the
1387 Slurm compute node daemons (slurmd, slurmstepd).
1388 RealMemory
1391 The total memory, in MB, on the node.
1392 State Identify the state(s) assigned to the node with '+' delimited
1395 state flags.
1396 States:
1399 ALLOCATED
1401 Indicates that the node has all CPUs allocated to job(s)
1402 running on the node.
1403 DOWN The node does not have any running jobs and is unavail‐
1405 able for new work.
1406 ERROR The node is in an error state. Consult the logs for more
1408 information about what caused this state.
1409 FUTURE The node is currently not fully configured, but expected
1411 to be available at some point in the indefinite future
1412 for use.
1413 IDLE Indicates that the node is available for work but does
1415 not currently have any jobs assigned to it.
1416 MIXED Indicates that the node is in multiple states. For in‐
1418 stance if only part of the node is ALLOCATED and the rest
1419 of the node is IDLE the state will be MIXED.
1420 UNKNOWN
1422 The node has not yet registered with the controller and
1423 its state is not known.
1424 Flags:
1427 CLOUD Indicates that the node is configured as a cloud node, to
1429 be brought up on demand, but not currently running.
1430 COMPLETING
1432 Indicates that the only job on the node or that all jobs
1433 on the node are in the process of completing.
1434 DRAIN The node is not accepting any new jobs and any currently
1436 running jobs will complete.
1437 DYNAMIC
1439 Slurm allows you to define multiple types of nodes in a
1440 FUTURE state. When starting slurmd on a node you can
1441 specify the -F flag to have the node match and use an ex‐
1442 isting definition in your slurm.conf file. The DYNAMIC
1443 state indicates that the node was started as a Dynamic
1444 Future node.
1445 INVALID_REG
1447 The node did not register correctly with the controller.
1448 MAINTENANCE
1450 The node is currently in a reservation that includes the
1451 maintenance flag.
1452 NOT_RESPONDING
1454 Node is not responding.
1455 PERFCTRS
1457 Indicates that Network Performance Counters associated
1458 with this node are in use, rendering this node as not us‐
1459 able for any other jobs.
1460 POWER_DOWN
1462 Node is pending power down.
1463 POWERED_DOWN
1465 Node is currently powered down and not capable of running
1466 any jobs.
1467 POWERING_DOWN
1469 Node is in the process of powering down.
1470 POWERING_UP
1472 Node is in the process of powering up.
1473 PLANNED
1475 The node is earmarked for a job that will start in the
1476 future.
1477 REBOOT_ISSUED
1479 A reboot request has been sent to the agent configured to
1480 handle this request.
1481 REBOOT_REQUESTED
1483 A request to reboot this node has been made, but hasn't
1484 been handled yet.
1485 RESERVED
1487 Indicates the node is in an advanced reservation and not
1488 generally available.
1489 The meaning of the energy information is as follows:
1492 CurrentWatts
1495 The instantaneous power consumption of the node at the
1496 time of the last node energy accounting sample, in watts.
1497 LowestJoules
1500 The energy consumed by the node between the last time it
1501 was powered on and the last time it was registered by
1502 slurmd, in joules.
1503 ConsumedJoules
1506 The energy consumed by the node between the last time it
1507 was registered by the slurmd daemon and the last node en‐
1508 ergy accounting sample, in joules.
1509 If the reported value is "n/s" (not supported), the node does
1512 not support the configured AcctGatherEnergyType plugin. If the
1513 reported value is zero, energy accounting for nodes is disabled.
1514 The meaning of the external sensors information is as follows:
1517 ExtSensorsJoules
1520 The energy consumed by the node between the last time it
1521 was powered on and the last external sensors plugin node
1522 sample, in joules.
1523 ExtSensorsWatts
1527 The instantaneous power consumption of the node at the
1528 time of the last external sensors plugin node sample, in
1529 watts.
1530 ExtSensorsTemp
1533 The temperature of the node at the time of the last ex‐
1534 ternal sensors plugin node sample, in celsius.
1535 If the reported value is "n/s" (not supported), the node does
1538 not support the configured ExtSensorsType plugin.
1539 FRONTEND - SPECIFICATIONS FOR UPDATE COMMAND
1542 FrontendName=<name>
1543 Identify the front end node to be updated. This specification is
1544 required.
1545 Reason=<reason>
1548 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1549 Use quotes to enclose a reason having more than one word.
1550 State=<state>
1553 Identify the state to be assigned to the front end node. Possi‐
1554 ble values are "DOWN", "DRAIN" or "RESUME". If you want to re‐
1555 move a front end node from service, you typically want to set
1556 its state to "DRAIN". "RESUME" is not an actual node state, but
1557 will return a "DRAINED", "DRAINING", or "DOWN" front end node to
1558 service, either "IDLE" or "ALLOCATED" state as appropriate.
1559 Setting a front end node "DOWN" will cause all running and sus‐
1560 pended jobs on that node to be terminated.
1561 PARTITIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
1564 AllowGroups=<name>
1565 Identify the user groups which may use this partition. Multiple
1566 groups may be specified in a comma separated list. To permit
1567 all groups to use the partition specify "AllowGroups=ALL".
1568 AllocNodes=<name>
1571 Comma separated list of nodes from which users can execute jobs
1572 in the partition. Node names may be specified using the node
1573 range expression syntax described above. The default value is
1574 "ALL".
1575 Alternate=<partition name>
1578 Alternate partition to be used if the state of this partition is
1579 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1580 set alternate partition.
1581 CpuBind=<node>
1584 Specify the task binding mode to be used by default for this
1585 partition. Supported options include: "none", "board",
1586 "socket", "ldom" (NUMA), "core", "thread" and "off" (remove pre‐
1587 vious binding mode).
1588 Default=<yes|no>
1591 Specify if this partition is to be used by jobs which do not ex‐
1592 plicitly identify a partition to use. Possible output values
1593 are "YES" and "NO". In order to change the default partition of
1594 a running system, use the scontrol update command and set De‐
1595 fault=yes for the partition that you want to become the new de‐
1596 fault.
1597 DefaultTime=<time>
1600 Run time limit used for jobs that don't specify a value. If not
1601 set then MaxTime will be used. Format is the same as for Max‐
1602 Time.
1603 DefMemPerCPU=<MB>
1606 Set the default memory to be allocated per CPU for jobs in this
1607 partition. The memory size is specified in megabytes.
1608 DefMemPerNode=<MB>
1610 Set the default memory to be allocated per node for jobs in this
1611 partition. The memory size is specified in megabytes.
1612 DisableRootJobs=<yes|no>
1615 Specify if jobs can be executed as user root. Possible values
1616 are "YES" and "NO".
1617 GraceTime=<seconds>
1620 Specifies, in units of seconds, the preemption grace time to be
1621 extended to a job which has been selected for preemption. The
1622 default value is zero, no preemption grace time is allowed on
1623 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1624 Hidden=<yes|no>
1627 Specify if the partition and its jobs should be hidden from
1628 view. Hidden partitions will by default not be reported by
1629 Slurm APIs or commands. Possible values are "YES" and "NO".
1630 JobDefaults=<specs>
1633 Specify job default values using a comma-delimited list of
1634 "key=value" pairs. Supported keys include
1635 DefCpuPerGPU Default number of CPUs per allocated GPU.
1637 DefMemPerGPU Default memory limit (in megabytes) per allocated
1639 GPU.
1640 MaxMemPerCPU=<MB>
1643 Set the maximum memory to be allocated per CPU for jobs in this
1644 partition. The memory size is specified in megabytes.
1645 MaxMemPerCNode=<MB>
1647 Set the maximum memory to be allocated per node for jobs in this
1648 partition. The memory size is specified in megabytes.
1649 MaxNodes=<count>
1652 Set the maximum number of nodes which will be allocated to any
1653 single job in the partition. Specify a number, "INFINITE" or
1654 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1655 upon jobs that have already begun execution.
1656 MaxTime=<time>
1659 The maximum run time for jobs. Output format is
1660 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1661 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1662 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1663 onds. Time resolution is one minute and second values are
1664 rounded up to the next minute. Changing the MaxTime of a parti‐
1665 tion has no effect upon jobs that have already begun execution.
1666 MinNodes=<count>
1669 Set the minimum number of nodes which will be allocated to any
1670 single job in the partition. Changing the MinNodes of a parti‐
1671 tion has no effect upon jobs that have already begun execution.
1672 Increasing this value may prevent pending jobs from starting,
1673 even if they were submitted without -N/--nodes specification.
1674 If you do get in that situation, updating the MinNodes value of
1675 a pending job using the scontrol command will allow that job to
1676 be scheduled.
1677 Nodes=<name>
1680 Identify the node(s) to be associated with this partition. Mul‐
1681 tiple node names may be specified using simple node range ex‐
1682 pressions (e.g. "lx[10-20]"). Note that jobs may only be asso‐
1683 ciated with one partition at any time. Specify a blank data
1684 value to remove all nodes from a partition: "Nodes=". Changing
1685 the Nodes in a partition has no effect upon jobs that have al‐
1686 ready begun execution.
1687 OverTimeLimit=<count>
1690 Number of minutes by which a job can exceed its time limit be‐
1691 fore being canceled. The configured job time limit is treated
1692 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1693 vides a hard limit, at which point the job is canceled. This is
1694 particularly useful for backfill scheduling, which bases upon
1695 each job's soft time limit. A partition-specific OverTimeLimit
1696 will override any global OverTimeLimit value. If not specified,
1697 the global OverTimeLimit value will take precedence. May not
1698 exceed 65533 minutes. An input value of "UNLIMITED" will clear
1699 any previously configured partition-specific OverTimeLimit
1700 value.
1701 OverSubscribe=<yes|no|exclusive|force>[:<job_count>]
1704 Specify if compute resources (i.e. individual CPUs) in this par‐
1705 tition can be shared by multiple jobs. Possible values are
1706 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1707 specifies how many jobs can be allocated to use each resource.
1708 PartitionName=<name>
1711 Identify the partition to be updated. This specification is re‐
1712 quired.
1713 PreemptMode=<mode>
1716 Reset the mechanism used to preempt jobs in this partition if
1717 PreemptType is configured to preempt/partition_prio. The default
1718 preemption mechanism is specified by the cluster-wide Preempt‐
1719 Mode configuration parameter. Possible values are "OFF", "CAN‐
1720 CEL", "REQUEUE" and "SUSPEND".
1721 Priority=<count>
1724 Jobs submitted to a higher priority partition will be dispatched
1725 before pending jobs in lower priority partitions and if possible
1726 they will preempt running jobs from lower priority partitions.
1727 Note that a partition's priority takes precedence over a job's
1728 priority. The value may not exceed 65533.
1729 PriorityJobFactor=<count>
1732 Partition factor used by priority/multifactor plugin in calcu‐
1733 lating job priority. The value may not exceed 65533. Also see
1734 PriorityTier.
1735 PriorityTier=<count>
1738 Jobs submitted to a partition with a higher priority tier value
1739 will be dispatched before pending jobs in partition with lower
1740 priority tier value and, if possible, they will preempt
1741 running jobs from partitions with lower priority tier values.
1742 Note that a partition's priority tier takes precedence over a
1743 job's priority. The value may not exceed 65533. Also see Pri‐
1744 orityJobFactor.
1745 QOS=<QOSname|blank to remove>
1748 Set the partition QOS with a QOS name or to remove the Partition
1749 QOS leave the option blank.
1750 RootOnly=<yes|no>
1753 Specify if only allocation requests initiated by user root will
1754 be satisfied. This can be used to restrict control of the par‐
1755 tition to some meta-scheduler. Possible values are "YES" and
1756 "NO".
1757 ReqResv=<yes|no>
1760 Specify if only allocation requests designating a reservation
1761 will be satisfied. This is used to restrict partition usage to
1762 be allowed only within a reservation. Possible values are "YES"
1763 and "NO".
1764 Shared=<yes|no|exclusive|force>[:<job_count>]
1767 Renamed to OverSubscribe, see option descriptions above.
1768 State=<up|down|drain|inactive>
1771 Specify if jobs can be allocated nodes or queued in this parti‐
1772 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1773 UP Designates that new jobs may queued on the partition,
1775 and that jobs may be allocated nodes and run from the
1776 partition.
1777 DOWN Designates that new jobs may be queued on the parti‐
1779 tion, but queued jobs may not be allocated nodes and
1780 run from the partition. Jobs already running on the
1781 partition continue to run. The jobs must be explicitly
1782 canceled to force their termination.
1783 DRAIN Designates that no new jobs may be queued on the par‐
1785 tition (job submission requests will be denied with an
1786 error message), but jobs already queued on the parti‐
1787 tion may be allocated nodes and run. See also the
1788 "Alternate" partition specification.
1789 INACTIVE Designates that no new jobs may be queued on the par‐
1791 tition, and jobs already queued may not be allocated
1792 nodes and run. See also the "Alternate" partition
1793 specification.
1794 TRESBillingWeights=<TRES Billing Weights>
1797 TRESBillingWeights is used to define the billing weights of each
1798 TRES type that will be used in calculating the usage of a job.
1799 The calculated usage is used when calculating fairshare and when
1800 enforcing the TRES billing limit on jobs. Updates affect new
1801 jobs and not existing jobs. See the slurm.conf man page for
1802 more information.
1803 RESERVATIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
1806 Reservation=<name>
1807 Identify the name of the reservation to be created, updated, or
1808 deleted. This parameter is required for update and is the only
1809 parameter for delete. For create, if you do not want to give a
1810 reservation name, use "scontrol create res ..." and a name will
1811 be created automatically.
1812 Accounts=<account list>
1815 List of accounts permitted to use the reserved nodes, for exam‐
1816 ple "Accounts=physcode1,physcode2". A user in any of the ac‐
1817 counts may use the reserved nodes. A new reservation must spec‐
1818 ify Users or Groups and/or Accounts. If both Users/Groups and
1819 Accounts are specified, a job must match both in order to use
1820 the reservation. Accounts can also be denied access to reserva‐
1821 tions by preceding all of the account names with '-'. Alter‐
1822 nately precede the equal sign with '-'. For example, "Ac‐
1823 counts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2"
1824 will permit any account except physcode1 and physcode2 to use
1825 the reservation. You can add or remove individual accounts from
1826 an existing reservation by using the update command and adding a
1827 '+' or '-' sign before the '=' sign. If accounts are denied ac‐
1828 cess to a reservation (account name preceded by a '-'), then all
1829 other accounts are implicitly allowed to use the reservation and
1830 it is not possible to also explicitly specify allowed accounts.
1831 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1834 Specification of burst buffer resources which are to be re‐
1835 served. "buffer_spec" consists of four elements:
1836 [plugin:][type:]#[units] "plugin" is the burst buffer plugin
1837 name, currently either "datawarp" or "generic". If no plugin is
1838 specified, the reservation applies to all configured burst buf‐
1839 fer plugins. "type" specifies a Cray generic burst buffer re‐
1840 source, for example "nodes". if "type" is not specified, the
1841 number is a measure of storage space. The "units" may be "N"
1842 (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers
1843 of 1024) and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).
1844 The default units are bytes for reservations of storage space.
1845 For example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage
1846 plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (re‐
1847 serve 100 GB of storage from all configured burst buffer plug‐
1848 ins). Jobs using this reservation are not restricted to these
1849 burst buffer resources, but may use these reserved resources
1850 plus any which are generally available. NOTE: Usually Slurm in‐
1851 terprets KB, MB, GB, TB, PB, TB units as powers of 1024, but for
1852 Burst Buffers size specifications Slurm supports both IEC/SI
1853 formats. This is because the CRAY API for managing DataWarps
1854 supports both formats.
1855 CoreCnt=<num>
1858 This option is only supported when SelectType=select/cons_res or
1859 select/cons_tres. Identify number of cores to be reserved. If
1860 NodeCnt is used without the FIRST_CORES flag, this is the total
1861 number of cores to reserve where cores per node is
1862 CoreCnt/NodeCnt. If a nodelist is used, or if NodeCnt is used
1863 with the FIRST_CORES flag, this should be an array of core num‐
1864 bers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or
1865 flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
1866 Licenses=<license>
1869 Specification of licenses (or other resources available on all
1870 nodes of the cluster) which are to be reserved. License names
1871 can be followed by a colon and count (the default count is one).
1872 Multiple license names should be comma separated (e.g. "Li‐
1873 censes=foo:4,bar"). A new reservation must specify one or more
1874 resource to be included: NodeCnt, Nodes and/or Licenses. If a
1875 reservation includes Licenses, but no NodeCnt or Nodes, then the
1876 option Flags=LICENSE_ONLY must also be specified. Jobs using
1877 this reservation are not restricted to these licenses, but may
1878 use these reserved licenses plus any which are generally avail‐
1879 able.
1880 MaxStartDelay[=<timespec>]
1883 Change MaxStartDelay value which specifies the maximum time an
1884 eligible job not requesting this reservation can delay a job re‐
1885 questing it. Default is none. Valid formats are minutes, min‐
1886 utes:seconds, hours:minutes:seconds, days-hours, days-hours:min‐
1887 utes, days-hours:minutes:seconds. Time resolution is one minute
1888 and second values are rounded up to the next minute. Output for‐
1889 mat is always [days-]hours:minutes:seconds.
1890 NodeCnt=<num>[,num,...]
1893 Identify number of nodes to be reserved. The number can include
1894 a suffix of "k" or "K", in which case the number specified is
1895 multiplied by 1024. A new reservation must specify one or more
1896 resource to be included: NodeCnt, Nodes and/or Licenses.
1897 Nodes=<name>
1900 Identify the node(s) to be reserved. Multiple node names may be
1901 specified using simple node range expressions (e.g.
1902 "Nodes=lx[10-20]"). Specify a blank data value to remove all
1903 nodes from a reservation: "Nodes=". A new reservation must
1904 specify one or more resource to be included: NodeCnt, Nodes
1905 and/or Licenses. A specification of "ALL" will reserve all
1906 nodes. Set Flags=PART_NODES and PartitionName= in order for
1907 changes in the nodes associated with a partition to also be re‐
1908 flected in the nodes associated with a reservation.
1909 StartTime=<time_spec>
1912 The start time for the reservation. A new reservation must
1913 specify a start time. It accepts times of the form HH:MM:SS for
1914 a specific time of day (seconds are optional). (If that time is
1915 already past, the next day is assumed.) You may also specify
1916 midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
1917 time-of-day suffixed with AM or PM for running in the morning or
1918 the evening. You can also say what day the job will be run, by
1919 specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or
1920 a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also give
1921 times like now + count time-units, where the time-units can be
1922 seconds (default), minutes, hours, days, or weeks and you can
1923 tell Slurm to run the job today with the keyword today and to
1924 run the job tomorrow with the keyword tomorrow. You cannot up‐
1925 date the StartTime of a reservation in ACTIVE state.
1926 EndTime=<time_spec>
1929 The end time for the reservation. A new reservation must spec‐
1930 ify an end time or a duration. Valid formats are the same as
1931 for StartTime.
1932 Duration=<time>
1935 The length of a reservation. A new reservation must specify an
1936 end time or a duration. Valid formats are minutes, minutes:sec‐
1937 onds, hours:minutes:seconds, days-hours, days-hours:minutes,
1938 days-hours:minutes:seconds, or UNLIMITED. Time resolution is
1939 one minute and second values are rounded up to the next minute.
1940 Output format is always [days-]hours:minutes:seconds.
1941 PartitionName=<name>
1944 Partition used to reserve nodes from. This will attempt to allo‐
1945 cate all nodes in the specified partition unless you request
1946 fewer resources than are available with CoreCnt, NodeCnt or
1947 TRES. Jobs will be allowed to use this reservation even if run‐
1948 ning in a different partition. There only needs to be overlap‐
1949 ping nodes from that different partition and the nodes used in
1950 the reservation.
1951 Flags=<flags>
1954 Flags associated with the reservation. You can add or remove
1955 individual flags from an existing reservation by adding a '+' or
1956 '-' sign before the '=' sign. For example: Flags-=DAILY (NOTE:
1957 this shortcut is not supported for all flags). Currently sup‐
1958 ported flags include:
1959 ANY_NODES This is a reservation for burst buffers and/or li‐
1961 censes only and not compute nodes. If this flag
1962 is set, a job using this reservation may use the
1963 associated burst buffers and/or licenses plus any
1964 compute nodes. If this flag is not set, a job us‐
1965 ing this reservation may use only the nodes and
1966 licenses associated with the reservation.
1967 DAILY Repeat the reservation at the same time every day.
1969 FLEX Permit jobs requesting the reservation to begin
1971 prior to the reservation's start time, end after
1972 the reservation's end time, and use any resources
1973 inside and/or outside of the reservation regard‐
1974 less of any constraints possibly set in the reser‐
1975 vation. A typical use case is to prevent jobs not
1976 explicitly requesting the reservation from using
1977 those reserved resources rather than forcing jobs
1978 requesting the reservation to use those resources
1979 in the time frame reserved. Another use case could
1980 be to always have a particular number of nodes
1981 with a specific feature reserved for a specific
1982 account so users in this account may use this
1983 nodes plus possibly other nodes without this fea‐
1984 ture.
1985 FIRST_CORES Use the lowest numbered cores on a node only. Flag
1987 removal with '-=' is not supported.
1988 IGNORE_JOBS Ignore currently running jobs when creating the
1990 reservation. This can be especially useful when
1991 reserving all nodes in the system for maintenance.
1992 LICENSE_ONLY See ANY_NODES.
1994 MAINT Maintenance mode, receives special accounting
1996 treatment. This reservation is permitted to use
1997 resources that are already in another reservation.
1998 MAGNETIC This flag allows jobs to be considered for this
2000 reservation even if they didn't request it.
2001 NO_HOLD_JOBS_AFTER
2003 By default, when a reservation ends the reserva‐
2004 tion request will be removed from any pending jobs
2005 submitted to the reservation and will be put into
2006 a held state. Use this flag to let jobs run out‐
2007 side of the reservation after the reservation is
2008 gone. Flag removal with '-=' is not supported.
2009 OVERLAP This reservation can be allocated resources that
2011 are already in another reservation. Flag removal
2012 with '-=' is not supported.
2013 PART_NODES This flag can be used to reserve all nodes within
2015 the specified partition. PartitionName and
2016 Nodes=ALL must be specified with this flag.
2017 PURGE_COMP[=<timespec>]
2019 Purge the reservation if it is ever idle for time‐
2020 spec (no jobs associated with it). If timespec
2021 isn't given then 5 minutes is the default. Valid
2022 timespec formats are minutes, minutes:seconds,
2023 hours:minutes:seconds, days-hours, days-hours:min‐
2024 utes, days-hours:minutes:seconds. Time resolution
2025 is one minute and second values are rounded up to
2026 the next minute. Output format is always
2027 [days-]hours:minutes:seconds.
2028 REPLACE Nodes which are DOWN, DRAINED, or allocated to
2030 jobs are automatically replenished using idle re‐
2031 sources. This option can be used to maintain a
2032 constant number of idle resources available for
2033 pending jobs (subject to availability of idle re‐
2034 sources). This should be used with the NodeCnt
2035 reservation option; do not identify specific nodes
2036 to be included in the reservation. Flag removal
2037 with '-=' is not supported.
2038 NOTE: Removing a node from the cluster while in a
2040 reservation with the REPLACE flag will not cause
2041 it to be replaced.
2042 REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically
2044 replenished using idle resources. This option can
2045 be used to maintain a constant sized pool of re‐
2046 sources available for pending jobs (subject to
2047 availability of idle resources). This should be
2048 used with the NodeCnt reservation option; do not
2049 identify specific nodes to be included in the
2050 reservation. Flag removal with '-=' is not sup‐
2051 ported.
2052 NOTE: Removing a node from the cluster while in a
2054 reservation with the REPLACE_DOWN flag will not
2055 cause it to be replaced.
2056 SPEC_NODES Reservation is for specific nodes (output only).
2058 STATIC_ALLOC Make it so after the nodes are selected for a
2060 reservation they don't change. Without this op‐
2061 tion when nodes are selected for a reservation and
2062 one goes down the reservation will select a new
2063 node to fill the spot.
2064 TIME_FLOAT The reservation start time is relative to the cur‐
2066 rent time and moves forward through time (e.g. a
2067 StartTime=now+10minutes will always be 10 minutes
2068 in the future). Repeating (e.g. DAILY) floating
2069 reservations are not supported. Flag cannot be
2070 added to or removed from an existing reservation.
2071 WEEKDAY Repeat the reservation at the same time on every
2073 weekday (Monday, Tuesday, Wednesday, Thursday and
2074 Friday).
2075 WEEKEND Repeat the reservation at the same time on every
2077 weekend day (Saturday and Sunday).
2078 WEEKLY Repeat the reservation at the same time every
2080 week.
2081 Features=<features>
2084 Set the reservation's required node features. Multiple values
2085 may be "&" separated if all features are required (AND opera‐
2086 tion) or separated by "|" if any of the specified features are
2087 required (OR operation). Parenthesis are also supported for
2088 features to be ANDed together with counts of nodes having the
2089 specified features. For example "Fea‐
2090 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
2091 reservation should include 4 nodes with ALL of the features
2092 "knl", "a2a", and "flat" plus 2 nodes with the feature
2093 "haswell".
2094 Value may be cleared with blank data value, "Features=".
2096 Groups=<group list>
2099 List of groups permitted to use the reserved nodes, for example
2100 "Group=bio,chem". A new reservation must specify Users or
2101 Groups and/or Accounts. If both Users/Groups and Accounts are
2102 specified, a job must match both in order to use the reserva‐
2103 tion. Unlike users groups do not allow denied access to reser‐
2104 vations. You can add or remove individual groups from an exist‐
2105 ing reservation by using the update command and adding a '+' or
2106 '-' sign before the '=' sign. NOTE: Groups and Users are mutu‐
2107 ally exclusive in reservations, if you want to switch between
2108 the 2 you must update the reservation with a group='' or user=''
2109 and fill in the opposite with the appropriate setting.
2110 Skip Used on a reoccurring reservation, skip to the next reservation
2113 iteration. NOTE: Only available for update.
2114 Users=<user list>
2117 List of users permitted to use the reserved nodes, for example
2118 "User=jones1,smith2". A new reservation must specify Users or
2119 Groups and/or Accounts. If both Users/Groups and Accounts are
2120 specified, a job must match both in order to use the reserva‐
2121 tion. Users can also be denied access to reservations by pre‐
2122 ceding all of the user names with '-'. Alternately precede the
2123 equal sign with '-'. For example, "User=-jones1,-smith2" or
2124 "User-=jones1,smith2" will permit any user except jones1 and
2125 smith2 to use the reservation. You can add or remove individual
2126 users from an existing reservation by using the update command
2127 and adding a '+' or '-' sign before the '=' sign. If users are
2128 denied access to a reservation (user name preceded by a '-'),
2129 then all other users are implicitly allowed to use the reserva‐
2130 tion and it is not possible to also explicitly specify allowed
2131 users. NOTE: Groups and Users are mutually exclusive in reser‐
2132 vations, if you want to switch between the 2 you must update the
2133 reservation with a group='' or user='' and fill in the opposite
2134 with the appropriate setting.
2135 TRES=<tres_spec>
2138 Comma-separated list of TRES required for the reservation. Cur‐
2139 rent supported TRES types with reservations are: CPU, Node, Li‐
2140 cense and BB. CPU and Node follow the same format as CoreCnt and
2141 NodeCnt parameters respectively. License names can be followed
2142 by an equal '=' and a count:
2143 License/<name1>=<count1>[,License/<name2>=<count2>,...]
2145 BurstBuffer can be specified in a similar way as BurstBuffer pa‐
2147 rameter. The only difference is that colon symbol ':' should be
2148 replaced by an equal '=' in order to follow the TRES format.
2149 Some examples of TRES valid specifications:
2151 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
2153 TRES=node=5k,license/iop1=2
2155 As specified in CoreCnt, if a nodelist is specified, cpu can be
2157 an array of core numbers by node: nodes=compute[1-3]
2158 TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
2159 Please note that CPU, Node, License and BB can override CoreCnt,
2161 NodeCnt, Licenses and BurstBuffer parameters respectively. Also
2162 CPU represents CoreCnt, in a reservation and will be adjusted if
2163 you have threads per core on your nodes.
2164 Note that a reservation that contains nodes or cores is associ‐
2166 ated with one partition, and can't span resources over multiple
2167 partitions. The only exception from this is when the reserva‐
2168 tion is created with explicitly requested nodes.
2169
2172 Executing scontrol sends a remote procedure call to slurmctld. If
2173 enough calls from scontrol or other Slurm client commands that send re‐
2174 mote procedure calls to the slurmctld daemon come in at once, it can
2175 result in a degradation of performance of the slurmctld daemon, possi‐
2176 bly resulting in a denial of service.
2177 Do not run scontrol or other Slurm client commands that send remote
2179 procedure calls to slurmctld from loops in shell scripts or other pro‐
2180 grams. Ensure that programs limit calls to scontrol to the minimum nec‐
2181 essary for the information you are trying to gather.
2182
2185 Some scontrol options may be set via environment variables. These envi‐
2186 ronment variables, along with their corresponding options, are listed
2187 below. (Note: Command line options will always override these set‐
2188 tings.)
2189 SCONTROL_ALL -a, --all
2191 SCONTROL_FEDERATION --federation
2193 SCONTROL_FUTURE -F, --future
2195 SCONTROL_LOCAL --local
2197 SCONTROL_SIBLING --sibling
2199 SLURM_BITSTR_LEN Specifies the string length to be used for holding
2201 a job array's task ID expression. The default
2202 value is 64 bytes. A value of 0 will print the
2203 full expression with any length required. Larger
2204 values may adversely impact the application perfor‐
2205 mance.
2206 SLURM_CLUSTERS Same as --clusters
2208 SLURM_CONF The location of the Slurm configuration file.
2210 SLURM_CONF_OUT When running 'write config', the location of the
2212 Slurm configuration file to be written.
2213 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
2215 value of standard, the default value, generates
2216 output in the form
2217 "year-month-dateThour:minute:second". A value of
2218 relative returns only "hour:minute:second" if the
2219 current day. For other dates in the current year
2220 it prints the "hour:minute" preceded by "Tomorr"
2221 (tomorrow), "Ystday" (yesterday), the name of the
2222 day for the coming week (e.g. "Mon", "Tue", etc.),
2223 otherwise the date (e.g. "25 Apr"). For other
2224 years it returns a date month and year without a
2225 time (e.g. "6 Jun 2012"). All of the time stamps
2226 use a 24 hour format.
2227 A valid strftime() format can also be specified.
2229 For example, a value of "%a %T" will report the day
2230 of the week and a time stamp (e.g. "Mon 12:34:56").
2231 SLURM_TOPO_LEN Specify the maximum size of the line when printing
2234 Topology. If not set, the default value is unlim‐
2235 ited.
2236
2239 When using SlurmDBD, users who have an AdminLevel defined (Operator or
2240 Admin) and users who are account coordinators are given the authority
2241 to view and modify jobs, reservations, nodes, etc., as defined in the
2242 following table - regardless of whether a PrivateData restriction has
2243 been defined in the slurm.conf file.
2244 scontrol show job(s): Admin, Operator, Coordinator
2246 scontrol update job: Admin, Operator, Coordinator
2247 scontrol requeue: Admin, Operator, Coordinator
2248 scontrol show step(s): Admin, Operator, Coordinator
2249 scontrol update step: Admin, Operator, Coordinator
2250 scontrol show node: Admin, Operator
2252 scontrol update node: Admin
2253 scontrol create partition: Admin
2255 scontrol show partition: Admin, Operator
2256 scontrol update partition: Admin
2257 scontrol delete partition: Admin
2258 scontrol create reservation: Admin, Operator
2260 scontrol show reservation: Admin, Operator
2261 scontrol update reservation: Admin, Operator
2262 scontrol delete reservation: Admin, Operator
2263 scontrol reconfig: Admin
2265 scontrol shutdown: Admin
2266 scontrol takeover: Admin
2267
2270 $ scontrol
2271 scontrol: show part debug
2272 PartitionName=debug
2273 AllocNodes=ALL AllowGroups=ALL Default=YES
2274 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2275 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2276 Nodes=snowflake[0-48]
2277 Priority=1 RootOnly=NO OverSubscribe=YES:4
2278 State=UP TotalCPUs=694 TotalNodes=49
2279 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2280 scontrol: show job 71701
2281 JobId=71701 Name=hostname
2282 UserId=da(1000) GroupId=da(1000)
2283 Priority=66264 Account=none QOS=normal WCKey=*123
2284 JobState=COMPLETED Reason=None Dependency=(null)
2285 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
2286 SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
2287 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2288 SuspendTime=None SecsPreSuspend=0
2289 Partition=debug AllocNode:Sid=snowflake:4702
2290 ReqNodeList=(null) ExcNodeList=(null)
2291 NodeList=snowflake0
2292 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2293 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2294 Features=(null) Reservation=(null)
2295 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2296 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2297 scontrol: show hostnames tux[1-3]
2298 tux1
2299 tux2
2300 tux3
2301 scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
2302 Reservation created: dbremer_1
2303 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2304 scontrol: delete Reservation=dbremer_1
2305 scontrol: quit
2306
2309 Copyright (C) 2002-2007 The Regents of the University of California.
2310 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2311 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2312 Copyright (C) 2010-2021 SchedMD LLC.
2313 This file is part of Slurm, a resource management program. For de‐
2315 tails, see <https://slurm.schedmd.com/>.
2316 Slurm is free software; you can redistribute it and/or modify it under
2318 the terms of the GNU General Public License as published by the Free
2319 Software Foundation; either version 2 of the License, or (at your op‐
2320 tion) any later version.
2321 Slurm is distributed in the hope that it will be useful, but WITHOUT
2323 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2324 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2325 for more details.
2326
2328 /etc/slurm.conf
2329
2331 scancel(1), sinfo(1), squeue(1), slurm_create_partition [22m(3),
2332 slurm_delete_partition [22m(3), slurm_load_ctl_conf [22m(3), slurm_load_jobs
2333 (3), slurm_load_node (3), slurm_load_partitions (3), slurm_reconfigure
2334 (3), slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2335 slurm_suspend (3), slurm_takeover (3), slurm_update_job (3), slurm_up‐
2336 date_node (3), slurm_update_partition (3), slurm.conf(5), slurmctld(8)
2337September 2021 Slurm Commands scontrol(1)