1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an Ad‐
17 ministrator. If an attempt to view or modify configuration information
18 is made by an unauthorized user, an error message will be printed and
19 the requested action will not occur. If no command is entered on the
20 execute line, scontrol will operate in an interactive mode and prompt
21 for input. It will continue prompting for input and executing commands
22 until explicitly terminated. If a command is entered on the execute
23 line, scontrol will execute that command and terminate. All commands
24 and options are case-insensitive, although node names, partition names,
25 and reservation names are case-sensitive (node names "LX" and "lx" are
26 distinct). All commands and options can be abbreviated to the extent
27 that the specification is unique. A modified Slurm configuration can be
28 written to a file using the scontrol write config command. The result‐
29 ing file will be named using the convention "slurm.conf.<datetime>" and
30 located in the same directory as the original "slurm.conf" file. The
31 directory containing the original slurm.conf must be writable for this
32 to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -d, --details
43 Causes the show command to provide additional details where
44 available.
45 --federation
47 Report jobs from federation if a member of one.
48 -F, --future
50 Report nodes in FUTURE state.
51 -h, --help
53 Print a help message describing the usage of scontrol.
54 --hide Do not display information about hidden partitions, their jobs
56 and job steps. By default, neither partitions that are config‐
57 ured as hidden nor those partitions unavailable to user's group
58 will be displayed (i.e. this is the default behavior).
59 --local
61 Show only information local to this cluster. Ignore other clus‐
62 ters in the federated if a member of one. Overrides --federa‐
63 tion.
64 -M, --clusters=<string>
66 The cluster to issue commands to. Only one cluster name may be
67 specified. Note that the SlurmDBD must be up for this option to
68 work properly. This option implicitly sets the --local option.
69 -o, --oneliner
72 Print information one line per record.
73 -Q, --quiet
75 Print no warning or informational messages, only fatal error
76 messages.
77 --sibling
79 Show all sibling jobs on a federated cluster. Implies --federa‐
80 tion.
81 -u, --uid=<uid>
83 Attempt to update a job as user <uid> instead of the invoking
84 user id.
85 -v, --verbose
87 Print detailed event logging. Multiple -v's will further in‐
88 crease the verbosity of logging. By default only errors will be
89 displayed.
90 -V , --version
93 Print version information and exit.
94 COMMANDS
97 abort Instruct the Slurm controller to terminate immediately and gen‐
98 erate a core file. See "man slurmctld" for information about
99 where the core file will be written.
100 cancel_reboot <NodeList>
103 Cancel pending reboots on nodes. The node will be undrain'ed and
104 the reason cleared if the node was drained by an ASAP reboot.
105 cluster CLUSTER_NAME
108 The cluster to issue commands to. Only one cluster name may be
109 specified.
110 create SPECIFICATION
113 Create a new partition or reservation. See the full list of pa‐
114 rameters below. Include the tag "res" to create a reservation
115 without specifying a reservation name.
116 completing
119 Display all jobs in a COMPLETING state along with associated
120 nodes in either a COMPLETING or DOWN state.
121 delete SPECIFICATION
124 Delete the entry with the specified SPECIFICATION. The two
125 SPECIFICATION choices are PartitionName=<name> and Reserva‐
126 tion=<name>. Reservations and partitions should have no associ‐
127 ated jobs at the time of their deletion (modify the jobs first).
128 If the specified partition is in use, the request is denied.
129 errnumstr ERRNO
132 Given a Slurm error number, return a descriptive string.
133 fsdampeningfactor FACTOR
136 Set the FairShareDampeningFactor in slurmctld.
137 help Display a description of scontrol options and commands.
140 hold job_list
143 Prevent a pending job from being started (sets its priority to
144 0). Use the release command to permit the job to be scheduled.
145 The job_list argument is a comma separated list of job IDs OR
146 "jobname=" with the job's name, which will attempt to hold all
147 jobs having that name. Note that when a job is held by a system
148 administrator using the hold command, only a system administra‐
149 tor may release the job for execution (also see the uhold com‐
150 mand). When the job is held by its owner, it may also be re‐
151 leased by the job's owner. Additionally, attempting to hold a
152 running job will have not suspend or cancel it. But, it will set
153 the job priority to 0 and update the job reason field, which
154 would hold the job if it was requeued at a later time.
155 notify job_id message
158 Send a message to standard error of the salloc or srun command
159 or batch job associated with the specified job_id.
160 pidinfo proc_id
163 Print the Slurm job id and scheduled termination time corre‐
164 sponding to the supplied process id, proc_id, on the current
165 node. This will work only with processes on node on which scon‐
166 trol is run, and only for those processes spawned by Slurm and
167 their descendants.
168 listpids [job_id[.step_id]] [NodeName]
171 Print a listing of the process IDs in a job step (if JO‐
172 BID.STEPID is provided), or all of the job steps in a job (if
173 job_id is provided), or all of the job steps in all of the jobs
174 on the local node (if job_id is not provided or job_id is "*").
175 This will work only with processes on the node on which scontrol
176 is run, and only for those processes spawned by Slurm and their
177 descendants. Note that some Slurm configurations (ProctrackType
178 value of pgid) are unable to identify all processes associated
179 with a job or job step.
180 Note that the NodeName option is only really useful when you
182 have multiple slurmd daemons running on the same host machine.
183 Multiple slurmd daemons on one host are, in general, only used
184 by Slurm developers.
185 ping Ping the primary and secondary slurmctld daemon and report if
188 they are responding.
189 reboot [ASAP] [nextstate=<RESUME|DOWN>] [reason=<reason>] <ALL|Node‐
192 List>
193 Reboot the nodes in the system when they become idle using the
194 RebootProgram as configured in Slurm's slurm.conf file. Each
195 node will have the "REBOOT" flag added to its node state. After
196 a node reboots and the slurmd daemon starts up again, the
197 HealthCheckProgram will run once. Then, the slurmd daemon will
198 register itself with the slurmctld daemon and the "REBOOT" flag
199 will be cleared. The node's "DRAIN" state flag will be cleared
200 if the reboot was "ASAP", nextstate=resume or down. The "ASAP"
201 option adds the "DRAIN" flag to each node's state, preventing
202 additional jobs from running on the node so it can be rebooted
203 and returned to service "As Soon As Possible" (i.e. ASAP).
204 "ASAP" will also set the node reason to "Reboot ASAP" if the
205 "reason" option isn't specified. If the "nextstate" option is
206 specified as "DOWN", then the node will remain in a down state
207 after rebooting. If "nextstate" is specified as "RESUME", then
208 the nodes will resume as normal and the node's reason and
209 "DRAIN" state will be cleared. Resuming nodes will be consid‐
210 ered as available in backfill future scheduling and won't be re‐
211 placed by idle nodes in a reservation. The "reason" option sets
212 each node's reason to a user-defined message. A default reason
213 of "reboot requested" is set if no other reason is set on the
214 node. The reason will be appended with "reboot issued" when the
215 reboot is issued and "reboot complete" when the node registers
216 and has a "nextstate" of "DOWN". You must specify either a list
217 of nodes or that ALL nodes are to be rebooted. NOTE: By de‐
218 fault, this command does not prevent additional jobs from being
219 scheduled on any nodes before reboot. To do this, you can ei‐
220 ther use the "ASAP" option or explicitly drain the nodes before‐
221 hand. You can alternately create an advanced reservation to
222 prevent additional jobs from being initiated on nodes to be re‐
223 booted. Pending reboots can be cancelled by using "scontrol
224 cancel_reboot <node>" or setting the node state to "CANCEL_RE‐
225 BOOT". A node will be marked "DOWN" if it doesn't reboot within
226 ResumeTimeout.
227 reconfigure
230 Instruct all Slurm daemons to re-read the configuration file.
231 This command does not restart the daemons. This mechanism would
232 be used to modify configuration parameters (Epilog, Prolog,
233 SlurmctldLogFile, SlurmdLogFile, etc.). The Slurm controller
234 (slurmctld) forwards the request to all other daemons (slurmd
235 daemon on each compute node). Running jobs continue execution.
236 Most configuration parameters can be changed by just running
237 this command, however, Slurm daemons should be shutdown and
238 restarted if any of these parameters are to be changed: Au‐
239 thType, ControlMach, PluginDir, StateSaveLocation, Slurmctld‐
240 Host, SlurmctldPort, or SlurmdPort. The slurmctld daemon and all
241 slurmd daemons must be restarted if nodes are added to or re‐
242 moved from the cluster.
243 release job_list
246 Release a previously held job to begin execution. The job_list
247 argument is a comma separated list of job IDs OR "jobname=" with
248 the job's name, which will attempt to hold all jobs having that
249 name. Also see hold.
250 requeue [option] job_list
253 Requeue a running, suspended or finished Slurm batch job into
254 pending state. The job_list argument is a comma separated list
255 of job IDs. The command accepts the following option:
256 Incomplete
258 Operate only on jobs (or tasks of a job array) which have
259 not completed. Specifically only jobs in the following
260 states will be requeued: CONFIGURING, RUNNING, STOPPED or
261 SUSPENDED.
262 requeuehold [option] job_list
265 Requeue a running, suspended or finished Slurm batch job into
266 pending state, moreover the job is put in held state (priority
267 zero). The job_list argument is a comma separated list of job
268 IDs. A held job can be released using scontrol to reset its
269 priority (e.g. "scontrol release <job_id>"). The command ac‐
270 cepts the following options:
271 Incomplete
273 Operate only on jobs (or tasks of a job array) which have
274 not completed. Specifically only jobs in the following
275 states will be requeued: CONFIGURING, RUNNING, STOPPED or
276 SUSPENDED.
277 State=SpecialExit
279 The "SpecialExit" keyword specifies that the job has to
280 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
281 trol show job" command will display the JobState as SPE‐
282 CIAL_EXIT, while the "squeue" command as SE.
283 resume job_list
286 Resume a previously suspended job. The job_list argument is a
287 comma separated list of job IDs. Also see suspend.
288 NOTE: A suspended job releases its CPUs for allocation to other
290 jobs. Resuming a previously suspended job may result in multi‐
291 ple jobs being allocated the same CPUs, which could trigger gang
292 scheduling with some configurations or severe degradation in
293 performance with other configurations. Use of the scancel com‐
294 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
295 out releasing its CPUs for allocation to other jobs and would be
296 a preferable mechanism in many cases. If performing system
297 maintenance you may want to use suspend/resume in the following
298 way. Before suspending set all nodes to draining or set all par‐
299 titions to down so that no new jobs can be scheduled. Then sus‐
300 pend jobs. Once maintenance is done resume jobs then resume
301 nodes and/or set all partitions back to up. Use with caution.
302 schedloglevel LEVEL
305 Enable or disable scheduler logging. LEVEL may be "0", "1",
306 "disable" or "enable". "0" has the same effect as "disable". "1"
307 has the same effect as "enable". This value is temporary and
308 will be overwritten when the slurmctld daemon reads the
309 slurm.conf configuration file (e.g. when the daemon is restarted
310 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
311 parameter is present.
312 setdebug LEVEL
315 Change the debug level of the slurmctld daemon. LEVEL may be an
316 integer value between zero and nine (using the same values as
317 SlurmctldDebug in the slurm.conf file) or the name of the most
318 detailed message type to be printed: "quiet", "fatal", "error",
319 "info", "verbose", "debug", "debug2", "debug3", "debug4", or
320 "debug5". This value is temporary and will be overwritten when‐
321 ever the slurmctld daemon reads the slurm.conf configuration
322 file (e.g. when the daemon is restarted or scontrol reconfigure
323 is executed).
324 setdebugflags [+|-]FLAG
327 Add or remove DebugFlags of the slurmctld daemon. See "man
328 slurm.conf" for a list of supported DebugFlags. NOTE: Changing
329 the value of some DebugFlags will have no effect without
330 restarting the slurmctld daemon, which would set DebugFlags
331 based upon the contents of the slurm.conf configuration file.
332 show ENTITY ID
335 or
336 show ENTITY=ID
338 Display the state of the specified entity with the specified
339 identification. ENTITY may be aliases, assoc_mgr, bbstat,
340 burstbuffer, config, daemons, dwstat, federation, frontend, job,
341 node, partition, reservation, slurmd, step, topology, hostlist,
342 hostlistsorted or hostnames ID can be used to identify a spe‐
343 cific element of the identified entity: job ID, node name, par‐
344 tition name, reservation name, or job step ID for job, node,
345 partition, or step respectively. For an ENTITY of bbstat or dw‐
346 stat (they are equivalent) optional arguments are the options of
347 the local status command. The status commands will be executed
348 by the slurmctld daemon and its response returned to the user.
349 For an ENTITY of topology, the ID may be a node or switch name.
350 If one node name is specified, all switches connected to that
351 node (and their parent switches) will be shown. If more than
352 one node name is specified, only switches that connect to all
353 named nodes will be shown. aliases will return all NodeName
354 values associated to a given NodeHostname (useful to get the
355 list of virtual nodes associated with a real node in a configu‐
356 ration where multiple slurmd daemons execute on a single compute
357 node). assoc_mgr displays the current contents of the slurm‐
358 ctld's internal cache for users, associations and/or qos. The ID
359 may be users=<user1>,[...,<userN>], accounts=<acct1>,[...,<ac‐
360 ctN>], qos=<qos1>,[...,<qosN>] and/or flags=<users,assoc,qos>,
361 used to filter the desired section to be displayed. If no flags
362 are specified, all sections are displayed. burstbuffer displays
363 the current status of the BurstBuffer plugin. config displays
364 parameter names from the configuration files in mixed case (e.g.
365 SlurmdPort=7003) while derived parameters names are in upper
366 case only (e.g. SLURM_VERSION). hostnames takes an optional
367 hostlist expression as input and writes a list of individual
368 host names to standard output (one per line). If no hostlist ex‐
369 pression is supplied, the contents of the SLURM_JOB_NODELIST en‐
370 vironment variable is used. For example "tux[1-3]" is mapped to
371 "tux1","tux2" and "tux3" (one hostname per line). hostlist
372 takes a list of host names and prints the hostlist expression
373 for them (the inverse of hostnames). hostlist can also take the
374 absolute pathname of a file (beginning with the character '/')
375 containing a list of hostnames. Multiple node names may be
376 specified using simple node range expressions (e.g.
377 "lx[10-20]"). All other ID values must identify a single ele‐
378 ment. The job step ID is of the form "job_id.step_id", (e.g.
379 "1234.1"). slurmd reports the current status of the slurmd dae‐
380 mon executing on the same node from which the scontrol command
381 is executed (the local host). It can be useful to diagnose prob‐
382 lems. By default hostlist does not sort the node list or make
383 it unique (e.g. tux2,tux1,tux2 = tux[2,1-2]). If you wanted a
384 sorted list use hostlistsorted (e.g. tux2,tux1,tux2 =
385 tux[1-2,2]). By default, all elements of the entity type speci‐
386 fied are printed. For an ENTITY of job, if the job does not
387 specify socket-per-node, cores-per-socket or threads-per-core
388 then it will display '*' in ReqS:C:T=*:*:* field. For an ENTITY
389 of federation, the federation name that the controller is part
390 of and the sibling clusters part of the federation will be
391 listed.
392 shutdown OPTION
395 Instruct Slurm daemons to save current state and terminate. By
396 default, the Slurm controller (slurmctld) forwards the request
397 all other daemons (slurmd daemon on each compute node). An OP‐
398 TION of slurmctld or controller results in only the slurmctld
399 daemon being shutdown and the slurmd daemons remaining active.
400 suspend job_list
403 Suspend a running job. The job_list argument is a comma sepa‐
404 rated list of job IDs. Use the resume command to resume its ex‐
405 ecution. User processes must stop on receipt of SIGSTOP signal
406 and resume upon receipt of SIGCONT for this operation to be ef‐
407 fective. Not all architectures and configurations support job
408 suspension. If a suspended job is requeued, it will be placed
409 in a held state. The time a job is suspended will not count
410 against a job's time limit. Only an operator, administrator,
411 SlurmUser, or root can suspend jobs.
412 takeover [INDEX]
415 Instruct one of Slurm's backup controllers (slurmctld) to take
416 over system control. By default the first backup controller (IN‐
417 DEX=1) requests control from the primary and waits for its ter‐
418 mination. After that, it switches from backup mode to controller
419 mode. If primary controller can not be contacted, it directly
420 switches to controller mode. This can be used to speed up the
421 Slurm controller fail-over mechanism when the primary node is
422 down. This can be used to minimize disruption if the computer
423 executing the primary Slurm controller is scheduled down.
424 (Note: Slurm's primary controller will take the control back at
425 startup.)
426 top job_list
429 Move the specified job IDs to the top of the queue of jobs be‐
430 longing to the identical user ID, partition name, account, and
431 QOS. The job_list argument is a comma separated ordered list of
432 job IDs. Any job not matching all of those fields will not be
433 effected. Only jobs submitted to a single partition will be ef‐
434 fected. This operation changes the order of jobs by adjusting
435 job nice values. The net effect on that user's throughput will
436 be negligible to slightly negative. This operation is disabled
437 by default for non-privileged (non-operator, admin, SlurmUser,
438 or root) users. This operation may be enabled for non-privileged
439 users by the system administrator by including the option "en‐
440 able_user_top" in the SchedulerParameters configuration parame‐
441 ter.
442 token [lifespan=<lifespan>] [username=<username>]
445 Return an auth token which can be used to support JWT authenti‐
446 cation if AuthAltTypes=auth/jwt has been enabled on the system.
447 Supports two optional arguments. lifespan= may be used to spec‐
448 ify the token's lifespan in seconds. username (only available to
449 SlurmUser/root) may be used to request a token for a different
450 username.
451 uhold job_list
454 Prevent a pending job from being started (sets its priority to
455 0). The job_list argument is a space separated list of job IDs
456 or job names. Use the release command to permit the job to be
457 scheduled. This command is designed for a system administrator
458 to hold a job so that the job owner may release it rather than
459 requiring the intervention of a system administrator (also see
460 the hold command).
461 update SPECIFICATION
464 Update job, step, node, partition, or reservation configuration
465 per the supplied specification. SPECIFICATION is in the same
466 format as the Slurm configuration file and the output of the
467 show command described above. It may be desirable to execute the
468 show command (described above) on the specific entity you want
469 to update, then use cut-and-paste tools to enter updated config‐
470 uration values to the update. Note that while most configuration
471 values can be changed using this command, not all can be changed
472 using this mechanism. In particular, the hardware configuration
473 of a node or the physical addition or removal of nodes from the
474 cluster may only be accomplished through editing the Slurm con‐
475 figuration file and executing the reconfigure command (described
476 above).
477 version
480 Display the version number of scontrol being executed.
481 wait_job job_id
484 Wait until a job and all of its nodes are ready for use or the
485 job has entered some termination state. This option is particu‐
486 larly useful in the Slurm Prolog or in the batch script itself
487 if nodes are powered down and restarted automatically as needed.
488 NOTE: Don't use scontrol wait_job in PrologCtld or Prolog with
490 PrologFlags=Alloc as this will result in deadlock.
491 write batch_script job_id optional_filename
494 Write the batch script for a given job_id to a file or to std‐
495 out. The file will default to slurm-<job_id>.sh if the optional
496 filename argument is not given. The script will be written to
497 stdout if - is given instead of a filename. The batch script
498 can only be retrieved by an admin or operator, or by the owner
499 of the job.
500 write config optional_filename
503 Write the current configuration to a file with the naming con‐
504 vention of "slurm.conf.<datetime>" in the same directory as the
505 original slurm.conf file. If a filename is given that file lo‐
506 cation with a .<datetime> suffix is created.
507 INTERACTIVE COMMANDS
510 NOTE: All commands listed below can be used in the interactive mode,
511 but NOT on the initial command line.
512 all Show all partitions, their jobs and jobs steps. This causes in‐
515 formation to be displayed about partitions that are configured
516 as hidden and partitions that are unavailable to user's group.
517 details
520 Causes the show command to provide additional details where
521 available. Job information will include CPUs and NUMA memory
522 allocated on each node. Note that on computers with hyper‐
523 threading enabled and Slurm configured to allocate cores, each
524 listed CPU represents one physical core. Each hyperthread on
525 that core can be allocated a separate task, so a job's CPU count
526 and task count may differ. See the --cpu-bind and --mem-bind
527 option descriptions in srun man pages for more information. The
528 details option is currently only supported for the show job com‐
529 mand.
530 exit Terminate scontrol interactive session.
533 hide Do not display partition, job or jobs step information for par‐
536 titions that are configured as hidden or partitions that are un‐
537 available to the user's group. This is the default behavior.
538 oneliner
541 Print information one line per record.
542 quiet Print no warning or informational messages, only fatal error
545 messages.
546 quit Terminate the execution of scontrol.
549 verbose
552 Print detailed event logging. This includes time-stamps on data
553 structures, record counts, etc.
554 !! Repeat the last command executed.
557 JOBS - SPECIFICATIONS FOR UPDATE COMMAND
560 Note that update requests done by either root, SlurmUser or Administra‐
561 tors are not subject to certain restrictions. For instance, if an Ad‐
562 ministrator changes the QOS on a pending job, certain limits such as
563 the TimeLimit will not be changed automatically as changes made by the
564 Administrators are allowed to violate these restrictions.
565 Account=<account>
568 Account name to be changed for this job's resource use. Value
569 may be cleared with blank data value, "Account=".
570 AdminComment=<spec>
572 Arbitrary descriptive string. Can only be set by a Slurm admin‐
573 istrator.
574 ArrayTaskThrottle=<count>
576 Specify the maximum number of tasks in a job array that can exe‐
577 cute at the same time. Set the count to zero in order to elimi‐
578 nate any limit. The task throttle count for a job array is re‐
579 ported as part of its ArrayTaskId field, preceded with a percent
580 sign. For example "ArrayTaskId=1-10%2" indicates the maximum
581 number of running tasks is limited to 2.
582 BurstBuffer=<spec>
584 Burst buffer specification to be changed for this job's resource
585 use. Value may be cleared with blank data value, "Burst‐
586 Buffer=". Format is burst buffer plugin specific.
587 Clusters=<spec>
589 Specifies the clusters that the federated job can run on.
590 ClusterFeatures=<spec>
592 Specifies features that a federated cluster must have to have a
593 sibling job submitted to it. Slurm will attempt to submit a sib‐
594 ling job to a cluster if it has at least one of the specified
595 features.
596 Comment=<spec>
598 Arbitrary descriptive string.
599 Contiguous=<yes|no>
601 Set the job's requirement for contiguous (consecutive) nodes to
602 be allocated. Possible values are "YES" and "NO". Only the
603 Slurm administrator or root can change this parameter.
604 CoreSpec=<count>
606 Number of cores to reserve per node for system use. The job
607 will be charged for these cores, but be unable to use them.
608 Will be reported as "*" if not constrained.
609 CPUsPerTask=<count>
611 Change the CPUsPerTask job's value.
612 Deadline=<time_spec>
614 It accepts times of the form HH:MM:SS to specify a deadline to a
615 job at a specific time of day (seconds are optional). You may
616 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
617 you can have a time-of-day suffixed with AM or PM for a deadline
618 in the morning or the evening. You can specify a deadline for
619 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
620 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
621 give times like now + count time-units, where the time-units can
622 be seconds (default), minutes, hours, days, or weeks and you can
623 tell Slurm to put a deadline for tomorrow with the keyword to‐
624 morrow. The specified deadline must be later than the current
625 time. Only pending jobs can have the deadline updated. Only
626 the Slurm administrator or root can change this parameter.
627 DelayBoot=<time_spec>
629 Change the time to decide whether to reboot nodes in order to
630 satisfy job's feature specification if the job has been eligible
631 to run for less than this time period. See salloc/sbatch man
632 pages option --delay-boot.
633 Dependency=<dependency_list>
635 Defer job's initiation until specified job dependency specifica‐
636 tion is satisfied. Cancel dependency with an empty depen‐
637 dency_list (e.g. "Dependency="). <dependency_list> is of the
638 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
639 can share the same dependency and these jobs may even belong to
640 different users.
641 after:job_id[:jobid...]
643 This job can begin execution after the specified jobs
644 have begun execution or been canceled.
645 afterany:job_id[:jobid...]
647 This job can begin execution after the specified jobs
648 have terminated.
649 afternotok:job_id[:jobid...]
651 This job can begin execution after the specified jobs
652 have terminated in some failed state (non-zero exit code,
653 node failure, timed out, etc).
654 afterok:job_id[:jobid...]
656 This job can begin execution after the specified jobs
657 have successfully executed (ran to completion with an
658 exit code of zero).
659 singleton
661 This job can begin execution after any previously
662 launched jobs sharing the same job name and user have
663 terminated. In other words, only one job by that name
664 and owned by that user can be running or suspended at any
665 point in time.
666 EligibleTime=<time_spec>
668 See StartTime.
669 EndTime
671 The time the job is expected to terminate based on the job's
672 time limit. When the job ends sooner, this field will be up‐
673 dated with the actual end time.
674 ExcNodeList=<nodes>
676 Set the job's list of excluded node. Multiple node names may be
677 specified using simple node range expressions (e.g.
678 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
679 NodeList=".
680 Features=<features>
682 Set the job's required node features. The list of features may
683 include multiple feature names separated by ampersand (AND)
684 and/or vertical bar (OR) operators. For example: Fea‐
685 tures="opteron&video" or Features="fast|faster". In the first
686 example, only nodes having both the feature "opteron" AND the
687 feature "video" will be used. There is no mechanism to specify
688 that you want one node with feature "opteron" and another node
689 with feature "video" in case no node has both features. If only
690 one of a set of possible options should be used for all allo‐
691 cated nodes, then use the OR operator and enclose the options
692 within square brackets. For example: "Fea‐
693 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
694 all nodes must be allocated on a single rack of the cluster, but
695 any of those four racks can be used. A request can also specify
696 the number of nodes needed with some feature by appending an as‐
697 terisk and count after the feature name. For example "Fea‐
698 tures=graphics*4" indicates that at least four allocated nodes
699 must have the feature "graphics." Parenthesis are also sup‐
700 ported for features to be ANDed together. For example "Fea‐
701 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
702 cation should include 4 nodes with ALL of the features "knl",
703 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
704 straints with node counts may only be combined with AND opera‐
705 tors. Value may be cleared with blank data value, for example
706 "Features=".
707 Gres=<list>
710 Specifies a comma delimited list of generic consumable re‐
711 sources. The format of each entry on the list is
712 "name[:count[*cpu]]". The name is that of the consumable re‐
713 source. The count is the number of those resources with a de‐
714 fault value of 1. The specified resources will be allocated to
715 the job on each node allocated unless "*cpu" is appended, in
716 which case the resources will be allocated on a per cpu basis.
717 The available generic consumable resources is configurable by
718 the system administrator. A list of available generic consum‐
719 able resources will be printed and the command will exit if the
720 option argument is "help". Examples of use include
721 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
722 JobId=<job_list>
725 Identify the job(s) to be updated. The job_list may be a comma
726 separated list of job IDs. Either JobId or JobName is required.
727 Licenses=<name>
729 Specification of licenses (or other resources available on all
730 nodes of the cluster) as described in salloc/sbatch/srun man
731 pages.
732 MailType=<types>
734 Set the mail event types. Valid type values are NONE, BEGIN,
735 END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, RE‐
736 QUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and
737 teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 per‐
738 cent of time limit), TIME_LIMIT_80 (reached 80 percent of time
739 limit), TIME_LIMIT_50 (reached 50 percent of time limit) and AR‐
740 RAY_TASKS (send emails for each array task). Multiple type val‐
741 ues may be specified in a comma separated list. Unless the AR‐
742 RAY_TASKS option is specified, mail notifications on job BEGIN,
743 END and FAIL apply to a job array as a whole rather than gener‐
744 ating individual email messages for each task in the job array.
745 MailUser=<name>
747 Set the user to receive email notification of state changes. A
748 blank string will set the mail user to the default which is the
749 submitting user.
750 MinCPUsNode=<count>
752 Set the job's minimum number of CPUs per node to the specified
753 value.
754 MinMemoryCPU=<megabytes>
756 Set the job's minimum real memory required per allocated CPU to
757 the specified value. Either MinMemoryCPU or MinMemoryNode may be
758 set, but not both.
759 MinMemoryNode=<megabytes>
761 Set the job's minimum real memory required per node to the spec‐
762 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
763 but not both.
764 MinTmpDiskNode=<megabytes>
766 Set the job's minimum temporary disk space required per node to
767 the specified value. Only the Slurm administrator or root can
768 change this parameter.
769 TimeMin=<timespec>
771 Change TimeMin value which specifies the minimum time limit min‐
772 utes of the job.
773 JobName=<name>
775 Identify the name of jobs to be modified or set the job's name
776 to the specified value. When used to identify jobs to be modi‐
777 fied, all jobs belonging to all users are modified unless the
778 UserID option is used to identify a specific user. Either JobId
779 or JobName is required.
780 Name[=<name>]
782 See JobName.
783 Nice[=<adjustment>]
785 Update the job with an adjusted scheduling priority within
786 Slurm. With no adjustment value the scheduling priority is de‐
787 creased by 100. A negative nice value increases the priority,
788 otherwise decreases it. The adjustment range is +/- 2147483645.
789 Only privileged users can specify a negative adjustment.
790 NodeList=<nodes>
792 Change the nodes allocated to a running job to shrink its size.
793 The specified list of nodes must be a subset of the nodes cur‐
794 rently allocated to the job. Multiple node names may be speci‐
795 fied using simple node range expressions (e.g. "lx[10-20]"). Af‐
796 ter a job's allocation is reduced, subsequent srun commands must
797 explicitly specify node and task counts which are valid for the
798 new allocation.
799 NumCPUs=<min_count>[-<max_count>]
801 Set the job's minimum and optionally maximum count of CPUs to be
802 allocated.
803 NumNodes=<min_count>[-<max_count>]
805 Set the job's minimum and optionally maximum count of nodes to
806 be allocated. If the job is already running, use this to spec‐
807 ify a node count less than currently allocated and resources
808 previously allocated to the job will be relinquished. After a
809 job's allocation is reduced, subsequent srun commands must ex‐
810 plicitly specify node and task counts which are valid for the
811 new allocation. Also see the NodeList parameter above. This is
812 the same as ReqNodes.
813 NumTasks=<count>
815 Set the job's count of required tasks to the specified value.
816 This is the same as ReqProcs.
817 OverSubscribe=<yes|no>
819 Set the job's ability to share compute resources (i.e. individ‐
820 ual CPUs) with other jobs. Possible values are "YES" and "NO".
821 This option can only be changed for pending jobs.
822 Partition=<name>
824 Set the job's partition to the specified value.
825 Priority=<number>
827 Set the job's priority to the specified value. Note that a job
828 priority of zero prevents the job from ever being scheduled. By
829 setting a job's priority to zero it is held. Set the priority
830 to a non-zero value to permit it to run. Explicitly setting a
831 job's priority clears any previously set nice value and removes
832 the priority/multifactor plugin's ability to manage a job's pri‐
833 ority. In order to restore the priority/multifactor plugin's
834 ability to manage a job's priority, hold and then release the
835 job. Only the Slurm administrator or root can increase job's
836 priority.
837 QOS=<name>
839 Set the job's QOS (Quality Of Service) to the specified value.
840 Value may be cleared with blank data value, "QOS=".
841 Reboot=<yes|no>
843 Set the job's flag that specifies whether to force the allocated
844 nodes to reboot before starting the job. This is only supported
845 with some system configurations and therefore it could be
846 silently ignored.
847 ReqCores=<count>
849 Change the job's requested Cores count.
850 ReqNodeList=<nodes>
852 Set the job's list of required node. Multiple node names may be
853 specified using simple node range expressions (e.g.
854 "lx[10-20]"). Value may be cleared with blank data value, "Re‐
855 qNodeList=".
856 ReqNodes=<min_count>[-<max_count>]
858 See NumNodes.
859 ReqProcs=<count>
861 See NumTasks.
862 ReqSockets=<count>
864 Change the job's requested socket count.
865 ReqThreads=<count>
867 Change the job's requested threads count.
868 Requeue=<0|1>
870 Stipulates whether a job should be requeued after a node fail‐
871 ure: 0 for no, 1 for yes.
872 ReservationName=<name>
874 Set the job's reservation to the specified value. Value may be
875 cleared with blank data value, "ReservationName=".
876 ResetAccrueTime
878 Reset the job's accrue time value to 0 meaning it will lose any
879 time previously accrued for priority. Helpful if you have a
880 large queue of jobs already in the queue and want to start lim‐
881 iting how many jobs can accrue time without waiting for the
882 queue to flush out.
883 SiteFactor=<account>
885 Specify the job's admin priority factor in the range of
886 +/-2147483645. Only privileged users can modify the value.
887 StdOut=<filepath>
889 Set the batch job's stdout file path.
890 Shared=<yes|no>
892 See OverSubscribe option above.
893 StartTime=<time_spec>
895 Set the job's earliest initiation time. It accepts times of the
896 form HH:MM:SS to run a job at a specific time of day (seconds
897 are optional). (If that time is already past, the next day is
898 assumed.) You may also specify midnight, noon, fika (3 PM) or
899 teatime (4 PM) and you can have a time-of-day suffixed with AM
900 or PM for running in the morning or the evening. You can also
901 say what day the job will be run, by specifying a date of the
902 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
903 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
904 count time-units, where the time-units can be seconds (default),
905 minutes, hours, days, or weeks and you can tell Slurm to run the
906 job today with the keyword today and to run the job tomorrow
907 with the keyword tomorrow.
908 Notes on date/time specifications:
910 - although the 'seconds' field of the HH:MM:SS time specifica‐
911 tion is allowed by the code, note that the poll time of the
912 Slurm scheduler is not precise enough to guarantee dispatch of
913 the job on the exact second. The job will be eligible to start
914 on the next poll following the specified time. The exact poll
915 interval depends on the Slurm scheduler (e.g., 60 seconds with
916 the default sched/builtin).
917 - if no time (HH:MM:SS) is specified, the default is
918 (00:00:00).
919 - if a date is specified without a year (e.g., MM/DD) then the
920 current year is assumed, unless the combination of MM/DD and
921 HH:MM:SS has already passed for that year, in which case the
922 next year is used.
923 Switches=<count>[@<max-time-to-wait>]
925 When a tree topology is used, this defines the maximum count of
926 switches desired for the job allocation. If Slurm finds an allo‐
927 cation containing more switches than the count specified, the
928 job remain pending until it either finds an allocation with de‐
929 sired switch count or the time limit expires. By default there
930 is no switch count limit and no time limit delay. Set the count
931 to zero in order to clean any previously set count (disabling
932 the limit). The job's maximum time delay may be limited by the
933 system administrator using the SchedulerParameters configuration
934 parameter with the max_switch_wait parameter option. Also see
935 wait-for-switch.
936 wait-for-switch=<seconds>
939 Change max time to wait for a switch <seconds> secs.
940 TasksPerNode=<count>
943 Change the job's requested TasksPerNode.
944 ThreadSpec=<count>
947 Number of threads to reserve per node for system use. The job
948 will be charged for these threads, but be unable to use them.
949 Will be reported as "*" if not constrained.
950 TimeLimit=<time>
953 The job's time limit. Output format is [days-]hours:min‐
954 utes:seconds or "UNLIMITED". Input format (for update command)
955 set is minutes, minutes:seconds, hours:minutes:seconds,
956 days-hours, days-hours:minutes or days-hours:minutes:seconds.
957 Time resolution is one minute and second values are rounded up
958 to the next minute. If changing the time limit of a job, either
959 specify a new time limit value or precede the time and equal
960 sign with a "+" or "-" to increment or decrement the current
961 time limit (e.g. "TimeLimit+=30"). In order to increment or
962 decrement the current time limit, the JobId specification must
963 precede the TimeLimit specification. Note that incrementing or
964 decrementing the time limit for a job array is only allowed be‐
965 fore the job array has been split into more than one job record.
966 Only the Slurm administrator or root can increase job's Time‐
967 Limit.
968 UserID=<UID or name>
971 Used with the JobName option to identify jobs to be modified.
972 Either a user name or numeric ID (UID), may be specified.
973 WCKey=<key>
976 Set the job's workload characterization key to the specified
977 value.
978 WorkDir=<directory_name>
981 Set the job's working directory to the specified value. Note
982 that this may only be set for jobs in the PENDING state, and
983 that jobs may fail to launch if they rely on relative paths to
984 the originally submitted WorkDir.
985 JOBS - SPECIFICATIONS FOR SHOW COMMAND
988 The "show" command, when used with the "job" or "job <jobid>" entity
989 displays detailed information about a job or jobs. Much of this infor‐
990 mation may be modified using the "update job" command as described
991 above. However, the following fields displayed by the show job command
992 are read-only and cannot be modified:
993 AllocNode:Sid
996 Local node and system id making the resource allocation.
997 BatchFlag
999 Jobs submitted using the sbatch command have BatchFlag set to 1.
1000 Jobs submitted using other commands have BatchFlag set to 0.
1001 ExitCode=<exit>:<sig>
1003 Exit status reported for the job by the wait() function. The
1004 first number is the exit code, typically as set by the exit()
1005 function. The second number of the signal that caused the
1006 process to terminate if it was terminated by a signal.
1007 GroupId
1009 The group under which the job was submitted.
1010 JobState
1012 The current state of the job.
1013 NodeListIndices
1015 The NodeIndices expose the internal indices into the node table
1016 associated with the node(s) allocated to the job.
1017 NtasksPerN:B:S:C=
1019 <tasks_per_node>:<tasks_per_base‐
1020 board>:<tasks_per_socket>:<tasks_per_core> Specifies the number
1021 of tasks to be started per hardware component (node, baseboard,
1022 socket and core). Unconstrained values may be shown as "0" or
1023 "*".
1024 PreemptEligibleTime
1026 Time the job becomes eligible for preemption. Modified by Pre‐
1027 emptExemptTime, either from the global option in slurm.conf or
1028 the job QOS. This is hidden if the job has not started or if
1029 PreemptMode=OFF.
1030 PreemptTime
1032 Time at which job was signaled that it was selected for preemp‐
1033 tion. (Meaningful only for PreemptMode=CANCEL and the partition
1034 or QOS with which the job is associated has a GraceTime value
1035 designated.) This is hidden if the job has not started or if
1036 PreemptMode=OFF.
1037 PreSusTime
1039 Time the job ran prior to last suspend.
1040 Reason The reason job is not running: e.g., waiting "Resources".
1042 ReqB:S:C:T=
1044 <baseboard_count>:<socket_per_base‐
1045 board_count>:<core_per_socket_count>:<thread_per_core_count>
1046 Specifies the count of various hardware components requested by
1047 the job. Unconstrained values may be shown as "0" or "*".
1048 SecsPreSuspend=<seconds>
1050 If the job is suspended, this is the run time accumulated by the
1051 job (in seconds) prior to being suspended.
1052 Socks/Node=<count>
1054 Count of desired sockets per node
1055 SubmitTime
1057 The time and date stamp (in localtime) the job was submitted.
1058 The format of the output is identical to that of the EndTime
1059 field.
1060 NOTE: If a job is requeued, the submit time is reset. To obtain
1062 the original submit time it is necessary to use the "sacct -j
1063 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1064 cate option to display all duplicate entries for a job.
1065 SuspendTime
1067 Time the job was last suspended or resumed.
1068 NOTE on information displayed for various job states: When you
1070 submit a request for the "show job" function the scontrol
1071 process makes an RPC request call to slurmctld with a RE‐
1072 QUEST_JOB_INFO message type. If the state of the job is PEND‐
1073 ING, then it returns some detail information such as: min_nodes,
1074 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1075 ING the code assumes that it is in a further state such as RUN‐
1076 NING, COMPLETE, etc. In these cases the code explicitly returns
1077 zero for these values. These values are meaningless once the job
1078 resources have been allocated and the job has started.
1079 STEPS - SPECIFICATIONS FOR UPDATE COMMAND
1082 StepId=<job_id>[.<step_id>]
1083 Identify the step to be updated. If the job_id is given, but no
1084 step_id is specified then all steps of the identified job will
1085 be modified. This specification is required.
1086 CompFile=<completion file>
1088 Update a step with information about a step's completion. Can
1089 be useful if step statistics aren't directly available through a
1090 jobacct_gather plugin. The file is a space-delimited file with
1091 format for Version 1 is as follows
1092 1 34461 0 2 0 3 1361906011 1361906015 1 1 3368 13357 /bin/sleep
1094 A B C D E F G H I J K L M
1095 Field Descriptions:
1097 A file version
1099 B ALPS apid
1100 C inblocks
1101 D outblocks
1102 E exit status
1103 F number of allocated CPUs
1104 G start time
1105 H end time
1106 I utime
1107 J stime
1108 K maxrss
1109 L uid
1110 M command name
1111 TimeLimit=<time>
1113 The job's time limit. Output format is [days-]hours:min‐
1114 utes:seconds or "UNLIMITED". Input format (for update command)
1115 set is minutes, minutes:seconds, hours:minutes:seconds,
1116 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1117 Time resolution is one minute and second values are rounded up
1118 to the next minute. If changing the time limit of a step, ei‐
1119 ther specify a new time limit value or precede the time with a
1120 "+" or "-" to increment or decrement the current time limit
1121 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1122 current time limit, the StepId specification must precede the
1123 TimeLimit specification.
1124 NODES - SPECIFICATIONS FOR UPDATE COMMAND
1127 NodeName=<name>
1128 Identify the node(s) to be updated. Multiple node names may be
1129 specified using simple node range expressions (e.g.
1130 "lx[10-20]"). This specification is required.
1131 ActiveFeatures=<features>
1134 Identify the feature(s) currently active on the specified node.
1135 Any previously active feature specification will be overwritten
1136 with the new value. Also see AvailableFeatures. Typically Ac‐
1137 tiveFeatures will be identical to AvailableFeatures; however Ac‐
1138 tiveFeatures may be configured as a subset of the AvailableFea‐
1139 tures. For example, a node may be booted in multiple configura‐
1140 tions. In that case, all possible configurations may be identi‐
1141 fied as AvailableFeatures, while ActiveFeatures would identify
1142 the current node configuration.
1143 AvailableFeatures=<features>
1146 Identify the feature(s) available on the specified node. Any
1147 previously defined available feature specification will be over‐
1148 written with the new value. AvailableFeatures assigned via
1149 scontrol will only persist across the restart of the slurmctld
1150 daemon with the -R option and state files preserved or slurm‐
1151 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1152 meant to be persistent across normal restarts of slurmctld or
1153 the execution of scontrol reconfig.
1154 Note: Available features being removed via scontrol must not be
1156 active (i.e. remove them from ActiveFeatures first).
1157 Comment=<comment>
1160 Arbitrary descriptive string. Use quotes to enclose a comment
1161 having more than one word
1162 CpuBind=<node>
1165 Specify the task binding mode to be used by default for this
1166 node. Supported options include: "none", "board", "socket",
1167 "ldom" (NUMA), "core", "thread" and "off" (remove previous bind‐
1168 ing mode).
1169 Gres=<gres>
1172 Identify generic resources to be associated with the specified
1173 node. Any previously defined generic resources will be over‐
1174 written with the new value. Specifications for multiple generic
1175 resources should be comma separated. Each resource specifica‐
1176 tion consists of a name followed by an optional colon with a nu‐
1177 meric value (default value is one) (e.g. "Gres=band‐
1178 width:10000"). Modification of GRES count associated with spe‐
1179 cific files (e.g. GPUs) is not allowed other than to set their
1180 count on a node to zero. In order to change the GRES count to
1181 another value, modify your slurm.conf and gres.conf files and
1182 restart daemons. If GRES as associated with specific sockets,
1183 that information will be reported For example if all 4 GPUs on a
1184 node are all associated with socket zero, then
1185 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1186 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1187 associated with specific GPUs is not reported, but only avail‐
1188 able by parsing the gres.conf file. Generic resources assigned
1189 via scontrol will only persist across the restart of the slurm‐
1190 ctld daemon with the -R option and state files preserved or
1191 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1192 changes meant to be persistent across normal restarts of slurm‐
1193 ctld or the execution of scontrol reconfig.
1194 NodeAddr=<node address>
1197 Name that a node should be referred to in establishing a commu‐
1198 nications path. This name will be used as an argument to the
1199 getaddrinfo() function for identification. If a node range ex‐
1200 pression is used to designate multiple nodes, they must exactly
1201 match the entries in the NodeName (e.g. "NodeName=lx[0-7]
1202 NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses.
1203 NodeHostname=<node hostname>
1206 Typically this would be the string that "/bin/hostname -s" re‐
1207 turns. It may also be the fully qualified domain name as re‐
1208 turned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid
1209 domain name associated with the host through the host database
1210 (/etc/hosts) or DNS, depending on the resolver settings. Note
1211 that if the short form of the hostname is not used, it may pre‐
1212 vent use of hostlist expressions (the numeric portion in brack‐
1213 ets must be at the end of the string). A node range expression
1214 can be used to specify a set of nodes. If an expression is used,
1215 the number of nodes identified by NodeHostname must be identical
1216 to the number of nodes identified by NodeName.
1217 Reason=<reason>
1220 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1221 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1222 having more than one word.
1223 State=<state>
1226 Identify the state to be assigned to the node. Possible node
1227 states are "NoResp", "ALLOC", "ALLOCATED", "COMPLETING", "DOWN",
1228 "DRAIN", "FAIL", "FAILING", "FUTURE" "IDLE", "MAINT", "MIXED",
1229 "PERFCTRS/NPC", "RESERVED", "POWER_DOWN", "POWER_UP", "RESUME"
1230 or "UNDRAIN". Not all of those states can be set using the scon‐
1231 trol command only the following can: "CANCEL_REBOOT", "DOWN",
1232 "DRAIN", "FAIL", "FUTURE", "RESUME", "NoResp", "POWER_DOWN",
1233 "POWER_UP" and "UNDRAIN". If a node is in a "MIXED" state it
1234 usually means the node is in multiple states. For instance if
1235 only part of the node is "ALLOCATED" and the rest of the node is
1236 "IDLE" the state will be "MIXED". If you want to remove a node
1237 from service, you typically want to set its state to "DRAIN".
1238 "CANCEL_REBOOT" cancels a pending reboot on the node (same as
1239 "scontrol cancel_reboot <node>"). "FAILING" is similar to
1240 "DRAIN" except that some applications will seek to relinquish
1241 those nodes before the job completes. "PERFCTRS/NPC" indicates
1242 that Network Performance Counters associated with this node are
1243 in use, rendering this node as not usable for any other jobs.
1244 "RESERVED" indicates the node is in an advanced reservation and
1245 not generally available. "RESUME" is not an actual node state,
1246 but will change a node state from "DRAINED", "DRAINING", "DOWN"
1247 or "REBOOT" to either "IDLE" or "ALLOCATED" state as appropri‐
1248 ate. "RESUME" will also clear the "POWERING_DOWN" state of a
1249 node and make it eligible to be allocted. "UNDRAIN" clears the
1250 node from being drained (like "RESUME"), but will not change the
1251 node's base state (e.g. "DOWN"). Setting a node "DOWN" will
1252 cause all running and suspended jobs on that node to be termi‐
1253 nated. "POWER_DOWN" and "POWER_UP" will use the configured Sus‐
1254 pendProg and ResumeProg programs to explicitly place a node in
1255 or out of a power saving mode. If a node is already in the
1256 process of being powered up or down, the command will only
1257 change the state of the node but won't have any effect until the
1258 configured ResumeTimeout or SuspendTimeout is reached. Use of
1259 this command can be useful in situations where a ResumeProg like
1260 capmc in Cray machines is stalled and one wants to restore the
1261 node to "IDLE" manually, in this case rebooting the node and
1262 setting the state to "POWER_DOWN" will cancel the previous
1263 "POWER_UP" state and the node will become "IDLE". The "NoResp"
1264 state will only set the "NoResp" flag for a node without chang‐
1265 ing its underlying state. While all of the above states are
1266 valid, some of them are not valid new node states given their
1267 prior state. If the node state code printed is followed by "~",
1268 this indicates the node is presently in a power saving mode
1269 (typically running at reduced frequency). If the node state
1270 code is followed by "#", this indicates the node is presently
1271 being powered up or configured. If the node state code is fol‐
1272 lowed by "$", this indicates the node is currently in a reserva‐
1273 tion with a flag value of "maintenance". If the node state code
1274 is followed by "@", this indicates the node is currently sched‐
1275 uled to be rebooted. Generally only "DRAIN", "FAIL" and "RE‐
1276 SUME" should be used. NOTE: The scontrol command should not be
1277 used to change node state on Cray systems. Use Cray tools such
1278 as xtprocadmin instead.
1279 Weight=<weight>
1282 Identify weight to be associated with specified nodes. This al‐
1283 lows dynamic changes to weight associated with nodes, which will
1284 be used for the subsequent node allocation decisions. Weight
1285 assigned via scontrol will only persist across the restart of
1286 the slurmctld daemon with the -R option and state files pre‐
1287 served or slurmctld's receipt of a SIGHUP. Update slurm.conf
1288 with any changes meant to be persistent across normal restarts
1289 of slurmctld or the execution of scontrol reconfig.
1290 NODES - SPECIFICATIONS FOR SHOW COMMAND
1293 The meaning of the energy information is as follows:
1294 CurrentWatts
1297 The instantaneous power consumption of the node at the time of
1298 the last node energy accounting sample, in watts.
1299 LowestJoules
1302 The energy consumed by the node between the last time it was
1303 powered on and the last time it was registered by slurmd, in
1304 joules.
1305 ConsumedJoules
1308 The energy consumed by the node between the last time it was
1309 registered by the slurmd daemon and the last node energy ac‐
1310 counting sample, in joules.
1311 If the reported value is "n/s" (not supported), the node does not sup‐
1314 port the configured AcctGatherEnergyType plugin. If the reported value
1315 is zero, energy accounting for nodes is disabled.
1316 The meaning of the external sensors information is as follows:
1319 ExtSensorsJoules
1322 The energy consumed by the node between the last time it was
1323 powered on and the last external sensors plugin node sample, in
1324 joules.
1325 ExtSensorsWatts
1329 The instantaneous power consumption of the node at the time of
1330 the last external sensors plugin node sample, in watts.
1331 ExtSensorsTemp
1334 The temperature of the node at the time of the last external
1335 sensors plugin node sample, in celsius.
1336 If the reported value is "n/s" (not supported), the node does not sup‐
1339 port the configured ExtSensorsType plugin.
1340 The meaning of the resource specialization information is as follows:
1343 CPUSpecList
1346 The list of Slurm abstract CPU IDs on this node reserved for ex‐
1347 clusive use by the Slurm compute node daemons (slurmd, slurm‐
1348 stepd).
1349 MemSpecLimit
1352 The combined memory limit, in megabytes, on this node for the
1353 Slurm compute node daemons (slurmd, slurmstepd).
1354 The meaning of the memory information is as follows:
1357 RealMemory
1360 The total memory, in MB, on the node.
1361 AllocMem
1364 The total memory, in MB, currently allocated by jobs on the
1365 node.
1366 FreeMem
1369 The total memory, in MB, currently free on the node as reported
1370 by the OS.
1371 FRONTEND - SPECIFICATIONS FOR UPDATE COMMAND
1374 FrontendName=<name>
1375 Identify the front end node to be updated. This specification is
1376 required.
1377 Reason=<reason>
1380 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1381 Use quotes to enclose a reason having more than one word.
1382 State=<state>
1385 Identify the state to be assigned to the front end node. Possi‐
1386 ble values are "DOWN", "DRAIN" or "RESUME". If you want to re‐
1387 move a front end node from service, you typically want to set
1388 its state to "DRAIN". "RESUME" is not an actual node state, but
1389 will return a "DRAINED", "DRAINING", or "DOWN" front end node to
1390 service, either "IDLE" or "ALLOCATED" state as appropriate.
1391 Setting a front end node "DOWN" will cause all running and sus‐
1392 pended jobs on that node to be terminated.
1393 PARTITIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
1396 AllowGroups=<name>
1397 Identify the user groups which may use this partition. Multiple
1398 groups may be specified in a comma separated list. To permit
1399 all groups to use the partition specify "AllowGroups=ALL".
1400 AllocNodes=<name>
1403 Comma separated list of nodes from which users can execute jobs
1404 in the partition. Node names may be specified using the node
1405 range expression syntax described above. The default value is
1406 "ALL".
1407 Alternate=<partition name>
1410 Alternate partition to be used if the state of this partition is
1411 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1412 set alternate partition.
1413 CpuBind=<node>
1416 Specify the task binding mode to be used by default for this
1417 partition. Supported options include: "none", "board",
1418 "socket", "ldom" (NUMA), "core", "thread" and "off" (remove pre‐
1419 vious binding mode).
1420 Default=<yes|no>
1423 Specify if this partition is to be used by jobs which do not ex‐
1424 plicitly identify a partition to use. Possible output values
1425 are "YES" and "NO". In order to change the default partition of
1426 a running system, use the scontrol update command and set De‐
1427 fault=yes for the partition that you want to become the new de‐
1428 fault.
1429 DefaultTime=<time>
1432 Run time limit used for jobs that don't specify a value. If not
1433 set then MaxTime will be used. Format is the same as for Max‐
1434 Time.
1435 DefMemPerCPU=<MB>
1438 Set the default memory to be allocated per CPU for jobs in this
1439 partition. The memory size is specified in megabytes.
1440 DefMemPerNode=<MB>
1442 Set the default memory to be allocated per node for jobs in this
1443 partition. The memory size is specified in megabytes.
1444 DisableRootJobs=<yes|no>
1447 Specify if jobs can be executed as user root. Possible values
1448 are "YES" and "NO".
1449 GraceTime=<seconds>
1452 Specifies, in units of seconds, the preemption grace time to be
1453 extended to a job which has been selected for preemption. The
1454 default value is zero, no preemption grace time is allowed on
1455 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1456 Hidden=<yes|no>
1459 Specify if the partition and its jobs should be hidden from
1460 view. Hidden partitions will by default not be reported by
1461 Slurm APIs or commands. Possible values are "YES" and "NO".
1462 JobDefaults=<specs>
1465 Specify job default values using a comma delimited list of
1466 "key=value" pairs. Supported keys include
1467 DefCpuPerGPU Default number of CPUs per allocated GPU.
1469 DefMemPerGPU Default memory limit (in megabytes) per allocated
1471 GPU.
1472 MaxMemPerCPU=<MB>
1475 Set the maximum memory to be allocated per CPU for jobs in this
1476 partition. The memory size is specified in megabytes.
1477 MaxMemPerCNode=<MB>
1479 Set the maximum memory to be allocated per node for jobs in this
1480 partition. The memory size is specified in megabytes.
1481 MaxNodes=<count>
1484 Set the maximum number of nodes which will be allocated to any
1485 single job in the partition. Specify a number, "INFINITE" or
1486 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1487 upon jobs that have already begun execution.
1488 MaxTime=<time>
1491 The maximum run time for jobs. Output format is
1492 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1493 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1494 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1495 onds. Time resolution is one minute and second values are
1496 rounded up to the next minute. Changing the MaxTime of a parti‐
1497 tion has no effect upon jobs that have already begun execution.
1498 MinNodes=<count>
1501 Set the minimum number of nodes which will be allocated to any
1502 single job in the partition. Changing the MinNodes of a parti‐
1503 tion has no effect upon jobs that have already begun execution.
1504 Increasing this value may prevent pending jobs from starting,
1505 even if they were submitted without -N/--nodes specification.
1506 If you do get in that situation, updating the MinNodes value of
1507 a pending job using the scontrol command will allow that job to
1508 be scheduled.
1509 Nodes=<name>
1512 Identify the node(s) to be associated with this partition. Mul‐
1513 tiple node names may be specified using simple node range ex‐
1514 pressions (e.g. "lx[10-20]"). Note that jobs may only be asso‐
1515 ciated with one partition at any time. Specify a blank data
1516 value to remove all nodes from a partition: "Nodes=". Changing
1517 the Nodes in a partition has no effect upon jobs that have al‐
1518 ready begun execution.
1519 OverTimeLimit=<count>
1522 Number of minutes by which a job can exceed its time limit be‐
1523 fore being canceled. The configured job time limit is treated
1524 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1525 vides a hard limit, at which point the job is canceled. This is
1526 particularly useful for backfill scheduling, which bases upon
1527 each job's soft time limit. A partition-specific OverTimeLimit
1528 will override any global OverTimeLimit value. If not specified,
1529 the global OverTimeLimit value will take precedence. May not
1530 exceed 65533 minutes. An input value of "UNLIMITED" will clear
1531 any previously configured partition-specific OverTimeLimit
1532 value.
1533 OverSubscribe=<yes|no|exclusive|force>[:<job_count>]
1536 Specify if compute resources (i.e. individual CPUs) in this par‐
1537 tition can be shared by multiple jobs. Possible values are
1538 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1539 specifies how many jobs can be allocated to use each resource.
1540 PartitionName=<name>
1543 Identify the partition to be updated. This specification is re‐
1544 quired.
1545 PreemptMode=<mode>
1548 Reset the mechanism used to preempt jobs in this partition if
1549 PreemptType is configured to preempt/partition_prio. The default
1550 preemption mechanism is specified by the cluster-wide Preempt‐
1551 Mode configuration parameter. Possible values are "OFF", "CAN‐
1552 CEL", "REQUEUE" and "SUSPEND".
1553 Priority=<count>
1556 Jobs submitted to a higher priority partition will be dispatched
1557 before pending jobs in lower priority partitions and if possible
1558 they will preempt running jobs from lower priority partitions.
1559 Note that a partition's priority takes precedence over a job's
1560 priority. The value may not exceed 65533.
1561 PriorityJobFactor=<count>
1564 Partition factor used by priority/multifactor plugin in calcu‐
1565 lating job priority. The value may not exceed 65533. Also see
1566 PriorityTier.
1567 PriorityTier=<count>
1570 Jobs submitted to a partition with a higher priority tier value
1571 will be dispatched before pending jobs in partition with lower
1572 priority tier value and, if possible, they will preempt
1573 running jobs from partitions with lower priority tier values.
1574 Note that a partition's priority tier takes precedence over a
1575 job's priority. The value may not exceed 65533. Also see Pri‐
1576 orityJobFactor.
1577 QOS=<QOSname|blank to remove>
1580 Set the partition QOS with a QOS name or to remove the Partition
1581 QOS leave the option blank.
1582 RootOnly=<yes|no>
1585 Specify if only allocation requests initiated by user root will
1586 be satisfied. This can be used to restrict control of the par‐
1587 tition to some meta-scheduler. Possible values are "YES" and
1588 "NO".
1589 ReqResv=<yes|no>
1592 Specify if only allocation requests designating a reservation
1593 will be satisfied. This is used to restrict partition usage to
1594 be allowed only within a reservation. Possible values are "YES"
1595 and "NO".
1596 Shared=<yes|no|exclusive|force>[:<job_count>]
1599 Renamed to OverSubscribe, see option descriptions above.
1600 State=<up|down|drain|inactive>
1603 Specify if jobs can be allocated nodes or queued in this parti‐
1604 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1605 UP Designates that new jobs may queued on the partition,
1607 and that jobs may be allocated nodes and run from the
1608 partition.
1609 DOWN Designates that new jobs may be queued on the parti‐
1611 tion, but queued jobs may not be allocated nodes and
1612 run from the partition. Jobs already running on the
1613 partition continue to run. The jobs must be explicitly
1614 canceled to force their termination.
1615 DRAIN Designates that no new jobs may be queued on the par‐
1617 tition (job submission requests will be denied with an
1618 error message), but jobs already queued on the parti‐
1619 tion may be allocated nodes and run. See also the
1620 "Alternate" partition specification.
1621 INACTIVE Designates that no new jobs may be queued on the par‐
1623 tition, and jobs already queued may not be allocated
1624 nodes and run. See also the "Alternate" partition
1625 specification.
1626 TRESBillingWeights=<TRES Billing Weights>
1629 TRESBillingWeights is used to define the billing weights of each
1630 TRES type that will be used in calculating the usage of a job.
1631 The calculated usage is used when calculating fairshare and when
1632 enforcing the TRES billing limit on jobs. Updates affect new
1633 jobs and not existing jobs. See the slurm.conf man page for
1634 more information.
1635 RESERVATIONS - SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS
1638 Reservation=<name>
1639 Identify the name of the reservation to be created, updated, or
1640 deleted. This parameter is required for update and is the only
1641 parameter for delete. For create, if you do not want to give a
1642 reservation name, use "scontrol create res ..." and a name will
1643 be created automatically.
1644 Accounts=<account list>
1647 List of accounts permitted to use the reserved nodes, for exam‐
1648 ple "Accounts=physcode1,physcode2". A user in any of the ac‐
1649 counts may use the reserved nodes. A new reservation must spec‐
1650 ify Users or Groups and/or Accounts. If both Users/Groups and
1651 Accounts are specified, a job must match both in order to use
1652 the reservation. Accounts can also be denied access to reserva‐
1653 tions by preceding all of the account names with '-'. Alter‐
1654 nately precede the equal sign with '-'. For example, "Ac‐
1655 counts=-physcode1,-physcode2" or "Accounts-=physcode1,physcode2"
1656 will permit any account except physcode1 and physcode2 to use
1657 the reservation. You can add or remove individual accounts from
1658 an existing reservation by using the update command and adding a
1659 '+' or '-' sign before the '=' sign. If accounts are denied ac‐
1660 cess to a reservation (account name preceded by a '-'), then all
1661 other accounts are implicitly allowed to use the reservation and
1662 it is not possible to also explicitly specify allowed accounts.
1663 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1666 Specification of burst buffer resources which are to be re‐
1667 served. "buffer_spec" consists of four elements:
1668 [plugin:][type:]#[units] "plugin" is the burst buffer plugin
1669 name, currently either "datawarp" or "generic". If no plugin is
1670 specified, the reservation applies to all configured burst buf‐
1671 fer plugins. "type" specifies a Cray generic burst buffer re‐
1672 source, for example "nodes". if "type" is not specified, the
1673 number is a measure of storage space. The "units" may be "N"
1674 (nodes), "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers
1675 of 1024) and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).
1676 The default units are bytes for reservations of storage space.
1677 For example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage
1678 plus 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (re‐
1679 serve 100 GB of storage from all configured burst buffer plug‐
1680 ins). Jobs using this reservation are not restricted to these
1681 burst buffer resources, but may use these reserved resources
1682 plus any which are generally available. NOTE: Usually Slurm in‐
1683 terprets KB, MB, GB, TB, PB, TB units as powers of 1024, but for
1684 Burst Buffers size specifications Slurm supports both IEC/SI
1685 formats. This is because the CRAY API for managing DataWarps
1686 supports both formats.
1687 CoreCnt=<num>
1690 This option is only supported when SelectType=select/cons_res or
1691 select/cons_tres. Identify number of cores to be reserved. If
1692 NodeCnt is used without the FIRST_CORES flag, this is the total
1693 number of cores to reserve where cores per node is
1694 CoreCnt/NodeCnt. If a nodelist is used, or if NodeCnt is used
1695 with the FIRST_CORES flag, this should be an array of core num‐
1696 bers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or
1697 flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
1698 Licenses=<license>
1701 Specification of licenses (or other resources available on all
1702 nodes of the cluster) which are to be reserved. License names
1703 can be followed by a colon and count (the default count is one).
1704 Multiple license names should be comma separated (e.g. "Li‐
1705 censes=foo:4,bar"). A new reservation must specify one or more
1706 resource to be included: NodeCnt, Nodes and/or Licenses. If a
1707 reservation includes Licenses, but no NodeCnt or Nodes, then the
1708 option Flags=LICENSE_ONLY must also be specified. Jobs using
1709 this reservation are not restricted to these licenses, but may
1710 use these reserved licenses plus any which are generally avail‐
1711 able.
1712 MaxStartDelay[=<timespec>]
1715 Change MaxStartDelay value which specifies the maximum time an
1716 eligible job not requesting this reservation can delay a job re‐
1717 questing it. Default is none. Valid formats are minutes, min‐
1718 utes:seconds, hours:minutes:seconds, days-hours, days-hours:min‐
1719 utes, days-hours:minutes:seconds. Time resolution is one minute
1720 and second values are rounded up to the next minute. Output for‐
1721 mat is always [days-]hours:minutes:seconds.
1722 NodeCnt=<num>[,num,...]
1725 Identify number of nodes to be reserved. The number can include
1726 a suffix of "k" or "K", in which case the number specified is
1727 multiplied by 1024. A new reservation must specify one or more
1728 resource to be included: NodeCnt, Nodes and/or Licenses.
1729 Nodes=<name>
1732 Identify the node(s) to be reserved. Multiple node names may be
1733 specified using simple node range expressions (e.g.
1734 "Nodes=lx[10-20]"). Specify a blank data value to remove all
1735 nodes from a reservation: "Nodes=". A new reservation must
1736 specify one or more resource to be included: NodeCnt, Nodes
1737 and/or Licenses. A specification of "ALL" will reserve all
1738 nodes. Set Flags=PART_NODES and PartitionName= in order for
1739 changes in the nodes associated with a partition to also be re‐
1740 flected in the nodes associated with a reservation.
1741 StartTime=<time_spec>
1744 The start time for the reservation. A new reservation must
1745 specify a start time. It accepts times of the form HH:MM:SS for
1746 a specific time of day (seconds are optional). (If that time is
1747 already past, the next day is assumed.) You may also specify
1748 midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
1749 time-of-day suffixed with AM or PM for running in the morning or
1750 the evening. You can also say what day the job will be run, by
1751 specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or
1752 a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also give
1753 times like now + count time-units, where the time-units can be
1754 seconds (default), minutes, hours, days, or weeks and you can
1755 tell Slurm to run the job today with the keyword today and to
1756 run the job tomorrow with the keyword tomorrow. You cannot up‐
1757 date the StartTime of a reservation in ACTIVE state.
1758 EndTime=<time_spec>
1761 The end time for the reservation. A new reservation must spec‐
1762 ify an end time or a duration. Valid formats are the same as
1763 for StartTime.
1764 Duration=<time>
1767 The length of a reservation. A new reservation must specify an
1768 end time or a duration. Valid formats are minutes, minutes:sec‐
1769 onds, hours:minutes:seconds, days-hours, days-hours:minutes,
1770 days-hours:minutes:seconds, or UNLIMITED. Time resolution is
1771 one minute and second values are rounded up to the next minute.
1772 Output format is always [days-]hours:minutes:seconds.
1773 PartitionName=<name>
1776 Partition used to reserve nodes from. This will attempt to allo‐
1777 cate all nodes in the specified partition unless you request
1778 fewer resources than are available with CoreCnt, NodeCnt or
1779 TRES. Jobs will be allowed to use this reservation even if run‐
1780 ning in a different partition. There only needs to be overlap‐
1781 ping nodes from that different partition and the nodes used in
1782 the reservation.
1783 Flags=<flags>
1786 Flags associated with the reservation. You can add or remove
1787 individual flags from an existing reservation by adding a '+' or
1788 '-' sign before the '=' sign. For example: Flags-=DAILY (NOTE:
1789 this shortcut is not supported for all flags). Currently sup‐
1790 ported flags include:
1791 ANY_NODES This is a reservation for burst buffers and/or li‐
1793 censes only and not compute nodes. If this flag
1794 is set, a job using this reservation may use the
1795 associated burst buffers and/or licenses plus any
1796 compute nodes. If this flag is not set, a job us‐
1797 ing this reservation may use only the nodes and
1798 licenses associated with the reservation.
1799 DAILY Repeat the reservation at the same time every day.
1801 FLEX Permit jobs requesting the reservation to begin
1803 prior to the reservation's start time, end after
1804 the reservation's end time, and use any resources
1805 inside and/or outside of the reservation regard‐
1806 less of any constraints possibly set in the reser‐
1807 vation. A typical use case is to prevent jobs not
1808 explicitly requesting the reservation from using
1809 those reserved resources rather than forcing jobs
1810 requesting the reservation to use those resources
1811 in the time frame reserved. Another use case could
1812 be to always have a particular number of nodes
1813 with a specific feature reserved for a specific
1814 account so users in this account may use this
1815 nodes plus possibly other nodes without this fea‐
1816 ture.
1817 FIRST_CORES Use the lowest numbered cores on a node only. Flag
1819 removal with '-=' is not supported.
1820 IGNORE_JOBS Ignore currently running jobs when creating the
1822 reservation. This can be especially useful when
1823 reserving all nodes in the system for maintenance.
1824 LICENSE_ONLY See ANY_NODES.
1826 MAINT Maintenance mode, receives special accounting
1828 treatment. This reservation is permitted to use
1829 resources that are already in another reservation.
1830 MAGNETIC This flag allows jobs to be considered for this
1832 reservation even if they didn't request it.
1833 NO_HOLD_JOBS_AFTER
1835 By default, when a reservation ends the reserva‐
1836 tion request will be removed from any pending jobs
1837 submitted to the reservation and will be put into
1838 a held state. Use this flag to let jobs run out‐
1839 side of the reservation after the reservation is
1840 gone. Flag removal with '-=' is not supported.
1841 OVERLAP This reservation can be allocated resources that
1843 are already in another reservation. Flag removal
1844 with '-=' is not supported.
1845 PART_NODES This flag can be used to reserve all nodes within
1847 the specified partition. PartitionName and
1848 Nodes=ALL must be specified with this flag.
1849 PURGE_COMP[=<timespec>]
1851 Purge the reservation if it is ever idle for time‐
1852 spec (no jobs associated with it). If timespec
1853 isn't given then 5 minutes is the default. Valid
1854 timespec formats are minutes, minutes:seconds,
1855 hours:minutes:seconds, days-hours, days-hours:min‐
1856 utes, days-hours:minutes:seconds. Time resolution
1857 is one minute and second values are rounded up to
1858 the next minute. Output format is always
1859 [days-]hours:minutes:seconds.
1860 REPLACE Nodes which are DOWN, DRAINED, or allocated to
1862 jobs are automatically replenished using idle re‐
1863 sources. This option can be used to maintain a
1864 constant number of idle resources available for
1865 pending jobs (subject to availability of idle re‐
1866 sources). This should be used with the NodeCnt
1867 reservation option; do not identify specific nodes
1868 to be included in the reservation. Flag removal
1869 with '-=' is not supported.
1870 NOTE: Removing a node from the cluster while in a
1872 reservation with the REPLACE flag will not cause
1873 it to be replaced.
1874 REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically
1876 replenished using idle resources. This option can
1877 be used to maintain a constant sized pool of re‐
1878 sources available for pending jobs (subject to
1879 availability of idle resources). This should be
1880 used with the NodeCnt reservation option; do not
1881 identify specific nodes to be included in the
1882 reservation. Flag removal with '-=' is not sup‐
1883 ported.
1884 NOTE: Removing a node from the cluster while in a
1886 reservation with the REPLACE_DOWN flag will not
1887 cause it to be replaced.
1888 SPEC_NODES Reservation is for specific nodes (output only).
1890 STATIC_ALLOC Make it so after the nodes are selected for a
1892 reservation they don't change. Without this op‐
1893 tion when nodes are selected for a reservation and
1894 one goes down the reservation will select a new
1895 node to fill the spot.
1896 TIME_FLOAT The reservation start time is relative to the cur‐
1898 rent time and moves forward through time (e.g. a
1899 StartTime=now+10minutes will always be 10 minutes
1900 in the future). Repeating (e.g. DAILY) floating
1901 reservations are not supported. Flag cannot be
1902 added to or removed from an existing reservation.
1903 WEEKDAY Repeat the reservation at the same time on every
1905 weekday (Monday, Tuesday, Wednesday, Thursday and
1906 Friday).
1907 WEEKEND Repeat the reservation at the same time on every
1909 weekend day (Saturday and Sunday).
1910 WEEKLY Repeat the reservation at the same time every
1912 week.
1913 Features=<features>
1916 Set the reservation's required node features. Multiple values
1917 may be "&" separated if all features are required (AND opera‐
1918 tion) or separated by "|" if any of the specified features are
1919 required (OR operation). Parenthesis are also supported for
1920 features to be ANDed together with counts of nodes having the
1921 specified features. For example "Fea‐
1922 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
1923 reservation should include 4 nodes with ALL of the features
1924 "knl", "a2a", and "flat" plus 2 nodes with the feature
1925 "haswell".
1926 Value may be cleared with blank data value, "Features=".
1928 Groups=<group list>
1931 List of groups permitted to use the reserved nodes, for example
1932 "Group=bio,chem". A new reservation must specify Users or
1933 Groups and/or Accounts. If both Users/Groups and Accounts are
1934 specified, a job must match both in order to use the reserva‐
1935 tion. Unlike users groups do not allow denied access to reser‐
1936 vations. You can add or remove individual groups from an exist‐
1937 ing reservation by using the update command and adding a '+' or
1938 '-' sign before the '=' sign. NOTE: Groups and Users are mutu‐
1939 ally exclusive in reservations, if you want to switch between
1940 the 2 you must update the reservation with a group='' or user=''
1941 and fill in the opposite with the appropriate setting.
1942 Skip Used on a reoccurring reservation, skip to the next reservation
1945 iteration. NOTE: Only available for update.
1946 Users=<user list>
1949 List of users permitted to use the reserved nodes, for example
1950 "User=jones1,smith2". A new reservation must specify Users or
1951 Groups and/or Accounts. If both Users/Groups and Accounts are
1952 specified, a job must match both in order to use the reserva‐
1953 tion. Users can also be denied access to reservations by pre‐
1954 ceding all of the user names with '-'. Alternately precede the
1955 equal sign with '-'. For example, "User=-jones1,-smith2" or
1956 "User-=jones1,smith2" will permit any user except jones1 and
1957 smith2 to use the reservation. You can add or remove individual
1958 users from an existing reservation by using the update command
1959 and adding a '+' or '-' sign before the '=' sign. If users are
1960 denied access to a reservation (user name preceded by a '-'),
1961 then all other users are implicitly allowed to use the reserva‐
1962 tion and it is not possible to also explicitly specify allowed
1963 users. NOTE: Groups and Users are mutually exclusive in reser‐
1964 vations, if you want to switch between the 2 you must update the
1965 reservation with a group='' or user='' and fill in the opposite
1966 with the appropriate setting.
1967 TRES=<tres_spec>
1970 Comma-separated list of TRES required for the reservation. Cur‐
1971 rent supported TRES types with reservations are: CPU, Node, Li‐
1972 cense and BB. CPU and Node follow the same format as CoreCnt and
1973 NodeCnt parameters respectively. License names can be followed
1974 by an equal '=' and a count:
1975 License/<name1>=<count1>[,License/<name2>=<count2>,...]
1977 BurstBuffer can be specified in a similar way as BurstBuffer pa‐
1979 rameter. The only difference is that colon symbol ':' should be
1980 replaced by an equal '=' in order to follow the TRES format.
1981 Some examples of TRES valid specifications:
1983 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
1985 TRES=node=5k,license/iop1=2
1987 As specified in CoreCnt, if a nodelist is specified, cpu can be
1989 an array of core numbers by node: nodes=compute[1-3]
1990 TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
1991 Please note that CPU, Node, License and BB can override CoreCnt,
1993 NodeCnt, Licenses and BurstBuffer parameters respectively. Also
1994 CPU represents CoreCnt, in a reservation and will be adjusted if
1995 you have threads per core on your nodes.
1996 Note that a reservation that contains nodes or cores is associ‐
1998 ated with one partition, and can't span resources over multiple
1999 partitions. The only exception from this is when the reserva‐
2000 tion is created with explicitly requested nodes.
2001
2004 Executing scontrol sends a remote procedure call to slurmctld. If
2005 enough calls from scontrol or other Slurm client commands that send re‐
2006 mote procedure calls to the slurmctld daemon come in at once, it can
2007 result in a degradation of performance of the slurmctld daemon, possi‐
2008 bly resulting in a denial of service.
2009 Do not run scontrol or other Slurm client commands that send remote
2011 procedure calls to slurmctld from loops in shell scripts or other pro‐
2012 grams. Ensure that programs limit calls to scontrol to the minimum nec‐
2013 essary for the information you are trying to gather.
2014
2017 Some scontrol options may be set via environment variables. These envi‐
2018 ronment variables, along with their corresponding options, are listed
2019 below. (Note: Commandline options will always override these settings.)
2020 SCONTROL_ALL -a, --all
2022 SCONTROL_FEDERATION --federation
2024 SCONTROL_FUTURE -F, --future
2026 SCONTROL_LOCAL --local
2028 SCONTROL_SIBLING --sibling
2030 SLURM_BITSTR_LEN Specifies the string length to be used for holding
2032 a job array's task ID expression. The default
2033 value is 64 bytes. A value of 0 will print the
2034 full expression with any length required. Larger
2035 values may adversely impact the application perfor‐
2036 mance.
2037 SLURM_CLUSTERS Same as --clusters
2039 SLURM_CONF The location of the Slurm configuration file.
2041 SLURM_CONF_OUT When running 'write config', the location of the
2043 Slurm configuration file to be written.
2044 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
2046 value of standard, the default value, generates
2047 output in the form
2048 "year-month-dateThour:minute:second". A value of
2049 relative returns only "hour:minute:second" if the
2050 current day. For other dates in the current year
2051 it prints the "hour:minute" preceded by "Tomorr"
2052 (tomorrow), "Ystday" (yesterday), the name of the
2053 day for the coming week (e.g. "Mon", "Tue", etc.),
2054 otherwise the date (e.g. "25 Apr"). For other
2055 years it returns a date month and year without a
2056 time (e.g. "6 Jun 2012"). All of the time stamps
2057 use a 24 hour format.
2058 A valid strftime() format can also be specified.
2060 For example, a value of "%a %T" will report the day
2061 of the week and a time stamp (e.g. "Mon 12:34:56").
2062 SLURM_TOPO_LEN Specify the maximum size of the line when printing
2065 Topology. If not set, the default value is unlim‐
2066 ited.
2067
2070 When using SlurmDBD, users who have an AdminLevel defined (Operator or
2071 Admin) and users who are account coordinators are given the authority
2072 to view and modify jobs, reservations, nodes, etc., as defined in the
2073 following table - regardless of whether a PrivateData restriction has
2074 been defined in the slurm.conf file.
2075 scontrol show job(s): Admin, Operator, Coordinator
2077 scontrol update job: Admin, Operator, Coordinator
2078 scontrol requeue: Admin, Operator, Coordinator
2079 scontrol show step(s): Admin, Operator, Coordinator
2080 scontrol update step: Admin, Operator, Coordinator
2081 scontrol show node: Admin, Operator
2083 scontrol update node: Admin
2084 scontrol create partition: Admin
2086 scontrol show partition: Admin, Operator
2087 scontrol update partition: Admin
2088 scontrol delete partition: Admin
2089 scontrol create reservation: Admin, Operator
2091 scontrol show reservation: Admin, Operator
2092 scontrol update reservation: Admin, Operator
2093 scontrol delete reservation: Admin, Operator
2094 scontrol reconfig: Admin
2096 scontrol shutdown: Admin
2097 scontrol takeover: Admin
2098
2101 $ scontrol
2102 scontrol: show part debug
2103 PartitionName=debug
2104 AllocNodes=ALL AllowGroups=ALL Default=YES
2105 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2106 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2107 Nodes=snowflake[0-48]
2108 Priority=1 RootOnly=NO OverSubscribe=YES:4
2109 State=UP TotalCPUs=694 TotalNodes=49
2110 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2111 scontrol: show job 71701
2112 JobId=71701 Name=hostname
2113 UserId=da(1000) GroupId=da(1000)
2114 Priority=66264 Account=none QOS=normal WCKey=*123
2115 JobState=COMPLETED Reason=None Dependency=(null)
2116 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
2117 SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
2118 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2119 SuspendTime=None SecsPreSuspend=0
2120 Partition=debug AllocNode:Sid=snowflake:4702
2121 ReqNodeList=(null) ExcNodeList=(null)
2122 NodeList=snowflake0
2123 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2124 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2125 Features=(null) Reservation=(null)
2126 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2127 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2128 scontrol: show hostnames tux[1-3]
2129 tux1
2130 tux2
2131 tux3
2132 scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00 Users=dbremer NodeCnt=10
2133 Reservation created: dbremer_1
2134 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2135 scontrol: delete Reservation=dbremer_1
2136 scontrol: quit
2137
2140 Copyright (C) 2002-2007 The Regents of the University of California.
2141 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2142 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2143 Copyright (C) 2010-2018 SchedMD LLC.
2144 This file is part of Slurm, a resource management program. For de‐
2146 tails, see <https://slurm.schedmd.com/>.
2147 Slurm is free software; you can redistribute it and/or modify it under
2149 the terms of the GNU General Public License as published by the Free
2150 Software Foundation; either version 2 of the License, or (at your op‐
2151 tion) any later version.
2152 Slurm is distributed in the hope that it will be useful, but WITHOUT
2154 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2155 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2156 for more details.
2157
2159 /etc/slurm.conf
2160
2162 scancel(1), sinfo(1), squeue(1), slurm_create_partition [22m(3),
2163 slurm_delete_partition [22m(3), slurm_load_ctl_conf [22m(3), slurm_load_jobs
2164 (3), slurm_load_node (3), slurm_load_partitions (3), slurm_reconfigure
2165 (3), slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2166 slurm_suspend (3), slurm_takeover (3), slurm_update_job (3), slurm_up‐
2167 date_node (3), slurm_update_partition (3), slurm.conf(5), slurmctld(8)
2168March 2021 Slurm Commands scontrol(1)