1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an
17 Administrator. If an attempt to view or modify configuration informa‐
18 tion is made by an unauthorized user, an error message will be printed
19 and the requested action will not occur. If no command is entered on
20 the execute line, scontrol will operate in an interactive mode and
21 prompt for input. It will continue prompting for input and executing
22 commands until explicitly terminated. If a command is entered on the
23 execute line, scontrol will execute that command and terminate. All
24 commands and options are case-insensitive, although node names, parti‐
25 tion names, and reservation names are case-sensitive (node names "LX"
26 and "lx" are distinct). All commands and options can be abbreviated to
27 the extent that the specification is unique. A modified Slurm configu‐
28 ration can be written to a file using the scontrol write config com‐
29 mand. The resulting file will be named using the convention
30 "slurm.conf.<datetime>" and located in the same directory as the origi‐
31 nal "slurm.conf" file. The directory containing the original slurm.conf
32 must be writable for this to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -d, --details
43 Causes the show command to provide additional details where
44 available.
45 --federation
47 Report jobs from federation if a member of one.
48 -F, --future
50 Report nodes in FUTURE state.
51 -h, --help
53 Print a help message describing the usage of scontrol.
54 --hide Do not display information about hidden partitions, their jobs
56 and job steps. By default, neither partitions that are config‐
57 ured as hidden nor those partitions unavailable to user's group
58 will be displayed (i.e. this is the default behavior).
59 --local
61 Show only information local to this cluster. Ignore other clus‐
62 ters in the federated if a member of one. Overrides --federa‐
63 tion.
64 -M, --clusters=<string>
66 The cluster to issue commands to. Only one cluster name may be
67 specified. Note that the SlurmDBD must be up for this option to
68 work properly. This option implicitly sets the --local option.
69 -o, --oneliner
72 Print information one line per record.
73 -Q, --quiet
75 Print no warning or informational messages, only fatal error
76 messages.
77 --sibling
79 Show all sibling jobs on a federated cluster. Implies --federa‐
80 tion.
81 -u, --uid=<uid>
83 Attempt to update a job as user <uid> instead of the invoking
84 user id.
85 -v, --verbose
87 Print detailed event logging. Multiple -v's will further
88 increase the verbosity of logging. By default only errors will
89 be displayed.
90 -V , --version
93 Print version information and exit.
94 COMMANDS
96 abort Instruct the Slurm controller to terminate immediately and gen‐
99 erate a core file. See "man slurmctld" for information about
100 where the core file will be written.
101 cancel_reboot <NodeList>
104 Cancel pending reboots on nodes. The node will be undrain'ed and
105 the reason cleared if the node was drained by an ASAP reboot.
106 cluster CLUSTER_NAME
109 The cluster to issue commands to. Only one cluster name may be
110 specified.
111 create SPECIFICATION
114 Create a new partition or reservation. See the full list of
115 parameters below. Include the tag "res" to create a reservation
116 without specifying a reservation name.
117 completing
120 Display all jobs in a COMPLETING state along with associated
121 nodes in either a COMPLETING or DOWN state.
122 delete SPECIFICATION
125 Delete the entry with the specified SPECIFICATION. The two
126 SPECIFICATION choices are PartitionName=<name> and Reserva‐
127 tion=<name>. Reservations and partitions should have no associ‐
128 ated jobs at the time of their deletion (modify the jobs first).
129 If the specified partition is in use, the request is denied.
130 errnumstr ERRNO
133 Given a Slurm error number, return a descriptive string.
134 fsdampeningfactor FACTOR
137 Set the FairShareDampeningFactor in slurmctld.
138 help Display a description of scontrol options and commands.
141 hold job_list
144 Prevent a pending job from being started (sets its priority to
145 0). Use the release command to permit the job to be scheduled.
146 The job_list argument is a comma separated list of job IDs OR
147 "jobname=" with the job's name, which will attempt to hold all
148 jobs having that name. Note that when a job is held by a system
149 administrator using the hold command, only a system administra‐
150 tor may release the job for execution (also see the uhold com‐
151 mand). When the job is held by its owner, it may also be
152 released by the job's owner. Additionally, attempting to hold a
153 running job will have not suspend or cancel it. But, it will set
154 the job priority to 0 and update the job reason field, which
155 would hold the job if it was requeued at a later time.
156 notify job_id message
159 Send a message to standard error of the salloc or srun command
160 or batch job associated with the specified job_id.
161 pidinfo proc_id
164 Print the Slurm job id and scheduled termination time corre‐
165 sponding to the supplied process id, proc_id, on the current
166 node. This will work only with processes on node on which scon‐
167 trol is run, and only for those processes spawned by Slurm and
168 their descendants.
169 listpids [job_id[.step_id]] [NodeName]
172 Print a listing of the process IDs in a job step (if
173 JOBID.STEPID is provided), or all of the job steps in a job (if
174 job_id is provided), or all of the job steps in all of the jobs
175 on the local node (if job_id is not provided or job_id is "*").
176 This will work only with processes on the node on which scontrol
177 is run, and only for those processes spawned by Slurm and their
178 descendants. Note that some Slurm configurations (ProctrackType
179 value of pgid) are unable to identify all processes associated
180 with a job or job step.
181 Note that the NodeName option is only really useful when you
183 have multiple slurmd daemons running on the same host machine.
184 Multiple slurmd daemons on one host are, in general, only used
185 by Slurm developers.
186 ping Ping the primary and secondary slurmctld daemon and report if
189 they are responding.
190 reboot [ASAP] [nextstate=<RESUME|DOWN>] [reason=<reason>]
193 <ALL|NodeList>
194 Reboot the nodes in the system when they become idle using the
195 RebootProgram as configured in Slurm's slurm.conf file. Each
196 node will have the "REBOOT" flag added to its node state. After
197 a node reboots and the slurmd daemon starts up again, the
198 HealthCheckProgram will run once. Then, the slurmd daemon will
199 register itself with the slurmctld daemon and the "REBOOT" flag
200 will be cleared. The node's "DRAIN" state flag will be cleared
201 if the reboot was "ASAP", nextstate=resume or down. The "ASAP"
202 option adds the "DRAIN" flag to each node's state, preventing
203 additional jobs from running on the node so it can be rebooted
204 and returned to service "As Soon As Possible" (i.e. ASAP).
205 "ASAP" will also set the node reason to "Reboot ASAP" if the
206 "reason" option isn't specified. If the "nextstate" option is
207 specified as "DOWN", then the node will remain in a down state
208 after rebooting. If "nextstate" is specified as "RESUME", then
209 the nodes will resume as normal and the node's reason and
210 "DRAIN" state will be cleared. Resuming nodes will be consid‐
211 ered as available in backfill future scheduling and won't be
212 replaced by idle nodes in a reservation. The "reason" option
213 sets each node's reason to a user-defined message. A default
214 reason of "reboot requested" is set if no other reason is set on
215 the node. The reason will be appended with "reboot issued" when
216 the reboot is issued and "reboot complete" when the node regis‐
217 ters and has a "nextstate" of "DOWN". You must specify either a
218 list of nodes or that ALL nodes are to be rebooted. NOTE: By
219 default, this command does not prevent additional jobs from
220 being scheduled on any nodes before reboot. To do this, you can
221 either use the "ASAP" option or explicitly drain the nodes
222 beforehand. You can alternately create an advanced reservation
223 to prevent additional jobs from being initiated on nodes to be
224 rebooted. Pending reboots can be cancelled by using "scontrol
225 cancel_reboot <node>" or setting the node state to "CAN‐
226 CEL_REBOOT". A node will be marked "DOWN" if it doesn't reboot
227 within ResumeTimeout.
228 reconfigure
231 Instruct all Slurm daemons to re-read the configuration file.
232 This command does not restart the daemons. This mechanism would
233 be used to modify configuration parameters (Epilog, Prolog,
234 SlurmctldLogFile, SlurmdLogFile, etc.). The Slurm controller
235 (slurmctld) forwards the request to all other daemons (slurmd
236 daemon on each compute node). Running jobs continue execution.
237 Most configuration parameters can be changed by just running
238 this command, however, Slurm daemons should be shutdown and
239 restarted if any of these parameters are to be changed:
240 AuthType, ControlMach, PluginDir, StateSaveLocation, Slurmctld‐
241 Host, SlurmctldPort, or SlurmdPort. The slurmctld daemon and all
242 slurmd daemons must be restarted if nodes are added to or
243 removed from the cluster.
244 release job_list
247 Release a previously held job to begin execution. The job_list
248 argument is a comma separated list of job IDs OR "jobname=" with
249 the job's name, which will attempt to hold all jobs having that
250 name. Also see hold.
251 requeue [option] job_list
254 Requeue a running, suspended or finished Slurm batch job into
255 pending state. The job_list argument is a comma separated list
256 of job IDs. The command accepts the following option:
257 Incomplete
259 Operate only on jobs (or tasks of a job array) which have
260 not completed. Specifically only jobs in the following
261 states will be requeued: CONFIGURING, RUNNING, STOPPED or
262 SUSPENDED.
263 requeuehold [option] job_list
266 Requeue a running, suspended or finished Slurm batch job into
267 pending state, moreover the job is put in held state (priority
268 zero). The job_list argument is a comma separated list of job
269 IDs. A held job can be released using scontrol to reset its
270 priority (e.g. "scontrol release <job_id>"). The command
271 accepts the following options:
272 Incomplete
274 Operate only on jobs (or tasks of a job array) which have
275 not completed. Specifically only jobs in the following
276 states will be requeued: CONFIGURING, RUNNING, STOPPED or
277 SUSPENDED.
278 State=SpecialExit
280 The "SpecialExit" keyword specifies that the job has to
281 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
282 trol show job" command will display the JobState as SPE‐
283 CIAL_EXIT, while the "squeue" command as SE.
284 resume job_list
287 Resume a previously suspended job. The job_list argument is a
288 comma separated list of job IDs. Also see suspend.
289 NOTE: A suspended job releases its CPUs for allocation to other
291 jobs. Resuming a previously suspended job may result in multi‐
292 ple jobs being allocated the same CPUs, which could trigger gang
293 scheduling with some configurations or severe degradation in
294 performance with other configurations. Use of the scancel com‐
295 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
296 out releasing its CPUs for allocation to other jobs and would be
297 a preferable mechanism in many cases. If performing system
298 maintenance you may want to use suspend/resume in the following
299 way. Before suspending set all nodes to draining or set all par‐
300 titions to down so that no new jobs can be scheduled. Then sus‐
301 pend jobs. Once maintenance is done resume jobs then resume
302 nodes and/or set all partitions back to up. Use with caution.
303 schedloglevel LEVEL
306 Enable or disable scheduler logging. LEVEL may be "0", "1",
307 "disable" or "enable". "0" has the same effect as "disable". "1"
308 has the same effect as "enable". This value is temporary and
309 will be overwritten when the slurmctld daemon reads the
310 slurm.conf configuration file (e.g. when the daemon is restarted
311 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
312 parameter is present.
313 setdebug LEVEL
316 Change the debug level of the slurmctld daemon. LEVEL may be an
317 integer value between zero and nine (using the same values as
318 SlurmctldDebug in the slurm.conf file) or the name of the most
319 detailed message type to be printed: "quiet", "fatal", "error",
320 "info", "verbose", "debug", "debug2", "debug3", "debug4", or
321 "debug5". This value is temporary and will be overwritten when‐
322 ever the slurmctld daemon reads the slurm.conf configuration
323 file (e.g. when the daemon is restarted or scontrol reconfigure
324 is executed).
325 setdebugflags [+|-]FLAG
328 Add or remove DebugFlags of the slurmctld daemon. See "man
329 slurm.conf" for a list of supported DebugFlags. NOTE: Changing
330 the value of some DebugFlags will have no effect without
331 restarting the slurmctld daemon, which would set DebugFlags
332 based upon the contents of the slurm.conf configuration file.
333 show ENTITY ID
336 or
337 show ENTITY=ID
339 Display the state of the specified entity with the specified
340 identification. ENTITY may be aliases, assoc_mgr, bbstat,
341 burstbuffer, config, daemons, dwstat, federation, frontend, job,
342 node, partition, reservation, slurmd, step, topology, hostlist,
343 hostlistsorted or hostnames ID can be used to identify a spe‐
344 cific element of the identified entity: job ID, node name, par‐
345 tition name, reservation name, or job step ID for job, node,
346 partition, or step respectively. For an ENTITY of bbstat or
347 dwstat (they are equivalent) optional arguments are the options
348 of the local status command. The status commands will be exe‐
349 cuted by the slurmctld daemon and its response returned to the
350 user. For an ENTITY of topology, the ID may be a node or switch
351 name. If one node name is specified, all switches connected to
352 that node (and their parent switches) will be shown. If more
353 than one node name is specified, only switches that connect to
354 all named nodes will be shown. aliases will return all NodeName
355 values associated to a given NodeHostname (useful to get the
356 list of virtual nodes associated with a real node in a configu‐
357 ration where multiple slurmd daemons execute on a single compute
358 node). assoc_mgr displays the current contents of the slurm‐
359 ctld's internal cache for users, associations and/or qos. The ID
360 may be users=<user1>,[...,<userN>],
361 accounts=<acct1>,[...,<acctN>], qos=<qos1>,[...,<qosN>] and/or
362 flags=<users,assoc,qos>, used to filter the desired section to
363 be displayed. If no flags are specified, all sections are dis‐
364 played. burstbuffer displays the current status of the Burst‐
365 Buffer plugin. config displays parameter names from the config‐
366 uration files in mixed case (e.g. SlurmdPort=7003) while derived
367 parameters names are in upper case only (e.g. SLURM_VERSION).
368 hostnames takes an optional hostlist expression as input and
369 writes a list of individual host names to standard output (one
370 per line). If no hostlist expression is supplied, the contents
371 of the SLURM_JOB_NODELIST environment variable is used. For
372 example "tux[1-3]" is mapped to "tux1","tux2" and "tux3" (one
373 hostname per line). hostlist takes a list of host names and
374 prints the hostlist expression for them (the inverse of host‐
375 names). hostlist can also take the absolute pathname of a file
376 (beginning with the character '/') containing a list of host‐
377 names. Multiple node names may be specified using simple node
378 range expressions (e.g. "lx[10-20]"). All other ID values must
379 identify a single element. The job step ID is of the form
380 "job_id.step_id", (e.g. "1234.1"). slurmd reports the current
381 status of the slurmd daemon executing on the same node from
382 which the scontrol command is executed (the local host). It can
383 be useful to diagnose problems. By default hostlist does not
384 sort the node list or make it unique (e.g. tux2,tux1,tux2 =
385 tux[2,1-2]). If you wanted a sorted list use hostlistsorted
386 (e.g. tux2,tux1,tux2 = tux[1-2,2]). By default, all elements of
387 the entity type specified are printed. For an ENTITY of job, if
388 the job does not specify socket-per-node, cores-per-socket or
389 threads-per-core then it will display '*' in ReqS:C:T=*:*:*
390 field. For an ENTITY of federation, the federation name that the
391 controller is part of and the sibling clusters part of the fed‐
392 eration will be listed.
393 shutdown OPTION
396 Instruct Slurm daemons to save current state and terminate. By
397 default, the Slurm controller (slurmctld) forwards the request
398 all other daemons (slurmd daemon on each compute node). An
399 OPTION of slurmctld or controller results in only the slurmctld
400 daemon being shutdown and the slurmd daemons remaining active.
401 suspend job_list
404 Suspend a running job. The job_list argument is a comma sepa‐
405 rated list of job IDs. Use the resume command to resume its
406 execution. User processes must stop on receipt of SIGSTOP sig‐
407 nal and resume upon receipt of SIGCONT for this operation to be
408 effective. Not all architectures and configurations support job
409 suspension. If a suspended job is requeued, it will be placed
410 in a held state. The time a job is suspended will not count
411 against a job's time limit. Only an operator, administrator,
412 SlurmUser, or root can suspend jobs.
413 takeover [INDEX]
416 Instruct one of Slurm's backup controllers (slurmctld) to take
417 over system control. By default the first backup controller
418 (INDEX=1) requests control from the primary and waits for its
419 termination. After that, it switches from backup mode to con‐
420 troller mode. If primary controller can not be contacted, it
421 directly switches to controller mode. This can be used to speed
422 up the Slurm controller fail-over mechanism when the primary
423 node is down. This can be used to minimize disruption if the
424 computer executing the primary Slurm controller is scheduled
425 down. (Note: Slurm's primary controller will take the control
426 back at startup.)
427 top job_list
430 Move the specified job IDs to the top of the queue of jobs
431 belonging to the identical user ID, partition name, account, and
432 QOS. The job_list argument is a comma separated ordered list of
433 job IDs. Any job not matching all of those fields will not be
434 effected. Only jobs submitted to a single partition will be
435 effected. This operation changes the order of jobs by adjusting
436 job nice values. The net effect on that user's throughput will
437 be negligible to slightly negative. This operation is disabled
438 by default for non-privileged (non-operator, admin, SlurmUser,
439 or root) users. This operation may be enabled for non-privileged
440 users by the system administrator by including the option
441 "enable_user_top" in the SchedulerParameters configuration
442 parameter.
443 token [lifespan=<lifespan>] [username=<username>]
446 Return an auth token which can be used to support JWT authenti‐
447 cation if AuthAltTypes=auth/jwt has been enabled on the system.
448 Supports two optional arguments. lifespan= may be used to spec‐
449 ify the token's lifespan. username (only available to Slur‐
450 mUser/root) may be used to request a token for a different user‐
451 name.
452 uhold job_list
455 Prevent a pending job from being started (sets its priority to
456 0). The job_list argument is a space separated list of job IDs
457 or job names. Use the release command to permit the job to be
458 scheduled. This command is designed for a system administrator
459 to hold a job so that the job owner may release it rather than
460 requiring the intervention of a system administrator (also see
461 the hold command).
462 update SPECIFICATION
465 Update job, step, node, partition, or reservation configuration
466 per the supplied specification. SPECIFICATION is in the same
467 format as the Slurm configuration file and the output of the
468 show command described above. It may be desirable to execute the
469 show command (described above) on the specific entity you want
470 to update, then use cut-and-paste tools to enter updated config‐
471 uration values to the update. Note that while most configuration
472 values can be changed using this command, not all can be changed
473 using this mechanism. In particular, the hardware configuration
474 of a node or the physical addition or removal of nodes from the
475 cluster may only be accomplished through editing the Slurm con‐
476 figuration file and executing the reconfigure command (described
477 above).
478 version
481 Display the version number of scontrol being executed.
482 wait_job job_id
485 Wait until a job and all of its nodes are ready for use or the
486 job has entered some termination state. This option is particu‐
487 larly useful in the Slurm Prolog or in the batch script itself
488 if nodes are powered down and restarted automatically as needed.
489 NOTE: Don't use scontrol wait_job in PrologCtld or Prolog with
491 PrologFlags=Alloc as this will result in deadlock.
492 write batch_script job_id optional_filename
495 Write the batch script for a given job_id to a file or to std‐
496 out. The file will default to slurm-<job_id>.sh if the optional
497 filename argument is not given. The script will be written to
498 stdout if - is given instead of a filename. The batch script
499 can only be retrieved by an admin or operator, or by the owner
500 of the job.
501 write config optional_filename
504 Write the current configuration to a file with the naming con‐
505 vention of "slurm.conf.<datetime>" in the same directory as the
506 original slurm.conf file. If a filename is given that file
507 location with a .<datetime> suffix is created.
508 INTERACTIVE COMMANDS
511 NOTE: All commands listed below can be used in the interactive
512 mode, but NOT on the initial command line.
513 all Show all partitions, their jobs and jobs steps. This causes
516 information to be displayed about partitions that are configured
517 as hidden and partitions that are unavailable to user's group.
518 details
521 Causes the show command to provide additional details where
522 available. Job information will include CPUs and NUMA memory
523 allocated on each node. Note that on computers with hyper‐
524 threading enabled and Slurm configured to allocate cores, each
525 listed CPU represents one physical core. Each hyperthread on
526 that core can be allocated a separate task, so a job's CPU count
527 and task count may differ. See the --cpu-bind and --mem-bind
528 option descriptions in srun man pages for more information. The
529 details option is currently only supported for the show job com‐
530 mand.
531 exit Terminate scontrol interactive session.
534 hide Do not display partition, job or jobs step information for par‐
537 titions that are configured as hidden or partitions that are
538 unavailable to the user's group. This is the default behavior.
539 oneliner
542 Print information one line per record.
543 quiet Print no warning or informational messages, only fatal error
546 messages.
547 quit Terminate the execution of scontrol.
550 verbose
553 Print detailed event logging. This includes time-stamps on data
554 structures, record counts, etc.
555 !! Repeat the last command executed.
558 SPECIFICATIONS FOR UPDATE COMMAND, JOBS
561 Note that update requests done by either root, SlurmUser or Administra‐
563 tors are not subject to certain restrictions. For instance, if an
564 Administrator changes the QOS on a pending job, certain limits such as
565 the TimeLimit will not be changed automatically as changes made by the
566 Administrators are allowed to violate these restrictions.
567 Account=<account>
570 Account name to be changed for this job's resource use. Value
571 may be cleared with blank data value, "Account=".
572 AdminComment=<spec>
574 Arbitrary descriptive string. Can only be set by a Slurm admin‐
575 istrator.
576 ArrayTaskThrottle=<count>
578 Specify the maximum number of tasks in a job array that can exe‐
579 cute at the same time. Set the count to zero in order to elimi‐
580 nate any limit. The task throttle count for a job array is
581 reported as part of its ArrayTaskId field, preceded with a per‐
582 cent sign. For example "ArrayTaskId=1-10%2" indicates the maxi‐
583 mum number of running tasks is limited to 2.
584 BurstBuffer=<spec>
586 Burst buffer specification to be changed for this job's resource
587 use. Value may be cleared with blank data value, "Burst‐
588 Buffer=". Format is burst buffer plugin specific.
589 Clusters=<spec>
591 Specifies the clusters that the federated job can run on.
592 ClusterFeatures=<spec>
594 Specifies features that a federated cluster must have to have a
595 sibling job submitted to it. Slurm will attempt to submit a sib‐
596 ling job to a cluster if it has at least one of the specified
597 features.
598 Comment=<spec>
600 Arbitrary descriptive string.
601 Contiguous=<yes|no>
603 Set the job's requirement for contiguous (consecutive) nodes to
604 be allocated. Possible values are "YES" and "NO". Only the
605 Slurm administrator or root can change this parameter.
606 CoreSpec=<count>
608 Number of cores to reserve per node for system use. The job
609 will be charged for these cores, but be unable to use them.
610 Will be reported as "*" if not constrained.
611 CPUsPerTask=<count>
613 Change the CPUsPerTask job's value.
614 Deadline=<time_spec>
616 It accepts times of the form HH:MM:SS to specify a deadline to a
617 job at a specific time of day (seconds are optional). You may
618 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
619 you can have a time-of-day suffixed with AM or PM for a deadline
620 in the morning or the evening. You can specify a deadline for
621 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
622 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
623 give times like now + count time-units, where the time-units can
624 be seconds (default), minutes, hours, days, or weeks and you can
625 tell Slurm to put a deadline for tomorrow with the keyword
626 tomorrow. The specified deadline must be later than the current
627 time. Only pending jobs can have the deadline updated. Only
628 the Slurm administrator or root can change this parameter.
629 DelayBoot=<time_spec>
631 Change the time to decide whether to reboot nodes in order to
632 satisfy job's feature specification if the job has been eligible
633 to run for less than this time period. See salloc/sbatch man
634 pages option --delay-boot.
635 Dependency=<dependency_list>
637 Defer job's initiation until specified job dependency specifica‐
638 tion is satisfied. Cancel dependency with an empty depen‐
639 dency_list (e.g. "Dependency="). <dependency_list> is of the
640 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
641 can share the same dependency and these jobs may even belong to
642 different users.
643 after:job_id[:jobid...]
645 This job can begin execution after the specified jobs
646 have begun execution or been canceled.
647 afterany:job_id[:jobid...]
649 This job can begin execution after the specified jobs
650 have terminated.
651 afternotok:job_id[:jobid...]
653 This job can begin execution after the specified jobs
654 have terminated in some failed state (non-zero exit code,
655 node failure, timed out, etc).
656 afterok:job_id[:jobid...]
658 This job can begin execution after the specified jobs
659 have successfully executed (ran to completion with an
660 exit code of zero).
661 singleton
663 This job can begin execution after any previously
664 launched jobs sharing the same job name and user have
665 terminated. In other words, only one job by that name
666 and owned by that user can be running or suspended at any
667 point in time.
668 EligibleTime=<time_spec>
670 See StartTime.
671 EndTime
673 The time the job is expected to terminate based on the job's
674 time limit. When the job ends sooner, this field will be
675 updated with the actual end time.
676 ExcNodeList=<nodes>
678 Set the job's list of excluded node. Multiple node names may be
679 specified using simple node range expressions (e.g.
680 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
681 NodeList=".
682 Features=<features>
684 Set the job's required node features. The list of features may
685 include multiple feature names separated by ampersand (AND)
686 and/or vertical bar (OR) operators. For example: Fea‐
687 tures="opteron&video" or Features="fast|faster". In the first
688 example, only nodes having both the feature "opteron" AND the
689 feature "video" will be used. There is no mechanism to specify
690 that you want one node with feature "opteron" and another node
691 with feature "video" in case no node has both features. If only
692 one of a set of possible options should be used for all allo‐
693 cated nodes, then use the OR operator and enclose the options
694 within square brackets. For example: "Fea‐
695 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
696 all nodes must be allocated on a single rack of the cluster, but
697 any of those four racks can be used. A request can also specify
698 the number of nodes needed with some feature by appending an
699 asterisk and count after the feature name. For example "Fea‐
700 tures=graphics*4" indicates that at least four allocated nodes
701 must have the feature "graphics." Parenthesis are also sup‐
702 ported for features to be ANDed together. For example "Fea‐
703 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
704 cation should include 4 nodes with ALL of the features "knl",
705 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
706 straints with node counts may only be combined with AND opera‐
707 tors. Value may be cleared with blank data value, for example
708 "Features=".
709 Gres=<list>
712 Specifies a comma delimited list of generic consumable
713 resources. The format of each entry on the list is
714 "name[:count[*cpu]]". The name is that of the consumable
715 resource. The count is the number of those resources with a
716 default value of 1. The specified resources will be allocated
717 to the job on each node allocated unless "*cpu" is appended, in
718 which case the resources will be allocated on a per cpu basis.
719 The available generic consumable resources is configurable by
720 the system administrator. A list of available generic consum‐
721 able resources will be printed and the command will exit if the
722 option argument is "help". Examples of use include
723 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
724 JobId=<job_list>
727 Identify the job(s) to be updated. The job_list may be a comma
728 separated list of job IDs. Either JobId or JobName is required.
729 Licenses=<name>
731 Specification of licenses (or other resources available on all
732 nodes of the cluster) as described in salloc/sbatch/srun man
733 pages.
734 MailType=<types>
736 Set the mail event types. Valid type values are NONE, BEGIN,
737 END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL,
738 REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and
739 teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 per‐
740 cent of time limit), TIME_LIMIT_80 (reached 80 percent of time
741 limit), TIME_LIMIT_50 (reached 50 percent of time limit) and
742 ARRAY_TASKS (send emails for each array task). Multiple type
743 values may be specified in a comma separated list. Unless the
744 ARRAY_TASKS option is specified, mail notifications on job
745 BEGIN, END and FAIL apply to a job array as a whole rather than
746 generating individual email messages for each task in the job
747 array.
748 MailUser=<name>
750 Set the user to receive email notification of state changes. A
751 blank string will set the mail user to the default which is the
752 submitting user.
753 MinCPUsNode=<count>
755 Set the job's minimum number of CPUs per node to the specified
756 value.
757 MinMemoryCPU=<megabytes>
759 Set the job's minimum real memory required per allocated CPU to
760 the specified value. Either MinMemoryCPU or MinMemoryNode may be
761 set, but not both.
762 MinMemoryNode=<megabytes>
764 Set the job's minimum real memory required per node to the spec‐
765 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
766 but not both.
767 MinTmpDiskNode=<megabytes>
769 Set the job's minimum temporary disk space required per node to
770 the specified value. Only the Slurm administrator or root can
771 change this parameter.
772 TimeMin=<timespec>
774 Change TimeMin value which specifies the minimum time limit min‐
775 utes of the job.
776 JobName=<name>
778 Identify the name of jobs to be modified or set the job's name
779 to the specified value. When used to identify jobs to be modi‐
780 fied, all jobs belonging to all users are modified unless the
781 UserID option is used to identify a specific user. Either JobId
782 or JobName is required.
783 Name[=<name>]
785 See JobName.
786 Nice[=<adjustment>]
788 Update the job with an adjusted scheduling priority within
789 Slurm. With no adjustment value the scheduling priority is
790 decreased by 100. A negative nice value increases the priority,
791 otherwise decreases it. The adjustment range is +/- 2147483645.
792 Only privileged users can specify a negative adjustment.
793 NodeList=<nodes>
795 Change the nodes allocated to a running job to shrink its size.
796 The specified list of nodes must be a subset of the nodes cur‐
797 rently allocated to the job. Multiple node names may be speci‐
798 fied using simple node range expressions (e.g. "lx[10-20]").
799 After a job's allocation is reduced, subsequent srun commands
800 must explicitly specify node and task counts which are valid for
801 the new allocation.
802 NumCPUs=<min_count>[-<max_count>]
804 Set the job's minimum and optionally maximum count of CPUs to be
805 allocated.
806 NumNodes=<min_count>[-<max_count>]
808 Set the job's minimum and optionally maximum count of nodes to
809 be allocated. If the job is already running, use this to spec‐
810 ify a node count less than currently allocated and resources
811 previously allocated to the job will be relinquished. After a
812 job's allocation is reduced, subsequent srun commands must
813 explicitly specify node and task counts which are valid for the
814 new allocation. Also see the NodeList parameter above. This is
815 the same than ReqNodes.
816 NumTasks=<count>
818 Set the job's count of required tasks to the specified value.
819 This is the same than ReqProcs.
820 OverSubscribe=<yes|no>
822 Set the job's ability to share compute resources (i.e. individ‐
823 ual CPUs) with other jobs. Possible values are "YES" and "NO".
824 This option can only be changed for pending jobs.
825 Partition=<name>
827 Set the job's partition to the specified value.
828 Priority=<number>
830 Set the job's priority to the specified value. Note that a job
831 priority of zero prevents the job from ever being scheduled. By
832 setting a job's priority to zero it is held. Set the priority
833 to a non-zero value to permit it to run. Explicitly setting a
834 job's priority clears any previously set nice value and removes
835 the priority/multifactor plugin's ability to manage a job's pri‐
836 ority. In order to restore the priority/multifactor plugin's
837 ability to manage a job's priority, hold and then release the
838 job. Only the Slurm administrator or root can increase job's
839 priority.
840 QOS=<name>
842 Set the job's QOS (Quality Of Service) to the specified value.
843 Value may be cleared with blank data value, "QOS=".
844 Reboot=<yes|no>
846 Set the job's flag that specifies whether to force the allocated
847 nodes to reboot before starting the job. This is only supported
848 with some system configurations and therefore it could be
849 silently ignored.
850 ReqCores=<count>
852 Change the job's requested Cores count.
853 ReqNodeList=<nodes>
855 Set the job's list of required node. Multiple node names may be
856 specified using simple node range expressions (e.g.
857 "lx[10-20]"). Value may be cleared with blank data value,
858 "ReqNodeList=".
859 ReqNodes=<min_count>[-<max_count>]
861 See NumNodes.
862 ReqProcs=<count>
864 See NumTasks.
865 ReqSockets=<count>
867 Change the job's requested socket count.
868 ReqThreads=<count>
870 Change the job's requested threads count.
871 Requeue=<0|1>
873 Stipulates whether a job should be requeued after a node fail‐
874 ure: 0 for no, 1 for yes.
875 ReservationName=<name>
877 Set the job's reservation to the specified value. Value may be
878 cleared with blank data value, "ReservationName=".
879 ResetAccrueTime
881 Reset the job's accrue time value to 0 meaning it will lose any
882 time previously accrued for priority. Helpful if you have a
883 large queue of jobs already in the queue and want to start lim‐
884 iting how many jobs can accrue time without waiting for the
885 queue to flush out.
886 SiteFactor=<account>
888 Specify the job's admin priority factor in the range of
889 +/-2147483645. Only privileged users can modify the value.
890 StdOut=<filepath>
892 Set the batch job's stdout file path.
893 Shared=<yes|no>
895 See OverSubscribe option above.
896 StartTime=<time_spec>
898 Set the job's earliest initiation time. It accepts times of the
899 form HH:MM:SS to run a job at a specific time of day (seconds
900 are optional). (If that time is already past, the next day is
901 assumed.) You may also specify midnight, noon, fika (3 PM) or
902 teatime (4 PM) and you can have a time-of-day suffixed with AM
903 or PM for running in the morning or the evening. You can also
904 say what day the job will be run, by specifying a date of the
905 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
906 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
907 count time-units, where the time-units can be seconds (default),
908 minutes, hours, days, or weeks and you can tell Slurm to run the
909 job today with the keyword today and to run the job tomorrow
910 with the keyword tomorrow.
911 Notes on date/time specifications:
913 - although the 'seconds' field of the HH:MM:SS time specifica‐
914 tion is allowed by the code, note that the poll time of the
915 Slurm scheduler is not precise enough to guarantee dispatch of
916 the job on the exact second. The job will be eligible to start
917 on the next poll following the specified time. The exact poll
918 interval depends on the Slurm scheduler (e.g., 60 seconds with
919 the default sched/builtin).
920 - if no time (HH:MM:SS) is specified, the default is
921 (00:00:00).
922 - if a date is specified without a year (e.g., MM/DD) then the
923 current year is assumed, unless the combination of MM/DD and
924 HH:MM:SS has already passed for that year, in which case the
925 next year is used.
926 Switches=<count>[@<max-time-to-wait>]
928 When a tree topology is used, this defines the maximum count of
929 switches desired for the job allocation. If Slurm finds an allo‐
930 cation containing more switches than the count specified, the
931 job remain pending until it either finds an allocation with
932 desired switch count or the time limit expires. By default there
933 is no switch count limit and no time limit delay. Set the count
934 to zero in order to clean any previously set count (disabling
935 the limit). The job's maximum time delay may be limited by the
936 system administrator using the SchedulerParameters configuration
937 parameter with the max_switch_wait parameter option. Also see
938 wait-for-switch.
939 wait-for-switch=<seconds>
942 Change max time to wait for a switch <seconds> secs.
943 TasksPerNode=<count>
946 Change the job's requested TasksPerNode.
947 ThreadSpec=<count>
950 Number of threads to reserve per node for system use. The job
951 will be charged for these threads, but be unable to use them.
952 Will be reported as "*" if not constrained.
953 TimeLimit=<time>
956 The job's time limit. Output format is [days-]hours:min‐
957 utes:seconds or "UNLIMITED". Input format (for update command)
958 set is minutes, minutes:seconds, hours:minutes:seconds,
959 days-hours, days-hours:minutes or days-hours:minutes:seconds.
960 Time resolution is one minute and second values are rounded up
961 to the next minute. If changing the time limit of a job, either
962 specify a new time limit value or precede the time and equal
963 sign with a "+" or "-" to increment or decrement the current
964 time limit (e.g. "TimeLimit+=30"). In order to increment or
965 decrement the current time limit, the JobId specification must
966 precede the TimeLimit specification. Note that incrementing or
967 decrementing the time limit for a job array is only allowed
968 before the job array has been split into more than one job
969 record. Only the Slurm administrator or root can increase job's
970 TimeLimit.
971 UserID=<UID or name>
974 Used with the JobName option to identify jobs to be modified.
975 Either a user name or numeric ID (UID), may be specified.
976 WCKey=<key>
979 Set the job's workload characterization key to the specified
980 value.
981 WorkDir=<directory_name>
984 Set the job's working directory to the specified value. Note
985 that this may only be set for jobs in the PENDING state, and
986 that jobs may fail to launch if they rely on relative paths to
987 the originally submitted WorkDir.
988 SPECIFICATIONS FOR SHOW COMMAND, JOBS
991 The "show" command, when used with the "job" or "job <jobid>" entity
993 displays detailed information about a job or jobs. Much of this infor‐
994 mation may be modified using the "update job" command as described
995 above. However, the following fields displayed by the show job command
996 are read-only and cannot be modified:
997 AllocNode:Sid
1000 Local node and system id making the resource allocation.
1001 BatchFlag
1003 Jobs submitted using the sbatch command have BatchFlag set to 1.
1004 Jobs submitted using other commands have BatchFlag set to 0.
1005 ExitCode=<exit>:<sig>
1007 Exit status reported for the job by the wait() function. The
1008 first number is the exit code, typically as set by the exit()
1009 function. The second number of the signal that caused the
1010 process to terminate if it was terminated by a signal.
1011 GroupId
1013 The group under which the job was submitted.
1014 JobState
1016 The current state of the job.
1017 NodeListIndices
1019 The NodeIndices expose the internal indices into the node table
1020 associated with the node(s) allocated to the job.
1021 NtasksPerN:B:S:C=
1023 <tasks_per_node>:<tasks_per_base‐
1024 board>:<tasks_per_socket>:<tasks_per_core> Specifies the number
1025 of tasks to be started per hardware component (node, baseboard,
1026 socket and core). Unconstrained values may be shown as "0" or
1027 "*".
1028 PreemptEligibleTime
1030 Time the job becomes eligible for preemption. Modified by Pre‐
1031 emptExemptTime, either from the global option in slurm.conf or
1032 the job QOS. This is hidden if the job has not started or if
1033 PreemptMode=OFF.
1034 PreemptTime
1036 Time at which job was signaled that it was selected for preemp‐
1037 tion. (Meaningful only for PreemptMode=CANCEL and the partition
1038 or QOS with which the job is associated has a GraceTime value
1039 designated.) This is hidden if the job has not started or if
1040 PreemptMode=OFF.
1041 PreSusTime
1043 Time the job ran prior to last suspend.
1044 Reason The reason job is not running: e.g., waiting "Resources".
1046 ReqB:S:C:T=
1048 <baseboard_count>:<socket_per_base‐
1049 board_count>:<core_per_socket_count>:<thread_per_core_count>
1050 Specifies the count of various hardware components requested by
1051 the job. Unconstrained values may be shown as "0" or "*".
1052 SecsPreSuspend=<seconds>
1054 If the job is suspended, this is the run time accumulated by the
1055 job (in seconds) prior to being suspended.
1056 Socks/Node=<count>
1058 Count of desired sockets per node
1059 SubmitTime
1061 The time and date stamp (in localtime) the job was submitted.
1062 The format of the output is identical to that of the EndTime
1063 field.
1064 NOTE: If a job is requeued, the submit time is reset. To obtain
1066 the original submit time it is necessary to use the "sacct -j
1067 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1068 cate option to display all duplicate entries for a job.
1069 SuspendTime
1071 Time the job was last suspended or resumed.
1072 NOTE on information displayed for various job states:
1074 When you submit a request for the "show job" function the scon‐
1075 trol process makes an RPC request call to slurmctld with a
1076 REQUEST_JOB_INFO message type. If the state of the job is PEND‐
1077 ING, then it returns some detail information such as: min_nodes,
1078 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1079 ING the code assumes that it is in a further state such as RUN‐
1080 NING, COMPLETE, etc. In these cases the code explicitly returns
1081 zero for these values. These values are meaningless once the job
1082 resources have been allocated and the job has started.
1083 SPECIFICATIONS FOR UPDATE COMMAND, STEPS
1086 StepId=<job_id>[.<step_id>]
1088 Identify the step to be updated. If the job_id is given, but no
1089 step_id is specified then all steps of the identified job will
1090 be modified. This specification is required.
1091 CompFile=<completion file>
1093 Update a step with information about a step's completion. Can
1094 be useful if step statistics aren't directly available through a
1095 jobacct_gather plugin. The file is a space-delimited file with
1096 format for Version 1 is as follows
1097 1 34461 0 2 0 3 1361906011 1361906015 1 1 3368 13357 /bin/sleep
1099 A B C D E F G H I J K L M
1100 Field Descriptions:
1102 A file version
1104 B ALPS apid
1105 C inblocks
1106 D outblocks
1107 E exit status
1108 F number of allocated CPUs
1109 G start time
1110 H end time
1111 I utime
1112 J stime
1113 K maxrss
1114 L uid
1115 M command name
1116 TimeLimit=<time>
1118 The job's time limit. Output format is [days-]hours:min‐
1119 utes:seconds or "UNLIMITED". Input format (for update command)
1120 set is minutes, minutes:seconds, hours:minutes:seconds,
1121 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1122 Time resolution is one minute and second values are rounded up
1123 to the next minute. If changing the time limit of a step,
1124 either specify a new time limit value or precede the time with a
1125 "+" or "-" to increment or decrement the current time limit
1126 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1127 current time limit, the StepId specification must precede the
1128 TimeLimit specification.
1129 SPECIFICATIONS FOR UPDATE COMMAND, NODES
1132 NodeName=<name>
1134 Identify the node(s) to be updated. Multiple node names may be
1135 specified using simple node range expressions (e.g.
1136 "lx[10-20]"). This specification is required.
1137 ActiveFeatures=<features>
1140 Identify the feature(s) currently active on the specified node.
1141 Any previously active feature specification will be overwritten
1142 with the new value. Also see AvailableFeatures. Typically
1143 ActiveFeatures will be identical to AvailableFeatures; however
1144 ActiveFeatures may be configured as a subset of the Available‐
1145 Features. For example, a node may be booted in multiple configu‐
1146 rations. In that case, all possible configurations may be iden‐
1147 tified as AvailableFeatures, while ActiveFeatures would identify
1148 the current node configuration.
1149 AvailableFeatures=<features>
1152 Identify the feature(s) available on the specified node. Any
1153 previously defined available feature specification will be over‐
1154 written with the new value. AvailableFeatures assigned via
1155 scontrol will only persist across the restart of the slurmctld
1156 daemon with the -R option and state files preserved or slurm‐
1157 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1158 meant to be persistent across normal restarts of slurmctld or
1159 the execution of scontrol reconfig.
1160 Note: Available features being removed via scontrol must not be
1162 active (i.e. remove them from ActiveFeatures first).
1163 Comment=<comment>
1166 Arbitrary descriptive string. Use quotes to enclose a comment
1167 having more than one word
1168 CpuBind=<node>
1171 Specify the task binding mode to be used by default for this
1172 node. Supported options include: "none", "board", "socket",
1173 "ldom" (NUMA), "core", "thread" and "off" (remove previous bind‐
1174 ing mode).
1175 Gres=<gres>
1178 Identify generic resources to be associated with the specified
1179 node. Any previously defined generic resources will be over‐
1180 written with the new value. Specifications for multiple generic
1181 resources should be comma separated. Each resource specifica‐
1182 tion consists of a name followed by an optional colon with a
1183 numeric value (default value is one) (e.g. "Gres=band‐
1184 width:10000"). Modification of GRES count associated with spe‐
1185 cific files (e.g. GPUs) is not allowed other than to set their
1186 count on a node to zero. In order to change the GRES count to
1187 another value, modify your slurm.conf and gres.conf files and
1188 restart daemons. If GRES as associated with specific sockets,
1189 that information will be reported For example if all 4 GPUs on a
1190 node are all associated with socket zero, then
1191 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1192 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1193 associated with specific GPUs is not reported, but only avail‐
1194 able by parsing the gres.conf file. Generic resources assigned
1195 via scontrol will only persist across the restart of the slurm‐
1196 ctld daemon with the -R option and state files preserved or
1197 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1198 changes meant to be persistent across normal restarts of slurm‐
1199 ctld or the execution of scontrol reconfig.
1200 NodeAddr=<node address>
1203 Name that a node should be referred to in establishing a commu‐
1204 nications path. This name will be used as an argument to the
1205 getaddrinfo() function for identification. If a node range
1206 expression is used to designate multiple nodes, they must
1207 exactly match the entries in the NodeName (e.g. "Node‐
1208 Name=lx[0-7] NodeAddr=elx[0-7]"). NodeAddr may also contain IP
1209 addresses.
1210 NodeHostname=<node hostname>
1213 Typically this would be the string that "/bin/hostname -s"
1214 returns. It may also be the fully qualified domain name as
1215 returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any
1216 valid domain name associated with the host through the host
1217 database (/etc/hosts) or DNS, depending on the resolver set‐
1218 tings. Note that if the short form of the hostname is not used,
1219 it may prevent use of hostlist expressions (the numeric portion
1220 in brackets must be at the end of the string). A node range
1221 expression can be used to specify a set of nodes. If an expres‐
1222 sion is used, the number of nodes identified by NodeHostname
1223 must be identical to the number of nodes identified by NodeName.
1224 Reason=<reason>
1227 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1228 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1229 having more than one word.
1230 State=<state>
1233 Identify the state to be assigned to the node. Possible node
1234 states are "NoResp", "ALLOC", "ALLOCATED", "COMPLETING", "DOWN",
1235 "DRAIN", "FAIL", "FAILING", "FUTURE" "IDLE", "MAINT", "MIXED",
1236 "PERFCTRS/NPC", "RESERVED", "POWER_DOWN", "POWER_UP", "RESUME"
1237 or "UNDRAIN". Not all of those states can be set using the scon‐
1238 trol command only the following can: "CANCEL_REBOOT", "DOWN",
1239 "DRAIN", "FAIL", "FUTURE", "RESUME", "NoResp", "POWER_DOWN",
1240 "POWER_UP" and "UNDRAIN". If a node is in a "MIXED" state it
1241 usually means the node is in multiple states. For instance if
1242 only part of the node is "ALLOCATED" and the rest of the node is
1243 "IDLE" the state will be "MIXED". If you want to remove a node
1244 from service, you typically want to set its state to "DRAIN".
1245 "CANCEL_REBOOT" cancels a pending reboot on the node (same as
1246 "scontrol cancel_reboot <node>"). "FAILING" is similar to
1247 "DRAIN" except that some applications will seek to relinquish
1248 those nodes before the job completes. "PERFCTRS/NPC" indicates
1249 that Network Performance Counters associated with this node are
1250 in use, rendering this node as not usable for any other jobs.
1251 "RESERVED" indicates the node is in an advanced reservation and
1252 not generally available. "RESUME" is not an actual node state,
1253 but will change a node state from "DRAINED", "DRAINING", "DOWN"
1254 or "REBOOT" to either "IDLE" or "ALLOCATED" state as appropri‐
1255 ate. "RESUME" will also clear the "POWERING_DOWN" state of a
1256 node and make it eligible to be allocted. "UNDRAIN" clears the
1257 node from being drained (like "RESUME"), but will not change the
1258 node's base state (e.g. "DOWN"). Setting a node "DOWN" will
1259 cause all running and suspended jobs on that node to be termi‐
1260 nated. "POWER_DOWN" and "POWER_UP" will use the configured Sus‐
1261 pendProg and ResumeProg programs to explicitly place a node in
1262 or out of a power saving mode. If a node is already in the
1263 process of being powered up or down, the command will only
1264 change the state of the node but won't have any effect until the
1265 configured ResumeTimeout or SuspendTimeout is reached. Use of
1266 this command can be useful in situations where a ResumeProg like
1267 capmc in Cray machines is stalled and one wants to restore the
1268 node to "IDLE" manually, in this case rebooting the node and
1269 setting the state to "POWER_DOWN" will cancel the previous
1270 "POWER_UP" state and the node will become "IDLE". The "NoResp"
1271 state will only set the "NoResp" flag for a node without chang‐
1272 ing its underlying state. While all of the above states are
1273 valid, some of them are not valid new node states given their
1274 prior state. If the node state code printed is followed by "~",
1275 this indicates the node is presently in a power saving mode
1276 (typically running at reduced frequency). If the node state
1277 code is followed by "#", this indicates the node is presently
1278 being powered up or configured. If the node state code is fol‐
1279 lowed by "$", this indicates the node is currently in a reserva‐
1280 tion with a flag value of "maintenance". If the node state code
1281 is followed by "@", this indicates the node is currently sched‐
1282 uled to be rebooted. Generally only "DRAIN", "FAIL" and
1283 "RESUME" should be used. NOTE: The scontrol command should not
1284 be used to change node state on Cray systems. Use Cray tools
1285 such as xtprocadmin instead.
1286 Weight=<weight>
1289 Identify weight to be associated with specified nodes. This
1290 allows dynamic changes to weight associated with nodes, which
1291 will be used for the subsequent node allocation decisions.
1292 Weight assigned via scontrol will only persist across the
1293 restart of the slurmctld daemon with the -R option and state
1294 files preserved or slurmctld's receipt of a SIGHUP. Update
1295 slurm.conf with any changes meant to be persistent across normal
1296 restarts of slurmctld or the execution of scontrol reconfig.
1297 SPECIFICATIONS FOR SHOW COMMAND, NODES
1300 The meaning of the energy information is as follows:
1302 CurrentWatts
1305 The instantaneous power consumption of the node at the time of
1306 the last node energy accounting sample, in watts.
1307 LowestJoules
1310 The energy consumed by the node between the last time it was
1311 powered on and the last time it was registered by slurmd, in
1312 joules.
1313 ConsumedJoules
1316 The energy consumed by the node between the last time it was
1317 registered by the slurmd daemon and the last node energy
1318 accounting sample, in joules.
1319 If the reported value is "n/s" (not supported), the node does not sup‐
1322 port the configured AcctGatherEnergyType plugin. If the reported value
1323 is zero, energy accounting for nodes is disabled.
1324 The meaning of the external sensors information is as follows:
1327 ExtSensorsJoules
1330 The energy consumed by the node between the last time it was
1331 powered on and the last external sensors plugin node sample, in
1332 joules.
1333 ExtSensorsWatts
1337 The instantaneous power consumption of the node at the time of
1338 the last external sensors plugin node sample, in watts.
1339 ExtSensorsTemp
1342 The temperature of the node at the time of the last external
1343 sensors plugin node sample, in celsius.
1344 If the reported value is "n/s" (not supported), the node does not sup‐
1347 port the configured ExtSensorsType plugin.
1348 The meaning of the resource specialization information is as follows:
1351 CPUSpecList
1354 The list of Slurm abstract CPU IDs on this node reserved for
1355 exclusive use by the Slurm compute node daemons (slurmd, slurm‐
1356 stepd).
1357 MemSpecLimit
1360 The combined memory limit, in megabytes, on this node for the
1361 Slurm compute node daemons (slurmd, slurmstepd).
1362 The meaning of the memory information is as follows:
1365 RealMemory
1368 The total memory, in MB, on the node.
1369 AllocMem
1372 The total memory, in MB, currently allocated by jobs on the
1373 node.
1374 FreeMem
1377 The total memory, in MB, currently free on the node as reported
1378 by the OS.
1379 SPECIFICATIONS FOR UPDATE COMMAND, FRONTEND
1382 FrontendName=<name>
1385 Identify the front end node to be updated. This specification is
1386 required.
1387 Reason=<reason>
1390 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1391 Use quotes to enclose a reason having more than one word.
1392 State=<state>
1395 Identify the state to be assigned to the front end node. Possi‐
1396 ble values are "DOWN", "DRAIN" or "RESUME". If you want to
1397 remove a front end node from service, you typically want to set
1398 its state to "DRAIN". "RESUME" is not an actual node state, but
1399 will return a "DRAINED", "DRAINING", or "DOWN" front end node to
1400 service, either "IDLE" or "ALLOCATED" state as appropriate.
1401 Setting a front end node "DOWN" will cause all running and sus‐
1402 pended jobs on that node to be terminated.
1403 SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, PARTITIONS
1406 AllowGroups=<name>
1408 Identify the user groups which may use this partition. Multiple
1409 groups may be specified in a comma separated list. To permit
1410 all groups to use the partition specify "AllowGroups=ALL".
1411 AllocNodes=<name>
1414 Comma separated list of nodes from which users can execute jobs
1415 in the partition. Node names may be specified using the node
1416 range expression syntax described above. The default value is
1417 "ALL".
1418 Alternate=<partition name>
1421 Alternate partition to be used if the state of this partition is
1422 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1423 set alternate partition.
1424 CpuBind=<node>
1427 Specify the task binding mode to be used by default for this
1428 partition. Supported options include: "none", "board",
1429 "socket", "ldom" (NUMA), "core", "thread" and "off" (remove pre‐
1430 vious binding mode).
1431 Default=<yes|no>
1434 Specify if this partition is to be used by jobs which do not
1435 explicitly identify a partition to use. Possible output values
1436 are "YES" and "NO". In order to change the default partition of
1437 a running system, use the scontrol update command and set
1438 Default=yes for the partition that you want to become the new
1439 default.
1440 DefaultTime=<time>
1443 Run time limit used for jobs that don't specify a value. If not
1444 set then MaxTime will be used. Format is the same as for Max‐
1445 Time.
1446 DefMemPerCPU=<MB>
1449 Set the default memory to be allocated per CPU for jobs in this
1450 partition. The memory size is specified in megabytes.
1451 DefMemPerNode=<MB>
1453 Set the default memory to be allocated per node for jobs in this
1454 partition. The memory size is specified in megabytes.
1455 DisableRootJobs=<yes|no>
1458 Specify if jobs can be executed as user root. Possible values
1459 are "YES" and "NO".
1460 GraceTime=<seconds>
1463 Specifies, in units of seconds, the preemption grace time to be
1464 extended to a job which has been selected for preemption. The
1465 default value is zero, no preemption grace time is allowed on
1466 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1467 Hidden=<yes|no>
1470 Specify if the partition and its jobs should be hidden from
1471 view. Hidden partitions will by default not be reported by
1472 Slurm APIs or commands. Possible values are "YES" and "NO".
1473 JobDefaults=<specs>
1476 Specify job default values using a comma delimited list of
1477 "key=value" pairs. Supported keys include
1478 DefCpuPerGPU Default number of CPUs per allocated GPU.
1480 DefMemPerGPU Default memory limit (in megabytes) per allocated
1482 GPU.
1483 MaxMemPerCPU=<MB>
1486 Set the maximum memory to be allocated per CPU for jobs in this
1487 partition. The memory size is specified in megabytes.
1488 MaxMemPerCNode=<MB>
1490 Set the maximum memory to be allocated per node for jobs in this
1491 partition. The memory size is specified in megabytes.
1492 MaxNodes=<count>
1495 Set the maximum number of nodes which will be allocated to any
1496 single job in the partition. Specify a number, "INFINITE" or
1497 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1498 upon jobs that have already begun execution.
1499 MaxTime=<time>
1502 The maximum run time for jobs. Output format is
1503 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1504 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1505 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1506 onds. Time resolution is one minute and second values are
1507 rounded up to the next minute. Changing the MaxTime of a parti‐
1508 tion has no effect upon jobs that have already begun execution.
1509 MinNodes=<count>
1512 Set the minimum number of nodes which will be allocated to any
1513 single job in the partition. Changing the MinNodes of a parti‐
1514 tion has no effect upon jobs that have already begun execution.
1515 Increasing this value may prevent pending jobs from starting,
1516 even if they were submitted without -N/--nodes specification.
1517 If you do get in that situation, updating the MinNodes value of
1518 a pending job using the scontrol command will allow that job to
1519 be scheduled.
1520 Nodes=<name>
1523 Identify the node(s) to be associated with this partition. Mul‐
1524 tiple node names may be specified using simple node range
1525 expressions (e.g. "lx[10-20]"). Note that jobs may only be
1526 associated with one partition at any time. Specify a blank data
1527 value to remove all nodes from a partition: "Nodes=". Changing
1528 the Nodes in a partition has no effect upon jobs that have
1529 already begun execution.
1530 OverTimeLimit=<count>
1533 Number of minutes by which a job can exceed its time limit
1534 before being canceled. The configured job time limit is treated
1535 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1536 vides a hard limit, at which point the job is canceled. This is
1537 particularly useful for backfill scheduling, which bases upon
1538 each job's soft time limit. A partition-specific OverTimeLimit
1539 will override any global OverTimeLimit value. If not specified,
1540 the global OverTimeLimit value will take precedence. May not
1541 exceed 65533 minutes. An input value of "UNLIMITED" will clear
1542 any previously configured partition-specific OverTimeLimit
1543 value.
1544 OverSubscribe=<yes|no|exclusive|force>[:<job_count>]
1547 Specify if compute resources (i.e. individual CPUs) in this par‐
1548 tition can be shared by multiple jobs. Possible values are
1549 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1550 specifies how many jobs can be allocated to use each resource.
1551 PartitionName=<name>
1554 Identify the partition to be updated. This specification is
1555 required.
1556 PreemptMode=<mode>
1559 Reset the mechanism used to preempt jobs in this partition if
1560 PreemptType is configured to preempt/partition_prio. The default
1561 preemption mechanism is specified by the cluster-wide Preempt‐
1562 Mode configuration parameter. Possible values are "OFF", "CAN‐
1563 CEL", "REQUEUE" and "SUSPEND".
1564 Priority=<count>
1567 Jobs submitted to a higher priority partition will be dispatched
1568 before pending jobs in lower priority partitions and if possible
1569 they will preempt running jobs from lower priority partitions.
1570 Note that a partition's priority takes precedence over a job's
1571 priority. The value may not exceed 65533.
1572 PriorityJobFactor=<count>
1575 Partition factor used by priority/multifactor plugin in calcu‐
1576 lating job priority. The value may not exceed 65533. Also see
1577 PriorityTier.
1578 PriorityTier=<count>
1581 Jobs submitted to a partition with a higher priority tier value
1582 will be dispatched before pending jobs in partition with lower
1583 priority tier value and, if possible, they will preempt
1584 running jobs from partitions with lower priority tier values.
1585 Note that a partition's priority tier takes precedence over a
1586 job's priority. The value may not exceed 65533. Also see Pri‐
1587 orityJobFactor.
1588 QOS=<QOSname|blank to remove>
1591 Set the partition QOS with a QOS name or to remove the Partition
1592 QOS leave the option blank.
1593 RootOnly=<yes|no>
1596 Specify if only allocation requests initiated by user root will
1597 be satisfied. This can be used to restrict control of the par‐
1598 tition to some meta-scheduler. Possible values are "YES" and
1599 "NO".
1600 ReqResv=<yes|no>
1603 Specify if only allocation requests designating a reservation
1604 will be satisfied. This is used to restrict partition usage to
1605 be allowed only within a reservation. Possible values are "YES"
1606 and "NO".
1607 Shared=<yes|no|exclusive|force>[:<job_count>]
1610 Renamed to OverSubscribe, see option descriptions above.
1611 State=<up|down|drain|inactive>
1614 Specify if jobs can be allocated nodes or queued in this parti‐
1615 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1616 UP Designates that new jobs may queued on the partition,
1618 and that jobs may be allocated nodes and run from the
1619 partition.
1620 DOWN Designates that new jobs may be queued on the parti‐
1622 tion, but queued jobs may not be allocated nodes and
1623 run from the partition. Jobs already running on the
1624 partition continue to run. The jobs must be explicitly
1625 canceled to force their termination.
1626 DRAIN Designates that no new jobs may be queued on the par‐
1628 tition (job submission requests will be denied with an
1629 error message), but jobs already queued on the parti‐
1630 tion may be allocated nodes and run. See also the
1631 "Alternate" partition specification.
1632 INACTIVE Designates that no new jobs may be queued on the par‐
1634 tition, and jobs already queued may not be allocated
1635 nodes and run. See also the "Alternate" partition
1636 specification.
1637 TRESBillingWeights=<TRES Billing Weights>
1640 TRESBillingWeights is used to define the billing weights of each
1641 TRES type that will be used in calculating the usage of a job.
1642 The calculated usage is used when calculating fairshare and when
1643 enforcing the TRES billing limit on jobs. Updates affect new
1644 jobs and not existing jobs. See the slurm.conf man page for
1645 more information.
1646 SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, RESERVATIONS
1649 Reservation=<name>
1653 Identify the name of the reservation to be created, updated, or
1654 deleted. This parameter is required for update and is the only
1655 parameter for delete. For create, if you do not want to give a
1656 reservation name, use "scontrol create res ..." and a name will
1657 be created automatically.
1658 Accounts=<account list>
1661 List of accounts permitted to use the reserved nodes, for exam‐
1662 ple "Accounts=physcode1,physcode2". A user in any of the
1663 accounts may use the reserved nodes. A new reservation must
1664 specify Users or Groups and/or Accounts. If both Users/Groups
1665 and Accounts are specified, a job must match both in order to
1666 use the reservation. Accounts can also be denied access to
1667 reservations by preceding all of the account names with '-'.
1668 Alternately precede the equal sign with '-'. For example,
1669 "Accounts=-physcode1,-physcode2" or
1670 "Accounts-=physcode1,physcode2" will permit any account except
1671 physcode1 and physcode2 to use the reservation. You can add or
1672 remove individual accounts from an existing reservation by using
1673 the update command and adding a '+' or '-' sign before the '='
1674 sign. If accounts are denied access to a reservation (account
1675 name preceded by a '-'), then all other accounts are implicitly
1676 allowed to use the reservation and it is not possible to also
1677 explicitly specify allowed accounts.
1678 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1681 Specification of burst buffer resources which are to be
1682 reserved. "buffer_spec" consists of four elements: [plug‐
1683 in:][type:]#[units] "plugin" is the burst buffer plugin name,
1684 currently either "datawarp" or "generic". If no plugin is spec‐
1685 ified, the reservation applies to all configured burst buffer
1686 plugins. "type" specifies a Cray generic burst buffer resource,
1687 for example "nodes". if "type" is not specified, the number is
1688 a measure of storage space. The "units" may be "N" (nodes),
1689 "K|KiB", "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers of 1024)
1690 and "KB", "MB", "GB", "TB", "PB" (for powers of 1000). The
1691 default units are bytes for reservations of storage space. For
1692 example "BurstBuffer=datawarp:2TB" (reserve 2TB of storage plus
1693 3 nodes from the Cray plugin) or "BurstBuffer=100GB" (reserve
1694 100 GB of storage from all configured burst buffer plugins).
1695 Jobs using this reservation are not restricted to these burst
1696 buffer resources, but may use these reserved resources plus any
1697 which are generally available. NOTE: Usually Slurm interprets
1698 KB, MB, GB, TB, PB, TB units as powers of 1024, but for Burst
1699 Buffers size specifications Slurm supports both IEC/SI formats.
1700 This is because the CRAY API for managing DataWarps supports
1701 both formats.
1702 CoreCnt=<num>
1705 This option is only supported when SelectType=select/cons_res or
1706 select/cons_tres. Identify number of cores to be reserved. If
1707 NodeCnt is used without the FIRST_CORES flag, this is the total
1708 number of cores to reserve where cores per node is
1709 CoreCnt/NodeCnt. If a nodelist is used, or if NodeCnt is used
1710 with the FIRST_CORES flag, this should be an array of core num‐
1711 bers by node: Nodes=node[1-5] CoreCnt=2,2,3,3,4 or
1712 flags=FIRST_CORES NodeCnt=5 CoreCnt=1,2,1,3,2.
1713 Licenses=<license>
1716 Specification of licenses (or other resources available on all
1717 nodes of the cluster) which are to be reserved. License names
1718 can be followed by a colon and count (the default count is one).
1719 Multiple license names should be comma separated (e.g.
1720 "Licenses=foo:4,bar"). A new reservation must specify one or
1721 more resource to be included: NodeCnt, Nodes and/or Licenses.
1722 If a reservation includes Licenses, but no NodeCnt or Nodes,
1723 then the option Flags=LICENSE_ONLY must also be specified. Jobs
1724 using this reservation are not restricted to these licenses, but
1725 may use these reserved licenses plus any which are generally
1726 available.
1727 MaxStartDelay[=<timespec>]
1730 Change MaxStartDelay value which specifies the maximum time an
1731 eligible job not requesting this reservation can delay a job
1732 requesting it. Default is none. Valid formats are minutes, min‐
1733 utes:seconds, hours:minutes:seconds, days-hours, days-hours:min‐
1734 utes, days-hours:minutes:seconds. Time resolution is one minute
1735 and second values are rounded up to the next minute. Output for‐
1736 mat is always [days-]hours:minutes:seconds.
1737 NodeCnt=<num>[,num,...]
1740 Identify number of nodes to be reserved. The number can include
1741 a suffix of "k" or "K", in which case the number specified is
1742 multiplied by 1024. A new reservation must specify one or more
1743 resource to be included: NodeCnt, Nodes and/or Licenses.
1744 Nodes=<name>
1747 Identify the node(s) to be reserved. Multiple node names may be
1748 specified using simple node range expressions (e.g.
1749 "Nodes=lx[10-20]"). Specify a blank data value to remove all
1750 nodes from a reservation: "Nodes=". A new reservation must
1751 specify one or more resource to be included: NodeCnt, Nodes
1752 and/or Licenses. A specification of "ALL" will reserve all
1753 nodes. Set Flags=PART_NODES and PartitionName= in order for
1754 changes in the nodes associated with a partition to also be
1755 reflected in the nodes associated with a reservation.
1756 StartTime=<time_spec>
1759 The start time for the reservation. A new reservation must
1760 specify a start time. It accepts times of the form HH:MM:SS for
1761 a specific time of day (seconds are optional). (If that time is
1762 already past, the next day is assumed.) You may also specify
1763 midnight, noon, fika (3 PM) or teatime (4 PM) and you can have a
1764 time-of-day suffixed with AM or PM for running in the morning or
1765 the evening. You can also say what day the job will be run, by
1766 specifying a date of the form MMDDYY or MM/DD/YY or MM.DD.YY, or
1767 a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also give
1768 times like now + count time-units, where the time-units can be
1769 seconds (default), minutes, hours, days, or weeks and you can
1770 tell Slurm to run the job today with the keyword today and to
1771 run the job tomorrow with the keyword tomorrow. You cannot
1772 update the StartTime of a reservation in ACTIVE state.
1773 EndTime=<time_spec>
1776 The end time for the reservation. A new reservation must spec‐
1777 ify an end time or a duration. Valid formats are the same as
1778 for StartTime.
1779 Duration=<time>
1782 The length of a reservation. A new reservation must specify an
1783 end time or a duration. Valid formats are minutes, minutes:sec‐
1784 onds, hours:minutes:seconds, days-hours, days-hours:minutes,
1785 days-hours:minutes:seconds, or UNLIMITED. Time resolution is
1786 one minute and second values are rounded up to the next minute.
1787 Output format is always [days-]hours:minutes:seconds.
1788 PartitionName=<name>
1791 Partition used to reserve nodes from. This will attempt to allo‐
1792 cate all nodes in the specified partition unless you request
1793 fewer resources than are available with CoreCnt, NodeCnt or
1794 TRES. Jobs will be allowed to use this reservation even if run‐
1795 ning in a different partition. There only needs to be overlap‐
1796 ping nodes from that different partition and the nodes used in
1797 the reservation.
1798 Flags=<flags>
1801 Flags associated with the reservation. You can add or remove
1802 individual flags from an existing reservation by adding a '+' or
1803 '-' sign before the '=' sign. For example: Flags-=DAILY (NOTE:
1804 this shortcut is not supported for all flags). Currently sup‐
1805 ported flags include:
1806 ANY_NODES This is a reservation for burst buffers and/or
1808 licenses only and not compute nodes. If this flag
1809 is set, a job using this reservation may use the
1810 associated burst buffers and/or licenses plus any
1811 compute nodes. If this flag is not set, a job
1812 using this reservation may use only the nodes and
1813 licenses associated with the reservation.
1814 DAILY Repeat the reservation at the same time every day.
1816 FLEX Permit jobs requesting the reservation to begin
1818 prior to the reservation's start time, end after
1819 the reservation's end time, and use any resources
1820 inside and/or outside of the reservation regard‐
1821 less of any constraints possibly set in the reser‐
1822 vation. A typical use case is to prevent jobs not
1823 explicitly requesting the reservation from using
1824 those reserved resources rather than forcing jobs
1825 requesting the reservation to use those resources
1826 in the time frame reserved. Another use case could
1827 be to always have a particular number of nodes
1828 with a specific feature reserved for a specific
1829 account so users in this account may use this
1830 nodes plus possibly other nodes without this fea‐
1831 ture.
1832 FIRST_CORES Use the lowest numbered cores on a node only. Flag
1834 removal with '-=' is not supported.
1835 IGNORE_JOBS Ignore currently running jobs when creating the
1837 reservation. This can be especially useful when
1838 reserving all nodes in the system for maintenance.
1839 LICENSE_ONLY See ANY_NODES.
1841 MAINT Maintenance mode, receives special accounting
1843 treatment. This reservation is permitted to use
1844 resources that are already in another reservation.
1845 MAGNETIC This flag allows jobs to be considered for this
1847 reservation even if they didn't request it.
1848 NO_HOLD_JOBS_AFTER
1850 By default, when a reservation ends the reserva‐
1851 tion request will be removed from any pending jobs
1852 submitted to the reservation and will be put into
1853 a held state. Use this flag to let jobs run out‐
1854 side of the reservation after the reservation is
1855 gone. Flag removal with '-=' is not supported.
1856 OVERLAP This reservation can be allocated resources that
1858 are already in another reservation. Flag removal
1859 with '-=' is not supported.
1860 PART_NODES This flag can be used to reserve all nodes within
1862 the specified partition. PartitionName and
1863 Nodes=ALL must be specified with this flag.
1864 PURGE_COMP[=<timespec>]
1866 Purge the reservation if it is ever idle for time‐
1867 spec (no jobs associated with it). If timespec
1868 isn't given then 5 minutes is the default. Valid
1869 timespec formats are minutes, minutes:seconds,
1870 hours:minutes:seconds, days-hours, days-hours:min‐
1871 utes, days-hours:minutes:seconds. Time resolution
1872 is one minute and second values are rounded up to
1873 the next minute. Output format is always
1874 [days-]hours:minutes:seconds.
1875 REPLACE Nodes which are DOWN, DRAINED, or allocated to
1877 jobs are automatically replenished using idle
1878 resources. This option can be used to maintain a
1879 constant number of idle resources available for
1880 pending jobs (subject to availability of idle
1881 resources). This should be used with the NodeCnt
1882 reservation option; do not identify specific nodes
1883 to be included in the reservation. Flag removal
1884 with '-=' is not supported.
1885 NOTE: Removing a node from the cluster while in a
1887 reservation with the REPLACE flag will not cause
1888 it to be replaced.
1889 REPLACE_DOWN Nodes which are DOWN or DRAINED are automatically
1891 replenished using idle resources. This option can
1892 be used to maintain a constant sized pool of
1893 resources available for pending jobs (subject to
1894 availability of idle resources). This should be
1895 used with the NodeCnt reservation option; do not
1896 identify specific nodes to be included in the
1897 reservation. Flag removal with '-=' is not sup‐
1898 ported.
1899 NOTE: Removing a node from the cluster while in a
1901 reservation with the REPLACE_DOWN flag will not
1902 cause it to be replaced.
1903 SPEC_NODES Reservation is for specific nodes (output only).
1905 STATIC_ALLOC Make it so after the nodes are selected for a
1907 reservation they don't change. Without this
1908 option when nodes are selected for a reservation
1909 and one goes down the reservation will select a
1910 new node to fill the spot.
1911 TIME_FLOAT The reservation start time is relative to the cur‐
1913 rent time and moves forward through time (e.g. a
1914 StartTime=now+10minutes will always be 10 minutes
1915 in the future). Repeating (e.g. DAILY) floating
1916 reservations are not supported. Flag cannot be
1917 added to or removed from an existing reservation.
1918 WEEKDAY Repeat the reservation at the same time on every
1920 weekday (Monday, Tuesday, Wednesday, Thursday and
1921 Friday).
1922 WEEKEND Repeat the reservation at the same time on every
1924 weekend day (Saturday and Sunday).
1925 WEEKLY Repeat the reservation at the same time every
1927 week.
1928 Features=<features>
1931 Set the reservation's required node features. Multiple values
1932 may be "&" separated if all features are required (AND opera‐
1933 tion) or separated by "|" if any of the specified features are
1934 required (OR operation). Parenthesis are also supported for
1935 features to be ANDed together with counts of nodes having the
1936 specified features. For example "Fea‐
1937 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the advanced
1938 reservation should include 4 nodes with ALL of the features
1939 "knl", "a2a", and "flat" plus 2 nodes with the feature
1940 "haswell".
1941 Value may be cleared with blank data value, "Features=".
1943 Groups=<group list>
1946 List of groups permitted to use the reserved nodes, for example
1947 "Group=bio,chem". A new reservation must specify Users or
1948 Groups and/or Accounts. If both Users/Groups and Accounts are
1949 specified, a job must match both in order to use the reserva‐
1950 tion. Unlike users groups do not allow denied access to reser‐
1951 vations. You can add or remove individual groups from an exist‐
1952 ing reservation by using the update command and adding a '+' or
1953 '-' sign before the '=' sign. NOTE: Groups and Users are mutu‐
1954 ally exclusive in reservations, if you want to switch between
1955 the 2 you must update the reservation with a group='' or user=''
1956 and fill in the opposite with the appropriate setting.
1957 Skip Used on a reoccurring reservation, skip to the next reservation
1960 iteration. NOTE: Only available for update.
1961 Users=<user list>
1964 List of users permitted to use the reserved nodes, for example
1965 "User=jones1,smith2". A new reservation must specify Users or
1966 Groups and/or Accounts. If both Users/Groups and Accounts are
1967 specified, a job must match both in order to use the reserva‐
1968 tion. Users can also be denied access to reservations by pre‐
1969 ceding all of the user names with '-'. Alternately precede the
1970 equal sign with '-'. For example, "User=-jones1,-smith2" or
1971 "User-=jones1,smith2" will permit any user except jones1 and
1972 smith2 to use the reservation. You can add or remove individual
1973 users from an existing reservation by using the update command
1974 and adding a '+' or '-' sign before the '=' sign. If users are
1975 denied access to a reservation (user name preceded by a '-'),
1976 then all other users are implicitly allowed to use the reserva‐
1977 tion and it is not possible to also explicitly specify allowed
1978 users. NOTE: Groups and Users are mutually exclusive in reser‐
1979 vations, if you want to switch between the 2 you must update the
1980 reservation with a group='' or user='' and fill in the opposite
1981 with the appropriate setting.
1982 TRES=<tres_spec>
1985 Comma-separated list of TRES required for the reservation. Cur‐
1986 rent supported TRES types with reservations are: CPU, Node,
1987 License and BB. CPU and Node follow the same format as CoreCnt
1988 and NodeCnt parameters respectively. License names can be fol‐
1989 lowed by an equal '=' and a count:
1990 License/<name1>=<count1>[,License/<name2>=<count2>,...]
1992 BurstBuffer can be specified in a similar way as BurstBuffer
1994 parameter. The only difference is that colon symbol ':' should
1995 be replaced by an equal '=' in order to follow the TRES format.
1996 Some examples of TRES valid specifications:
1998 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
2000 TRES=node=5k,license/iop1=2
2002 As specified in CoreCnt, if a nodelist is specified, cpu can be
2004 an array of core numbers by node: nodes=compute[1-3]
2005 TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
2006 Please note that CPU, Node, License and BB can override CoreCnt,
2008 NodeCnt, Licenses and BurstBuffer parameters respectively. Also
2009 CPU represents CoreCnt, in a reservation and will be adjusted if
2010 you have threads per core on your nodes.
2011 Note that a reservation that contains nodes or cores is associ‐
2013 ated with one partition, and can't span resources over multiple
2014 partitions. The only exception from this is when the reserva‐
2015 tion is created with explicitly requested nodes.
2016
2019 Executing scontrol sends a remote procedure call to slurmctld. If
2020 enough calls from scontrol or other Slurm client commands that send
2021 remote procedure calls to the slurmctld daemon come in at once, it can
2022 result in a degradation of performance of the slurmctld daemon, possi‐
2023 bly resulting in a denial of service.
2024 Do not run scontrol or other Slurm client commands that send remote
2026 procedure calls to slurmctld from loops in shell scripts or other pro‐
2027 grams. Ensure that programs limit calls to scontrol to the minimum nec‐
2028 essary for the information you are trying to gather.
2029
2032 Some scontrol options may be set via environment variables. These envi‐
2033 ronment variables, along with their corresponding options, are listed
2034 below. (Note: Commandline options will always override these settings.)
2035 SCONTROL_ALL -a, --all
2037 SCONTROL_FEDERATION --federation
2039 SCONTROL_FUTURE -F, --future
2041 SCONTROL_LOCAL --local
2043 SCONTROL_SIBLING --sibling
2045 SLURM_BITSTR_LEN Specifies the string length to be used for holding
2047 a job array's task ID expression. The default
2048 value is 64 bytes. A value of 0 will print the
2049 full expression with any length required. Larger
2050 values may adversely impact the application perfor‐
2051 mance.
2052 SLURM_CLUSTERS Same as --clusters
2054 SLURM_CONF The location of the Slurm configuration file.
2056 SLURM_CONF_OUT When running 'write config', the location of the
2058 Slurm configuration file to be written.
2059 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
2061 value of standard, the default value, generates
2062 output in the form
2063 "year-month-dateThour:minute:second". A value of
2064 relative returns only "hour:minute:second" if the
2065 current day. For other dates in the current year
2066 it prints the "hour:minute" preceded by "Tomorr"
2067 (tomorrow), "Ystday" (yesterday), the name of the
2068 day for the coming week (e.g. "Mon", "Tue", etc.),
2069 otherwise the date (e.g. "25 Apr"). For other
2070 years it returns a date month and year without a
2071 time (e.g. "6 Jun 2012"). All of the time stamps
2072 use a 24 hour format.
2073 A valid strftime() format can also be specified.
2075 For example, a value of "%a %T" will report the day
2076 of the week and a time stamp (e.g. "Mon 12:34:56").
2077 SLURM_TOPO_LEN Specify the maximum size of the line when printing
2080 Topology. If not set, the default value is unlim‐
2081 ited.
2082
2085 When using SlurmDBD, users who have an AdminLevel defined (Operator or
2086 Admin) and users who are account coordinators are given the authority
2087 to view and modify jobs, reservations, nodes, etc., as defined in the
2088 following table - regardless of whether a PrivateData restriction has
2089 been defined in the slurm.conf file.
2090 scontrol show job(s): Admin, Operator, Coordinator
2092 scontrol update job: Admin, Operator, Coordinator
2093 scontrol requeue: Admin, Operator, Coordinator
2094 scontrol show step(s): Admin, Operator, Coordinator
2095 scontrol update step: Admin, Operator, Coordinator
2096 scontrol show node: Admin, Operator
2098 scontrol update node: Admin
2099 scontrol create partition: Admin
2101 scontrol show partition: Admin, Operator
2102 scontrol update partition: Admin
2103 scontrol delete partition: Admin
2104 scontrol create reservation: Admin, Operator
2106 scontrol show reservation: Admin, Operator
2107 scontrol update reservation: Admin, Operator
2108 scontrol delete reservation: Admin, Operator
2109 scontrol reconfig: Admin
2111 scontrol shutdown: Admin
2112 scontrol takeover: Admin
2113
2116 # scontrol
2117 scontrol: show part debug
2118 PartitionName=debug
2119 AllocNodes=ALL AllowGroups=ALL Default=YES
2120 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2121 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2122 Nodes=snowflake[0-48]
2123 Priority=1 RootOnly=NO OverSubscribe=YES:4
2124 State=UP TotalCPUs=694 TotalNodes=49
2125 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2126 scontrol: show job 71701
2127 JobId=71701 Name=hostname
2128 UserId=da(1000) GroupId=da(1000)
2129 Priority=66264 Account=none QOS=normal WCKey=*123
2130 JobState=COMPLETED Reason=None Dependency=(null)
2131 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 ExitCode=0:0
2132 SubmitTime=2010-01-05T10:58:40 EligibleTime=2010-01-05T10:58:40
2133 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2134 SuspendTime=None SecsPreSuspend=0
2135 Partition=debug AllocNode:Sid=snowflake:4702
2136 ReqNodeList=(null) ExcNodeList=(null)
2137 NodeList=snowflake0
2138 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2139 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2140 Features=(null) Reservation=(null)
2141 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2142 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2143 scontrol: show hostnames tux[1-3]
2144 tux1
2145 tux2
2146 tux3
2147 scontrol: create res StartTime=2009-04-01T08:00:00 Duration=5:00:00
2148 Users=dbremer NodeCnt=10
2149 Reservation created: dbremer_1
2150 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2151 scontrol: delete Reservation=dbremer_1
2152 scontrol: quit
2153
2156 Copyright (C) 2002-2007 The Regents of the University of California.
2157 Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2158 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2159 Copyright (C) 2010-2018 SchedMD LLC.
2160 This file is part of Slurm, a resource management program. For
2162 details, see <https://slurm.schedmd.com/>.
2163 Slurm is free software; you can redistribute it and/or modify it under
2165 the terms of the GNU General Public License as published by the Free
2166 Software Foundation; either version 2 of the License, or (at your
2167 option) any later version.
2168 Slurm is distributed in the hope that it will be useful, but WITHOUT
2170 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2171 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2172 for more details.
2173
2175 /etc/slurm.conf
2176
2178 scancel(1), sinfo(1), squeue(1), slurm_create_partition [22m(3),
2179 slurm_delete_partition (3), slurm_load_ctl_conf [22m(3), slurm_load_jobs
2180 (3), slurm_load_node (3), slurm_load_partitions (3), slurm_reconfigure
2181 (3), slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2182 slurm_suspend [22m(3), slurm_takeover [22m(3), slurm_update_job [22m(3),
2183 slurm_update_node [22m(3), slurm_update_partition [22m(3), slurm.conf(5),
2184 slurmctld(8)
2185October 2020 Slurm Commands scontrol(1)