1scontrol(1) Slurm Commands scontrol(1)
2
6 scontrol - view or modify Slurm configuration and state.
7
10 scontrol [OPTIONS...] [COMMAND...]
11
14 scontrol is used to view or modify Slurm configuration including: job,
15 job step, node, partition, reservation, and overall system configura‐
16 tion. Most of the commands can only be executed by user root or an
17 Administrator. If an attempt to view or modify configuration informa‐
18 tion is made by an unauthorized user, an error message will be printed
19 and the requested action will not occur. If no command is entered on
20 the execute line, scontrol will operate in an interactive mode and
21 prompt for input. It will continue prompting for input and executing
22 commands until explicitly terminated. If a command is entered on the
23 execute line, scontrol will execute that command and terminate. All
24 commands and options are case-insensitive, although node names, parti‐
25 tion names, and reservation names are case-sensitive (node names "LX"
26 and "lx" are distinct). All commands and options can be abbreviated to
27 the extent that the specification is unique. A modified Slurm configu‐
28 ration can be written to a file using the scontrol write config com‐
29 mand. The resulting file will be named using the convention
30 "slurm.conf.<datetime>" and located in the same directory as the origi‐
31 nal "slurm.conf" file. The directory containing the original slurm.conf
32 must be writable for this to occur.
33
36 -a, --all
37 When the show command is used, then display all partitions,
38 their jobs and jobs steps. This causes information to be dis‐
39 played about partitions that are configured as hidden and parti‐
40 tions that are unavailable to user's group.
41 -d, --details
43 Causes the show command to provide additional details where
44 available.
45 --federation
47 Report jobs from from federation if a member of one.
48 -F, --future
50 Report nodes in FUTURE state.
51 -h, --help
53 Print a help message describing the usage of scontrol.
54 --hide Do not display information about hidden partitions, their jobs
56 and job steps. By default, neither partitions that are config‐
57 ured as hidden nor those partitions unavailable to user's group
58 will be displayed (i.e. this is the default behavior).
59 --local
61 Show only information local to this cluster. Ignore other clus‐
62 ters in the federated if a member of one. Overrides --federa‐
63 tion.
64 -M, --clusters=<string>
66 The cluster to issue commands to. Only one cluster name may be
67 specified. Note that the SlurmDBD must be up for this option to
68 work properly. This option implicitly sets the --local option.
69 -o, --oneliner
72 Print information one line per record.
73 -Q, --quiet
75 Print no warning or informational messages, only fatal error
76 messages.
77 --sibling
79 Show all sibling jobs on a federated cluster. Implies --federa‐
80 tion.
81 -u, --uid=<uid>
83 Attempt to update a job as user <uid> instead of the invoking
84 user id.
85 -v, --verbose
87 Print detailed event logging. Multiple -v's will further
88 increase the verbosity of logging. By default only errors will
89 be displayed.
90 -V , --version
93 Print version information and exit.
94 COMMANDS
96 abort Instruct the Slurm controller to terminate immediately and gen‐
99 erate a core file. See "man slurmctld" for information about
100 where the core file will be written.
101 cancel_reboot <NodeList>
104 Cancel pending reboots on nodes.
105 checkpoint CKPT_OP ID
108 Perform a checkpoint activity on the job step(s) with the speci‐
109 fied identification. ID can be used to identify a specific job
110 (e.g. "<job_id>", which applies to all of its existing steps) or
111 a specific job step (e.g. "<job_id>.<step_id>"). Acceptable
112 values for CKPT_OP include:
113 able Test if presently not disabled, report start time if
115 checkpoint in progress
116 create Create a checkpoint and continue the job or job step
118 disable Disable future checkpoints
120 enable Enable future checkpoints
122 error Report the result for the last checkpoint request,
124 error code and message
125 restart Restart execution of the previously checkpointed job
127 or job step
128 requeue Create a checkpoint and requeue the batch job, com‐
130 bines vacate and restart operations
131 vacate Create a checkpoint and terminate the job or job
133 step
134 Acceptable values for CKPT_OP include:
135 MaxWait=<seconds> Maximum time for checkpoint to be written.
137 Default value is 10 seconds. Valid with
138 create and vacate options only.
139 ImageDir=<directory_name>
141 Location of checkpoint file. Valid with
142 create, vacate and restart options only.
143 This value takes precedent over any --check‐
144 point-dir value specified at job submission
145 time.
146 StickToNodes If set, resume job on the same nodes are
148 previously used. Valid with the restart
149 option only.
150 cluster CLUSTER_NAME
153 The cluster to issue commands to. Only one cluster name may be
154 specified.
155 create SPECIFICATION
158 Create a new partition or reservation. See the full list of
159 parameters below. Include the tag "res" to create a reservation
160 without specifying a reservation name.
161 completing
164 Display all jobs in a COMPLETING state along with associated
165 nodes in either a COMPLETING or DOWN state.
166 delete SPECIFICATION
169 Delete the entry with the specified SPECIFICATION. The two
170 SPECIFICATION choices are PartitionName=<name> and Reserva‐
171 tion=<name>. Reservations and partitions should have no associ‐
172 ated jobs at the time of their deletion (modify the jobs first).
173 If the specified partition is in use, the request is denied.
174 errnumstr ERRNO
177 Given a Slurm error number, return a descriptive string.
178 fsdampeningfactor FACTOR
181 Set the FairShareDampeningFactor in slurmctld.
182 help Display a description of scontrol options and commands.
185 hold job_list
188 Prevent a pending job from being started (sets it's priority to
189 0). Use the release command to permit the job to be scheduled.
190 The job_list argument is a comma separated list of job IDs OR
191 "jobname=" with the job's name, which will attempt to hold all
192 jobs having that name. Note that when a job is held by a system
193 administrator using the hold command, only a system administra‐
194 tor may release the job for execution (also see the uhold com‐
195 mand). When the job is held by its owner, it may also be
196 released by the job's owner. Additionally, attempting to hold a
197 running job will have not suspend or cancel it. But, it will set
198 the job priority to 0 and update the job reason field, which
199 would hold the job if it was requeued at a later time.
200 notify job_id message
203 Send a message to standard error of the salloc or srun command
204 or batch job associated with the specified job_id.
205 pidinfo proc_id
208 Print the Slurm job id and scheduled termination time corre‐
209 sponding to the supplied process id, proc_id, on the current
210 node. This will work only with processes on node on which scon‐
211 trol is run, and only for those processes spawned by Slurm and
212 their descendants.
213 listpids [job_id[.step_id]] [NodeName]
216 Print a listing of the process IDs in a job step (if
217 JOBID.STEPID is provided), or all of the job steps in a job (if
218 job_id is provided), or all of the job steps in all of the jobs
219 on the local node (if job_id is not provided or job_id is "*").
220 This will work only with processes on the node on which scontrol
221 is run, and only for those processes spawned by Slurm and their
222 descendants. Note that some Slurm configurations (ProctrackType
223 value of pgid) are unable to identify all processes associated
224 with a job or job step.
225 Note that the NodeName option is only really useful when you
227 have multiple slurmd daemons running on the same host machine.
228 Multiple slurmd daemons on one host are, in general, only used
229 by Slurm developers.
230 ping Ping the primary and secondary slurmctld daemon and report if
233 they are responding.
234 reboot [ASAP] [nextstate=<RESUME|DOWN>] [reason=<reason>]
237 <ALL|NodeList>
238 Reboot the nodes in the system when they become idle using the
239 RebootProgram as configured in Slurm's slurm.conf file. Each
240 node will have the "REBOOT" flag added to its node state. After
241 a node reboots and the slurmd daemon starts up again, the
242 HealthCheckProgram will run once. Then, the slurmd daemon will
243 register itself with the slurmctld daemon and the "REBOOT" flag
244 will be cleared. If the node reason is "Reboot ASAP", Slurm
245 will clear the node's "DRAIN" state flag as well. The "ASAP"
246 option adds the "DRAIN" flag to each node's state, preventing
247 additional jobs from running on the node so it can be rebooted
248 and returned to service "As Soon As Possible" (i.e. ASAP).
249 "ASAP" will also set the node reason to "Reboot ASAP" if the
250 "reason" option isn't specified. If the "nextstate" option is
251 specified as "DOWN", then the node will remain in a down state
252 after rebooting. If "nextstate" is specified as "RESUME", then
253 the nodes will resume as normal when the node registers and the
254 node reason will be cleared. Resuming nodes will be considered
255 as available in backfill future scheduling and won't be replaced
256 by idle nodes in a reservation. When using the "nextstate" and
257 "reason" options together the reason will be appended with
258 "reboot issued" when the reboot is issued and "reboot complete"
259 when the node registers with a "nextstate" of "DOWN". The "rea‐
260 son" option sets each node's reason to a user-defined message.
261 You must specify either a list of nodes or that ALL nodes are to
262 be rebooted. NOTE: By default, this command does not prevent
263 additional jobs from being scheduled on any nodes before reboot.
264 To do this, you can either use the "ASAP" option or explicitly
265 drain the nodes beforehand. You can alternately create an
266 advanced reservation to prevent additional jobs from being ini‐
267 tiated on nodes to be rebooted. Pending reboots can be can‐
268 celled by using "scontrol cancel_reboot <node>" or setting the
269 node state to "CANCEL_REBOOT". A node will be marked "DOWN" if
270 it doesn't reboot within ResumeTimeout.
271 reconfigure
274 Instruct all Slurm daemons to re-read the configuration file.
275 This command does not restart the daemons. This mechanism would
276 be used to modify configuration parameters (Epilog, Prolog,
277 SlurmctldLogFile, SlurmdLogFile, etc.). The Slurm controller
278 (slurmctld) forwards the request all other daemons (slurmd dae‐
279 mon on each compute node). Running jobs continue execution.
280 Most configuration parameters can be changed by just running
281 this command, however, Slurm daemons should be shutdown and
282 restarted if any of these parameters are to be changed:
283 AuthType, BackupAddr, BackupController, ControlAddr, Control‐
284 Mach, PluginDir, StateSaveLocation, SlurmctldPort or SlurmdPort.
285 The slurmctld daemon and all slurmd daemons must be restarted if
286 nodes are added to or removed from the cluster.
287 release job_list
290 Release a previously held job to begin execution. The job_list
291 argument is a comma separated list of job IDs OR "jobname=" with
292 the job's name, which will attempt to hold all jobs having that
293 name. Also see hold.
294 requeue [option] job_list
297 Requeue a running, suspended or finished Slurm batch job into
298 pending state. The job_list argument is a comma separated list
299 of job IDs. The command accepts the following option:
300 Incomplete
302 Operate only on jobs (or tasks of a job array) which have
303 not completed. Specifically only jobs in the following
304 states will be requeued: CONFIGURING, RUNNING, STOPPED or
305 SUSPENDED.
306 requeuehold [option] job_list
309 Requeue a running, suspended or finished Slurm batch job into
310 pending state, moreover the job is put in held state (priority
311 zero). The job_list argument is a comma separated list of job
312 IDs. A held job can be released using scontrol to reset its
313 priority (e.g. "scontrol release <job_id>"). The command
314 accepts the following options:
315 Incomplete
317 Operate only on jobs (or tasks of a job array) which have
318 not completed. Specifically only jobs in the following
319 states will be requeued: CONFIGURING, RUNNING, STOPPED or
320 SUSPENDED.
321 State=SpecialExit
323 The "SpecialExit" keyword specifies that the job has to
324 be put in a special state JOB_SPECIAL_EXIT. The "scon‐
325 trol show job" command will display the JobState as SPE‐
326 CIAL_EXIT, while the "squeue" command as SE.
327 resume job_list
330 Resume a previously suspended job. The job_list argument is a
331 comma separated list of job IDs. Also see suspend.
332 NOTE: A suspended job releases its CPUs for allocation to other
334 jobs. Resuming a previously suspended job may result in multi‐
335 ple jobs being allocated the same CPUs, which could trigger gang
336 scheduling with some configurations or severe degradation in
337 performance with other configurations. Use of the scancel com‐
338 mand to send SIGSTOP and SIGCONT signals would stop a job with‐
339 out releasing its CPUs for allocation to other jobs and would be
340 a preferable mechanism in many cases. If performing system
341 maintenance you may want to use suspend/resume in the following
342 way. Before suspending set all nodes to draining or set all par‐
343 titions to down so that no new jobs can be scheduled. Then sus‐
344 pend jobs. Once maintenance is done resume jobs then resume
345 nodes and/or set all partitions back to up. Use with caution.
346 schedloglevel LEVEL
349 Enable or disable scheduler logging. LEVEL may be "0", "1",
350 "disable" or "enable". "0" has the same effect as "disable". "1"
351 has the same effect as "enable". This value is temporary and
352 will be overwritten when the slurmctld daemon reads the
353 slurm.conf configuration file (e.g. when the daemon is restarted
354 or scontrol reconfigure is executed) if the SlurmSchedLogLevel
355 parameter is present.
356 setdebug LEVEL
359 Change the debug level of the slurmctld daemon. LEVEL may be an
360 integer value between zero and nine (using the same values as
361 SlurmctldDebug in the slurm.conf file) or the name of the most
362 detailed message type to be printed: "quiet", "fatal", "error",
363 "info", "verbose", "debug", "debug2", "debug3", "debug4", or
364 "debug5". This value is temporary and will be overwritten when‐
365 ever the slurmctld daemon reads the slurm.conf configuration
366 file (e.g. when the daemon is restarted or scontrol reconfigure
367 is executed).
368 setdebugflags [+|-]FLAG
371 Add or remove DebugFlags of the slurmctld daemon. See "man
372 slurm.conf" for a list of supported DebugFlags. NOTE: Changing
373 the value of some DebugFlags will have no effect without
374 restarting the slurmctld daemon, which would set DebugFlags
375 based upon the contents of the slurm.conf configuration file.
376 show ENTITY ID
379 or
380 show ENTITY=ID
382 Display the state of the specified entity with the specified
383 identification. ENTITY may be aliases, assoc_mgr, bbstat,
384 burstbuffer, config, daemons, dwstat, federation, frontend, job,
385 node, partition, powercap, reservation, slurmd, step, topology,
386 hostlist, hostlistsorted or hostnames ID can be used to identify
387 a specific element of the identified entity: job ID, node name,
388 partition name, reservation name, or job step ID for job, node,
389 partition, or step respectively. For an ENTITY of bbstat or
390 dwstat (they are equivalent) optional arguments are the options
391 of the local status command. The status commands will be exe‐
392 cuted by the slurmctld daemon and its response returned to the
393 user. For an ENTITY of topology, the ID may be a node or switch
394 name. If one node name is specified, all switches connected to
395 that node (and their parent switches) will be shown. If more
396 than one node name is specified, only switches that connect to
397 all named nodes will be shown. aliases will return all NodeName
398 values associated to a given NodeHostname (useful to get the
399 list of virtual nodes associated with a real node in a configu‐
400 ration where multiple slurmd daemons execute on a single compute
401 node). assoc_mgr displays the current contents of the slurm‐
402 ctld's internal cache for users, associations and/or qos. The ID
403 may be users=<user1>,[...,<userN>],
404 accounts=<acct1>,[...,<acctN>], qos=<qos1>,[...,<qosN>] and/or
405 flags=<users,assoc,qos>, used to filter the desired section to
406 be displayed. If no flags are specified, all sections are dis‐
407 played. burstbuffer displays the current status of the Burst‐
408 Buffer plugin. config displays parameter names from the config‐
409 uration files in mixed case (e.g. SlurmdPort=7003) while derived
410 parameters names are in upper case only (e.g. SLURM_VERSION).
411 hostnames takes an optional hostlist expression as input and
412 writes a list of individual host names to standard output (one
413 per line). If no hostlist expression is supplied, the contents
414 of the SLURM_JOB_NODELIST environment variable is used. For
415 example "tux[1-3]" is mapped to "tux1","tux2" and "tux3" (one
416 hostname per line). hostlist takes a list of host names and
417 prints the hostlist expression for them (the inverse of host‐
418 names). hostlist can also take the absolute pathname of a file
419 (beginning with the character '/') containing a list of host‐
420 names. Multiple node names may be specified using simple node
421 range expressions (e.g. "lx[10-20]"). All other ID values must
422 identify a single element. The job step ID is of the form
423 "job_id.step_id", (e.g. "1234.1"). slurmd reports the current
424 status of the slurmd daemon executing on the same node from
425 which the scontrol command is executed (the local host). It can
426 be useful to diagnose problems. By default hostlist does not
427 sort the node list or make it unique (e.g. tux2,tux1,tux2 =
428 tux[2,1-2]). If you wanted a sorted list use hostlistsorted
429 (e.g. tux2,tux1,tux2 = tux[1-2,2]). By default, all elements of
430 the entity type specified are printed. For an ENTITY of job, if
431 the job does not specify socket-per-node, cores-per-socket or
432 threads-per-core then it will display '*' in ReqS:C:T=*:*:*
433 field. For an ENTITY of federation, the federation name that the
434 controller is part of and the sibling clusters part of the fed‐
435 eration will be listed.
436 shutdown OPTION
439 Instruct Slurm daemons to save current state and terminate. By
440 default, the Slurm controller (slurmctld) forwards the request
441 all other daemons (slurmd daemon on each compute node). An
442 OPTION of slurmctld or controller results in only the slurmctld
443 daemon being shutdown and the slurmd daemons remaining active.
444 suspend job_list
447 Suspend a running job. The job_list argument is a comma sepa‐
448 rated list of job IDs. Use the resume command to resume its
449 execution. User processes must stop on receipt of SIGSTOP sig‐
450 nal and resume upon receipt of SIGCONT for this operation to be
451 effective. Not all architectures and configurations support job
452 suspension. If a suspended job is requeued, it will be placed
453 in a held state. The time a job is suspended will not count
454 against a job's time limit. Only an operator, administrator,
455 SlurmUser, or root can suspend jobs.
456 takeover
459 Instruct Slurm's backup controller (slurmctld) to take over sys‐
460 tem control. Slurm's backup controller requests control from
461 the primary and waits for its termination. After that, it
462 switches from backup mode to controller mode. If primary con‐
463 troller can not be contacted, it directly switches to controller
464 mode. This can be used to speed up the Slurm controller
465 fail-over mechanism when the primary node is down. This can be
466 used to minimize disruption if the computer executing the pri‐
467 mary Slurm controller is scheduled down. (Note: Slurm's primary
468 controller will take the control back at startup.)
469 top job_list
472 Move the specified job IDs to the top of the queue of jobs
473 belonging to the identical user ID, partition name, account, and
474 QOS. The job_list argument is a comma separated ordered list of
475 job IDs. Any job not matching all of those fields will not be
476 effected. Only jobs submitted to a single partition will be
477 effected. This operation changes the order of jobs by adjusting
478 job nice values. The net effect on that user's throughput will
479 be negligible to slightly negative. This operation is disabled
480 by default for non-privileged (non-operator, admin, SlurmUser,
481 or root) users. This operation may be enabled for non-privileged
482 users by the system administrator by including the option
483 "enable_user_top" in the SchedulerParameters configuration
484 parameter.
485 uhold job_list
488 Prevent a pending job from being started (sets it's priority to
489 0). The job_list argument is a space separated list of job IDs
490 or job names. Use the release command to permit the job to be
491 scheduled. This command is designed for a system administrator
492 to hold a job so that the job owner may release it rather than
493 requiring the intervention of a system administrator (also see
494 the hold command).
495 update SPECIFICATION
498 Update job, step, node, partition, powercapping or reservation
499 configuration per the supplied specification. SPECIFICATION is
500 in the same format as the Slurm configuration file and the out‐
501 put of the show command described above. It may be desirable to
502 execute the show command (described above) on the specific
503 entity you want to update, then use cut-and-paste tools to enter
504 updated configuration values to the update. Note that while most
505 configuration values can be changed using this command, not all
506 can be changed using this mechanism. In particular, the hardware
507 configuration of a node or the physical addition or removal of
508 nodes from the cluster may only be accomplished through editing
509 the Slurm configuration file and executing the reconfigure com‐
510 mand (described above).
511 version
514 Display the version number of scontrol being executed.
515 wait_job job_id
518 Wait until a job and all of its nodes are ready for use or the
519 job has entered some termination state. This option is particu‐
520 larly useful in the Slurm Prolog or in the batch script itself
521 if nodes are powered down and restarted automatically as needed.
522 NOTE: Don't use scontrol wait_job in PrologCtld or Prolog with
524 PrologFlags=Alloc as this will result in deadlock.
525 write batch_script job_id optional_filename
528 Write the batch script for a given job_id to a file or to std‐
529 out. The file will default to slurm-<job_id>.sh if the optional
530 filename argument is not given. The script will be written to
531 stdout if - is given instead of a filename. The batch script
532 can only be retrieved by an admin or operator, or by the owner
533 of the job.
534 write config
537 Write the current configuration to a file with the naming con‐
538 vention of "slurm.conf.<datetime>" in the same directory as the
539 original slurm.conf file.
540 INTERACTIVE COMMANDS
543 NOTE: All commands listed below can be used in the interactive
544 mode, but NOT on the initial command line.
545 all Show all partitions, their jobs and jobs steps. This causes
548 information to be displayed about partitions that are configured
549 as hidden and partitions that are unavailable to user's group.
550 details
553 Causes the show command to provide additional details where
554 available. Job information will include CPUs and NUMA memory
555 allocated on each node. Note that on computers with hyper‐
556 threading enabled and Slurm configured to allocate cores, each
557 listed CPU represents one physical core. Each hyperthread on
558 that core can be allocated a separate task, so a job's CPU count
559 and task count may differ. See the --cpu-bind and --mem-bind
560 option descriptions in srun man pages for more information. The
561 details option is currently only supported for the show job com‐
562 mand.
563 exit Terminate scontrol interactive session.
566 hide Do not display partition, job or jobs step information for par‐
569 titions that are configured as hidden or partitions that are
570 unavailable to the user's group. This is the default behavior.
571 oneliner
574 Print information one line per record.
575 quiet Print no warning or informational messages, only fatal error
578 messages.
579 quit Terminate the execution of scontrol.
582 verbose
585 Print detailed event logging. This includes time-stamps on data
586 structures, record counts, etc.
587 !! Repeat the last command executed.
590 SPECIFICATIONS FOR UPDATE COMMAND, JOBS
593 Note that update requests done by either root, SlurmUser or Administra‐
595 tors are not subject to certain restrictions. For instance, if an
596 Administrator changes the QOS on a pending job, certain limits such as
597 the TimeLimit will not be changed automatically as changes made by the
598 Administrators are allowed to violate these restrictions.
599 Account=<account>
602 Account name to be changed for this job's resource use. Value
603 may be cleared with blank data value, "Account=".
604 AdminComment=<spec>
606 Arbitrary descriptive string. Can only be set by a Slurm admin‐
607 istrator.
608 ArrayTaskThrottle=<count>
610 Specify the maximum number of tasks in a job array that can exe‐
611 cute at the same time. Set the count to zero in order to elimi‐
612 nate any limit. The task throttle count for a job array is
613 reported as part of its ArrayTaskId field, preceded with a per‐
614 cent sign. For example "ArrayTaskId=1-10%2" indicates the maxi‐
615 mum number of running tasks is limited to 2.
616 BurstBuffer=<spec>
618 Burst buffer specification to be changed for this job's resource
619 use. Value may be cleared with blank data value, "Burst‐
620 Buffer=". Format is burst buffer plugin specific.
621 Clusters=<spec>
623 Specifies the clusters that the federated job can run on.
624 ClusterFeatures=<spec>
626 Specifies features that a federated cluster must have to have a
627 sibling job submitted to it. Slurm will attempt to submit a sib‐
628 ling job to a cluster if it has at least one of the specified
629 features.
630 Comment=<spec>
632 Arbitrary descriptive string.
633 Contiguous=<yes|no>
635 Set the job's requirement for contiguous (consecutive) nodes to
636 be allocated. Possible values are "YES" and "NO". Only the
637 Slurm administrator or root can change this parameter.
638 CoreSpec=<count>
640 Number of cores to reserve per node for system use. The job
641 will be charged for these cores, but be unable to use them.
642 Will be reported as "*" if not constrained.
643 CPUsPerTask=<count>
645 Change the CPUsPerTask job's value.
646 Deadline=<time_spec>
648 It accepts times of the form HH:MM:SS to specify a deadline to a
649 job at a specific time of day (seconds are optional). You may
650 also specify midnight, noon, fika (3 PM) or teatime (4 PM) and
651 you can have a time-of-day suffixed with AM or PM for a deadline
652 in the morning or the evening. You can specify a deadline for
653 the job with a date of the form MMDDYY or MM/DD/YY or MM.DD.YY,
654 or a date and time as YYYY-MM-DD[THH:MM[:SS]]. You can also
655 give times like now + count time-units, where the time-units can
656 be minutes, hours, days, or weeks and you can tell Slurm to put
657 a deadline for tomorrow with the keyword tomorrow. The speci‐
658 fied deadline must be later than the current time. Only pending
659 jobs can have the deadline updated. Only the Slurm administra‐
660 tor or root can change this parameter.
661 DelayBoot=<time_spec>
663 Change the time to decide whether to reboot nodes in order to
664 satisfy job's feature specification if the job has been eligible
665 to run for less than this time period. See salloc/sbatch man
666 pages option --delay-boot.
667 Dependency=<dependency_list>
669 Defer job's initiation until specified job dependency specifica‐
670 tion is satisfied. Cancel dependency with an empty depen‐
671 dency_list (e.g. "Dependency="). <dependency_list> is of the
672 form <type:job_id[:job_id][,type:job_id[:job_id]]>. Many jobs
673 can share the same dependency and these jobs may even belong to
674 different users.
675 after:job_id[:jobid...]
677 This job can begin execution after the specified jobs
678 have begun execution.
679 afterany:job_id[:jobid...]
681 This job can begin execution after the specified jobs
682 have terminated.
683 afternotok:job_id[:jobid...]
685 This job can begin execution after the specified jobs
686 have terminated in some failed state (non-zero exit code,
687 node failure, timed out, etc).
688 afterok:job_id[:jobid...]
690 This job can begin execution after the specified jobs
691 have successfully executed (ran to completion with an
692 exit code of zero).
693 singleton
695 This job can begin execution after any previously
696 launched jobs sharing the same job name and user have
697 terminated. In other words, only one job by that name
698 and owned by that user can be running or suspended at any
699 point in time.
700 EligibleTime=<time_spec>
702 See StartTime.
703 EndTime
705 The time the job is expected to terminate based on the job's
706 time limit. When the job ends sooner, this field will be
707 updated with the actual end time.
708 ExcNodeList=<nodes>
710 Set the job's list of excluded node. Multiple node names may be
711 specified using simple node range expressions (e.g.
712 "lx[10-20]"). Value may be cleared with blank data value, "Exc‐
713 NodeList=".
714 Features=<features>
716 Set the job's required node features. The list of features may
717 include multiple feature names separated by ampersand (AND)
718 and/or vertical bar (OR) operators. For example: Fea‐
719 tures="opteron&video" or Features="fast|faster". In the first
720 example, only nodes having both the feature "opteron" AND the
721 feature "video" will be used. There is no mechanism to specify
722 that you want one node with feature "opteron" and another node
723 with feature "video" in case no node has both features. If only
724 one of a set of possible options should be used for all allo‐
725 cated nodes, then use the OR operator and enclose the options
726 within square brackets. For example: "Fea‐
727 tures=[rack1|rack2|rack3|rack4]" might be used to specify that
728 all nodes must be allocated on a single rack of the cluster, but
729 any of those four racks can be used. A request can also specify
730 the number of nodes needed with some feature by appending an
731 asterisk and count after the feature name. For example "Fea‐
732 tures=graphics*4" indicates that at least four allocated nodes
733 must have the feature "graphics." Parenthesis are also sup‐
734 ported for features to be ANDed together. For example "Fea‐
735 tures=[(knl&a2a&flat)*4&haswell*2]" indicates the resource allo‐
736 cation should include 4 nodes with ALL of the features "knl",
737 "a2a", and "flat" plus 2 nodes with the feature "haswell". Con‐
738 straints with node counts may only be combined with AND opera‐
739 tors. Value may be cleared with blank data value, for example
740 "Features=".
741 Gres=<list>
744 Specifies a comma delimited list of generic consumable
745 resources. The format of each entry on the list is
746 "name[:count[*cpu]]". The name is that of the consumable
747 resource. The count is the number of those resources with a
748 default value of 1. The specified resources will be allocated
749 to the job on each node allocated unless "*cpu" is appended, in
750 which case the resources will be allocated on a per cpu basis.
751 The available generic consumable resources is configurable by
752 the system administrator. A list of available generic consum‐
753 able resources will be printed and the command will exit if the
754 option argument is "help". Examples of use include
755 "Gres=gpus:2*cpu,disk=40G" and "Gres=help".
756 JobId=<job_list>
759 Identify the job(s) to be updated. The job_list may be a comma
760 separated list of job IDs. Either JobId or JobName is required.
761 Licenses=<name>
763 Specification of licenses (or other resources available on all
764 nodes of the cluster) as described in salloc/sbatch/srun man
765 pages.
766 MinCPUsNode=<count>
768 Set the job's minimum number of CPUs per node to the specified
769 value.
770 MinMemoryCPU=<megabytes>
772 Set the job's minimum real memory required per allocated CPU to
773 the specified value. Either MinMemoryCPU or MinMemoryNode may be
774 set, but not both.
775 MinMemoryNode=<megabytes>
777 Set the job's minimum real memory required per node to the spec‐
778 ified value. Either MinMemoryCPU or MinMemoryNode may be set,
779 but not both.
780 MinTmpDiskNode=<megabytes>
782 Set the job's minimum temporary disk space required per node to
783 the specified value. Only the Slurm administrator or root can
784 change this parameter.
785 imeMin=<timespec>
787 Change TimeMin value which specifies the minimum time limit min‐
788 utes of the job.
789 JobName=<name>
791 Identify the name of jobs to be modified or set the job's name
792 to the specified value. When used to identify jobs to be modi‐
793 fied, all jobs belonging to all users are modified unless the
794 UserID option is used to identify a specific user. Either JobId
795 or JobName is required.
796 Name[=<name>]
798 See JobName.
799 Nice[=<adjustment>]
801 Update the job with an adjusted scheduling priority within
802 Slurm. With no adjustment value the scheduling priority is
803 decreased by 100. A negative nice value increases the priority,
804 otherwise decreases it. The adjustment range is +/- 2147483645.
805 Only privileged users can specify a negative adjustment.
806 NodeList=<nodes>
808 Change the nodes allocated to a running job to shrink it's size.
809 The specified list of nodes must be a subset of the nodes cur‐
810 rently allocated to the job. Multiple node names may be speci‐
811 fied using simple node range expressions (e.g. "lx[10-20]").
812 After a job's allocation is reduced, subsequent srun commands
813 must explicitly specify node and task counts which are valid for
814 the new allocation.
815 NumCPUs=<min_count>[-<max_count>]
817 Set the job's minimum and optionally maximum count of CPUs to be
818 allocated.
819 NumNodes=<min_count>[-<max_count>]
821 Set the job's minimum and optionally maximum count of nodes to
822 be allocated. If the job is already running, use this to spec‐
823 ify a node count less than currently allocated and resources
824 previously allocated to the job will be relinquished. After a
825 job's allocation is reduced, subsequent srun commands must
826 explicitly specify node and task counts which are valid for the
827 new allocation. Also see the NodeList parameter above. This is
828 the same than ReqNodes.
829 NumTasks=<count>
831 Set the job's count of required tasks to the specified value.
832 This is the same than ReqProcs.
833 OverSubscribe=<yes|no>
835 Set the job's ability to share compute resources (i.e. individ‐
836 ual CPUs) with other jobs. Possible values are "YES" and "NO".
837 This option can only be changed for pending jobs.
838 Partition=<name>
840 Set the job's partition to the specified value.
841 Priority=<number>
843 Set the job's priority to the specified value. Note that a job
844 priority of zero prevents the job from ever being scheduled. By
845 setting a job's priority to zero it is held. Set the priority
846 to a non-zero value to permit it to run. Explicitly setting a
847 job's priority clears any previously set nice value and removes
848 the priority/multifactor plugin's ability to manage a job's pri‐
849 ority. In order to restore the priority/multifactor plugin's
850 ability to manage a job's priority, hold and then release the
851 job. Only the Slurm administrator or root can increase job's
852 priority.
853 QOS=<name>
855 Set the job's QOS (Quality Of Service) to the specified value.
856 Value may be cleared with blank data value, "QOS=".
857 Reboot=<yes|no>
859 Set the job's flag that specifies whether to force the allocated
860 nodes to reboot before starting the job. This is only supported
861 with some system configurations and therefore it could be
862 silently ignored.
863 ReqCores=<count>
865 Change the job's requested Cores count.
866 ReqNodeList=<nodes>
868 Set the job's list of required node. Multiple node names may be
869 specified using simple node range expressions (e.g.
870 "lx[10-20]"). Value may be cleared with blank data value,
871 "ReqNodeList=".
872 ReqNodes=<min_count>[-<max_count>]
874 See NumNodes.
875 ReqProcs=<count>
877 See NumTasks.
878 ReqSockets=<count>
880 Change the job's requested socket count.
881 ReqThreads=<count>
883 Change the job's requested threads count.
884 Requeue=<0|1>
886 Stipulates whether a job should be requeued after a node fail‐
887 ure: 0 for no, 1 for yes.
888 ReservationName=<name>
890 Set the job's reservation to the specified value. Value may be
891 cleared with blank data value, "ReservationName=".
892 ResetAccrueTime
894 Reset the job's accrue time value to 0 meaning it will loose any
895 time previously accrued for priority. Helpful if you have a
896 large queue of jobs already in the queue and want to start lim‐
897 iting how many jobs can accrue time without waiting for the
898 queue to flush out.
899 SiteFactor=<account>
901 Specify the job's admin priority factor in the range of
902 +/-2147483645. Only privileged users can modify the value.
903 StdOut=<filepath>
905 Set the batch job's stdout file path.
906 Shared=<yes|no>
908 See OverSubscribe option above.
909 StartTime=<time_spec>
911 Set the job's earliest initiation time. It accepts times of the
912 form HH:MM:SS to run a job at a specific time of day (seconds
913 are optional). (If that time is already past, the next day is
914 assumed.) You may also specify midnight, noon, fika (3 PM) or
915 teatime (4 PM) and you can have a time-of-day suffixed with AM
916 or PM for running in the morning or the evening. You can also
917 say what day the job will be run, by specifying a date of the
918 form MMDDYY or MM/DD/YY or MM.DD.YY, or a date and time as
919 YYYY-MM-DD[THH:MM[:SS]]. You can also give times like now +
920 count time-units, where the time-units can be minutes, hours,
921 days, or weeks and you can tell Slurm to run the job today with
922 the keyword today and to run the job tomorrow with the keyword
923 tomorrow.
924 Notes on date/time specifications:
926 - although the 'seconds' field of the HH:MM:SS time specifica‐
927 tion is allowed by the code, note that the poll time of the
928 Slurm scheduler is not precise enough to guarantee dispatch of
929 the job on the exact second. The job will be eligible to start
930 on the next poll following the specified time. The exact poll
931 interval depends on the Slurm scheduler (e.g., 60 seconds with
932 the default sched/builtin).
933 - if no time (HH:MM:SS) is specified, the default is
934 (00:00:00).
935 - if a date is specified without a year (e.g., MM/DD) then the
936 current year is assumed, unless the combination of MM/DD and
937 HH:MM:SS has already passed for that year, in which case the
938 next year is used.
939 Switches=<count>[@<max-time-to-wait>]
941 When a tree topology is used, this defines the maximum count of
942 switches desired for the job allocation. If Slurm finds an allo‐
943 cation containing more switches than the count specified, the
944 job remain pending until it either finds an allocation with
945 desired switch count or the time limit expires. By default there
946 is no switch count limit and no time limit delay. Set the count
947 to zero in order to clean any previously set count (disabling
948 the limit). The job's maximum time delay may be limited by the
949 system administrator using the SchedulerParameters configuration
950 parameter with the max_switch_wait parameter option. Also see
951 wait-for-switch.
952 wait-for-switch=<seconds>
955 Change max time to wait for a switch <seconds> secs.
956 TasksPerNode=<count>
959 Change the job's requested TasksPerNode.
960 ThreadSpec=<count>
963 Number of threads to reserve per node for system use. The job
964 will be charged for these threads, but be unable to use them.
965 Will be reported as "*" if not constrained.
966 TimeLimit=<time>
969 The job's time limit. Output format is [days-]hours:min‐
970 utes:seconds or "UNLIMITED". Input format (for update command)
971 set is minutes, minutes:seconds, hours:minutes:seconds,
972 days-hours, days-hours:minutes or days-hours:minutes:seconds.
973 Time resolution is one minute and second values are rounded up
974 to the next minute. If changing the time limit of a job, either
975 specify a new time limit value or precede the time and equal
976 sign with a "+" or "-" to increment or decrement the current
977 time limit (e.g. "TimeLimit+=30"). In order to increment or
978 decrement the current time limit, the JobId specification must
979 precede the TimeLimit specification. Note that incrementing or
980 decrementing the time limit for a job array is only allowed
981 before the job array has been split into more than one job
982 record. Only the Slurm administrator or root can increase job's
983 TimeLimit.
984 UserID=<UID or name>
987 Used with the JobName option to identify jobs to be modified.
988 Either a user name or numeric ID (UID), may be specified.
989 WCKey=<key>
992 Set the job's workload characterization key to the specified
993 value.
994 WorkDir=<directory_name>
997 Set the job's working directory to the specified value. Note
998 that this may only be set for jobs in the PENDING state, and
999 that jobs may fail to launch if they rely on relative paths to
1000 the originally submitted WorkDir.
1001 SPECIFICATIONS FOR SHOW COMMAND, JOBS
1004 The "show" command, when used with the "job" or "job <jobid>" entity
1006 displays detailed information about a job or jobs. Much of this infor‐
1007 mation may be modified using the "update job" command as described
1008 above. However, the following fields displayed by the show job command
1009 are read-only and cannot be modified:
1010 AllocNode:Sid
1013 Local node and system id making the resource allocation.
1014 BatchFlag
1016 Jobs submitted using the sbatch command have BatchFlag set to 1.
1017 Jobs submitted using other commands have BatchFlag set to 0.
1018 ExitCode=<exit>:<sig>
1020 Exit status reported for the job by the wait() function. The
1021 first number is the exit code, typically as set by the exit()
1022 function. The second number of the signal that caused the
1023 process to terminate if it was terminated by a signal.
1024 GroupId
1026 The group under which the job was submitted.
1027 JobState
1029 The current state of the job.
1030 NodeListIndices
1032 The NodeIndices expose the internal indices into the node table
1033 associated with the node(s) allocated to the job.
1034 NtasksPerN:B:S:C=
1036 <tasks_per_node>:<tasks_per_base‐
1037 board>:<tasks_per_socket>:<tasks_per_core> Specifies the number
1038 of tasks to be started per hardware component (node, baseboard,
1039 socket and core). Unconstrained values may be shown as "0" or
1040 "*".
1041 PreemptEligibleTime
1043 Time the job becomes eligible for preemption. Modified by Pre‐
1044 emptExemptTime, either from the global option in slurm.conf or
1045 the job QOS. This is hidden if the job has not started or if
1046 PreemptMode=OFF.
1047 PreemptTime
1049 Time at which job was signaled that it was selected for preemp‐
1050 tion. (Meaningful only for PreemptMode=CANCEL and the partition
1051 or QOS with which the job is associated has a GraceTime value
1052 designated.) This is hidden if the job has not started or if
1053 PreemptMode=OFF.
1054 PreSusTime
1056 Time the job ran prior to last suspend.
1057 Reason The reason job is not running: e.g., waiting "Resources".
1059 ReqB:S:C:T=
1061 <baseboard_count>:<socket_per_base‐
1062 board_count>:<core_per_socket_count>:<thread_per_core_count>
1063 Specifies the count of various hardware components requested by
1064 the job. Unconstrained values may be shown as "0" or "*".
1065 SecsPreSuspend=<seconds>
1067 If the job is suspended, this is the run time accumulated by the
1068 job (in seconds) prior to being suspended.
1069 Socks/Node=<count>
1071 Count of desired sockets per node
1072 SubmitTime
1074 The time and date stamp (in localtime) the job was submitted.
1075 The format of the output is identical to that of the EndTime
1076 field.
1077 NOTE: If a job is requeued, the submit time is reset. To obtain
1079 the original submit time it is necessary to use the "sacct -j
1080 <job_id[.<step_id>]" command also designating the -D or --dupli‐
1081 cate option to display all duplicate entries for a job.
1082 SuspendTime
1084 Time the job was last suspended or resumed.
1085 NOTE on information displayed for various job states:
1087 When you submit a request for the "show job" function the scon‐
1088 trol process makes an RPC request call to slurmctld with a
1089 REQUEST_JOB_INFO message type. If the state of the job is PEND‐
1090 ING, then it returns some detail information such as: min_nodes,
1091 min_procs, cpus_per_task, etc. If the state is other than PEND‐
1092 ING the code assumes that it is in a further state such as RUN‐
1093 NING, COMPLETE, etc. In these cases the code explicitly returns
1094 zero for these values. These values are meaningless once the job
1095 resources have been allocated and the job has started.
1096 SPECIFICATIONS FOR UPDATE COMMAND, STEPS
1099 StepId=<job_id>[.<step_id>]
1101 Identify the step to be updated. If the job_id is given, but no
1102 step_id is specified then all steps of the identified job will
1103 be modified. This specification is required.
1104 CompFile=<completion file>
1106 Update a step with information about a steps completion. Can be
1107 useful if step statistics aren't directly available through a
1108 jobacct_gather plugin. The file is a space-delimited file with
1109 format for Version 1 is as follows
1110 1 34461 0 2 0 3 1361906011 1361906015 1 1 3368 13357 /bin/sleep
1112 A B C D E F G H I J K L M
1113 Field Descriptions:
1115 A file version
1117 B ALPS apid
1118 C inblocks
1119 D outblocks
1120 E exit status
1121 F number of allocated CPUs
1122 G start time
1123 H end time
1124 I utime
1125 J stime
1126 K maxrss
1127 L uid
1128 M command name
1129 TimeLimit=<time>
1131 The job's time limit. Output format is [days-]hours:min‐
1132 utes:seconds or "UNLIMITED". Input format (for update command)
1133 set is minutes, minutes:seconds, hours:minutes:seconds,
1134 days-hours, days-hours:minutes or days-hours:minutes:seconds.
1135 Time resolution is one minute and second values are rounded up
1136 to the next minute. If changing the time limit of a step,
1137 either specify a new time limit value or precede the time with a
1138 "+" or "-" to increment or decrement the current time limit
1139 (e.g. "TimeLimit=+30"). In order to increment or decrement the
1140 current time limit, the StepId specification must precede the
1141 TimeLimit specification.
1142 SPECIFICATIONS FOR UPDATE COMMAND, NODES
1145 NodeName=<name>
1147 Identify the node(s) to be updated. Multiple node names may be
1148 specified using simple node range expressions (e.g.
1149 "lx[10-20]"). This specification is required.
1150 ActiveFeatures=<features>
1153 Identify the feature(s) currently active on the specified node.
1154 Any previously active feature specification will be overwritten
1155 with the new value. Also see AvailableFeatures. Typically
1156 ActiveFeatures will be identical to AvailableFeatures; however
1157 ActiveFeatures may be configured as a subset of the Available‐
1158 Features. For example, a node may be booted in multiple configu‐
1159 rations. In that case, all possible configurations may be iden‐
1160 tified as AvailableFeatures, while ActiveFeatures would identify
1161 the current node configuration.
1162 AvailableFeatures=<features>
1165 Identify the feature(s) available on the specified node. Any
1166 previously defined available feature specification will be over‐
1167 written with the new value. AvailableFeatures assigned via
1168 scontrol will only persist across the restart of the slurmctld
1169 daemon with the -R option and state files preserved or slurm‐
1170 ctld's receipt of a SIGHUP. Update slurm.conf with any changes
1171 meant to be persistent across normal restarts of slurmctld or
1172 the execution of scontrol reconfig. Also see ActiveFeatures.
1173 CpuBind=<node>
1176 Specify the task binding mode to be used by default for this
1177 node. Supported options include: "none", "board", "socket",
1178 "ldom" (NUMA), "core", "thread" and "off" (remove previous bind‐
1179 ing mode).
1180 Gres=<gres>
1183 Identify generic resources to be associated with the specified
1184 node. Any previously defined generic resources will be over‐
1185 written with the new value. Specifications for multiple generic
1186 resources should be comma separated. Each resource specifica‐
1187 tion consists of a name followed by an optional colon with a
1188 numeric value (default value is one) (e.g. "Gres=band‐
1189 width:10000"). Modification of GRES count associated with spe‐
1190 cific files (e.g. GPUs) is not allowed other than to set their
1191 count on a node to zero. In order to change the GRES count to
1192 another value, modify your slurm.conf and gres.conf files and
1193 restart daemons. If GRES as associated with specific sockets,
1194 that information will be reported For example if all 4 GPUs on a
1195 node are all associated with socket zero, then
1196 "Gres=gpu:4(S:0)". If associated with sockets 0 and 1 then
1197 "Gres=gpu:4(S:0-1)". The information of which specific GPUs are
1198 associated with specific GPUs is not reported, but only avail‐
1199 able by parsing the gres.conf file. Generic resources assigned
1200 via scontrol will only persist across the restart of the slurm‐
1201 ctld daemon with the -R option and state files preserved or
1202 slurmctld's receipt of a SIGHUP. Update slurm.conf with any
1203 changes meant to be persistent across normal restarts of slurm‐
1204 ctld or the execution of scontrol reconfig.
1205 Reason=<reason>
1208 Identify the reason the node is in a "DOWN", "DRAINED", "DRAIN‐
1209 ING", "FAILING" or "FAIL" state. Use quotes to enclose a reason
1210 having more than one word.
1211 State=<state>
1214 Identify the state to be assigned to the node. Possible node
1215 states are "NoResp", "ALLOC", "ALLOCATED", "COMPLETING", "DOWN",
1216 "DRAIN", "FAIL", "FAILING", "FUTURE" "IDLE", "MAINT", "MIXED",
1217 "PERFCTRS/NPC", "RESERVED", "POWER_DOWN", "POWER_UP", "RESUME"
1218 or "UNDRAIN". Not all of those states can be set using the scon‐
1219 trol command only the following can: "CANCEL_REBOOT", "DOWN",
1220 "DRAIN", "FAIL", "FUTURE", "RESUME", "NoResp", "POWER_DOWN",
1221 "POWER_UP" and "UNDRAIN". If a node is in a "MIXED" state it
1222 usually means the node is in multiple states. For instance if
1223 only part of the node is "ALLOCATED" and the rest of the node is
1224 "IDLE" the state will be "MIXED". If you want to remove a node
1225 from service, you typically want to set it's state to "DRAIN".
1226 "CANCEL_REBOOT" cancels a pending reboot on the node (same as
1227 "scontrol cancel_reboot <node>"). "FAILING" is similar to
1228 "DRAIN" except that some applications will seek to relinquish
1229 those nodes before the job completes. "PERFCTRS/NPC" indicates
1230 that Network Performance Counters associated with this node are
1231 in use, rendering this node as not usable for any other jobs.
1232 "RESERVED" indicates the node is in an advanced reservation and
1233 not generally available. "RESUME" is not an actual node state,
1234 but will change a node state from "DRAINED", "DRAINING", "DOWN"
1235 or "REBOOT" to either "IDLE" or "ALLOCATED" state as appropri‐
1236 ate. "RESUME" will also clear the "POWERING_DOWN" state of a
1237 node and make it eligible to be allocted. "UNDRAIN" clears the
1238 node from being drained (like "RESUME"), but will not change the
1239 node's base state (e.g. "DOWN"). Setting a node "DOWN" will
1240 cause all running and suspended jobs on that node to be termi‐
1241 nated. "POWER_DOWN" and "POWER_UP" will use the configured Sus‐
1242 pendProg and ResumeProg programs to explicitly place a node in
1243 or out of a power saving mode. If a node is already in the
1244 process of being powered up or down, the command will only
1245 change the state of the node but won't have any effect until the
1246 configured ResumeTimeout or SuspendTimeout is reached. Use of
1247 this command can be useful in situations where a ResumeProg like
1248 capmc in Cray machines is stalled and one wants to restore the
1249 node to "IDLE" manually, in this case rebooting the node and
1250 setting the state to "POWER_DOWN" will cancel the previous
1251 "POWER_UP" state and the node will become "IDLE". The "NoResp"
1252 state will only set the "NoResp" flag for a node without chang‐
1253 ing its underlying state. While all of the above states are
1254 valid, some of them are not valid new node states given their
1255 prior state. If the node state code printed is followed by "~",
1256 this indicates the node is presently in a power saving mode
1257 (typically running at reduced frequency). If the node state
1258 code is followed by "#", this indicates the node is presently
1259 being powered up or configured. If the node state code is fol‐
1260 lowed by "$", this indicates the node is currently in a reserva‐
1261 tion with a flag value of "maintenance". If the node state code
1262 is followed by "@", this indicates the node is currently sched‐
1263 uled to be rebooted. Generally only "DRAIN", "FAIL" and
1264 "RESUME" should be used. NOTE: The scontrol command should not
1265 be used to change node state on Cray systems. Use Cray tools
1266 such as xtprocadmin instead.
1267 Weight=<weight>
1270 Identify weight to be associated with specified nodes. This
1271 allows dynamic changes to weight associated with nodes, which
1272 will be used for the subsequent node allocation decisions.
1273 Weight assigned via scontrol will only persist across the
1274 restart of the slurmctld daemon with the -R option and state
1275 files preserved or slurmctld's receipt of a SIGHUP. Update
1276 slurm.conf with any changes meant to be persistent across normal
1277 restarts of slurmctld or the execution of scontrol reconfig.
1278 SPECIFICATIONS FOR SHOW COMMAND, NODES
1281 The meaning of the energy information is as follows:
1283 CurrentWatts
1286 The instantaneous power consumption of the node at the time of
1287 the last node energy accounting sample, in watts.
1288 LowestJoules
1291 The energy consumed by the node between the last time it was
1292 powered on and the last time it was registered by slurmd, in
1293 joules.
1294 ConsumedJoules
1297 The energy consumed by the node between the last time it was
1298 registered by the slurmd daemon and the last node energy
1299 accounting sample, in joules.
1300 If the reported value is "n/s" (not supported), the node does not sup‐
1303 port the configured AcctGatherEnergyType plugin. If the reported value
1304 is zero, energy accounting for nodes is disabled.
1305 The meaning of the external sensors information is as follows:
1308 ExtSensorsJoules
1311 The energy consumed by the node between the last time it was
1312 powered on and the last external sensors plugin node sample, in
1313 joules.
1314 ExtSensorsWatts
1318 The instantaneous power consumption of the node at the time of
1319 the last external sensors plugin node sample, in watts.
1320 ExtSensorsTemp
1323 The temperature of the node at the time of the last external
1324 sensors plugin node sample, in celsius.
1325 If the reported value is "n/s" (not supported), the node does not sup‐
1328 port the configured ExtSensorsType plugin.
1329 The meaning of the resource specialization information is as follows:
1332 CPUSpecList
1335 The list of Slurm abstract CPU IDs on this node reserved for
1336 exclusive use by the Slurm compute node daemons (slurmd, slurm‐
1337 stepd).
1338 MemSpecLimit
1341 The combined memory limit, in megabytes, on this node for the
1342 Slurm compute node daemons (slurmd, slurmstepd).
1343 The meaning of the memory information is as follows:
1346 RealMemory
1349 The total memory, in MB, on the node.
1350 AllocMem
1353 The total memory, in MB, currently allocated by jobs on the
1354 node.
1355 FreeMem
1358 The total memory, in MB, currently free on the node as reported
1359 by the OS.
1360 SPECIFICATIONS FOR UPDATE COMMAND, FRONTEND
1363 FrontendName=<name>
1366 Identify the front end node to be updated. This specification is
1367 required.
1368 Reason=<reason>
1371 Identify the reason the node is in a "DOWN" or "DRAIN" state.
1372 Use quotes to enclose a reason having more than one word.
1373 State=<state>
1376 Identify the state to be assigned to the front end node. Possi‐
1377 ble values are "DOWN", "DRAIN" or "RESUME". If you want to
1378 remove a front end node from service, you typically want to set
1379 it's state to "DRAIN". "RESUME" is not an actual node state,
1380 but will return a "DRAINED", "DRAINING", or "DOWN" front end
1381 node to service, either "IDLE" or "ALLOCATED" state as appropri‐
1382 ate. Setting a front end node "DOWN" will cause all running and
1383 suspended jobs on that node to be terminated.
1384 SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, PARTITIONS
1387 AllowGroups=<name>
1389 Identify the user groups which may use this partition. Multiple
1390 groups may be specified in a comma separated list. To permit
1391 all groups to use the partition specify "AllowGroups=ALL".
1392 AllocNodes=<name>
1395 Comma separated list of nodes from which users can execute jobs
1396 in the partition. Node names may be specified using the node
1397 range expression syntax described above. The default value is
1398 "ALL".
1399 Alternate=<partition name>
1402 Alternate partition to be used if the state of this partition is
1403 "DRAIN" or "INACTIVE." The value "NONE" will clear a previously
1404 set alternate partition.
1405 CpuBind=<node>
1408 Specify the task binding mode to be used by default for this
1409 partition. Supported options include: "none", "board",
1410 "socket", "ldom" (NUMA), "core", "thread" and "off" (remove pre‐
1411 vious binding mode).
1412 Default=<yes|no>
1415 Specify if this partition is to be used by jobs which do not
1416 explicitly identify a partition to use. Possible output values
1417 are "YES" and "NO". In order to change the default partition of
1418 a running system, use the scontrol update command and set
1419 Default=yes for the partition that you want to become the new
1420 default.
1421 DefaultTime=<time>
1424 Run time limit used for jobs that don't specify a value. If not
1425 set then MaxTime will be used. Format is the same as for Max‐
1426 Time.
1427 DefMemPerCPU=<MB>
1430 Set the default memory to be allocated per CPU for jobs in this
1431 partition. The memory size is specified in megabytes.
1432 DefMemPerNode=<MB>
1434 Set the default memory to be allocated per node for jobs in this
1435 partition. The memory size is specified in megabytes.
1436 DisableRootJobs=<yes|no>
1439 Specify if jobs can be executed as user root. Possible values
1440 are "YES" and "NO".
1441 GraceTime=<seconds>
1444 Specifies, in units of seconds, the preemption grace time to be
1445 extended to a job which has been selected for preemption. The
1446 default value is zero, no preemption grace time is allowed on
1447 this partition or qos. (Meaningful only for PreemptMode=CANCEL)
1448 Hidden=<yes|no>
1451 Specify if the partition and its jobs should be hidden from
1452 view. Hidden partitions will by default not be reported by
1453 Slurm APIs or commands. Possible values are "YES" and "NO".
1454 JobDefaults=<specs>
1457 Specify job default values using a comma delimited list of
1458 "key=value" pairs. Supported keys include
1459 DefCpuPerGPU Default number of CPUs per allocated GPU.
1461 DefMemPerGPU Default memory limit (in megabytes) per allocated
1463 GPU.
1464 MaxMemPerCPU=<MB>
1467 Set the maximum memory to be allocated per CPU for jobs in this
1468 partition. The memory size is specified in megabytes.
1469 MaxMemPerCNode=<MB>
1471 Set the maximum memory to be allocated per node for jobs in this
1472 partition. The memory size is specified in megabytes.
1473 MaxNodes=<count>
1476 Set the maximum number of nodes which will be allocated to any
1477 single job in the partition. Specify a number, "INFINITE" or
1478 "UNLIMITED". Changing the MaxNodes of a partition has no effect
1479 upon jobs that have already begun execution.
1480 MaxTime=<time>
1483 The maximum run time for jobs. Output format is
1484 [days-]hours:minutes:seconds or "UNLIMITED". Input format (for
1485 update command) is minutes, minutes:seconds, hours:minutes:sec‐
1486 onds, days-hours, days-hours:minutes or days-hours:minutes:sec‐
1487 onds. Time resolution is one minute and second values are
1488 rounded up to the next minute. Changing the MaxTime of a parti‐
1489 tion has no effect upon jobs that have already begun execution.
1490 MinNodes=<count>
1493 Set the minimum number of nodes which will be allocated to any
1494 single job in the partition. Changing the MinNodes of a parti‐
1495 tion has no effect upon jobs that have already begun execution.
1496 Nodes=<name>
1499 Identify the node(s) to be associated with this partition. Mul‐
1500 tiple node names may be specified using simple node range
1501 expressions (e.g. "lx[10-20]"). Note that jobs may only be
1502 associated with one partition at any time. Specify a blank data
1503 value to remove all nodes from a partition: "Nodes=". Changing
1504 the Nodes in a partition has no effect upon jobs that have
1505 already begun execution.
1506 OverTimeLimit=<count>
1509 Number of minutes by which a job can exceed its time limit
1510 before being canceled. The configured job time limit is treated
1511 as a soft limit. Adding OverTimeLimit to the soft limit pro‐
1512 vides a hard limit, at which point the job is canceled. This is
1513 particularly useful for backfill scheduling, which bases upon
1514 each job's soft time limit. A partition-specific OverTimeLimit
1515 will override any global OverTimeLimit value. If not specified,
1516 the global OverTimeLimit value will take precedence. May not
1517 exceed exceed 65533 minutes. An input value of "UNLIMITED" will
1518 clear any previously configured partition-specific OverTimeLimit
1519 value.
1520 OverSubscribe=<yes|no|exclusive|force>[:<job_count>]
1523 Specify if compute resources (i.e. individual CPUs) in this par‐
1524 tition can be shared by multiple jobs. Possible values are
1525 "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count
1526 specifies how many jobs can be allocated to use each resource.
1527 PartitionName=<name>
1530 Identify the partition to be updated. This specification is
1531 required.
1532 PreemptMode=<mode>
1535 Reset the mechanism used to preempt jobs in this partition if
1536 PreemptType is configured to preempt/partition_prio. The default
1537 preemption mechanism is specified by the cluster-wide Preempt‐
1538 Mode configuration parameter. Possible values are "OFF", "CAN‐
1539 CEL", "CHECKPOINT", "REQUEUE" and "SUSPEND".
1540 Priority=<count>
1543 Jobs submitted to a higher priority partition will be dispatched
1544 before pending jobs in lower priority partitions and if possible
1545 they will preempt running jobs from lower priority partitions.
1546 Note that a partition's priority takes precedence over a job's
1547 priority. The value may not exceed 65533.
1548 PriorityJobFactor=<count>
1551 Partition factor used by priority/multifactor plugin in calcu‐
1552 lating job priority. The value may not exceed 65533. Also see
1553 PriorityTier.
1554 PriorityTier=<count>
1557 Jobs submitted to a partition with a higher priority tier value
1558 will be dispatched before pending jobs in partition with lower
1559 priority tier value and, if possible, they will preempt
1560 running jobs from partitions with lower priority tier values.
1561 Note that a partition's priority tier takes precedence over a
1562 job's priority. The value may not exceed 65533. Also see Pri‐
1563 orityJobFactor.
1564 QOS=<QOSname|blank to remove>
1567 Set the partition QOS with a QOS name or to remove the Partition
1568 QOS leave the option blank.
1569 RootOnly=<yes|no>
1572 Specify if only allocation requests initiated by user root will
1573 be satisfied. This can be used to restrict control of the par‐
1574 tition to some meta-scheduler. Possible values are "YES" and
1575 "NO".
1576 ReqResv=<yes|no>
1579 Specify if only allocation requests designating a reservation
1580 will be satisfied. This is used to restrict partition usage to
1581 be allowed only within a reservation. Possible values are "YES"
1582 and "NO".
1583 Shared=<yes|no|exclusive|force>[:<job_count>]
1586 Renamed to OverSubscribe, see option descriptions above.
1587 State=<up|down|drain|inactive>
1590 Specify if jobs can be allocated nodes or queued in this parti‐
1591 tion. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE".
1592 UP Designates that new jobs may queued on the partition,
1594 and that jobs may be allocated nodes and run from the
1595 partition.
1596 DOWN Designates that new jobs may be queued on the parti‐
1598 tion, but queued jobs may not be allocated nodes and
1599 run from the partition. Jobs already running on the
1600 partition continue to run. The jobs must be explicitly
1601 canceled to force their termination.
1602 DRAIN Designates that no new jobs may be queued on the par‐
1604 tition (job submission requests will be denied with an
1605 error message), but jobs already queued on the parti‐
1606 tion may be allocated nodes and run. See also the
1607 "Alternate" partition specification.
1608 INACTIVE Designates that no new jobs may be queued on the par‐
1610 tition, and jobs already queued may not be allocated
1611 nodes and run. See also the "Alternate" partition
1612 specification.
1613 TRESBillingWeights=<TRES Billing Weights>
1616 TRESBillingWeights is used to define the billing weights of each
1617 TRES type that will be used in calculating the usage of a job.
1618 The calculated usage is used when calculating fairshare and when
1619 enforcing the TRES billing limit on jobs. Updates affect new
1620 jobs and not existing jobs. See the slurm.conf man page for
1621 more information.
1622 SPECIFICATIONS FOR UPDATE COMMAND, POWERCAP
1626 PowerCap=<count>
1629 Set the amount of watts the cluster is limited to. Specify a
1630 number, "INFINITE" to enable the power capping logic without
1631 power restriction or "0" to disable the power capping logic.
1632 Update slurm.conf with any changes meant to be persistent across
1633 normal restarts of slurmctld or the execution of scontrol recon‐
1634 fig.
1635 SPECIFICATIONS FOR CREATE, UPDATE, AND DELETE COMMANDS, RESERVATIONS
1638 Reservation=<name>
1642 Identify the name of the reservation to be created,
1643 updated, or deleted. This parameter is required for
1644 update and is the only parameter for delete. For create,
1645 if you do not want to give a reservation name, use "scon‐
1646 trol create res ..." and a name will be created automati‐
1647 cally.
1648 Accounts=<account list>
1651 List of accounts permitted to use the reserved nodes, for
1652 example "Accounts=physcode1,physcode2". A user in any of
1653 the accounts may use the reserved nodes. A new reserva‐
1654 tion must specify Users and/or Accounts. If both Users
1655 and Accounts are specified, a job must match both in
1656 order to use the reservation. Accounts can also be
1657 denied access to reservations by preceding all of the
1658 account names with '-'. Alternately precede the equal
1659 sign with '-'. For example,
1660 "Accounts=-physcode1,-physcode2" or
1661 "Accounts-=physcode1,physcode2" will permit any account
1662 except physcode1 and physcode2 to use the reservation.
1663 You can add or remove individual accounts from an exist‐
1664 ing reservation by using the update command and adding a
1665 '+' or '-' sign before the '=' sign. If accounts are
1666 denied access to a reservation (account name preceded by
1667 a '-'), then all other accounts are implicitly allowed to
1668 use the reservation and it is not possible to also
1669 explicitly specify allowed accounts.
1670 BurstBuffer=<buffer_spec>[,<buffer_spec>,...]
1673 Specification of burst buffer resources which are to be
1674 reserved. "buffer_spec" consists of four elements:
1675 [plugin:][type:]#[units] "plugin" is the burst buffer
1676 plugin name, currently either "datawarp" or "generic".
1677 If no plugin is specified, the reservation applies to all
1678 configured burst buffer plugins. "type" specifies a Cray
1679 generic burst buffer resource, for example "nodes". if
1680 "type" is not specified, the number is a measure of stor‐
1681 age space. The "units" may be "N" (nodes), "K|KiB",
1682 "M|MiB", "G|GiB", "T|TiB", "P|PiB" (for powers of 1024)
1683 and "KB", "MB", "GB", "TB", "PB" (for powers of 1000).
1684 The default units are bytes for reservations of storage
1685 space. For example "BurstBuffer=datawarp:2TB" (reserve
1686 2TB of storage plus 3 nodes from the Cray plugin) or
1687 "BurstBuffer=100GB" (reserve 100 GB of storage from all
1688 configured burst buffer plugins). Jobs using this reser‐
1689 vation are not restricted to these burst buffer
1690 resources, but may use these reserved resources plus any
1691 which are generally available. NOTE: Usually Slurm
1692 interprets KB, MB, GB, TB, PB, TB units as powers of
1693 1024, but for Burst Buffers size specifications Slurm
1694 supports both IEC/SI formats. This is because the CRAY
1695 API for managing DataWarps supports both formats.
1696 CoreCnt=<num>
1699 This option is only supported when Select‐
1700 Type=select/cons_res. Identify number of cores to be
1701 reserved. If NodeCnt is used without the FIRST_CORES
1702 flag, this is the total number of cores to reserve where
1703 cores per node is CoreCnt/NodeCnt. If a nodelist is
1704 used, or if NodeCnt is used with the FIRST_CORES flag,
1705 this should be an array of core numbers by node:
1706 Nodes=node[1-5] CoreCnt=2,2,3,3,4 or flags=FIRST_CORES
1707 NodeCnt=5 CoreCnt=1,2,1,3,2.
1708 Licenses=<license>
1711 Specification of licenses (or other resources available
1712 on all nodes of the cluster) which are to be reserved.
1713 License names can be followed by a colon and count (the
1714 default count is one). Multiple license names should be
1715 comma separated (e.g. "Licenses=foo:4,bar"). A new
1716 reservation must specify one or more resource to be
1717 included: NodeCnt, Nodes and/or Licenses. If a reserva‐
1718 tion includes Licenses, but no NodeCnt or Nodes, then the
1719 option Flags=LICENSE_ONLY must also be specified. Jobs
1720 using this reservation are not restricted to these
1721 licenses, but may use these reserved licenses plus any
1722 which are generally available.
1723 NodeCnt=<num>[,num,...]
1726 Identify number of nodes to be reserved. The number can
1727 include a suffix of "k" or "K", in which case the number
1728 specified is multiplied by 1024. A new reservation must
1729 specify one or more resource to be included: NodeCnt,
1730 Nodes and/or Licenses.
1731 Nodes=<name>
1734 Identify the node(s) to be reserved. Multiple node names
1735 may be specified using simple node range expressions
1736 (e.g. "Nodes=lx[10-20]"). Specify a blank data value to
1737 remove all nodes from a reservation: "Nodes=". A new
1738 reservation must specify one or more resource to be
1739 included: NodeCnt, Nodes and/or Licenses. A specification
1740 of "ALL" will reserve all nodes. Set Flags=PART_NODES and
1741 PartitionName= in order for changes in the nodes associ‐
1742 ated with a partition to also be reflected in the nodes
1743 associated with a reservation.
1744 StartTime=<time_spec>
1747 The start time for the reservation. A new reservation
1748 must specify a start time. It accepts times of the form
1749 HH:MM:SS for a specific time of day (seconds are
1750 optional). (If that time is already past, the next day
1751 is assumed.) You may also specify midnight, noon, fika
1752 (3 PM) or teatime (4 PM) and you can have a time-of-day
1753 suffixed with AM or PM for running in the morning or the
1754 evening. You can also say what day the job will be run,
1755 by specifying a date of the form MMDDYY or MM/DD/YY or
1756 MM.DD.YY, or a date and time as YYYY-MM-DD[THH:MM[:SS]].
1757 You can also give times like now + count time-units,
1758 where the time-units can be minutes, hours, days, or
1759 weeks and you can tell Slurm to run the job today with
1760 the keyword today and to run the job tomorrow with the
1761 keyword tomorrow. You cannot update the StartTime of a
1762 reservation in ACTIVE state.
1763 EndTime=<time_spec>
1766 The end time for the reservation. A new reservation must
1767 specify an end time or a duration. Valid formats are the
1768 same as for StartTime.
1769 Duration=<time>
1772 The length of a reservation. A new reservation must
1773 specify an end time or a duration. Valid formats are
1774 minutes, minutes:seconds, hours:minutes:seconds,
1775 days-hours, days-hours:minutes, days-hours:minutes:sec‐
1776 onds, or UNLIMITED. Time resolution is one minute and
1777 second values are rounded up to the next minute. Output
1778 format is always [days-]hours:minutes:seconds.
1779 PartitionName=<name>
1782 Identify the partition to be reserved.
1783 Flags=<flags>
1786 Flags associated with the reservation. You can add or
1787 remove individual flags from an existing reservation by
1788 adding a '+' or '-' sign before the '=' sign. For exam‐
1789 ple: Flags-=DAILY (NOTE: this shortcut is not supported
1790 for all flags). Currently supported flags include:
1791 ANY_NODES This is a reservation for burst buffers
1793 and/or licenses only and not compute nodes.
1794 If this flag is set, a job using this
1795 reservation may use the associated burst
1796 buffers and/or licenses plus any compute
1797 nodes. If this flag is not set, a job
1798 using this reservation may use only the
1799 nodes and licenses associated with the
1800 reservation.
1801 DAILY Repeat the reservation at the same time
1803 every day.
1804 FLEX Permit jobs requesting the reservation to
1806 begin prior to the reservation's start
1807 time, end after the reservation's end time,
1808 and use any resources inside and/or outside
1809 of the reservation regardless of any con‐
1810 straints possibly set in the reservation. A
1811 typical use case is to prevent jobs not
1812 explicitly requesting the reservation from
1813 using those reserved resources rather than
1814 forcing jobs requesting the reservation to
1815 use those resources in the time frame
1816 reserved. Another use case could be to
1817 always have a particular number of nodes
1818 with a specific feature reserved for a spe‐
1819 cific account so users in this account may
1820 use this nodes plus possibly other nodes
1821 without this feature.
1822 FIRST_CORES Use the lowest numbered cores on a node
1824 only.
1825 IGNORE_JOBS Ignore currently running jobs when creating
1827 the reservation. This can be especially
1828 useful when reserving all nodes in the sys‐
1829 tem for maintenance.
1830 LICENSE_ONLY See ANY_NODES.
1832 MAINT Maintenance mode, receives special account‐
1834 ing treatment. This partition is permitted
1835 to use resources that are already in
1836 another reservation.
1837 NO_HOLD_JOBS_AFTER
1839 By default, when a reservation ends the
1840 reservation request will be removed from
1841 any pending jobs submitted to the reserva‐
1842 tion and will be put into a held state.
1843 Use this flag to let jobs run outside of
1844 the reservation after the reservation is
1845 gone.
1846 OVERLAP This reservation can be allocated resources
1848 that are already in another reservation.
1849 PART_NODES This flag can be used to reserve all nodes
1851 within the specified partition. Partition‐
1852 Name and Nodes=ALL must be specified or
1853 this option is ignored.
1854 PURGE_COMP Purge the reservation once the last associ‐
1856 ated job has completed. Once the reserva‐
1857 tion has been created, it must be populated
1858 within 5 minutes of its start time or it
1859 will be purged before any jobs have been
1860 run.
1861 REPLACE Nodes which are DOWN, DRAINED, or allocated
1863 to jobs are automatically replenished using
1864 idle resources. This option can be used to
1865 maintain a constant number of idle
1866 resources available for pending jobs (sub‐
1867 ject to availability of idle resources).
1868 This should be used with the NodeCnt reser‐
1869 vation option; do not identify specific
1870 nodes to be included in the reservation.
1871 REPLACE_DOWN Nodes which are DOWN or DRAINED are auto‐
1873 matically replenished using idle resources.
1874 This option can be used to maintain a con‐
1875 stant sized pool of resources available for
1876 pending jobs (subject to availability of
1877 idle resources). This should be used with
1878 the NodeCnt reservation option; do not
1879 identify specific nodes to be included in
1880 the reservation.
1881 SPEC_NODES Reservation is for specific nodes (output
1883 only)
1884 STATIC_ALLOC Make it so after the nodes are selected for
1886 a reservation they don't change. Without
1887 this option when nodes are selected for a
1888 reservation and one goes down the reserva‐
1889 tion will select a new node to fill the
1890 spot.
1891 TIME_FLOAT The reservation start time is relative to
1893 the current time and moves forward through
1894 time (e.g. a StartTime=now+10minutes will
1895 always be 10 minutes in the future).
1896 WEEKDAY Repeat the reservation at the same time on
1898 every weekday (Monday, Tuesday, Wednesday,
1899 Thursday and Friday).
1900 WEEKEND Repeat the reservation at the same time on
1902 every weekend day (Saturday and Sunday).
1903 WEEKLY Repeat the reservation at the same time
1905 every week.
1906 Features=<features>
1909 Set the reservation's required node features. Multiple
1910 values may be "&" separated if all features are required
1911 (AND operation) or separated by "|" if any of the speci‐
1912 fied features are required (OR operation). Parenthesis
1913 are also supported for features to be ANDed together with
1914 counts of nodes having the specified features. For exam‐
1915 ple "Features=[(knl&a2a&flat)*4&haswell*2]" indicates the
1916 advanced reservation should include 4 nodes with ALL of
1917 the features "knl", "a2a", and "flat" plus 2 nodes with
1918 the feature "haswell".
1919 Value may be cleared with blank data value, "Features=".
1921 Users=<user list>
1924 List of users permitted to use the reserved nodes, for
1925 example "User=jones1,smith2". A new reservation must
1926 specify Users and/or Accounts. If both Users and
1927 Accounts are specified, a job must match both in order to
1928 use the reservation. Users can also be denied access to
1929 reservations by preceding all of the user names with '-'.
1930 Alternately precede the equal sign with '-'. For exam‐
1931 ple, "User=-jones1,-smith2" or "User-=jones1,smith2" will
1932 permit any user except jones1 and smith2 to use the
1933 reservation. You can add or remove individual users from
1934 an existing reservation by using the update command and
1935 adding a '+' or '-' sign before the '=' sign. If users
1936 are denied access to a reservation (user name preceded by
1937 a '-'), then all other users are implicitly allowed to
1938 use the reservation and it is not possible to also
1939 explicitly specify allowed users.
1940 TRES=<tres_spec>
1943 Comma-separated list of TRES required for the reserva‐
1944 tion. Current supported TRES types with reservations are:
1945 CPU, Node, License and BB. CPU and Node follow the same
1946 format as CoreCnt and NodeCnt parameters respectively.
1947 License names can be followed by an equal '=' and a
1948 count:
1949 License/<name1>=<count1>[,License/<name2>=<count2>,...]
1951 BurstBuffer can be specified in a similar way as Burst‐
1953 Buffer parameter. The only difference is that colon sym‐
1954 bol ':' should be replaced by an equal '=' in order to
1955 follow the TRES format.
1956 Some examples of TRES valid specifications:
1958 TRES=cpu=5,bb/cray=4,license/iop1=1,license/iop2=3
1960 TRES=node=5k,license/iop1=2
1962 As specified in CoreCnt, if a nodelist is specified, cpu
1964 can be an array of core numbers by node: nodes=com‐
1965 pute[1-3] TRES=cpu=2,2,1,bb/cray=4,license/iop1=2
1966 Please note that CPU, Node, License and BB can override
1968 CoreCnt, NodeCnt, Licenses and BurstBuffer parameters
1969 respectively. Also CPU represents CoreCnt, in a reserva‐
1970 tion and will be adjusted if you have threads per core on
1971 your nodes.
1972 SPECIFICATIONS FOR SHOW COMMAND, LAYOUTS
1975 Without options, lists all configured layouts. With a layout
1977 specified, shows entities with following options:
1978 Key=<value>
1981 Keys/Values to update for the entities. The format must
1982 respect the layout.d configuration files. Key=Type cannot
1983 be updated. One Key/Value is required, several can be
1984 set.
1985 Entity=<value>
1987 Entities to show, default is not used. Can be set to "*".
1988 Type=<value>
1990 Type of entities to show, default is not used.
1991 nolayout
1993 If not used, only entities with defining the tree are
1994 shown. With the option, only leaves are shown.
1995
1998 Some scontrol options may be set via environment variables.
1999 These environment variables, along with their corresponding
2000 options, are listed below. (Note: Commandline options will
2001 always override these settings.)
2002 SCONTROL_ALL -a, --all
2004 SCONTROL_FEDERATION --federation
2006 SCONTROL_FUTURE -F, --future
2008 SCONTROL_LOCAL --local
2010 SCONTROL_SIBLING --sibling
2012 SLURM_BITSTR_LEN Specifies the string length to be used for
2014 holding a job array's task ID expression.
2015 The default value is 64 bytes. A value of 0
2016 will print the full expression with any
2017 length required. Larger values may
2018 adversely impact the application perfor‐
2019 mance.
2020 SLURM_CLUSTERS Same as --clusters
2022 SLURM_CONF The location of the Slurm configuration
2024 file.
2025 SLURM_TIME_FORMAT Specify the format used to report time
2027 stamps. A value of standard, the default
2028 value, generates output in the form
2029 "year-month-dateThour:minute:second". A
2030 value of relative returns only
2031 "hour:minute:second" if the current day.
2032 For other dates in the current year it
2033 prints the "hour:minute" preceded by
2034 "Tomorr" (tomorrow), "Ystday" (yesterday),
2035 the name of the day for the coming week
2036 (e.g. "Mon", "Tue", etc.), otherwise the
2037 date (e.g. "25 Apr"). For other years it
2038 returns a date month and year without a time
2039 (e.g. "6 Jun 2012"). All of the time stamps
2040 use a 24 hour format.
2041 A valid strftime() format can also be speci‐
2043 fied. For example, a value of "%a %T" will
2044 report the day of the week and a time stamp
2045 (e.g. "Mon 12:34:56").
2046 SLURM_TOPO_LEN Specify the maximum size of the line when
2049 printing Topology. If not set, the default
2050 value is unlimited.
2051
2054 When using the Slurm db, users who have AdminLevel's defined
2055 (Operator or Admin) and users who are account coordinators are
2056 given the authority to view and modify jobs, reservations,
2057 nodes, etc., as defined in the following table - regardless of
2058 whether a PrivateData restriction has been defined in the
2059 slurm.conf file.
2060 scontrol show job(s): Admin, Operator, Coordinator
2062 scontrol update job: Admin, Operator, Coordinator
2063 scontrol requeue: Admin, Operator, Coordinator
2064 scontrol show step(s): Admin, Operator, Coordinator
2065 scontrol update step: Admin, Operator, Coordinator
2066 scontrol show node: Admin, Operator
2068 scontrol update node: Admin
2069 scontrol create partition: Admin
2071 scontrol show partition: Admin, Operator
2072 scontrol update partition: Admin
2073 scontrol delete partition: Admin
2074 scontrol create reservation: Admin, Operator
2076 scontrol show reservation: Admin, Operator
2077 scontrol update reservation: Admin, Operator
2078 scontrol delete reservation: Admin, Operator
2079 scontrol reconfig: Admin
2081 scontrol shutdown: Admin
2082 scontrol takeover: Admin
2083
2086 # scontrol
2087 scontrol: show part debug
2088 PartitionName=debug
2089 AllocNodes=ALL AllowGroups=ALL Default=YES
2090 DefaultTime=NONE DisableRootJobs=NO Hidden=NO
2091 MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=1
2092 Nodes=snowflake[0-48]
2093 Priority=1 RootOnly=NO OverSubscribe=YES:4
2094 State=UP TotalCPUs=694 TotalNodes=49
2095 scontrol: update PartitionName=debug MaxTime=60:00 MaxNodes=4
2096 scontrol: show job 71701
2097 JobId=71701 Name=hostname
2098 UserId=da(1000) GroupId=da(1000)
2099 Priority=66264 Account=none QOS=normal WCKey=*123
2100 JobState=COMPLETED Reason=None Dependency=(null)
2101 TimeLimit=UNLIMITED Requeue=1 Restarts=0 BatchFlag=0 Exit‐
2102 Code=0:0
2103 SubmitTime=2010-01-05T10:58:40 Eligible‐
2104 Time=2010-01-05T10:58:40
2105 StartTime=2010-01-05T10:58:40 EndTime=2010-01-05T10:58:40
2106 SuspendTime=None SecsPreSuspend=0
2107 Partition=debug AllocNode:Sid=snowflake:4702
2108 ReqNodeList=(null) ExcNodeList=(null)
2109 NodeList=snowflake0
2110 NumNodes=1 NumCPUs=10 CPUs/Task=2 ReqS:C:T=1:1:1
2111 MinCPUsNode=2 MinMemoryNode=0 MinTmpDiskNode=0
2112 Features=(null) Reservation=(null)
2113 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
2114 scontrol: update JobId=71701 TimeLimit=30:00 Priority=500
2115 scontrol: show hostnames tux[1-3]
2116 tux1
2117 tux2
2118 tux3
2119 scontrol: create res StartTime=2009-04-01T08:00:00 Dura‐
2120 tion=5:00:00 Users=dbremer NodeCnt=10
2121 Reservation created: dbremer_1
2122 scontrol: update Reservation=dbremer_1 Flags=Maint NodeCnt=20
2123 scontrol: delete Reservation=dbremer_1
2124 scontrol: quit
2125
2128 Copyright (C) 2002-2007 The Regents of the University of Cali‐
2129 fornia. Produced at Lawrence Livermore National Laboratory (cf,
2130 DISCLAIMER).
2131 Copyright (C) 2008-2010 Lawrence Livermore National Security.
2132 Copyright (C) 2010-2018 SchedMD LLC.
2133 This file is part of Slurm, a resource management program. For
2135 details, see <https://slurm.schedmd.com/>.
2136 Slurm is free software; you can redistribute it and/or modify it
2138 under the terms of the GNU General Public License as published
2139 by the Free Software Foundation; either version 2 of the
2140 License, or (at your option) any later version.
2141 Slurm is distributed in the hope that it will be useful, but
2143 WITHOUT ANY WARRANTY; without even the implied warranty of MER‐
2144 CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
2145 General Public License for more details.
2146
2148 /etc/slurm.conf
2149
2151 scancel(1), sinfo(1), squeue(1), slurm_checkpoint [22m(3),
2152 slurm_create_partition [22m(3), slurm_delete_partition [22m(3),
2153 slurm_load_ctl_conf (3), slurm_load_jobs [22m(3), slurm_load_node
2154 (3), slurm_load_partitions [22m(3), slurm_reconfigure [22m(3),
2155 slurm_requeue [22m(3), slurm_resume [22m(3), slurm_shutdown [22m(3),
2156 slurm_suspend [22m(3), slurm_takeover [22m(3), slurm_update_job (3),
2157 slurm_update_node [22m(3), slurm_update_partition [22m(3),
2158 slurm.conf(5), slurmctld(8)
2159August 2019 Slurm Commands scontrol(1)