1sacctmgr(1) Slurm Commands sacctmgr(1)
2
6 sacctmgr - Used to view and modify Slurm account information.
7
10 sacctmgr [OPTIONS...] [COMMAND...]
11
14 sacctmgr is used to view or modify Slurm account information. The ac‐
15 count information is maintained within a database with the interface
16 being provided by slurmdbd (Slurm Database daemon). This database can
17 serve as a central storehouse of user and computer information for mul‐
18 tiple computers at a single site. Slurm account information is
19 recorded based upon four parameters that form what is referred to as an
20 association. These parameters are user, cluster, partition, and ac‐
21 count. user is the login name. cluster is the name of a Slurm managed
22 cluster as specified by the ClusterName parameter in the slurm.conf
23 configuration file. partition is the name of a Slurm partition on that
24 cluster. account is the bank account for a job. The intended mode of
25 operation is to initiate the sacctmgr command, add, delete, modify,
26 and/or list association records then commit the changes and exit.
27 NOTE: The contents of Slurm's database are maintained in lower case.
29 This may result in some sacctmgr output differing from that of other
30 Slurm commands.
31
34 -h, --help
35 Print a help message describing the usage of sacctmgr. This is
36 equivalent to the help command.
37 -i, --immediate
40 commit changes immediately without asking for confirmation.
41 -n, --noheader
44 No header will be added to the beginning of the output.
45 -p, --parsable
48 Output will be '|' delimited with a '|' at the end.
49 -P, --parsable2
52 Output will be '|' delimited without a '|' at the end.
53 -Q, --quiet
56 Print no messages other than error messages. This is equivalent
57 to the quiet command.
58 -r, --readonly
61 Makes it so the running sacctmgr cannot modify accounting infor‐
62 mation. The readonly option is for use within interactive mode.
63 -s, --associations
66 Use with show or list to display associations with the entity.
67 This is equivalent to the associations command.
68 -v, --verbose
71 Enable detailed logging. This is equivalent to the verbose com‐
72 mand.
73 -V , --version
76 Display version number. This is equivalent to the version com‐
77 mand.
78
81 add <ENTITY> <SPECS>
82 Add an entity. Identical to the create command.
83 archive {dump|load} <SPECS>
86 Write database information to a flat file or load information
87 that has previously been written to a file.
88 clear stats
91 Clear the server statistics.
92 create <ENTITY> <SPECS>
95 Add an entity. Identical to the add command.
96 delete <ENTITY> where <SPECS>
99 Delete the specified entities. Identical to the remove command.
100 dump <ENTITY> [File=FILENAME]
103 Dump cluster data to the specified file. If the filename is not
104 specified it uses clustername.cfg filename by default.
105 help Display a description of sacctmgr options and commands.
108 list <ENTITY> [<SPECS>]
111 Display information about the specified entity. By default, all
112 entries are displayed, you can narrow results by specifying
113 SPECS in your query. Identical to the show command.
114 load <FILENAME>
117 Load cluster data from the specified file. This is a configura‐
118 tion file generated by running the sacctmgr dump command. This
119 command does not load archive data, see the sacctmgr archive
120 load option instead.
121 modify <ENTITY> where <SPECS> set <SPECS>
124 Modify an entity.
125 reconfigure
128 Reconfigures the SlurmDBD if running with one.
129 remove <ENTITY> where <SPECS>
132 Delete the specified entities. Identical to the delete command.
133 show <ENTITY> [<SPECS>]
136 Display information about the specified entity. By default, all
137 entries are displayed, you can narrow results by specifying
138 SPECS in your query. Identical to the list command.
139 shutdown
142 Shutdown the server.
143 version
146 Display the version number of sacctmgr.
147
150 NOTE: All commands listed below can be used in the interactive mode,
151 but NOT on the initial command line.
152 exit Terminate sacctmgr interactive mode. Identical to the quit com‐
155 mand.
156 quiet Print no messages other than error messages.
159 quit Terminate the execution of sacctmgr interactive mode. Identical
162 to the exit command.
163 verbose
166 Enable detailed logging. This includes time-stamps on data
167 structures, record counts, etc. This is an independent command
168 with no options meant for use in interactive mode.
169 !! Repeat the last command.
172
175 account
176 A bank account, typically specified at job submit time using the
177 --account= option. These may be arranged in a hierarchical
178 fashion, for example accounts 'chemistry' and 'physics' may be
179 children of the account 'science'. The hierarchy may have an
180 arbitrary depth.
181 association
184 The entity used to group information consisting of four parame‐
185 ters: account, cluster, partition (optional), and user. Used
186 only with the list or show command. Add, modify, and delete
187 should be done to a user, account or cluster entity. This will
188 in turn update the underlying associations.
189 cluster
192 The ClusterName parameter in the slurm.conf configuration file,
193 used to differentiate accounts on different machines.
194 configuration
197 Used only with the list or show command to report current system
198 configuration.
199 coordinator
202 A special privileged user, usually an account manager, that can
203 add users or sub-accounts to the account they are coordinator
204 over. This should be a trusted person since they can change
205 limits on account and user associations, as well as cancel, re‐
206 queue or reassign accounts of jobs inside their realm.
207 event Events like downed or draining nodes on clusters.
210 federation
213 A group of clusters that work together to schedule jobs.
214 job Used to modify specific fields of a job: Derived Exit Code, the
217 Comment String, or wckey.
218 problem
221 Use with show or list to display entity problems.
222 qos Quality of Service.
225 reservation
228 A collection of resources set apart for use by a particular ac‐
229 count, user or group of users for a given period of time.
230 resource
233 Software resources for the system. Those are software licenses
234 shared among clusters.
235 RunawayJobs
238 Used only with the list or show command to report current jobs
239 that have been orphaned on the local cluster and are now run‐
240 away. If there are jobs in this state it will also give you an
241 option to "fix" them. NOTE: You must have an AdminLevel of at
242 least Operator to perform this.
243 stats Used with list or show command to view server statistics. Ac‐
246 cepts optional argument of ave_time or total_time to sort on
247 those fields. By default, sorts on increasing RPC count field.
248 transaction
251 List of transactions that have occurred during a given time pe‐
252 riod.
253 tres Used with list or show command to view a list of Trackable RE‐
256 Sources configured on the system.
257 user The login name. Usernames are case-insensitive (forced to lower‐
260 case) unless the PreserveCaseUser option has been set in the
261 SlurmDBD configuration file.
262 wckeys Workload Characterization Key. An arbitrary string for
265 grouping orthogonal accounts.
266
269 NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job
270 is being considered for being allocated resources. If starting a job
271 would cause any of its group limit to be exceeded, that job will not be
272 considered for scheduling even if that job might preempt other jobs
273 which would release sufficient group resources for the pending job to
274 be initiated.
275 DefaultQOS=<default qos>
278 The default QOS this association and its children should have.
279 This is overridden if set directly on a user. To clear a previ‐
280 ously set value use the modify command with a new value of -1.
281 Fairshare=<fairshare number | parent>
284 Number used in conjunction with other accounts to determine job
285 priority. Can also be the string parent, when used on a user
286 this means that the parent association is used for fairshare.
287 If Fairshare=parent is set on an account, that account's chil‐
288 dren will be effectively reparented for fairshare calculations
289 to the first parent of their parent that is not Fairshare=par‐
290 ent. Limits remain the same, only its fairshare value is af‐
291 fected. To clear a previously set value use the modify command
292 with a new value of -1.
293 GrpTRESMins=<TRES=max TRES minutes,...>
296 The total number of TRES minutes that can possibly be used by
297 past, present and future jobs running from this association and
298 its children. To clear a previously set value use the modify
299 command with a new value of -1 for each TRES id.
300 NOTE: This limit is not enforced if set on the root association
302 of a cluster. So even though it may appear in sacctmgr output,
303 it will not be enforced.
304 ALSO NOTE: This limit only applies when using the Priority Mul‐
306 tifactor plugin. The time is decayed using the value of Priori‐
307 tyDecayHalfLife or PriorityUsageResetPeriod as set in the
308 slurm.conf. When this limit is reached all associated jobs run‐
309 ning will be killed and all future jobs submitted with associa‐
310 tions in the group will be delayed until they are able to run
311 inside the limit.
312 GrpTRESRunMins=<TRES=max TRES run minutes,...>
315 Used to limit the combined total number of TRES minutes used by
316 all jobs running with this association and its children. This
317 takes into consideration time limit of running jobs and consumes
318 it, if the limit is reached no new jobs are started until other
319 jobs finish to allow time to free up.
320 GrpTRES=<TRES=max TRES,...>
323 Maximum number of TRES running jobs are able to be allocated in
324 aggregate for this association and all associations which are
325 children of this association. To clear a previously set value
326 use the modify command with a new value of -1 for each TRES id.
327 NOTE: This limit only applies fully when using the Select Con‐
329 sumable Resource plugin.
330 GrpJobs=<max jobs>
333 Maximum number of running jobs in aggregate for this association
334 and all associations which are children of this association. To
335 clear a previously set value use the modify command with a new
336 value of -1.
337 GrpJobsAccrue=<max jobs>
340 Maximum number of pending jobs in aggregate able to accrue age
341 priority for this association and all associations which are
342 children of this association. To clear a previously set value
343 use the modify command with a new value of -1.
344 GrpSubmitJobs=<max jobs>
347 Maximum number of jobs which can be in a pending or running
348 state at any time in aggregate for this association and all as‐
349 sociations which are children of this association. To clear a
350 previously set value use the modify command with a new value of
351 -1.
352 NOTE: This setting shows up in the sacctmgr output as GrpSubmit.
354 GrpWall=<max wall>
357 Maximum wall clock time running jobs are able to be allocated in
358 aggregate for this association and all associations which are
359 children of this association. To clear a previously set value
360 use the modify command with a new value of -1.
361 NOTE: This limit is not enforced if set on the root association
363 of a cluster. So even though it may appear in sacctmgr output,
364 it will not be enforced.
365 ALSO NOTE: This limit only applies when using the Priority Mul‐
367 tifactor plugin. The time is decayed using the value of Priori‐
368 tyDecayHalfLife or PriorityUsageResetPeriod as set in the
369 slurm.conf. When this limit is reached all associated jobs run‐
370 ning will be killed and all future jobs submitted with associa‐
371 tions in the group will be delayed until they are able to run
372 inside the limit.
373 MaxTRESMinsPerJob=<max TRES minutes>
376 Maximum number of TRES minutes each job is able to use in this
377 association. This is overridden if set directly on a user. De‐
378 fault is the cluster's limit. To clear a previously set value
379 use the modify command with a new value of -1 for each TRES id.
380 NOTE: This setting shows up in the sacctmgr output as Max‐
382 TRESMins.
383 MaxTRESPerJob=<max TRES>
386 Maximum number of TRES each job is able to use in this associa‐
387 tion. This is overridden if set directly on a user. Default is
388 the cluster's limit. To clear a previously set value use the
389 modify command with a new value of -1 for each TRES id.
390 NOTE: This setting shows up in the sacctmgr output as MaxTRES.
392 NOTE: This limit only applies fully when using cons_res or
394 cons_tres select type plugins.
395 MaxJobs=<max jobs>
398 Maximum number of jobs each user is allowed to run at one time
399 in this association. This is overridden if set directly on a
400 user. Default is the cluster's limit. To clear a previously
401 set value use the modify command with a new value of -1.
402 MaxJobsAccrue=<max jobs>
405 Maximum number of pending jobs able to accrue age priority at
406 any given time for the given association. This is overridden if
407 set directly on a user. Default is the cluster's limit. To
408 clear a previously set value use the modify command with a new
409 value of -1.
410 MaxSubmitJobs=<max jobs>
413 Maximum number of jobs which can this association can have in a
414 pending or running state at any time. Default is the cluster's
415 limit. To clear a previously set value use the modify command
416 with a new value of -1.
417 NOTE: This setting shows up in the sacctmgr output as MaxSubmit.
419 MaxWallDurationPerJob=<max wall>
422 Maximum wall clock time each job is able to use in this associa‐
423 tion. This is overridden if set directly on a user. Default is
424 the cluster's limit. <max wall> format is <min> or <min>:<sec>
425 or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>.
426 The value is recorded in minutes with rounding as needed. To
427 clear a previously set value use the modify command with a new
428 value of -1.
429 NOTE: Changing this value will have no effect on any running or
431 pending job.
432 NOTE: This setting shows up in the sacctmgr output as MaxWall.
434 Priority
437 What priority will be added to a job's priority when using this
438 association. This is overridden if set directly on a user. De‐
439 fault is the cluster's limit. To clear a previously set value
440 use the modify command with a new value of -1.
441 QosLevel<operator><comma separated list of qos names>
444 Specify the default Quality of Service's that jobs are able to
445 run at for this association. To get a list of valid QOS's use
446 'sacctmgr list qos'. This value will override its parents value
447 and push down to its children as the new default. Setting a
448 QosLevel to '' (two single quotes with nothing between them) re‐
449 stores its default setting. You can also use the operator +=
450 and -= to add or remove certain QOS's from a QOS list.
451 Valid <operator> values include:
453 =
454 Set QosLevel to the specified value. Note: the QOS that can
455 be used at a given account in the hierarchy are inherited
456 by the children of that account. By assigning QOS with the
457 = sign only the assigned QOS can be used by the account and
458 its children.
459 +=
460 Add the specified <qos> value to the current QosLevel.
461 The account will have access to this QOS and the other
462 previously assigned to it.
463 -=
464 Remove the specified <qos> value from the current
465 QosLevel.
466 See the EXAMPLES section below.
469
472 Cluster=<cluster>
473 Specific cluster to add account to. Default is all in system.
474 Description=<description>
477 An arbitrary string describing an account.
478 Name=<name>
481 The name of a bank account. Note the name must be unique and
482 can not be represent different bank accounts at different points
483 in the account hierarchy.
484 Organization=<org>
487 Organization to which the account belongs.
488 Parent=<parent>
491 Parent account of this account. Default is the root account, a
492 top level account.
493 RawUsage=<value>
496 This allows an administrator to reset the raw usage accrued to
497 an account. The only value currently supported is 0 (zero).
498 This is a settable specification only - it cannot be used as a
499 filter to list accounts.
500 WithAssoc
503 Display all associations for this account.
504 WithCoord
507 Display all coordinators for this account.
508 WithDeleted
511 Display information with previously deleted data.
512 NOTE: If using the WithAssoc option you can also query against associa‐
514 tion specific information to view only certain associations this ac‐
515 count may have. These extra options can be found in the SPECIFICATIONS
516 FOR ASSOCIATIONS section. You can also use the general specifications
517 list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES
518 section.
519
522 Account
523 The name of a bank account.
524 Description
527 An arbitrary string describing an account.
528 Organization
531 Organization to which the account belongs.
532 Coordinators
535 List of users that are a coordinator of the account. (Only
536 filled in when using the WithCoordinator option.)
537 NOTE: If using the WithAssoc option you can also view the information
539 about the various associations the account may have on all the clusters
540 in the system. The association information can be filtered. Note that
541 all the accounts in the database will always be shown as filter only
542 takes effect over the association data. The Association format fields
543 are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
544
548 Clusters=<comma separated list of cluster names>
549 List the associations of the cluster(s).
550 Accounts=<comma separated list of account names>
553 List the associations of the account(s).
554 Users=<comma separated list of user names>
557 List the associations of the user(s).
558 Partition=<comma separated list of partition names>
561 List the associations of the partition(s).
562 NOTE: You can also use the general specifications list above in the
564 GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.
565 Other options unique for listing associations:
567 OnlyDefaults
570 Display only associations that are default associations
571 Tree Display account names in a hierarchical fashion.
574 WithDeleted
577 Display information with previously deleted data.
578 WithSubAccounts
581 Display information with subaccounts. Only really valuable when
582 used with the account= option. This will display all the subac‐
583 count associations along with the accounts listed in the option.
584 WOLimits
587 Display information without limit information. This is for a
588 smaller default format of "Cluster,Account,User,Partition".
589 WOPInfo
592 Display information without parent information (i.e. parent id,
593 and parent account name). This option also implicitly sets the
594 WOPLimits option.
595 WOPLimits
598 Display information without hierarchical parent limits (i.e.
599 will only display limits where they are set instead of propagat‐
600 ing them from the parent).
601
605 Account
606 The name of a bank account in the association.
607 Cluster
610 The name of a cluster in the association.
611 DefaultQOS
614 The QOS the association will use by default if it as access to
615 it in the QOS list mentioned below.
616 Fairshare
619 Number used in conjunction with other accounts to determine job
620 priority. Can also be the string parent, when used on a user
621 this means that the parent association is used for fairshare.
622 If Fairshare=parent is set on an account, that account's chil‐
623 dren will be effectively reparented for fairshare calculations
624 to the first parent of their parent that is not Fairshare=par‐
625 ent. Limits remain the same, only its fairshare value is af‐
626 fected.
627 GrpTRESMins
630 The total number of TRES minutes that can possibly be used by
631 past, present and future jobs running from this association and
632 its children.
633 GrpTRESRunMins
636 Used to limit the combined total number of TRES minutes used by
637 all jobs running with this association and its children. This
638 takes into consideration time limit of running jobs and consumes
639 it, if the limit is reached no new jobs are started until other
640 jobs finish to allow time to free up.
641 GrpTRES
644 Maximum number of TRES running jobs are able to be allocated in
645 aggregate for this association and all associations which are
646 children of this association.
647 GrpJobs
650 Maximum number of running jobs in aggregate for this association
651 and all associations which are children of this association.
652 GrpJobsAccrue
655 Maximum number of pending jobs in aggregate able to accrue age
656 priority for this association and all associations which are
657 children of this association.
658 GrpSubmitJobs
661 Maximum number of jobs which can be in a pending or running
662 state at any time in aggregate for this association and all as‐
663 sociations which are children of this association.
664 NOTE: This setting shows up in the sacctmgr output as GrpSubmit.
666 GrpWall
669 Maximum wall clock time running jobs are able to be allocated in
670 aggregate for this association and all associations which are
671 children of this association.
672 ID The id of the association.
675 LFT Associations are kept in a hierarchy: this is the left most spot
678 in the hierarchy. When used with the RGT variable, all associa‐
679 tions with a LFT inside this LFT and before the RGT are children
680 of this association.
681 MaxTRESPerJob
684 Maximum number of TRES each job is able to use.
685 NOTE: This setting shows up in the sacctmgr output as MaxTRES.
687 MaxTRESMinsPerJob
690 Maximum number of TRES minutes each job is able to use.
691 NOTE: This setting shows up in the sacctmgr output as Max‐
693 TRESMins.
694 MaxTRESPerNode
697 Maximum number of TRES each node in a job allocation can use.
698 MaxJobs
701 Maximum number of jobs each user is allowed to run at one time.
702 MaxJobsAccrue
705 Maximum number of pending jobs able to accrue age priority at
706 any given time.
707 MaxSubmitJobs
710 Maximum number of jobs pending or running state at any time.
711 NOTE: This setting shows up in the sacctmgr output as MaxSubmit.
713 MaxWallDurationPerJob
716 Maximum wall clock time each job is able to use.
717 NOTE: This setting shows up in the sacctmgr output as MaxWall.
719 Qos
722 Valid QOS' for this association.
723 QosRaw
726 QOS' ID.
727 ParentID
730 The association id of the parent of this association.
731 ParentName
734 The account name of the parent of this association.
735 Partition
738 The name of a partition in the association.
739 Priority
742 What priority will be added to a job's priority when using this
743 association.
744 WithRawQOSLevel
747 Display QosLevel in an unevaluated raw format, consisting of a
748 comma separated list of QOS names prepended with '' (nothing),
749 '+' or '-' for the association. QOS names without +/- prepended
750 were assigned (ie, sacctmgr modify ... set QosLevel=qos_name)
751 for the entity listed or on one of its parents in the hierarchy.
752 QOS names with +/- prepended indicate the QOS was added/filtered
753 (ie, sacctmgr modify ... set QosLevel=[+-]qos_name) for the en‐
754 tity listed or on one of its parents in the hierarchy. Including
755 WOPLimits will show exactly where each QOS was assigned, added
756 or filtered in the hierarchy.
757 RGT Associations are kept in a hierarchy: this is the right most
760 spot in the hierarchy. When used with the LFT variable, all as‐
761 sociations with a LFT inside this RGT and after the LFT are
762 children of this association.
763 User The name of a user in the association.
766
769 Classification=<classification>
770 Type of machine, current classifications are capability, capac‐
771 ity and capapacity.
772 Features=<comma separated list of feature names>
775 Features that are specific to the cluster. Federated jobs can be
776 directed to clusters that contain the job requested features.
777 Federation=<federation>
780 The federation that this cluster should be a member of. A clus‐
781 ter can only be a member of one federation at a time.
782 FedState=<state>
785 The state of the cluster in the federation.
786 Valid states are:
787 ACTIVE Cluster will actively accept and schedule federated jobs.
789 INACTIVE
792 Cluster will not schedule or accept any jobs.
793 DRAIN Cluster will not accept any new jobs and will let exist‐
796 ing federated jobs complete.
797 DRAIN+REMOVE
800 Cluster will not accept any new jobs and will remove it‐
801 self from the federation once all federated jobs have
802 completed. When removed from the federation, the cluster
803 will accept jobs as a non-federated cluster.
804 Name=<name>
807 The name of a cluster. This should be equal to the ClusterName
808 parameter in the slurm.conf configuration file for some
809 Slurm-managed cluster.
810 RPC=<rpc list>
813 Comma separated list of numeric RPC values.
814 WithFed
817 Appends federation related columns to default format options
818 (e.g. Federation,ID,Features,FedState).
819 WOLimits
822 Display information without limit information. This is for a
823 smaller default format of Cluster,ControlHost,ControlPort,RPC
824 NOTE: You can also use the general specifications list above in the
826 GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.
827
831 Classification
832 Type of machine, i.e. capability, capacity or capapacity.
833 Cluster
836 The name of the cluster.
837 ControlHost
840 When a slurmctld registers with the database the ip address of
841 the controller is placed here.
842 ControlPort
845 When a slurmctld registers with the database the port the con‐
846 troller is listening on is placed here.
847 Features
850 The list of features on the cluster (if any).
851 Federation
854 The name of the federation this cluster is a member of (if any).
855 FedState
858 The state of the cluster in the federation (if a member of one).
859 FedStateRaw
862 Numeric value of the name of the FedState.
863 Flags Attributes possessed by the cluster. Current flags include Cray,
866 External and MultipleSlurmd.
867 External clusters are registration only clusters. A slurmctld
869 can designate an external slurmdbd with the AccountingStorageEx‐
870 ternalHost slurm.conf option. This allows a slurmctld to regis‐
871 ter to an external slurmdbd so that clusters attached to the ex‐
872 ternal slurmdbd can communicate with the external cluster with
873 Slurm commands.
874 ID The ID assigned to the cluster when a member of a federation.
877 This ID uniquely identifies the cluster and its jobs in the fed‐
878 eration.
879 NodeCount
882 The current count of nodes associated with the cluster.
883 NodeNames
886 The current Nodes associated with the cluster.
887 PluginIDSelect
890 The numeric value of the select plugin the cluster is using.
891 RPC When a slurmctld registers with the database the rpc version the
894 controller is running is placed here.
895 TRES Trackable RESources (Billing, BB (Burst buffer), CPU, Energy,
898 GRES, License, Memory, and Node) this cluster is accounting for.
899 NOTE: You can also view the information about the root association for
902 the cluster. The Association format fields are described in the
903 LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
904
908 Account=<comma separated list of account names>
909 Account name to add this user as a coordinator to.
910 Names=<comma separated list of user names>
912 Names of coordinators.
913 NOTE: To list coordinators use the WithCoordinator options with list
915 account or list user.
916
920 All_Clusters
921 Get information on all cluster shortcut.
922 All_Time
925 Get time period for all time shortcut.
926 Clusters=<comma separated list of cluster names>
929 List the events of the cluster(s). Default is the cluster where
930 the command was run.
931 End=<OPT>
934 Period ending of events. Default is now.
935 Valid time formats are...
937 HH:MM[:SS] [AM|PM]
939 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
940 MM/DD[/YY]-HH:MM[:SS]
941 YYYY-MM-DD[THH:MM[:SS]]
942 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
943 Event=<OPT>
946 Specific events to look for, valid options are Cluster or Node,
947 default is both.
948 MaxTRES=<OPT>
951 Max number of TRES affected by an event.
952 MinTRES=<OPT>
955 Min number of TRES affected by an event.
956 Nodes=<comma separated list of node names>
959 Node names affected by an event.
960 Reason=<comma separated list of reasons>
963 Reason an event happened.
964 Start=<OPT>
967 Period start of events. Default is 00:00:00 of previous day,
968 unless states are given with the States= spec events. If this
969 is the case the default behavior is to return events currently
970 in the states specified.
971 Valid time formats are...
973 HH:MM[:SS] [AM|PM]
975 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
976 MM/DD[/YY]-HH:MM[:SS]
977 YYYY-MM-DD[THH:MM[:SS]]
978 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
979 States=<comma separated list of states>
982 State of a node in a node event. If this is set, the event type
983 is set automatically to Node.
984 User=<comma separated list of users>
987 Query against users who set the event. If this is set, the
988 event type is set automatically to Node since only user slurm
989 can perform a cluster event.
990
994 Cluster
995 The name of the cluster event happened on.
996 ClusterNodes
999 The hostlist of nodes on a cluster in a cluster event.
1000 Duration
1003 Time period the event was around for.
1004 End Period when event ended.
1007 Event Name of the event.
1010 EventRaw
1013 Numeric value of the name of the event.
1014 NodeName
1017 The node affected by the event. In a cluster event, this is
1018 blank.
1019 Reason The reason an event happened.
1022 Start Period when event started.
1025 State On a node event this is the formatted state of the node during
1028 the event.
1029 StateRaw
1032 On a node event this is the numeric value of the state of the
1033 node during the event.
1034 TRES Number of TRES involved with the event.
1037 User On a node event this is the user who caused the event to happen.
1040
1044 Clusters[+|-]=<comma separated list of cluster names>
1045 List of clusters to add/remove to a federation. A blank value
1046 (e.g. clusters=) will remove all federations for the federation.
1047 NOTE: a cluster can only be a member of one federation.
1048 Name=<name>
1051 The name of the federation.
1052 Tree Display federations in a hierarchical fashion.
1055
1058 Features
1059 The list of features on the cluster.
1060 Federation
1063 The name of the federation.
1064 Cluster
1067 Name of the cluster that is a member of the federation.
1068 FedState
1071 The state of the cluster in the federation.
1072 FedStateRaw
1075 Numeric value of the name of the FedState.
1076 Index The index of the cluster in the federation.
1079
1083 Comment=<comment>
1084 The job's comment string when the AccountingStoreJobComment pa‐
1085 rameter in the slurm.conf file is set (or defaults) to YES. The
1086 user can only modify the comment string of their own job.
1087 Cluster=<cluster_list>
1090 List of clusters to alter jobs on, defaults to local cluster.
1091 DerivedExitCode=<derived_exit_code>
1094 The derived exit code can be modified after a job completes
1095 based on the user's judgment of whether the job succeeded or
1096 failed. The user can only modify the derived exit code of their
1097 own job.
1098 EndTime
1101 Jobs must end before this time to be modified. Format output is,
1102 YYYY-MM-DDTHH:MM:SS, unless changed through the SLURM_TIME_FOR‐
1103 MAT environment variable.
1104 JobID=<jobid_list>
1107 The id of the job to change. Not needed if altering multiple
1108 jobs using wckey specification.
1109 NewWCKey=<newwckey>
1112 Use to rename a wckey on job(s) in the accounting database
1113 StartTime
1116 Jobs must start at or after this time to be modified in the same
1117 format as EndTime.
1118 User=<user_list>
1121 Used to specify the jobs of users jobs to alter.
1122 WCKey=<wckey_list>
1125 Used to specify the wckeys to alter.
1126 The DerivedExitCode, Comment and WCKey fields are the only
1129 fields of a job record in the database that can be modified af‐
1130 ter job completion.
1131
1134 The sacct command is the exclusive command to display job records from
1135 the Slurm database.
1136
1139 NOTE: The group limits (GrpJobs, GrpNodes, etc.) are tested when a job
1140 is being considered for being allocated resources. If starting a job
1141 would cause any of its group limit to be exceeded, that job will not be
1142 considered for scheduling even if that job might preempt other jobs
1143 which would release sufficient group resources for the pending job to
1144 be initiated.
1145 Flags Used by the slurmctld to override or enforce certain character‐
1148 istics.
1149 Valid options are
1150 DenyOnLimit
1152 If set, jobs using this QOS will be rejected at submis‐
1153 sion time if they do not conform to the QOS 'Max' limits.
1154 Group limits will also be treated like 'Max' limits as
1155 well and will be denied if they go over. By default jobs
1156 that go over these limits will pend until they conform.
1157 This currently only applies to QOS and Association lim‐
1158 its.
1159 EnforceUsageThreshold
1161 If set, and the QOS also has a UsageThreshold, any jobs
1162 submitted with this QOS that fall below the UsageThresh‐
1163 old will be held until their Fairshare Usage goes above
1164 the Threshold.
1165 NoDecay
1167 If set, this QOS will not have its GrpTRESMins, GrpWall
1168 and UsageRaw decayed by the slurm.conf PriorityDecay‐
1169 HalfLife or PriorityUsageResetPeriod settings. This al‐
1170 lows a QOS to provide aggregate limits that, once con‐
1171 sumed, will not be replenished automatically. Such a QOS
1172 will act as a time-limited quota of resources for an as‐
1173 sociation that has access to it. Account/user usage will
1174 still be decayed for associations using the QOS. The QOS
1175 GrpTRESMins and GrpWall limits can be increased or the
1176 QOS RawUsage value reset to 0 (zero) to again allow jobs
1177 submitted with this QOS to be queued (if DenyOnLimit is
1178 set) or run (pending with QOSGrp{TRES}MinutesLimit or
1179 QOSGrpWallLimit reasons, where {TRES} is some type of
1180 trackable resource).
1181 NoReserve
1183 If this flag is set and backfill scheduling is used, jobs
1184 using this QOS will not reserve resources in the backfill
1185 schedule's map of resources allocated through time. This
1186 flag is intended for use with a QOS that may be preempted
1187 by jobs associated with all other QOS (e.g use with a
1188 "standby" QOS). If this flag is used with a QOS which can
1189 not be preempted by all other QOS, it could result in
1190 starvation of larger jobs.
1191 PartitionMaxNodes
1193 If set jobs using this QOS will be able to override the
1194 requested partition's MaxNodes limit.
1195 PartitionMinNodes
1197 If set jobs using this QOS will be able to override the
1198 requested partition's MinNodes limit.
1199 OverPartQOS
1201 If set jobs using this QOS will be able to override any
1202 limits used by the requested partition's QOS limits.
1203 PartitionTimeLimit
1205 If set jobs using this QOS will be able to override the
1206 requested partition's TimeLimit.
1207 RequiresReservation
1209 If set jobs using this QOS must designate a reservation
1210 when submitting a job. This option can be useful in re‐
1211 stricting usage of a QOS that may have greater preemptive
1212 capability or additional resources to be allowed only
1213 within a reservation.
1214 UsageFactorSafe
1216 If set, and AccountingStorageEnforce includes Safe, jobs
1217 will only be able to run if the job can run to completion
1218 with the UsageFactor applied.
1219 GraceTime
1222 Preemption grace time to be extended to a job which has been se‐
1223 lected for preemption.
1224 GrpTRESMins
1227 The total number of TRES minutes that can possibly be used by
1228 past, present and future jobs running from this QOS.
1229 GrpTRESRunMins
1232 Used to limit the combined total number of TRES minutes used by
1233 all jobs running with this QOS. This takes into consideration
1234 time limit of running jobs and consumes it, if the limit is
1235 reached no new jobs are started until other jobs finish to allow
1236 time to free up.
1237 GrpTRES
1240 Maximum number of TRES running jobs are able to be allocated in
1241 aggregate for this QOS.
1242 GrpJobs
1245 Maximum number of running jobs in aggregate for this QOS.
1246 GrpJobsAccrue
1249 Maximum number of pending jobs in aggregate able to accrue age
1250 priority for this QOS.
1251 GrpSubmitJobs
1254 Maximum number of jobs which can be in a pending or running
1255 state at any time in aggregate for this QOS.
1256 NOTE: This setting shows up in the sacctmgr output as GrpSubmit.
1258 GrpWall
1261 Maximum wall clock time running jobs are able to be allocated in
1262 aggregate for this QOS. If this limit is reached submission re‐
1263 quests will be denied and the running jobs will be killed.
1264 ID The id of the QOS.
1266 MaxJobsAccruePerAccount
1269 Maximum number of pending jobs an account (or subacct) can have
1270 accruing age priority at any given time.
1271 MaxJobsAccruePerUser
1274 Maximum number of pending jobs a user can have accruing age pri‐
1275 ority at any given time.
1276 MaxJobsPerAccount
1279 Maximum number of jobs each account is allowed to run at one
1280 time.
1281 MaxJobsPerUser
1284 Maximum number of jobs each user is allowed to run at one time.
1285 MaxSubmitJobsPerAccount
1288 Maximum number of jobs pending or running state at any time per
1289 account.
1290 MaxSubmitJobsPerUser
1293 Maximum number of jobs pending or running state at any time per
1294 user.
1295 MaxTRESMinsPerJob
1298 Maximum number of TRES minutes each job is able to use.
1299 NOTE: This setting shows up in the sacctmgr output as Max‐
1301 TRESMins.
1302 MaxTRESPerAccount
1305 Maximum number of TRES each account is able to use.
1306 MaxTRESPerJob
1309 Maximum number of TRES each job is able to use.
1310 NOTE: This setting shows up in the sacctmgr output as MaxTRES.
1312 MaxTRESPerNode
1315 Maximum number of TRES each node in a job allocation can use.
1316 MaxTRESPerUser
1319 Maximum number of TRES each user is able to use.
1320 MaxWallDurationPerJob
1323 Maximum wall clock time each job is able to use.
1324 NOTE: This setting shows up in the sacctmgr output as MaxWall.
1326 MinPrioThreshold
1329 Minimum priority required to reserve resources when scheduling.
1330 MinTRESPerJob
1333 Minimum number of TRES each job running under this QOS must re‐
1334 quest. Otherwise the job will pend until modified.
1335 NOTE: This setting shows up in the sacctmgr output as MinTRES.
1337 Name Name of the QOS.
1340 Preempt
1343 Other QOS' this QOS can preempt.
1344 NOTE: The Priority of a QOS is NOT related to QOS preemption,
1346 only Preempt is used to define which QOS can preempt others.
1347 PreemptExemptTime
1350 Specifies a minimum run time for jobs of this QOS before they
1351 are considered for preemption. This QOS option takes precedence
1352 over the global PreemptExemptTime. Setting to -1 disables the
1353 option, allowing another QOS or the global option to take ef‐
1354 fect. Setting to 0 indicates no minimum run time and supersedes
1355 the lower priority QOS (see OverPartQOS) and/or the global op‐
1356 tion in slurm.conf.
1357 PreemptMode
1360 Mechanism used to preempt jobs or enable gang scheduling for
1361 this QOS when the cluster PreemptType is set to preempt/qos.
1362 This QOS-specific PreemptMode will override the cluster-wide
1363 PreemptMode for this QOS. Unsetting the QOS specific Preempt‐
1364 Mode, by specifying "OFF", "" or "Cluster", makes it use the de‐
1365 fault cluster-wide PreemptMode.
1366 See the description of the cluster-wide PreemptMode parameter
1367 for further details of the available modes.
1368 Priority
1371 What priority will be added to a job's priority when using this
1372 QOS.
1373 NOTE: The Priority of a QOS is NOT related to QOS preemption,
1375 see Preempt instead.
1376 RawUsage=<value>
1379 This allows an administrator to reset the raw usage accrued to a
1380 QOS. The only value currently supported is 0 (zero). This is a
1381 settable specification only - it cannot be used as a filter to
1382 list accounts.
1383 UsageFactor
1386 Usage factor when running with this QOS. See below for more de‐
1387 tails.
1388 UsageThreshold
1391 A float representing the lowest fairshare of an association al‐
1392 lowable to run a job. If an association falls below this
1393 threshold and has pending jobs or submits new jobs those jobs
1394 will be held until the usage goes back above the threshold. Use
1395 sshare to see current shares on the system.
1396 WithDeleted
1399 Display information with previously deleted data.
1400
1404 Description
1405 An arbitrary string describing a QOS.
1406 GraceTime
1409 Preemption grace time to be extended to a job which has been se‐
1410 lected for preemption in the format of hh:mm:ss. The default
1411 value is zero, no preemption grace time is allowed on this par‐
1412 tition. NOTE: This value is only meaningful for QOS Preempt‐
1413 Mode=CANCEL.
1414 GrpTRESMins
1417 The total number of TRES minutes that can possibly be used by
1418 past, present and future jobs running from this QOS. To clear a
1419 previously set value use the modify command with a new value of
1420 -1 for each TRES id. NOTE: This limit only applies when using
1421 the Priority Multifactor plugin. The time is decayed using the
1422 value of PriorityDecayHalfLife or PriorityUsageResetPeriod as
1423 set in the slurm.conf. When this limit is reached all associ‐
1424 ated jobs running will be killed and all future jobs submitted
1425 with this QOS will be delayed until they are able to run inside
1426 the limit.
1427 GrpTRES
1430 Maximum number of TRES running jobs are able to be allocated in
1431 aggregate for this QOS. To clear a previously set value use the
1432 modify command with a new value of -1 for each TRES id.
1433 GrpJobs
1436 Maximum number of running jobs in aggregate for this QOS. To
1437 clear a previously set value use the modify command with a new
1438 value of -1.
1439 GrpJobsAccrue
1442 Maximum number of pending jobs in aggregate able to accrue age
1443 priority for this QOS. To clear a previously set value use the
1444 modify command with a new value of -1.
1445 GrpSubmitJobs
1448 Maximum number of jobs which can be in a pending or running
1449 state at any time in aggregate for this QOS. To clear a previ‐
1450 ously set value use the modify command with a new value of -1.
1451 NOTE: This setting shows up in the sacctmgr output as GrpSubmit.
1453 GrpWall
1456 Maximum wall clock time running jobs are able to be allocated in
1457 aggregate for this QOS. To clear a previously set value use the
1458 modify command with a new value of -1. NOTE: This limit only
1459 applies when using the Priority Multifactor plugin. The time is
1460 decayed using the value of PriorityDecayHalfLife or Priori‐
1461 tyUsageResetPeriod as set in the slurm.conf. When this limit is
1462 reached all associated jobs running will be killed and all fu‐
1463 ture jobs submitted with this QOS will be delayed until they are
1464 able to run inside the limit.
1465 MaxTRESMinsPerJob
1468 Maximum number of TRES minutes each job is able to use. To
1469 clear a previously set value use the modify command with a new
1470 value of -1 for each TRES id.
1471 NOTE: This setting shows up in the sacctmgr output as Max‐
1473 TRESMins.
1474 MaxTRESPerAccount
1477 Maximum number of TRES each account is able to use. To clear a
1478 previously set value use the modify command with a new value of
1479 -1 for each TRES id.
1480 MaxTRESPerJob
1483 Maximum number of TRES each job is able to use. To clear a pre‐
1484 viously set value use the modify command with a new value of -1
1485 for each TRES id.
1486 NOTE: This setting shows up in the sacctmgr output as MaxTRES.
1488 MaxTRESPerNode
1491 Maximum number of TRES each node in a job allocation can use.
1492 To clear a previously set value use the modify command with a
1493 new value of -1 for each TRES id.
1494 MaxTRESPerUser
1497 Maximum number of TRES each user is able to use. To clear a
1498 previously set value use the modify command with a new value of
1499 -1 for each TRES id.
1500 MaxJobsPerAccount
1503 Maximum number of jobs each account is allowed to run at one
1504 time. To clear a previously set value use the modify command
1505 with a new value of -1.
1506 MaxJobsPerUser
1509 Maximum number of jobs each user is allowed to run at one time.
1510 To clear a previously set value use the modify command with a
1511 new value of -1.
1512 MaxSubmitJobsPerAccount
1515 Maximum number of jobs pending or running state at any time per
1516 account. To clear a previously set value use the modify command
1517 with a new value of -1.
1518 MaxSubmitJobsPerUser
1521 Maximum number of jobs pending or running state at any time per
1522 user. To clear a previously set value use the modify command
1523 with a new value of -1.
1524 MaxWallDurationPerJob
1527 Maximum wall clock time each job is able to use. <max wall>
1528 format is <min> or <min>:<sec> or <hr>:<min>:<sec> or
1529 <days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded
1530 in minutes with rounding as needed. To clear a previously set
1531 value use the modify command with a new value of -1.
1532 NOTE: This setting shows up in the sacctmgr output as MaxWall.
1534 MinPrioThreshold
1537 Minimum priority required to reserve resources when scheduling.
1538 To clear a previously set value use the modify command with a
1539 new value of -1.
1540 MinTRES
1543 Minimum number of TRES each job running under this QOS must re‐
1544 quest. Otherwise the job will pend until modified. To clear a
1545 previously set value use the modify command with a new value of
1546 -1 for each TRES id.
1547 Name Name of the QOS. Needed for creation.
1550 Preempt
1553 Other QOS' this QOS can preempt. Setting a Preempt to '' (two
1554 single quotes with nothing between them) restores its default
1555 setting. You can also use the operator += and -= to add or re‐
1556 move certain QOS's from a QOS list.
1557 PreemptMode
1560 Mechanism used to preempt jobs of this QOS if the clusters Pre‐
1561 emptType is configured to preempt/qos. The default preemption
1562 mechanism is specified by the cluster-wide PreemptMode configu‐
1563 ration parameter. Possible values are "Cluster" (meaning use
1564 cluster default), "Cancel", and "Requeue". This option is not
1565 compatible with PreemptMode=OFF or PreemptMode=SUSPEND (i.e.
1566 preempted jobs must be removed from the resources).
1567 Priority
1570 What priority will be added to a job's priority when using this
1571 QOS. To clear a previously set value use the modify command
1572 with a new value of -1.
1573 UsageFactor
1576 A float that is factored into a job’s TRES usage (e.g. RawUsage,
1577 TRESMins, TRESRunMins). For example, if the usagefactor was 2,
1578 for every TRESBillingUnit second a job ran it would count for 2.
1579 If the usagefactor was .5, every second would only count for
1580 half of the time. A setting of 0 would add no timed usage from
1581 the job.
1582 The usage factor only applies to the job's QOS and not the par‐
1584 tition QOS.
1585 If the UsageFactorSafe flag is set and AccountingStorageEnforce
1587 includes Safe, jobs will only be able to run if the job can run
1588 to completion with the UsageFactor applied.
1589 If the UsageFactorSafe flag is not set and AccountingStorageEn‐
1591 force includes Safe, a job will be able to be scheduled without
1592 the UsageFactor applied and will be able to run without being
1593 killed due to limits.
1594 If the UsageFactorSafe flag is not set and AccountingStorageEn‐
1596 force does not include Safe, a job will be able to be scheduled
1597 without the UsageFactor applied and could be killed due to lim‐
1598 its.
1599 See AccountingStorageEnforce in slurm.conf man page.
1601 Default is 1. To clear a previously set value use the modify
1603 command with a new value of -1.
1604
1607 Clusters=<comma separated list of cluster names>
1608 List the reservations of the cluster(s). Default is the cluster
1609 where the command was run.
1610 End=<OPT>
1613 Period ending of reservations. Default is now.
1614 Valid time formats are...
1616 HH:MM[:SS] [AM|PM]
1618 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
1619 MM/DD[/YY]-HH:MM[:SS]
1620 YYYY-MM-DD[THH:MM[:SS]]
1621 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
1622 ID=<OPT>
1625 Comma separated list of reservation ids.
1626 Names=<OPT>
1629 Comma separated list of reservation names.
1630 Nodes=<comma separated list of node names>
1633 Node names where reservation ran.
1634 Start=<OPT>
1637 Period start of reservations. Default is 00:00:00 of current
1638 day.
1639 Valid time formats are...
1641 HH:MM[:SS] [AM|PM]
1643 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
1644 MM/DD[/YY]-HH:MM[:SS]
1645 YYYY-MM-DD[THH:MM[:SS]]
1646 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
1647
1650 Associations
1651 The id's of the associations able to run in the reservation.
1652 Cluster
1655 Name of cluster reservation was on.
1656 End End time of reservation.
1659 Flags Flags on the reservation.
1662 ID Reservation ID.
1665 Name Name of this reservation.
1668 NodeNames
1671 List of nodes in the reservation.
1672 Start Start time of reservation.
1675 TRES List of TRES in the reservation.
1678 UnusedWall
1681 Wall clock time in seconds unused by any job. A job's allocated
1682 usage is its run time multiplied by the ratio of its CPUs to the
1683 total number of CPUs in the reservation. For example, a job us‐
1684 ing all the CPUs in the reservation running for 1 minute would
1685 reduce unused_wall by 1 minute.
1686
1690 Clusters=<name list> Comma separated list of cluster names on which
1691 specified resources are to be available. If no names are designated
1692 then the clusters already allowed to use this resource will be altered.
1693 Count=<OPT>
1696 Number of software resources of a specific name configured on
1697 the system being controlled by a resource manager.
1698 Descriptions=
1701 A brief description of the resource.
1702 Flags=<OPT>
1705 Flags that identify specific attributes of the system resource.
1706 At this time no flags have been defined.
1707 ServerType=<OPT>
1710 The type of a software resource manager providing the licenses.
1711 For example FlexNext Publisher Flexlm license server or Reprise
1712 License Manager RLM.
1713 Names=<OPT>
1716 Comma separated list of the name of a resource configured on the
1717 system being controlled by a resource manager. If this resource
1718 is seen on the slurmctld its name will be name@server to distin‐
1719 guish it from local resources defined in a slurm.conf.
1720 PercentAllowed=<percent allowed>
1723 Percentage of a specific resource that can be used on specified
1724 cluster.
1725 Server=<OPT>
1728 The name of the server serving up the resource. Default is
1729 'slurmdb' indicating the licenses are being served by the data‐
1730 base.
1731 Type=<OPT>
1734 The type of the resource represented by this record. Currently
1735 the only valid type is License.
1736 WithClusters
1739 Display the clusters percentage of resources. If a resource
1740 hasn't been given to a cluster the resource will not be dis‐
1741 played with this flag.
1742 NOTE: Resource is used to define each resource configured on a system
1745 available for usage by Slurm clusters.
1746
1749 Cluster
1750 Name of cluster resource is given to.
1751 Count The count of a specific resource configured on the system glob‐
1754 ally.
1755 Allocated
1758 The percent of licenses allocated to a cluster.
1759 Description
1762 Description of the resource.
1763 ServerType
1766 The type of the server controlling the licenses.
1767 Name Name of this resource.
1770 Server Server serving up the resource.
1773 Type Type of resource this record represents.
1776
1779 Cluster
1780 Name of cluster job ran on.
1781 ID Id of the job.
1784 Name Name of the job.
1787 Partition
1790 Partition job ran on.
1791 State Current State of the job in the database.
1794 TimeStart
1797 Time job started running.
1798 TimeEnd
1801 Current recorded time of the end of the job.
1802
1805 Accounts=<comma separated list of account names>
1806 Only print out the transactions affecting specified accounts.
1807 Action=<Specific action the list will display>
1810 Only display transactions of the specified action type.
1811 Actor=<Specific name the list will display>
1814 Only display transactions done by a certain person.
1815 Clusters=<comma separated list of cluster names>
1818 Only print out the transactions affecting specified clusters.
1819 End=<Date and time of last transaction to return>
1822 Return all transactions before this Date and time. Default is
1823 now.
1824 Start=<Date and time of first transaction to return>
1827 Return all transactions after this Date and time. Default is
1828 epoch.
1829 Valid time formats for End and Start are...
1831 HH:MM[:SS] [AM|PM]
1833 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
1834 MM/DD[/YY]-HH:MM[:SS]
1835 YYYY-MM-DD[THH:MM[:SS]]
1836 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
1837 Users=<comma separated list of user names>
1840 Only print out the transactions affecting specified users.
1841 WithAssoc
1844 Get information about which associations were affected by the
1845 transactions.
1846
1850 Action Displays the type of Action that took place.
1851 Actor Displays the Actor to generate a transaction.
1854 Info Displays details of the transaction.
1857 TimeStamp
1860 Displays when the transaction occurred.
1861 Where Displays details of the constraints for the transaction.
1864 NOTE: If using the WithAssoc option you can also view the information
1866 about the various associations the transaction affected. The Associa‐
1867 tion format fields are described in the LIST/SHOW ASSOCIATION FORMAT
1868 OPTIONS section.
1869
1873 Account=<account>
1874 Account name to add this user to.
1875 AdminLevel=<level>
1878 Admin level of user. Valid levels are None, Operator, and Ad‐
1879 min.
1880 Cluster=<cluster>
1883 Specific cluster to add user to the account on. Default is all
1884 in system.
1885 DefaultAccount=<account>
1888 Identify the default bank account name to be used for a job if
1889 none is specified at submission time.
1890 DefaultWCKey=<defaultwckey>
1893 Identify the default Workload Characterization Key.
1894 Name=<name>
1897 Name of user.
1898 NewName=<newname>
1901 Use to rename a user in the accounting database
1902 Partition=<name>
1905 Partition name.
1906 RawUsage=<value>
1909 This allows an administrator to reset the raw usage accrued to a
1910 user. The only value currently supported is 0 (zero). This is
1911 a settable specification only - it cannot be used as a filter to
1912 list users.
1913 WCKeys=<wckeys>
1916 Workload Characterization Key values.
1917 WithAssoc
1920 Display all associations for this user.
1921 WithCoord
1924 Display all accounts a user is coordinator for.
1925 WithDeleted
1928 Display information with previously deleted data.
1929 NOTE: If using the WithAssoc option you can also query against associa‐
1931 tion specific information to view only certain associations this user
1932 may have. These extra options can be found in the SPECIFICATIONS FOR
1933 ASSOCIATIONS section. You can also use the general specifications list
1934 above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES sec‐
1935 tion.
1936
1940 AdminLevel
1941 Admin level of user.
1942 DefaultAccount
1945 The user's default account.
1946 Coordinators
1949 List of users that are a coordinator of the account. (Only
1950 filled in when using the WithCoordinator option.)
1951 User The name of a user.
1954 NOTE: If using the WithAssoc option you can also view the information
1956 about the various associations the user may have on all the clusters in
1957 the system. The association information can be filtered. Note that all
1958 the users in the database will always be shown as filter only takes ef‐
1959 fect over the association data. The Association format fields are de‐
1960 scribed in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
1961
1965 WCKey Workload Characterization Key.
1966 Cluster
1969 Specific cluster for the WCKey.
1970 User The name of a user for the WCKey.
1973 NOTE: If using the WithAssoc option you can also view the information
1975 about the various associations the user may have on all the clusters in
1976 the system. The Association format fields are described in the
1977 LIST/SHOW ASSOCIATION FORMAT OPTIONS section.
1978
1981 Name The name of the trackable resource. This option is required for
1982 TRES types BB (Burst buffer), GRES, and License. Types CPU, En‐
1983 ergy, Memory, and Node do not have Names. For example if GRES
1984 is the type then name is the denomination of the GRES itself
1985 e.g. GPU.
1986 ID The identification number of the trackable resource as it ap‐
1989 pears in the database.
1990 Type The type of the trackable resource. Current types are BB (Burst
1993 buffer), CPU, Energy, GRES, License, Memory, and Node.
1994
1997 Trackable RESources (TRES) are used in many QOS or Association limits.
1998 When setting the limits they are comma separated list. Each TRES has a
1999 different limit, i.e. GrpTRESMins=cpu=10,mem=20 would make 2 different
2000 limits 1 for 10 cpu minutes and 1 for 20 MB memory minutes. This is
2001 the case for each limit that deals with TRES. To remove the limit -1
2002 is used i.e. GrpTRESMins=cpu=-1 would remove only the cpu TRES limit.
2003 NOTE: When dealing with Memory as a TRES all limits are in MB.
2005 NOTE: The Billing TRES is calculated from a partition's TRESBilling‐
2007 Weights. It is temporarily calculated during scheduling for each parti‐
2008 tion to enforce billing TRES limits. The final Billing TRES is calcu‐
2009 lated after the job has been allocated resources. The final number can
2010 be seen in scontrol show jobs and sacct output.
2011
2014 When using the format option for listing various fields you can put a
2015 %NUMBER afterwards to specify how many characters should be printed.
2016 e.g. format=name%30 will print 30 characters of field name right justi‐
2018 fied. A -30 will print 30 characters left justified.
2019
2022 sacctmgr has the capability to load and dump Slurm association data to
2023 and from a file. This method can easily add a new cluster or copy an
2024 existing cluster's associations into a new cluster with similar ac‐
2025 counts. Each file contains Slurm association data for a single cluster.
2026 Comments can be put into the file with the # character. Each line of
2027 information must begin with one of the four titles; Cluster, Parent,
2028 Account or User. Following the title is a space, dash, space, entity
2029 value, then specifications. Specifications are colon separated. If any
2030 variable, such as an Organization name, has a space in it, surround the
2031 name with single or double quotes.
2032 To create a file of associations you can run
2034 sacctmgr dump tux file=tux.cfg
2035 To load a previously created file you can run
2037 sacctmgr load file=tux.cfg
2038 sacctmgr dump/load must be run as a Slurm administrator or root. If us‐
2040 ing sacctmgr load on a database without any associations, it must be
2041 run as root (because there aren't any users in the database yet).
2042 Other options for load are:
2044 clean - delete what was already there and start from scratch
2045 with this information.
2046 Cluster= - specify a different name for the cluster than that
2047 which is in the file.
2048 Since the associations in the system follow a hierarchy, so does the
2050 file. Anything that is a parent needs to be defined before any chil‐
2051 dren. The only exception is the understood 'root' account. This is
2052 always a default for any cluster and does not need to be defined.
2053 To edit/create a file start with a cluster line for the new cluster:
2055 Cluster - cluster_name:MaxTRESPerJob=node=15
2057 Anything included on this line will be the default for all associations
2059 on this cluster. The options for the cluster are:
2060 GrpTRESMins=
2062 The total number of TRES minutes that can possibly be
2063 used by past, present and future jobs running from this
2064 association and its children.
2065 GrpTRESRunMins=
2067 Used to limit the combined total number of TRES minutes
2068 used by all jobs running with this association and its
2069 children. This takes into consideration time limit of
2070 running jobs and consumes it, if the limit is reached no
2071 new jobs are started until other jobs finish to allow
2072 time to free up.
2073 GrpTRES=
2075 Maximum number of TRES running jobs are able to be allo‐
2076 cated in aggregate for this association and all associa‐
2077 tions which are children of this association.
2078 GrpJobs=
2080 Maximum number of running jobs in aggregate for this as‐
2081 sociation and all associations which are children of this
2082 association.
2083 GrpJobsAccrue=
2085 Maximum number of pending jobs in aggregate able to ac‐
2086 crue age priority for this association and all associa‐
2087 tions which are children of this association.
2088 GrpNodes=
2090 Maximum number of nodes running jobs are able to be allo‐
2091 cated in aggregate for this association and all associa‐
2092 tions which are children of this association.
2093 GrpSubmitJobs=
2095 Maximum number of jobs which can be in a pending or run‐
2096 ning state at any time in aggregate for this association
2097 and all associations which are children of this associa‐
2098 tion.
2099 GrpWall=
2101 Maximum wall clock time running jobs are able to be allo‐
2102 cated in aggregate for this association and all associa‐
2103 tions which are children of this association.
2104 FairShare=
2106 Number used in conjunction with other associations to de‐
2107 termine job priority.
2108 MaxJobs=
2110 Maximum number of jobs the children of this association
2111 can run.
2112 MaxTRESPerJob=
2114 Maximum number of trackable resources per job the chil‐
2115 dren of this association can run.
2116 MaxWallDurationPerJob=
2118 Maximum time (not related to job size) children of this
2119 accounts jobs can run.
2120 QOS=
2122 Comma separated list of Quality of Service names (Defined
2123 in sacctmgr).
2124 After the entry for the root account you will have entries for the
2126 other accounts on the system. The entries will look similar to this ex‐
2127 ample:
2128 Parent - root
2130 Account - cs:MaxTRESPerJob=node=5:MaxJobs=4:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC'
2131 Parent - cs
2132 Account - test:MaxTRESPerJob=node=1:MaxJobs=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test'
2133 Any of the options after a ':' can be left out and they can be in any
2135 order. If you want to add any sub accounts just list the Parent THAT
2136 HAS ALREADY BEEN CREATED before the account you are adding.
2137 Account options are:
2139 Description=
2141 A brief description of the account.
2142 GrpTRESMins=
2144 Maximum number of TRES hours running jobs are able to be
2145 allocated in aggregate for this association and all asso‐
2146 ciations which are children of this association. Grp‐
2147 TRESRunMins= Used to limit the combined total number of
2148 TRES minutes used by all jobs running with this associa‐
2149 tion and its children. This takes into consideration
2150 time limit of running jobs and consumes it, if the limit
2151 is reached no new jobs are started until other jobs fin‐
2152 ish to allow time to free up.
2153 GrpTRES=
2155 Maximum number of TRES running jobs are able to be allo‐
2156 cated in aggregate for this association and all associa‐
2157 tions which are children of this association.
2158 GrpJobs=
2160 Maximum number of running jobs in aggregate for this as‐
2161 sociation and all associations which are children of this
2162 association.
2163 GrpJobsAccrue
2165 Maximum number of pending jobs in aggregate able to ac‐
2166 crue age priority for this association and all associa‐
2167 tions which are children of this association.
2168 GrpNodes=
2170 Maximum number of nodes running jobs are able to be allo‐
2171 cated in aggregate for this association and all associa‐
2172 tions which are children of this association.
2173 GrpSubmitJobs=
2175 Maximum number of jobs which can be in a pending or run‐
2176 ning state at any time in aggregate for this association
2177 and all associations which are children of this associa‐
2178 tion.
2179 GrpWall=
2181 Maximum wall clock time running jobs are able to be allo‐
2182 cated in aggregate for this association and all associa‐
2183 tions which are children of this association.
2184 FairShare=
2186 Number used in conjunction with other associations to de‐
2187 termine job priority.
2188 MaxJobs=
2190 Maximum number of jobs the children of this association
2191 can run.
2192 MaxNodesPerJob=
2194 Maximum number of nodes per job the children of this as‐
2195 sociation can run.
2196 MaxWallDurationPerJob=
2198 Maximum time (not related to job size) children of this
2199 accounts jobs can run.
2200 Organization=
2202 Name of organization that owns this account.
2203 QOS(=,+=,-=)
2205 Comma separated list of Quality of Service names (Defined
2206 in sacctmgr).
2207 To add users to an account add a line after the Parent line, similar to
2210 this:
2211 Parent - test
2213 User - adam:MaxTRESPerJob=node:2:MaxJobs=3:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test'
2214 User options are:
2217 AdminLevel=
2219 Type of admin this user is (Administrator, Operator)
2220 Must be defined on the first occurrence of the user.
2221 Coordinator=
2223 Comma separated list of accounts this user is coordinator
2224 over
2225 Must be defined on the first occurrence of the user.
2226 DefaultAccount=
2228 System wide default account name
2229 Must be defined on the first occurrence of the user.
2230 FairShare=
2232 Number used in conjunction with other associations to de‐
2233 termine job priority.
2234 MaxJobs=
2236 Maximum number of jobs this user can run.
2237 MaxTRESPerJob=
2239 Maximum number of trackable resources per job this user
2240 can run.
2241 MaxWallDurationPerJob=
2243 Maximum time (not related to job size) this user can run.
2244 QOS(=,+=,-=)
2246 Comma separated list of Quality of Service names (Defined
2247 in sacctmgr).
2248
2251 Sacctmgr has the capability to archive to a flatfile and or load that
2252 data if needed later. The archiving is usually done by the slurmdbd
2253 and it is highly recommended you only do it through sacctmgr if you
2254 completely understand what you are doing. For slurmdbd options see
2255 "man slurmdbd" for more information. Loading data into the database
2256 can be done from these files to either view old data or regenerate
2257 rolled up data.
2258 archive dump
2261 Dump accounting data to file. Data will not be archived unless the cor‐
2262 responding purge option is included in this command or in slur‐
2263 mdbd.conf. This operation cannot be rolled back once executed. If one
2264 of the following options is not specified when sacctmgr is called, the
2265 value configured in slurmdbd.conf is used.
2266 Directory=
2269 Directory to store the archive data.
2270 Events Archive Events. If not specified and PurgeEventAfter is set all
2272 event data removed will be lost permanently.
2273 Jobs Archive Jobs. If not specified and PurgeJobAfter is set all job
2275 data removed will be lost permanently.
2276 PurgeEventAfter=
2278 Purge cluster event records older than time stated in months.
2279 If you want to purge on a shorter time period you can include
2280 hours, or days behind the numeric value to get those more fre‐
2281 quent purges. (e.g. a value of '12hours' would purge everything
2282 older than 12 hours.)
2283 PurgeJobAfter=
2285 Purge job records older than time stated in months. If you want
2286 to purge on a shorter time period you can include hours, or days
2287 behind the numeric value to get those more frequent purges.
2288 (e.g. a value of '12hours' would purge everything older than 12
2289 hours.)
2290 PurgeStepAfter=
2292 Purge step records older than time stated in months. If you
2293 want to purge on a shorter time period you can include hours, or
2294 days behind the numeric value to get those more frequent purges.
2295 (e.g. a value of '12hours' would purge everything older than 12
2296 hours.)
2297 PurgeSuspendAfter=
2299 Purge job suspend records older than time stated in months. If
2300 you want to purge on a shorter time period you can include
2301 hours, or days behind the numeric value to get those more fre‐
2302 quent purges. (e.g. a value of '12hours' would purge everything
2303 older than 12 hours.)
2304 Script=
2306 Run this script instead of the generic form of archive to flat
2307 files.
2308 Steps Archive Steps. If not specified and PurgeStepAfter is set all
2310 step data removed will be lost permanently.
2311 Suspend
2313 Archive Suspend Data. If not specified and PurgeSuspendAfter is
2314 set all suspend data removed will be lost permanently.
2315 archive load
2318 Load in to the database previously archived data. The archive file will
2319 not be loaded if the records already exist in the database - therefore,
2320 trying to load an archive file more than once will result in an error.
2321 When this data is again archived and purged from the database, if the
2322 old archive file is still in the directory ArchiveDir, a new archive
2323 file will be created (see ArchiveDir in the slurmdbd.conf man page), so
2324 the old file will not be overwritten and these files will have dupli‐
2325 cate records.
2326 File= File to load into database. The specified file must exist on the
2329 slurmdbd host, which is not necessarily the machine running the
2330 command.
2331 Insert=
2333 SQL to insert directly into the database. This should be used
2334 very cautiously since this is writing your sql into the data‐
2335 base.
2336
2339 Executing sacctmgr sends a remote procedure call to slurmdbd. If enough
2340 calls from sacctmgr or other Slurm client commands that send remote
2341 procedure calls to the slurmdbd daemon come in at once, it can result
2342 in a degradation of performance of the slurmdbd daemon, possibly re‐
2343 sulting in a denial of service.
2344 Do not run sacctmgr or other Slurm client commands that send remote
2346 procedure calls to slurmdbd from loops in shell scripts or other pro‐
2347 grams. Ensure that programs limit calls to sacctmgr to the minimum
2348 necessary for the information you are trying to gather.
2349
2352 Some sacctmgr options may be set via environment variables. These envi‐
2353 ronment variables, along with their corresponding options, are listed
2354 below. (Note: commandline options will always override these settings)
2355 SLURM_CONF The location of the Slurm configuration file.
2357
2360 NOTE: There is an order to set up accounting associations. You must
2361 define clusters before you add accounts and you must add accounts be‐
2362 fore you can add users.
2363 $ sacctmgr create cluster tux
2365 $ sacctmgr create account name=science fairshare=50
2366 $ sacctmgr create account name=chemistry parent=science fairshare=30
2367 $ sacctmgr create account name=physics parent=science fairshare=20
2368 $ sacctmgr create user name=adam cluster=tux account=physics fairshare=10
2369 $ sacctmgr delete user name=adam cluster=tux account=physics
2370 $ sacctmgr delete account name=physics cluster=tux
2371 $ sacctmgr modify user where name=adam cluster=tux account=physics set maxjobs=2 maxwall=30:00
2372 $ sacctmgr add user brian account=chemistry
2373 $ sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd
2374 $ sacctmgr list transactions Action="Add Users" Start=11/03-10:30:00 format=Where,Time
2375 $ sacctmgr dump cluster=tux file=tux_data_file
2376 $ sacctmgr load tux_data_file
2377 A user's account can not be changed directly. A new association needs
2379 to be created for the user with the new account. Then the association
2380 with the old account can be deleted.
2381 When modifying an object placing the key words 'set' and the optional
2383 'where' is critical to perform correctly below are examples to produce
2384 correct results. As a rule of thumb anything you put in front of the
2385 set will be used as a quantifier. If you want to put a quantifier af‐
2386 ter the key word 'set' you should use the key word 'where'. The follow‐
2387 ing is wrong:
2388 $ sacctmgr modify user name=adam set fairshare=10 cluster=tux
2390 This will produce an error as the above line reads modify user adam set
2392 fairshare=10 and cluster=tux. Either of the following is correct:
2393 $ sacctmgr modify user name=adam cluster=tux set fairshare=10
2395 $ sacctmgr modify user name=adam set fairshare=10 where cluster=tux
2396 When changing qos for something only use the '=' operator when wanting
2398 to explicitly set the qos to something. In most cases you will want to
2399 use the '+=' or '-=' operator to either add to or remove from the ex‐
2400 isting qos already in place.
2401 If a user already has qos of normal,standby for a parent or it was ex‐
2403 plicitly set you should use qos+=expedite to add this to the list in
2404 this fashion.
2405 If you are looking to only add the qos expedite to only a certain ac‐
2407 count and or cluster you can do that by specifying them in the sacctmgr
2408 line.
2409 $ sacctmgr modify user name=adam set qos+=expedite
2411 or
2413 $ sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite
2415 Let's give an example how to add QOS to user accounts. List all avail‐
2417 able QOSs in the cluster.
2418 $ sacctmgr show qos format=name
2420 Name
2421 ---------
2422 normal
2423 expedite
2424 List all the associations in the cluster.
2426 $ sacctmgr show assoc format=cluster,account,qos
2428 Cluster Account QOS
2429 -------- ---------- -----
2430 zebra root normal
2431 zebra root normal
2432 zebra g normal
2433 zebra g1 normal
2434 Add the QOS expedite to account G1 and display the result. Using the
2436 operator += the QOS will be added together with the existing QOS to
2437 this account.
2438 $ sacctmgr modify account name=g1 set qos+=expedite
2440 $ sacctmgr show assoc format=cluster,account,qos
2441 Cluster Account QOS
2442 -------- -------- -------
2443 zebra root normal
2444 zebra root normal
2445 zebra g normal
2446 zebra g1 expedite,normal
2447 Now set the QOS expedite as the only QOS for the account G and display
2449 the result. Using the operator = that expedite is the only usable QOS
2450 by account G
2451 $ sacctmgr modify account name=G set qos=expedite
2453 $ sacctmgr show assoc format=cluster,account,user,qos
2454 Cluster Account QOS
2455 --------- -------- -----
2456 zebra root normal
2457 zebra root normal
2458 zebra g expedite
2459 zebra g1 expedite,normal
2460 If a new account is added under the account G it will inherit the QOS
2462 expedite and it will not have access to QOS normal.
2463 $ sacctmgr add account banana parent=G
2465 $ sacctmgr show assoc format=cluster,account,qos
2466 Cluster Account QOS
2467 --------- -------- -----
2468 zebra root normal
2469 zebra root normal
2470 zebra g expedite
2471 zebra banana expedite
2472 zebra g1 expedite,normal
2473 An example of listing trackable resources:
2475 $ sacctmgr show tres
2477 Type Name ID
2478 ---------- ----------------- --------
2479 cpu 1
2480 mem 2
2481 energy 3
2482 node 4
2483 billing 5
2484 gres gpu:tesla 1001
2485 license vcs 1002
2486 bb cray 1003
2487
2490 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
2491 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
2492 Copyright (C) 2010-2016 SchedMD LLC.
2493 This file is part of Slurm, a resource management program. For de‐
2495 tails, see <https://slurm.schedmd.com/>.
2496 Slurm is free software; you can redistribute it and/or modify it under
2498 the terms of the GNU General Public License as published by the Free
2499 Software Foundation; either version 2 of the License, or (at your op‐
2500 tion) any later version.
2501 Slurm is distributed in the hope that it will be useful, but WITHOUT
2503 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
2504 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2505 for more details.
2506
2509 slurm.conf(5), slurmdbd(8)
2510April 2021 Slurm Commands sacctmgr(1)