1sacct(1) Slurm Commands sacct(1)
2
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
11 sacct [OPTIONS...]
12
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18 The sacct command displays job accounting data stored in the job ac‐
20 counting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24 For the root user, the sacct command displays job accounting data for
26 all users, although there are options to filter the output to report
27 only the jobs from a specified user or group.
28 For the non-root user, the sacct command limits the display of job ac‐
30 counting data to jobs that were launched with their own user identifier
31 (UID) by default. Data for other users can be displayed with the --al‐
32 lusers, --user, or --uid options.
33 Elapsed time fields are presented as [days-]hours:minutes:seconds[.mi‐
35 croseconds]. Only 'CPU' fields will ever have microseconds.
36 The default input file is the file named in the AccountingStorageLoc
38 parameter in slurm.conf.
39 NOTE: If designated, the slurmdbd.conf option PrivateData may further
41 restrict the accounting data visible to users which are not SlurmUser,
42 root, or a user with AdminLevel=Admin. See the slurmdbd.conf man page
43 for additional details on restricting access to accounting data.
44 NOTE: The contents of Slurm's database are maintained in lower case.
46 This may result in some sacct output differing from that of other Slurm
47 commands.
48 NOTE: Much of the data reported by sacct has been generated by the
50 wait3() and getrusage() system calls. Some systems gather and report
51 incomplete information for these calls; sacct reports values of 0 for
52 this missing data. See your systems getrusage [4m(3) man page for informa‐
53 tion about which data are actually available on your system.
54
57 -A, --accounts=<account_list>
58 Displays jobs when a comma separated list of accounts are
59 given as the argument.
60 -L, --allclusters
62 Display jobs ran on all clusters. By default, only jobs ran
63 on the cluster from where sacct is called are displayed.
64 -X, --allocations
67 Only show statistics relevant to the job allocation itself,
68 not taking steps into consideration.
69 NOTE: Without including steps, utilization statistics for job
71 allocation(s) will be reported as zero.
72 -a, --allusers
75 Displays all users' jobs when run by user root or if Private‐
76 Data is not configured to jobs. Otherwise display the cur‐
77 rent user's jobs
78 -x, --associations=<assoc_list>
80 Displays the statistics only for the jobs running under the
81 association ids specified by the assoc_list operand, which is
82 a comma-separated list of association ids. Space characters
83 are not allowed in the assoc_list. Default is all associa‐
84 tions.
85 -B, --batch-script
88 This option will print the batch script of job if the job
89 used one. If the job didn't have a script 'NONE' is output.
90 NOTE: AccountingStoreFlags=job_script is required for this.
91 NOTE: Requesting specific job(s) with '-j' is required for
92 this.
93 -b, --brief
95 Displays a brief listing consisting of JobID, State, and Ex‐
96 itCode.
97 -M, --clusters=<cluster_list>
99 Displays the statistics only for the jobs started on the
100 clusters specified by the cluster_list operand, which is a
101 comma-separated list of clusters. Space characters are not
102 allowed in the cluster_list. A value of 'all' will query to
103 run on all clusters. The default is current cluster you are
104 executing the sacct command on or all clusters in the federa‐
105 tion when executed on a federated cluster. This option im‐
106 plicitly sets the --local option.
107 -c, --completion
110 Use job completion data instead of job accounting. The Job‐
111 CompType parameter in the slurm.conf file must be defined to
112 a non-none option. Does not support federated cluster infor‐
113 mation (local data only).
114 -C, --constraints=<constraint_list>
116 Comma separated list to filter jobs based on what con‐
117 straints/features the job requested. Multiple options will
118 be treated as 'and' not 'or', so the job would need all con‐
119 straints specified to be returned not one or the other.
120 --delimiter=<characters>
123 ASCII characters used to separate the fields when specifying
124 the -p or -P options. The default delimiter is a '|'. This
125 option is ignored if -p or -P options are not specified.
126 -D, --duplicates
129 If Slurm job ids are reset, some job numbers will probably
130 appear more than once in the accounting log file but refer to
131 different jobs. Such jobs can be distinguished by the "sub‐
132 mit" time stamp in the data records.
133 When data for specific jobs are requested with the --jobs op‐
136 tion, sacct returns the most recent job with that number.
137 This behavior can be overridden by specifying --duplicates,
138 in which case all records that match the selection criteria
139 will be returned.
140 NOTE: Revoked federated sibling jobs are hidden unless the
143 --duplicates option is specified.
144 -E, --endtime=<end_time>
147 Select jobs in any state before the specified time. If
149 states are given with the -s option return jobs in this state
150 before this period. See the DEFAULT TIME WINDOW for more de‐
151 tails.
152 Valid time formats are:
154 HH:MM[:SS][AM|PM]
155 MMDD[YY][-HH:MM[:SS]]
156 MM.DD[.YY][-HH:MM[:SS]]
157 MM/DD[/YY][-HH:MM[:SS]]
158 YYYY-MM-DD[THH:MM[:SS]]
159 today, midnight, noon, fika (3 PM), teatime (4 PM)
160 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
161 --env-vars
163 This option will print the running environment of a batch
164 job, otherwise 'NONE' is output.
165 NOTE: AccountingStoreFlags=job_env is required for this.
166 NOTE: Requesting specific job(s) with '-j' is required for
167 this.
168 --federation
170 Show jobs from the federation if a member of one.
171 -f, --file=<file>
174 Causes the sacct command to read job accounting data from the
175 named file instead of the current Slurm job accounting log
176 file. Only applicable when running the jobcomp/filetxt
177 plugin.
178 -F, --flags=<flag_list>
181 Comma separated list to filter jobs based on what various
182 ways the jobs were handled. Current flags are SchedSubmit,
183 SchedMain, SchedBackfill. These particular options describe
184 the scheduler that started the job.
185 -o, --format
188 Comma separated list of fields. (use "--helpformat" for a
189 list of available fields).
190 NOTE: When using the format option for listing various fields
192 you can put a %NUMBER afterwards to specify how many charac‐
193 ters should be printed.
194 e.g. format=name%30 will print 30 characters of field name
196 right justified. A %-30 will print 30 characters left justi‐
197 fied.
198 When set, the SACCT_FORMAT environment variable will override
200 the default format. For example:
201 SACCT_FORMAT="jobid,user,account,cluster"
203 -g, --gid=, --group=<gid_or_group_list>
206 Displays the statistics only for the jobs started with the
207 GID or the GROUP specified by the gid_list or the group_list
208 operand, which is a comma-separated list. Space characters
209 are not allowed. Default is no restrictions.
210 -h, --help
213 Displays a general help message.
214 -e, --helpformat
217 Print a list of fields that can be specified with the --for‐
219 mat option.
220 Fields available:
223 Account AdminComment AllocCPUS AllocNodes
225 AllocTRES AssocID AveCPU AveCPUFreq
226 AveDiskRead AveDiskWrite AvePages AveRSS
227 AveVMSize BlockID Cluster Comment
228 Constraints Container ConsumedEnergy ConsumedEnergyRaw
229 CPUTime CPUTimeRAW DBIndex DerivedExitCode
230 Elapsed ElapsedRaw Eligible End
231 ExitCode Flags GID Group
232 JobID JobIDRaw JobName Layout
233 MaxDiskRead MaxDiskReadNode MaxDiskReadTask MaxDiskWrite
234 MaxDiskWriteNode MaxDiskWriteTask MaxPages MaxPagesNode
235 MaxPagesTask MaxRSS MaxRSSNode MaxRSSTask
236 MaxVMSize MaxVMSizeNode MaxVMSizeTask McsLabel
237 MinCPU MinCPUNode MinCPUTask NCPUS
238 NNodes NodeList NTasks Priority
239 Partition QOS QOSRAW Reason
240 ReqCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov
241 ReqCPUS ReqMem ReqNodes ReqTRES
242 Reservation ReservationId Reserved ResvCPU
243 ResvCPURAW Start State Submit
244 SubmitLine Suspended SystemCPU SystemComment
245 Timelimit TimelimitRaw TotalCPU TRESUsageInAve
246 TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin
247 TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve
248 TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
249 TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot UID
250 User UserCPU WCKey WCKeyID
251 WorkDir
252 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
254 SUsageIn[Ave|Tot]. They represent the average/total of the
255 highest watermarks over all ranks in the step. When using
256 sstat they represent the average/total at the moment the com‐
257 mand was run.
258 NOTE: TRESUsage*Min* values represent the lowest highwater
260 mark in the step.
261 The section titled "Job Accounting Fields" describes these
263 fields.
264 -j, --jobs=<job[.step]>
267 Displays information about the specified job[.step] or list
268 of job[.step]s.
269 The job[.step] parameter is a comma-separated list of jobs.
271 Space characters are not permitted in this list.
272 NOTE: A step id of 'batch' will display the information about
273 the batch step.
274 By default sacct shows only jobs with Eligible time, but with
276 this option the non-eligible will be also shown.
277 NOTE: If --state is also specified, as non-eligible are not
278 PD, then non-eligible jobs will not be displayed. See the
279 DEFAULT TIME WINDOW for details about how this option changes
280 the default -S and -E options.
281 --json Dump job information as JSON. All other formating arugments
284 will be ignored.
285 --local Show only jobs local to this cluster. Ignore other clusters
288 in this federation (if any). Overrides --federation.
289 -l, --long
292 Equivalent to specifying:
293 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
295 sizenode, maxvmsizetask,avevm‐
296 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
297 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
298 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
299 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
300 gov,reqmem,consumeden‐
301 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
302 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
303 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
304 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
305 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
306 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
307 outave,tresusageouttot
308 --name=<jobname_list>
311 Display jobs that have any of these name(s).
312 -i, --nnodes=<min[-max]>
314 Return jobs that ran on the specified number of nodes.
315 --noconvert
318 Don't convert units from their original type (e.g. 2048M
319 won't be converted to 2G).
320 -N, --nodelist=<node_list>
322 Display jobs that ran on any of these node(s). node_list can
323 be a ranged string.
324 -n, --noheader
326 No heading will be added to the output. The default action is
327 to display a header.
328 -p, --parsable
330 Output will be '|' delimited with a '|' at the end. See also
331 the --delimiter option.
332 -P, --parsable2
335 Output will be '|' delimited without a '|' at the end. See
336 also the --delimiter option.
337 -r, --partition
340 Comma separated list of partitions to select jobs and job
341 steps from. The default is all partitions.
342 -q, --qos Only send data about jobs using these qos. Default is all.
345 -R, --reason=<reason_list>
348 Comma separated list to filter jobs based on what reason the
349 job wasn't scheduled outside resources/priority.
350 -S, --starttime
353 Select jobs in any state after the specified time. Default is
354 00:00:00 of the current day, unless the '-s' or '-j' options
355 are used. If the '-s' option is used, then the default is
356 'now'. If states are given with the '-s' option then only
357 jobs in this state at this time will be returned. If the '-j'
358 option is used, then the default time is Unix Epoch 0. See
359 the DEFAULT TIME WINDOW for more details.
360 Valid time formats are:
362 HH:MM[:SS][AM|PM]
363 MMDD[YY][-HH:MM[:SS]]
364 MM.DD[.YY][-HH:MM[:SS]]
365 MM/DD[/YY][-HH:MM[:SS]]
366 YYYY-MM-DD[THH:MM[:SS]]
367 today, midnight, noon, fika (3 PM), teatime (4 PM)
368 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
369 -s, --state=<state_list>
372 Selects jobs based on their state during the time period
373 given. Unless otherwise specified, the start and end time
374 will be the current time when the --state option is specified
375 and only currently running jobs can be displayed. A start
376 and/or end time must be specified to view information about
377 jobs not currently running. See the JOB STATE CODES section
378 below for a list of state designators. Multiple state names
379 may be specified using comma separators. Either the short or
380 long form of the state name may be used (e.g. CA or CAN‐
381 CELLED) and the name is case insensitive (i.e. ca and CA both
382 work).
383 NOTE: Note for a job to be selected in the PENDING state it
385 must have "EligibleTime" in the requested time interval or
386 different from "Unknown". The "EligibleTime" is displayed by
387 the "scontrol show job" command. For example jobs submitted
388 with the "--hold" option will have "EligibleTime=Unknown" as
389 they are pending indefinitely.
390 NOTE: When specifying states and no start time is given the
392 default start time is 'now'. This is only when -j is not
393 used. If -j is used the start time will default to 'Epoch'.
394 In both cases if no end time is given it will default to
395 'now'. See the DEFAULT TIME WINDOW for more details.
396 -K, --timelimit-max
399 Ignored by itself, but if timelimit_min is set this will be
400 the maximum timelimit of the range. Default is no restric‐
401 tion.
402 -k, --timelimit-min
405 Only send data about jobs with this timelimit. If used with
406 timelimit_max this will be the minimum timelimit of the
407 range. Default is no restriction.
408 -T, --truncate
411 Truncate time. So if a job started before --starttime the
412 start time would be truncated to --starttime. The same for
413 end time and --endtime.
414 -u, --uid=, --user=<uid_or_user_list>
417 Use this comma separated list of UIDs or user names to select
418 jobs to display. By default, the running user's UID is used.
419 --units=[KMGTP]
422 Display values in specified unit type. Takes precedence over
423 --noconvert option.
424 --usage Display a command usage summary.
427 --use-local-uid
430 When displaying UID, sacct uses the UID stored in Slurm's ac‐
431 counting database by default. Use this command to make Slurm
432 use a system call to get the UID from the username. This op‐
433 tion may be useful in an environment with multiple clusters
434 and one database where the UID's aren't the same on all clus‐
435 ters.
436 -v, --verbose
439 Primarily for debugging purposes, report the state of various
440 variables during processing.
441 -V, --version
444 Print version.
445 -W, --wckeys=<wckey_list>
448 Displays the statistics only for the jobs started on the wck‐
449 eys specified by the wckey_list operand, which is a
450 comma-separated list of wckey names. Space characters are
451 not allowed in the wckey_list. Default is all wckeys.
452 --whole-hetjob[=yes|no]
455 When querying and filtering heterogeneous jobs with --jobs,
456 Slurm will default to retrieving information about all the
457 components of the job if the het_job_id (leader id) is se‐
458 lected. If a non-leader heterogeneous job component id is se‐
459 lected then only that component is retrieved by default. This
460 behavior can be changed by using this option. If set to 'yes'
461 (or no argument), then information about all the components
462 will be retrieved no matter which component is selected in
463 the job filter. If set to 'no' then only the selected hetero‐
464 geneous job component(s) will be retrieved, even when select‐
465 ing the leader.
466 --yaml Dump job information as YAML. All other formating arugments
469 will be ignored.
470 Job Accounting Fields
473 The following describes each job accounting field:
474 ALL Print all fields listed below.
476 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
479 AllocNodes
482 Number of nodes allocated to the job/step. 0 if the
483 job is pending.
484 AllocTres Trackable resources. These are the resources allocated
487 to the job/step after the job started running. For
488 pending jobs this should be blank. For more details
489 see AccountingStorageTRES in slurm.conf.
490 NOTE: When a generic resource is configured with the
492 no_consume flag, the allocation will be printed with a
493 zero.
494 Account Account the job ran under.
497 AssocID Reference to the association of user, account and
500 cluster.
501 AveCPU Average (system + user) CPU time of all tasks in job.
504 AveCPUFreq
507 Average weighted CPU frequency of all tasks in job, in
508 kHz.
509 AveDiskRead
512 Average number of bytes read by all tasks in job.
513 AveDiskWrite
516 Average number of bytes written by all tasks in job.
517 AvePages Average number of page faults of all tasks in job.
520 AveRSS Average resident set size of all tasks in job.
523 AveVMSize Average Virtual Memory size of all tasks in job.
526 Cluster Cluster name.
529 Comment The job's comment string when the AccountingStoreFlags
532 parameter in the slurm.conf file contains 'job_com‐
533 ment'. The Comment string can be modified by invoking
534 sacctmgr modify job or the specialized sjobexitmod
535 command.
536 ConsumedEnergy
539 Total energy consumed by all tasks in job, in joules.
540 Note: Only in case of exclusive job allocation this
541 value reflects the jobs' real energy consumption.
542 Container Path to OCI Container Bundle requested.
545 CPUTime Time used (Elapsed time * CPU count) by a job or step
548 in HH:MM:SS format.
549 CPUTimeRAW
552 Time used (Elapsed time * CPU count) by a job or step
553 in cpu-seconds.
554 DerivedExitCode
557 The highest exit code returned by the job's job steps
558 (srun invocations). Following the colon is the signal
559 that caused the process to terminate if it was termi‐
560 nated by a signal. The DerivedExitCode can be modi‐
561 fied by invoking sacctmgr modify job or the special‐
562 ized sjobexitmod command.
563 Elapsed The job's elapsed time.
566 The format of this field's output is as follows:
568 [DD-[HH:]]MM:SS
569 as defined by the following:
571 DD days
573 hh hours
575 mm minutes
577 ss seconds
579 Eligible When the job became eligible to run in the same format
582 as End.
583 End Termination time of the job. Format output is,
586 YYYY-MM-DDTHH:MM:SS, unless changed through the
587 SLURM_TIME_FORMAT environment variable.
588 ExitCode The exit code returned by the job script or salloc,
591 typically as set by the exit() function. Following
592 the colon is the signal that caused the process to
593 terminate if it was terminated by a signal.
594 GID The group identifier of the user who ran the job.
597 Group The group name of the user who ran the job.
600 JobID The identification number of the job or job step.
603 Regular jobs are in the form:
605 JobID[.JobStep]
607 Array jobs are in the form:
609 ArrayJobID_ArrayTaskID
611 Heterogeneous jobs are in the form:
613 HetJobID+HetJobOffset
615 When printing job arrays, performance of the command
617 can be measurably improved for systems with large num‐
618 bers of jobs when a single job ID is specified. By de‐
619 fault, this field size will be limited to 64 bytes.
620 Use the environment variable SLURM_BITSTR_LEN to spec‐
621 ify larger field sizes.
622 JobIDRaw The identification number of the job or job step.
625 Prints the JobID in the form JobID[.JobStep] for regu‐
626 lar, heterogeneous and array jobs.
627 JobName The name of the job or job step. The slurm_account‐
630 ing.log file is a space delimited file. Because of
631 this if a space is used in the jobname an underscore
632 is substituted for the space before the record is
633 written to the accounting file. So when the jobname is
634 displayed by sacct the jobname that had a space in it
635 will now have an underscore in place of the space.
636 Layout What the layout of a step was when it was running.
639 This can be used to give you an idea of which node ran
640 which rank in your job.
641 MaxDiskRead
644 Maximum number of bytes read by all tasks in job.
645 MaxDiskReadNode
648 The node on which the maxdiskread occurred.
649 MaxDiskReadTask
652 The task ID where the maxdiskread occurred.
653 MaxDiskWrite
656 Maximum number of bytes written by all tasks in job.
657 MaxDiskWriteNode
660 The node on which the maxdiskwrite occurred.
661 MaxDiskWriteTask
664 The task ID where the maxdiskwrite occurred.
665 MaxPages Maximum number of page faults of all tasks in job.
668 MaxPagesNode
671 The node on which the maxpages occurred.
672 MaxPagesTask
675 The task ID where the maxpages occurred.
676 MaxRSS Maximum resident set size of all tasks in job.
679 MaxRSSNode
682 The node on which the maxrss occurred.
683 MaxRSSTask
686 The task ID where the maxrss occurred.
687 MaxVMSize Maximum Virtual Memory size of all tasks in job.
690 MaxVMSizeNode
693 The node on which the maxvmsize occurred.
694 MaxVMSizeTask
697 The task ID where the maxvmsize occurred.
698 MinCPU Minimum (system + user) CPU time of all tasks in job.
701 MinCPUNode
704 The node on which the mincpu occurred.
705 MinCPUTask
708 The task ID where the mincpu occurred.
709 NCPUS Total number of CPUs allocated to the job. Equivalent
712 to AllocCPUS.
713 NodeList List of nodes in job/step.
716 NNodes Number of nodes in a job or step. If the job is run‐
719 ning, or ran, this count will be the number allocated,
720 else the number will be the number requested.
721 NTasks Total number of tasks in a job or step.
724 Priority Slurm priority.
727 Partition Identifies the partition on which the job ran.
730 QOS Name of Quality of Service.
733 QOSRAW Numeric id of Quality of Service.
736 ReqCPUFreq
739 Requested CPU frequency for the step, in kHz. Note:
740 This value applies only to a job step. No value is re‐
741 ported for the job.
742 ReqCPUFreqGov
745 Requested CPU frequency governor for the step, in kHz.
746 Note: This value applies only to a job step. No value
747 is reported for the job.
748 ReqCPUFreqMax
751 Maximum requested CPU frequency for the step, in kHz.
752 Note: This value applies only to a job step. No value
753 is reported for the job.
754 ReqCPUFreqMin
757 Minimum requested CPU frequency for the step, in kHz.
758 Note: This value applies only to a job step. No value
759 is reported for the job.
760 ReqCPUS Number of requested CPUs.
763 ReqMem Minimum required memory for the job. It may have a
766 letter appended to it indicating units (M for
767 megabytes, G for gigabytes, etc.). Note: This value
768 is only from the job allocation, not the step.
769 ReqNodes Requested minimum Node count for the job/step.
772 ReqTres Trackable resources. These are the minimum resource
775 counts requested by the job/step at submission time.
776 For more details see AccountingStorageTRES in
777 slurm.conf.
778 Reservation
781 Reservation Name.
782 ReservationId
785 Reservation Id.
786 Reserved How much wall clock time was used as reserved time for
789 this job. This is derived from how long a job was
790 waiting from eligible time to when it actually
791 started. Format is the same as Elapsed.
792 ResvCPU How many CPU seconds were used as reserved time for
795 this job. Format is the same as Elapsed.
796 ResvCPURAW
799 How many CPU seconds were used as reserved time for
800 this job. Format is in processor seconds.
801 Start Initiation time of the job in the same format as End.
804 State Displays the job status, or state. See the JOB STATE
807 CODES section below for a list of possible states.
808 If more information is available on the job state than
810 will fit into the current field width (for example,
811 the UID that CANCELLED a job) the state will be fol‐
812 lowed by a "+". You can increase the size of the dis‐
813 played state using the "%NUMBER" format modifier de‐
814 scribed earlier.
815 NOTE: The RUNNING state will return suspended jobs as
817 well. In order to print suspended jobs you must re‐
818 quest SUSPENDED at a different call from RUNNING.
819 NOTE: The RUNNING state will return any jobs completed
821 (cancelled or otherwise) in the time period requested
822 as the job was also RUNNING during that time. If you
823 are only looking for jobs that finished, please choose
824 the appropriate state(s) without the RUNNING state.
825 Submit The time the job was submitted in the same format as
828 End.
829 NOTE: If a job is requeued, the submit time is reset.
831 To obtain the original submit time it is necessary to
832 use the -D or --duplicate option to display all dupli‐
833 cate entries for a job.
834 SubmitLine
837 The full command issued to submit the job.
838 Suspended The amount of time a job or job step was suspended.
841 Format is the same as Elapsed.
842 SystemCPU The amount of system CPU time used by the job or job
845 step. Format is the same as Elapsed.
846 NOTE: SystemCPU provides a measure of the task's par‐
848 ent process and does not include CPU time of child
849 processes.
850 Timelimit What the timelimit was/is for the job. Format is the
853 same as Elapsed.
854 TimelimitRaw
857 What the timelimit was/is for the job. Format is in
858 number of minutes.
859 TotalCPU The sum of the SystemCPU and UserCPU time used by the
862 job or job step. The total CPU time of the job may
863 exceed the job's elapsed time for jobs that include
864 multiple job steps. Format is the same as Elapsed.
865 NOTE: TotalCPU provides a measure of the task's parent
867 process and does not include CPU time of child pro‐
868 cesses.
869 TresUsageInAve
872 Tres average usage in by all tasks in job. NOTE: If
873 corresponding TresUsageInMaxTask is -1 the metric is
874 node centric instead of task.
875 TresUsageInMax
878 Tres maximum usage in by all tasks in job. NOTE: If
879 corresponding TresUsageInMaxTask is -1 the metric is
880 node centric instead of task.
881 TresUsageInMaxNode
884 Node for which each maximum TRES usage out occurred.
885 TresUsageInMaxTask
888 Task for which each maximum TRES usage out occurred.
889 TresUsageInMin
892 Tres minimum usage in by all tasks in job. NOTE: If
893 corresponding TresUsageInMinTask is -1 the metric is
894 node centric instead of task.
895 TresUsageInMinNode
898 Node for which each minimum TRES usage out occurred.
899 TresUsageInMinTask
902 Task for which each minimum TRES usage out occurred.
903 TresUsageInTot
906 Tres total usage in by all tasks in job.
907 TresUsageOutAve
910 Tres average usage out by all tasks in job. NOTE: If
911 corresponding TresUsageOutMaxTask is -1 the metric is
912 node centric instead of task.
913 TresUsageOutMax
916 Tres maximum usage out by all tasks in job. NOTE: If
917 corresponding TresUsageOutMaxTask is -1 the metric is
918 node centric instead of task.
919 TresUsageOutMaxNode
922 Node for which each maximum TRES usage out occurred.
923 TresUsageOutMaxTask
926 Task for which each maximum TRES usage out occurred.
927 TresUsageOutTot
930 Tres total usage out by all tasks in job.
931 UID The user identifier of the user who ran the job.
934 User The user name of the user who ran the job.
937 UserCPU The amount of user CPU time used by the job or job
940 step. Format is the same as Elapsed.
941 NOTE: UserCPU provides a measure of the task's parent
943 process and does not include CPU time of child pro‐
944 cesses.
945 WCKey Workload Characterization Key. Arbitrary string
948 for grouping orthogonal accounts together.
949 WCKeyID Reference to the wckey.
952
956 BF BOOT_FAIL Job terminated due to launch failure, typically due
957 to a hardware failure (e.g. unable to boot the node
958 or block and the job can not be requeued).
959 CA CANCELLED Job was explicitly cancelled by the user or system
961 administrator. The job may or may not have been
962 initiated.
963 CD COMPLETED Job has terminated all processes on all nodes with
965 an exit code of zero.
966 DL DEADLINE Job terminated on deadline.
968 F FAILED Job terminated with non-zero exit code or other
970 failure condition.
971 NF NODE_FAIL Job terminated due to failure of one or more allo‐
973 cated nodes.
974 OOM OUT_OF_MEMORY Job experienced out of memory error.
976 PD PENDING Job is awaiting resource allocation.
978 PR PREEMPTED Job terminated due to preemption.
980 R RUNNING Job currently has an allocation.
982 RQ REQUEUED Job was requeued.
984 RS RESIZING Job is about to change size.
986 RV REVOKED Sibling was removed from cluster due to other clus‐
988 ter starting the job.
989 S SUSPENDED Job has an allocation, but execution has been sus‐
991 pended and CPUs have been released for other jobs.
992 TO TIMEOUT Job terminated upon reaching its time limit.
994
997 The options --starttime and --endtime define the time window between
998 which sacct is going to search. For historical and practical reasons
999 their default values (i.e. the default time window) depends on other
1000 options: --jobs and --state.
1001 Depending on if --jobs and/or --state are specified, the default values
1003 of --starttime and --endtime options are:
1004 WITHOUT EITHER --jobs NOR --state specified:
1007 --starttime defaults to Midnight.
1008 --endtime defaults to Now.
1009 WITH --jobs AND WITHOUT --state specified:
1012 --starttime defaults to Epoch 0.
1013 --endtime defaults to Now.
1014 WITHOUT --jobs AND WITH --state specified:
1017 --starttime defaults to Now.
1018 --endtime defaults to --starttime and to Now if --starttime is
1019 not specified.
1020 WITH BOTH --jobs AND --state specified:
1023 --starttime defaults to Epoch 0.
1024 --endtime defaults to --starttime or to Now if --starttime is
1025 not specified.
1026 NOTE: With -v/--verbose a message about the actual time window in use
1029 is shown.
1030
1033 Executing sacct sends a remote procedure call to slurmdbd. If enough
1034 calls from sacct or other Slurm client commands that send remote proce‐
1035 dure calls to the slurmdbd daemon come in at once, it can result in a
1036 degradation of performance of the slurmdbd daemon, possibly resulting
1037 in a denial of service.
1038 Do not run sacct or other Slurm client commands that send remote proce‐
1040 dure calls to slurmdbd from loops in shell scripts or other programs.
1041 Ensure that programs limit calls to sacct to the minimum necessary for
1042 the information you are trying to gather.
1043
1046 Some sacct options may be set via environment variables. These environ‐
1047 ment variables, along with their corresponding options, are listed be‐
1048 low. (Note: Command line options will always override these settings.)
1049 SACCT_FEDERATION Same as --federation
1051 SACCT_LOCAL Same as --local
1053 SLURM_BITSTR_LEN Specifies the string length to be used for holding
1056 a job array's task ID expression. The default value
1057 is 64 bytes. A value of 0 will print the full ex‐
1058 pression with any length required. Larger values
1059 may adversely impact the application performance.
1060 SLURM_CONF The location of the Slurm configuration file.
1063 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
1065 value of standard, the default value, generates
1066 output in the form
1067 "year-month-dateThour:minute:second". A value of
1068 relative returns only "hour:minute:second" if the
1069 current day. For other dates in the current year
1070 it prints the "hour:minute" preceded by "Tomorr"
1071 (tomorrow), "Ystday" (yesterday), the name of the
1072 day for the coming week (e.g. "Mon", "Tue", etc.),
1073 otherwise the date (e.g. "25 Apr"). For other
1074 years it returns a date month and year without a
1075 time (e.g. "6 Jun 2012"). All of the time stamps
1076 use a 24 hour format.
1077 A valid strftime() format can also be specified.
1079 For example, a value of "%a %T" will report the day
1080 of the week and a time stamp (e.g. "Mon 12:34:56").
1081
1084 This example illustrates the default invocation of the sacct command:
1085 # sacct
1087 Jobid Jobname Partition Account AllocCPUS State ExitCode
1088 ---------- ---------- ---------- ---------- ---------- ---------- --------
1089 2 script01 srun acct1 1 RUNNING 0
1090 3 script02 srun acct1 1 RUNNING 0
1091 4 endscript srun acct1 1 RUNNING 0
1092 4.0 srun acct1 1 COMPLETED 0
1093 This example shows the same job accounting information with the brief
1096 option.
1097 # sacct --brief
1099 Jobid State ExitCode
1100 ---------- ---------- --------
1101 2 RUNNING 0
1102 3 RUNNING 0
1103 4 RUNNING 0
1104 4.0 COMPLETED 0
1105 # sacct --allocations
1107 Jobid Jobname Partition Account AllocCPUS State ExitCode
1108 ---------- ---------- ---------- ---------- ------- ---------- --------
1109 3 sja_init andy acct1 1 COMPLETED 0
1110 4 sjaload andy acct1 2 COMPLETED 0
1111 5 sja_scr1 andy acct1 1 COMPLETED 0
1112 6 sja_scr2 andy acct1 18 COMPLETED 2
1113 7 sja_scr3 andy acct1 18 COMPLETED 0
1114 8 sja_scr5 andy acct1 2 COMPLETED 0
1115 9 sja_scr7 andy acct1 90 COMPLETED 1
1116 10 endscript andy acct1 186 COMPLETED 0
1117 This example demonstrates the ability to customize the output of the
1120 sacct command. The fields are displayed in the order designated on the
1121 command line.
1122 # sacct --format=jobid,elapsed,ncpus,ntasks,state
1124 Jobid Elapsed Ncpus Ntasks State
1125 ---------- ---------- ---------- -------- ----------
1126 3 00:01:30 2 1 COMPLETED
1127 3.0 00:01:30 2 1 COMPLETED
1128 4 00:00:00 2 2 COMPLETED
1129 4.0 00:00:01 2 2 COMPLETED
1130 5 00:01:23 2 1 COMPLETED
1131 5.0 00:01:31 2 1 COMPLETED
1132 This example demonstrates the use of the -T (--truncate) option when
1135 used with -S (--starttime) and -E (--endtime). When the -T option is
1136 used, the start time of the job will be the specified -S value if the
1137 job was started before the specified time, otherwise the time will be
1138 the job's start time. The end time will be the specified -E option if
1139 the job ends after the specified time, otherwise it will be the jobs
1140 end time.
1141 Without -T (normal operation) sacct output would be like this.
1143 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1145 JobID Start End State
1146 --------- --------------------- -------------------- ------------
1147 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1148 3 2014-07-03T11:35:21 Unknown RUNNING
1149 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1150 5 2014-07-03T11:41:01 Unknown RUNNING
1151 By adding the -T option the job's start and end times are truncated to
1153 reflect only the time requested. If a job started after the start time
1154 requested or finished before the end time requested those times are not
1155 altered. The -T option is useful when determining exact run times dur‐
1156 ing any given period.
1157 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1159 JobID Start End State
1160 --------- --------------------- -------------------- ------------
1161 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1162 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1163 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1164 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1165 NOTE: If no -s (--state) option is given sacct will display eligible
1168 jobs during the specified period of time, otherwise it will return jobs
1169 that were in the state requested during that period of time.
1170 This example demonstrates the differences running sacct with and with‐
1172 out the --state flag for the same time period. Without the --state op‐
1173 tion, all eligible jobs in that time period are shown.
1174 # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1176 JobID Start End State
1177 ------------ ------------------- ------------------- ----------
1178 2955 11:15:12 11:20:12 COMPLETED
1179 2956 11:20:13 11:25:13 COMPLETED
1180 With the --state=pending option, only job 2956 will be shown because it
1182 had a dependency on 2955 and was still PENDING from 11:20:00 until it
1183 started at 11:21:13. Note that even though we requested PENDING jobs,
1184 the State shows as COMPLETED because that is the current State of the
1185 job.
1186 # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1188 JobID Start End State
1189 ------------ ------------------- ------------------- ----------
1190 2956 11:20:13 11:25:13 COMPLETED
1191
1193 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1194 L.P.
1195 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1196 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1197 Copyright (C) 2010-2021 SchedMD LLC.
1198 This file is part of Slurm, a resource management program. For de‐
1200 tails, see <https://slurm.schedmd.com/>.
1201 Slurm is free software; you can redistribute it and/or modify it under
1203 the terms of the GNU General Public License as published by the Free
1204 Software Foundation; either version 2 of the License, or (at your op‐
1205 tion) any later version.
1206 Slurm is distributed in the hope that it will be useful, but WITHOUT
1208 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1209 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1210 for more details.
1211
1213 /etc/slurm.conf
1214 Entries to this file enable job accounting and designate the
1215 job accounting log file that collects system job accounting.
1216 /var/log/slurm_accounting.log
1218 The default job accounting log file. By default, this file
1219 is set to read and write permission for root only.
1220
1222 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1223September 2021 Slurm Commands sacct(1)