1sacct(1) Slurm Commands sacct(1)
2
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
11 sacct [OPTIONS...]
12
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18 The sacct command displays job accounting data stored in the job ac‐
20 counting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24 For the root user, the sacct command displays job accounting data for
26 all users, although there are options to filter the output to report
27 only the jobs from a specified user or group.
28 For the non-root user, the sacct command limits the display of job ac‐
30 counting data to jobs that were launched with their own user identifier
31 (UID) by default. Data for other users can be displayed with the --al‐
32 lusers, --user, or --uid options.
33 Elapsed time fields are presented as [days-]hours:minutes:seconds[.mi‐
35 croseconds]. Only 'CPU' fields will ever have microseconds.
36 The default input file is the file named in the AccountingStorageLoc
38 parameter in slurm.conf.
39 NOTE: If designated, the slurmdbd.conf option PrivateData may further
41 restrict the accounting data visible to users which are not SlurmUser,
42 root, or a user with AdminLevel=Admin. See the slurmdbd.conf man page
43 for additional details on restricting access to accounting data.
44 NOTE: The contents of Slurm's database are maintained in lower case.
46 This may result in some sacct output differing from that of other Slurm
47 commands.
48 NOTE: Much of the data reported by sacct has been generated by the
50 wait3() and getrusage() system calls. Some systems gather and report
51 incomplete information for these calls; sacct reports values of 0 for
52 this missing data. See your systems getrusage [4m(3) man page for informa‐
53 tion about which data are actually available on your system.
54
57 -A, --accounts=<account_list>
58 Displays jobs when a comma separated list of accounts are
59 given as the argument.
60 -L, --allclusters
62 Display jobs ran on all clusters. By default, only jobs ran
63 on the cluster from where sacct is called are displayed.
64 -X, --allocations
66 Only show statistics relevant to the job allocation itself,
67 not taking steps into consideration.
68 NOTE: Without including steps, utilization statistics for job
70 allocation(s) will be reported as zero.
71 -a, --allusers
73 Displays all users' jobs when run by user root or if Private‐
74 Data is not configured to jobs. Otherwise display the cur‐
75 rent user's jobs
76 -x, --associations=<assoc_list>
78 Displays the statistics only for the jobs running under the
79 association ids specified by the assoc_list operand, which is
80 a comma-separated list of association ids. Space characters
81 are not allowed in the assoc_list. Default is all associa‐
82 tions.
83 -B, --batch-script
85 This option will print the batch script of job if the job
86 used one. If the job didn't have a script 'NONE' is output.
87 NOTE: AccountingStoreFlags=job_script is required for this.
88 NOTE: Requesting specific job(s) with '-j' is required for
89 this.
90 -b, --brief
92 Displays a brief listing consisting of JobID, State, and Ex‐
93 itCode.
94 -M, --clusters=<cluster_list>
96 Displays the statistics only for the jobs started on the
97 clusters specified by the cluster_list operand, which is a
98 comma-separated list of clusters. Space characters are not
99 allowed in the cluster_list. A value of 'all' will query to
100 run on all clusters. The default is current cluster you are
101 executing the sacct command on or all clusters in the federa‐
102 tion when executed on a federated cluster. This option im‐
103 plicitly sets the --local option.
104 -c, --completion
106 Use job completion data instead of job accounting. The Job‐
107 CompType parameter in the slurm.conf file must be defined to
108 a non-none option. Does not support federated cluster infor‐
109 mation (local data only).
110 -C, --constraints=<constraint_list>
112 Comma separated list to filter jobs based on what con‐
113 straints/features the job requested. Multiple options will
114 be treated as 'and' not 'or', so the job would need all con‐
115 straints specified to be returned not one or the other.
116 --delimiter=<characters>
118 ASCII characters used to separate the fields when specifying
119 the -p or -P options. The default delimiter is a '|'. This
120 option is ignored if -p or -P options are not specified.
121 -D, --duplicates
123 If Slurm job ids are reset, some job numbers will probably
124 appear more than once in the accounting log file but refer to
125 different jobs. Such jobs can be distinguished by the "sub‐
126 mit" time stamp in the data records.
127 When data for specific jobs are requested with the --jobs op‐
129 tion, sacct returns the most recent job with that number.
130 This behavior can be overridden by specifying --duplicates,
131 in which case all records that match the selection criteria
132 will be returned.
133 NOTE: Revoked federated sibling jobs are hidden unless the
135 --duplicates option is specified.
136 -E, --endtime=<end_time>
138 Select jobs in any state before the specified time. If
139 states are given with the -s option return jobs in this state
140 before this period. See the DEFAULT TIME WINDOW for more de‐
141 tails.
142 Valid time formats are:
144 HH:MM[:SS][AM|PM]
145 MMDD[YY][-HH:MM[:SS]]
146 MM.DD[.YY][-HH:MM[:SS]]
147 MM/DD[/YY][-HH:MM[:SS]]
148 YYYY-MM-DD[THH:MM[:SS]]
149 today, midnight, noon, fika (3 PM), teatime (4 PM)
150 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
151 --env-vars
153 This option will print the running environment of a batch
154 job, otherwise 'NONE' is output.
155 NOTE: AccountingStoreFlags=job_env is required for this.
156 NOTE: Requesting specific job(s) with '-j' is required for
157 this.
158 --federation
160 Show jobs from the federation if a member of one.
161 -f, --file=<file>
163 Causes the sacct command to read job accounting data from the
164 named file instead of the current Slurm job accounting log
165 file. Only applicable when running the jobcomp/filetxt
166 plugin.
167 -F, --flags=<flag_list>
169 Comma separated list to filter jobs based on what various
170 ways the jobs were handled. Current flags are SchedSubmit,
171 SchedMain, SchedBackfill. These particular options describe
172 the scheduler that started the job.
173 -o, --format
175 Comma separated list of fields. (use "--helpformat" for a
176 list of available fields).
177 NOTE: When using the format option for listing various fields
179 you can put a %NUMBER afterwards to specify how many charac‐
180 ters should be printed.
181 e.g. format=name%30 will print 30 characters of field name
183 right justified. A %-30 will print 30 characters left justi‐
184 fied.
185 When set, the SACCT_FORMAT environment variable will override
187 the default format. For example:
188 SACCT_FORMAT="jobid,user,account,cluster"
190 -g, --gid=, --group=<gid_or_group_list>
192 Displays the statistics only for the jobs started with the
193 GID or the GROUP specified by the gid_list or the group_list
194 operand, which is a comma-separated list. Space characters
195 are not allowed. Default is no restrictions.
196 -h, --help
198 Displays a general help message.
199 -e, --helpformat
201 Print a list of fields that can be specified with the --for‐
202 mat option.
203 Fields available:
205 Account AdminComment AllocCPUS AllocNodes
207 AllocTRES AssocID AveCPU AveCPUFreq
208 AveDiskRead AveDiskWrite AvePages AveRSS
209 AveVMSize BlockID Cluster Comment
210 Constraints Container ConsumedEnergy ConsumedEnergyRaw
211 CPUTime CPUTimeRAW DBIndex DerivedExitCode
212 Elapsed ElapsedRaw Eligible End
213 ExitCode Flags GID Group
214 JobID JobIDRaw JobName Layout
215 MaxDiskRead MaxDiskReadNode MaxDiskReadTask MaxDiskWrite
216 MaxDiskWriteNode MaxDiskWriteTask MaxPages MaxPagesNode
217 MaxPagesTask MaxRSS MaxRSSNode MaxRSSTask
218 MaxVMSize MaxVMSizeNode MaxVMSizeTask McsLabel
219 MinCPU MinCPUNode MinCPUTask NCPUS
220 NNodes NodeList NTasks Priority
221 Partition QOS QOSRAW Reason
222 ReqCPUFreq ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov
223 ReqCPUS ReqMem ReqNodes ReqTRES
224 Reservation ReservationId Reserved ResvCPU
225 ResvCPURAW Start State Submit
226 SubmitLine Suspended SystemCPU SystemComment
227 Timelimit TimelimitRaw TotalCPU TRESUsageInAve
228 TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin
229 TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve
230 TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
231 TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot UID
232 User UserCPU WCKey WCKeyID
233 WorkDir
234 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
236 SUsageIn[Ave|Tot]. They represent the average/total of the highest wa‐
237 termarks over all ranks in the step. When using sstat they represent
238 the average/total at the moment the command was run.
239 NOTE: TRESUsage*Min* values represent the lowest highwater mark in the
241 step.
242 The section titled "Job Accounting Fields" describes these fields.
244 -j, --jobs=<job[.step]>
246 Displays information about the specified job[.step] or list
247 of job[.step]s.
248 The job[.step] parameter is a comma-separated list of jobs.
250 Space characters are not permitted in this list.
251 NOTE: A step id of 'batch' will display the information about
252 the batch step.
253 By default sacct shows only jobs with Eligible time, but with
254 this option the non-eligible will be also shown.
255 NOTE: If --state is also specified, as non-eligible are not
256 PD, then non-eligible jobs will not be displayed. See the
257 DEFAULT TIME WINDOW for details about how this option changes
258 the default -S and -E options.
259 --json Dump job information as JSON. All other formatting arguments
261 will be ignored.
262 --local Show only jobs local to this cluster. Ignore other clusters
264 in this federation (if any). Overrides --federation.
265 -l, --long
267 Equivalent to specifying:
268 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
270 sizenode, maxvmsizetask,avevm‐
271 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
272 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
273 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
274 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
275 gov,reqmem,consumeden‐
276 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
277 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
278 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
279 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
280 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
281 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
282 outave,tresusageouttot
283 --name=<jobname_list>
285 Display jobs that have any of these name(s).
286 -i, --nnodes=<min[-max]>
288 Return jobs that ran on the specified number of nodes.
289 --noconvert
291 Don't convert units from their original type (e.g. 2048M
292 won't be converted to 2G).
293 -N, --nodelist=<node_list>
295 Display jobs that ran on any of these node(s). node_list can
296 be a ranged string.
297 -n, --noheader
299 No heading will be added to the output. The default action is
300 to display a header.
301 -p, --parsable
303 Output will be '|' delimited with a '|' at the end. See also
304 the --delimiter option.
305 -P, --parsable2
307 Output will be '|' delimited without a '|' at the end. See
308 also the --delimiter option.
309 -r, --partition
311 Comma separated list of partitions to select jobs and job
312 steps from. The default is all partitions.
313 -q, --qos Only send data about jobs using these qos. Default is all.
315 -R, --reason=<reason_list>
317 Comma separated list to filter jobs based on what reason the
318 job wasn't scheduled outside resources/priority.
319 -S, --starttime
321 Select jobs in any state after the specified time. Default is
322 00:00:00 of the current day, unless the '-s' or '-j' options
323 are used. If the '-s' option is used, then the default is
324 'now'. If states are given with the '-s' option then only
325 jobs in this state at this time will be returned. If the '-j'
326 option is used, then the default time is Unix Epoch 0. See
327 the DEFAULT TIME WINDOW for more details.
328 Valid time formats are:
330 HH:MM[:SS][AM|PM]
331 MMDD[YY][-HH:MM[:SS]]
332 MM.DD[.YY][-HH:MM[:SS]]
333 MM/DD[/YY][-HH:MM[:SS]]
334 YYYY-MM-DD[THH:MM[:SS]]
335 today, midnight, noon, fika (3 PM), teatime (4 PM)
336 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
337 -s, --state=<state_list>
339 Selects jobs based on their state during the time period
340 given. Unless otherwise specified, the start and end time
341 will be the current time when the --state option is specified
342 and only currently running jobs can be displayed. A start
343 and/or end time must be specified to view information about
344 jobs not currently running. See the JOB STATE CODES section
345 below for a list of state designators. Multiple state names
346 may be specified using comma separators. Either the short or
347 long form of the state name may be used (e.g. CA or CAN‐
348 CELLED) and the name is case insensitive (i.e. ca and CA both
349 work).
350 NOTE: Note for a job to be selected in the PENDING state it
352 must have "EligibleTime" in the requested time interval or
353 different from "Unknown". The "EligibleTime" is displayed by
354 the "scontrol show job" command. For example jobs submitted
355 with the "--hold" option will have "EligibleTime=Unknown" as
356 they are pending indefinitely.
357 NOTE: When specifying states and no start time is given the
359 default start time is 'now'. This is only when -j is not
360 used. If -j is used the start time will default to 'Epoch'.
361 In both cases if no end time is given it will default to
362 'now'. See the DEFAULT TIME WINDOW for more details.
363 -K, --timelimit-max
365 Ignored by itself, but if timelimit_min is set this will be
366 the maximum timelimit of the range. Default is no restric‐
367 tion.
368 -k, --timelimit-min
370 Only send data about jobs with this timelimit. If used with
371 timelimit_max this will be the minimum timelimit of the
372 range. Default is no restriction.
373 -T, --truncate
375 Truncate time. So if a job started before --starttime the
376 start time would be truncated to --starttime. The same for
377 end time and --endtime.
378 -u, --uid=, --user=<uid_or_user_list>
380 Use this comma separated list of UIDs or user names to select
381 jobs to display. By default, the running user's UID is used.
382 --units=[KMGTP]
384 Display values in specified unit type. Takes precedence over
385 --noconvert option.
386 --usage Display a command usage summary.
388 --use-local-uid
390 When displaying UID, sacct uses the UID stored in Slurm's ac‐
391 counting database by default. Use this command to make Slurm
392 use a system call to get the UID from the username. This op‐
393 tion may be useful in an environment with multiple clusters
394 and one database where the UID's aren't the same on all clus‐
395 ters.
396 -v, --verbose
398 Primarily for debugging purposes, report the state of various
399 variables during processing.
400 -V, --version
402 Print version.
403 -W, --wckeys=<wckey_list>
405 Displays the statistics only for the jobs started on the wck‐
406 eys specified by the wckey_list operand, which is a
407 comma-separated list of wckey names. Space characters are
408 not allowed in the wckey_list. Default is all wckeys.
409 --whole-hetjob[=yes|no]
411 When querying and filtering heterogeneous jobs with --jobs,
412 Slurm will default to retrieving information about all the
413 components of the job if the het_job_id (leader id) is se‐
414 lected. If a non-leader heterogeneous job component id is se‐
415 lected then only that component is retrieved by default. This
416 behavior can be changed by using this option. If set to 'yes'
417 (or no argument), then information about all the components
418 will be retrieved no matter which component is selected in
419 the job filter. If set to 'no' then only the selected hetero‐
420 geneous job component(s) will be retrieved, even when select‐
421 ing the leader.
422 --yaml Dump job information as YAML. All other formatting arguments
424 will be ignored.
425 Job Accounting Fields
427 Descriptions of each job accounting field can be found below. Note
428 that the Ave*, Max* and Min* accounting fields look at the values for
429 all the tasks of each step in a job and return the average, maximum or
430 minimum values for the job step.
431 ALL Print all fields listed below.
433 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
435 AllocNodes
437 Number of nodes allocated to the job/step. 0 if the
438 job is pending.
439 AllocTres Trackable resources. These are the resources allocated
441 to the job/step after the job started running. For
442 pending jobs this should be blank. For more details
443 see AccountingStorageTRES in slurm.conf.
444 NOTE: When a generic resource is configured with the
446 no_consume flag, the allocation will be printed with a
447 zero.
448 Account Account the job ran under.
450 AssocID Reference to the association of user, account and
452 cluster.
453 AveCPU Average (system + user) CPU time of all tasks in job.
455 AveCPUFreq
457 Average weighted CPU frequency of all tasks in job, in
458 kHz.
459 AveDiskRead
461 Average number of bytes read by all tasks in job.
462 AveDiskWrite
464 Average number of bytes written by all tasks in job.
465 AvePages Average number of page faults of all tasks in job.
467 AveRSS Average resident set size of all tasks in job.
469 AveVMSize Average Virtual Memory size of all tasks in job.
471 Cluster Cluster name.
473 Comment The job's comment string when the AccountingStoreFlags
475 parameter in the slurm.conf file contains 'job_com‐
476 ment'. The Comment string can be modified by invoking
477 sacctmgr modify job or the specialized sjobexitmod
478 command.
479 ConsumedEnergy
481 Total energy consumed by all tasks in job, in joules.
482 Note: Only in case of exclusive job allocation this
483 value reflects the jobs' real energy consumption.
484 Container Path to OCI Container Bundle requested.
486 CPUTime Time used (Elapsed time * CPU count) by a job or step
488 in HH:MM:SS format.
489 CPUTimeRAW
491 Time used (Elapsed time * CPU count) by a job or step
492 in cpu-seconds.
493 DerivedExitCode
495 The highest exit code returned by the job's job steps
496 (srun invocations). Following the colon is the signal
497 that caused the process to terminate if it was termi‐
498 nated by a signal. The DerivedExitCode can be modi‐
499 fied by invoking sacctmgr modify job or the special‐
500 ized sjobexitmod command.
501 Elapsed The job's elapsed time.
503 The format of this field's output is as follows:
505 [DD-[HH:]]MM:SS
507 as defined by the following:
509 DD days
511 hh hours
513 mm minutes
515 ss seconds
517 Eligible When the job became eligible to run. In the same for‐
519 mat as End.
520 End Termination time of the job. The output is of the for‐
522 mat YYYY-MM-DDTHH:MM:SS, unless changed through the
523 SLURM_TIME_FORMAT environment variable.
524 ExitCode The exit code returned by the job script or salloc,
526 typically as set by the exit() function. Following
527 the colon is the signal that caused the process to
528 terminate if it was terminated by a signal.
529 GID The group identifier of the user who ran the job.
531 Group The group name of the user who ran the job.
533 JobID The identification number of the job or job step.
535 Regular jobs are in the form:
537 JobID[.JobStep]
539 Array jobs are in the form:
541 ArrayJobID_ArrayTaskID
543 Heterogeneous jobs are in the form:
545 HetJobID+HetJobOffset
547 When printing job arrays, performance of the command
549 can be measurably improved for systems with large num‐
550 bers of jobs when a single job ID is specified. By de‐
551 fault, this field size will be limited to 64 bytes.
552 Use the environment variable SLURM_BITSTR_LEN to spec‐
553 ify larger field sizes.
554 JobIDRaw The identification number of the job or job step.
556 Prints the JobID in the form JobID[.JobStep] for regu‐
557 lar, heterogeneous and array jobs.
558 JobName The name of the job or job step. The slurm_account‐
560 ing.log file is a space delimited file. Because of
561 this if a space is used in the jobname an underscore
562 is substituted for the space before the record is
563 written to the accounting file. So when the jobname is
564 displayed by sacct the jobname that had a space in it
565 will now have an underscore in place of the space.
566 Layout What the layout of a step was when it was running.
568 This can be used to give you an idea of which node ran
569 which rank in your job.
570 MaxDiskRead
572 Maximum number of bytes read by all tasks in job.
573 MaxDiskReadNode
575 The node on which the maxdiskread occurred.
576 MaxDiskReadTask
578 The task ID where the maxdiskread occurred.
579 MaxDiskWrite
581 Maximum number of bytes written by all tasks in job.
582 MaxDiskWriteNode
584 The node on which the maxdiskwrite occurred.
585 MaxDiskWriteTask
587 The task ID where the maxdiskwrite occurred.
588 MaxPages Maximum number of page faults of all tasks in job.
590 MaxPagesNode
592 The node on which the maxpages occurred.
593 MaxPagesTask
595 The task ID where the maxpages occurred.
596 MaxRSS Maximum resident set size of all tasks in job.
598 MaxRSSNode
600 The node on which the maxrss occurred.
601 MaxRSSTask
603 The task ID where the maxrss occurred.
604 MaxVMSize Maximum Virtual Memory size of all tasks in job.
606 MaxVMSizeNode
608 The node on which the maxvmsize occurred.
609 MaxVMSizeTask
611 The task ID where the maxvmsize occurred.
612 MinCPU Minimum (system + user) CPU time of all tasks in job.
614 MinCPUNode
616 The node on which the mincpu occurred.
617 MinCPUTask
619 The task ID where the mincpu occurred.
620 NCPUS Total number of CPUs allocated to the job. Equivalent
622 to AllocCPUS.
623 NodeList List of nodes in job/step.
625 NNodes Number of nodes in a job or step. If the job is run‐
627 ning, or ran, this count will be the number allocated,
628 else the number will be the number requested.
629 NTasks Total number of tasks in a job or step.
631 Priority Slurm priority.
633 Partition Identifies the partition on which the job ran.
635 QOS Name of Quality of Service.
637 QOSRAW Numeric id of Quality of Service.
639 ReqCPUFreq
641 Requested CPU frequency for the step, in kHz. Note:
642 This value applies only to a job step. No value is re‐
643 ported for the job.
644 ReqCPUFreqGov
646 Requested CPU frequency governor for the step, in kHz.
647 Note: This value applies only to a job step. No value
648 is reported for the job.
649 ReqCPUFreqMax
651 Maximum requested CPU frequency for the step, in kHz.
652 Note: This value applies only to a job step. No value
653 is reported for the job.
654 ReqCPUFreqMin
656 Minimum requested CPU frequency for the step, in kHz.
657 Note: This value applies only to a job step. No value
658 is reported for the job.
659 ReqCPUS Number of requested CPUs.
661 ReqMem Minimum required memory for the job. It may have a
663 letter appended to it indicating units (M for
664 megabytes, G for gigabytes, etc.). Note: This value
665 is only from the job allocation, not the step.
666 ReqNodes Requested minimum Node count for the job/step.
668 ReqTres Trackable resources. These are the minimum resource
670 counts requested by the job/step at submission time.
671 For more details see AccountingStorageTRES in
672 slurm.conf.
673 Reservation
675 Reservation Name.
676 ReservationId
678 Reservation Id.
679 Reserved How much wall clock time was used as reserved time for
681 this job. This is derived from how long a job was
682 waiting from eligible time to when it actually
683 started. Format is the same as Elapsed.
684 ResvCPU How many CPU seconds were used as reserved time for
686 this job. Format is the same as Elapsed.
687 ResvCPURAW
689 How many CPU seconds were used as reserved time for
690 this job. Format is in processor seconds.
691 Start Initiation time of the job. In the same format as End.
693 State Displays the job status, or state. See the JOB STATE
695 CODES section below for a list of possible states.
696 If more information is available on the job state than
698 will fit into the current field width (for example,
699 the UID that CANCELLED a job) the state will be fol‐
700 lowed by a "+". You can increase the size of the dis‐
701 played state using the "%NUMBER" format modifier de‐
702 scribed earlier.
703 NOTE: The RUNNING state will return suspended jobs as
705 well. In order to print suspended jobs you must re‐
706 quest SUSPENDED at a different call from RUNNING.
707 NOTE: The RUNNING state will return any jobs completed
709 (cancelled or otherwise) in the time period requested
710 as the job was also RUNNING during that time. If you
711 are only looking for jobs that finished, please choose
712 the appropriate state(s) without the RUNNING state.
713 Submit The time the job was submitted. In the same format as
715 End.
716 NOTE: If a job is requeued, the submit time is reset.
718 To obtain the original submit time it is necessary to
719 use the -D or --duplicate option to display all dupli‐
720 cate entries for a job.
721 SubmitLine
723 The full command issued to submit the job.
724 Suspended The amount of time a job or job step was suspended.
726 Format is the same as Elapsed.
727 SystemCPU The amount of system CPU time used by the job or job
729 step. Format is the same as Elapsed.
730 NOTE: SystemCPU provides a measure of the task's par‐
732 ent process and does not include CPU time of child
733 processes.
734 Timelimit What the timelimit was/is for the job. Format is the
736 same as Elapsed.
737 TimelimitRaw
739 What the timelimit was/is for the job. Format is in
740 number of minutes.
741 TotalCPU The sum of the SystemCPU and UserCPU time used by the
743 job or job step. The total CPU time of the job may
744 exceed the job's elapsed time for jobs that include
745 multiple job steps. Format is the same as Elapsed.
746 NOTE: TotalCPU provides a measure of the task's parent
748 process and does not include CPU time of child pro‐
749 cesses.
750 TresUsageInAve
752 Tres average usage in by all tasks in job. NOTE: If
753 corresponding TresUsageInMaxTask is -1 the metric is
754 node centric instead of task.
755 TresUsageInMax
757 Tres maximum usage in by all tasks in job. NOTE: If
758 corresponding TresUsageInMaxTask is -1 the metric is
759 node centric instead of task.
760 TresUsageInMaxNode
762 Node for which each maximum TRES usage out occurred.
763 TresUsageInMaxTask
765 Task for which each maximum TRES usage out occurred.
766 TresUsageInMin
768 Tres minimum usage in by all tasks in job. NOTE: If
769 corresponding TresUsageInMinTask is -1 the metric is
770 node centric instead of task.
771 TresUsageInMinNode
773 Node for which each minimum TRES usage out occurred.
774 TresUsageInMinTask
776 Task for which each minimum TRES usage out occurred.
777 TresUsageInTot
779 Tres total usage in by all tasks in job.
780 TresUsageOutAve
782 Tres average usage out by all tasks in job. NOTE: If
783 corresponding TresUsageOutMaxTask is -1 the metric is
784 node centric instead of task.
785 TresUsageOutMax
787 Tres maximum usage out by all tasks in job. NOTE: If
788 corresponding TresUsageOutMaxTask is -1 the metric is
789 node centric instead of task.
790 TresUsageOutMaxNode
792 Node for which each maximum TRES usage out occurred.
793 TresUsageOutMaxTask
795 Task for which each maximum TRES usage out occurred.
796 TresUsageOutTot
798 Tres total usage out by all tasks in job.
799 UID The user identifier of the user who ran the job.
801 User The user name of the user who ran the job.
803 UserCPU The amount of user CPU time used by the job or job
805 step. Format is the same as Elapsed.
806 NOTE: UserCPU provides a measure of the task's parent
808 process and does not include CPU time of child pro‐
809 cesses.
810 WCKey Workload Characterization Key. Arbitrary string
812 for grouping orthogonal accounts together.
813 WCKeyID Reference to the wckey.
815
817 BF BOOT_FAIL Job terminated due to launch failure, typically due
818 to a hardware failure (e.g. unable to boot the node
819 or block and the job can not be requeued).
820 CA CANCELLED Job was explicitly cancelled by the user or system
822 administrator. The job may or may not have been
823 initiated.
824 CD COMPLETED Job has terminated all processes on all nodes with
826 an exit code of zero.
827 DL DEADLINE Job terminated on deadline.
829 F FAILED Job terminated with non-zero exit code or other
831 failure condition.
832 NF NODE_FAIL Job terminated due to failure of one or more allo‐
834 cated nodes.
835 OOM OUT_OF_MEMORY Job experienced out of memory error.
837 PD PENDING Job is awaiting resource allocation.
839 PR PREEMPTED Job terminated due to preemption.
841 R RUNNING Job currently has an allocation.
843 RQ REQUEUED Job was requeued.
845 RS RESIZING Job is about to change size.
847 RV REVOKED Sibling was removed from cluster due to other clus‐
849 ter starting the job.
850 S SUSPENDED Job has an allocation, but execution has been sus‐
852 pended and CPUs have been released for other jobs.
853 TO TIMEOUT Job terminated upon reaching its time limit.
855
857 The options --starttime and --endtime define the time window between
858 which sacct is going to search. For historical and practical reasons
859 their default values (i.e. the default time window) depends on other
860 options: --jobs and --state.
861 Depending on if --jobs and/or --state are specified, the default values
863 of --starttime and --endtime options are:
864 WITHOUT EITHER --jobs NOR --state specified:
867 --starttime defaults to Midnight.
868 --endtime defaults to Now.
869 WITH --jobs AND WITHOUT --state specified:
872 --starttime defaults to Epoch 0.
873 --endtime defaults to Now.
874 WITHOUT --jobs AND WITH --state specified:
877 --starttime defaults to Now.
878 --endtime defaults to --starttime and to Now if --starttime is not
879 specified.
880 WITH BOTH --jobs AND --state specified:
883 --starttime defaults to Epoch 0.
884 --endtime defaults to --starttime or to Now if --starttime is not spec‐
885 ified.
886 NOTE: With -v/--verbose a message about the actual time window in use
889 is shown.
890
893 Executing sacct sends a remote procedure call to slurmdbd. If enough
894 calls from sacct or other Slurm client commands that send remote proce‐
895 dure calls to the slurmdbd daemon come in at once, it can result in a
896 degradation of performance of the slurmdbd daemon, possibly resulting
897 in a denial of service.
898 Do not run sacct or other Slurm client commands that send remote proce‐
900 dure calls to slurmdbd from loops in shell scripts or other programs.
901 Ensure that programs limit calls to sacct to the minimum necessary for
902 the information you are trying to gather.
903
906 Some sacct options may be set via environment variables. These environ‐
907 ment variables, along with their corresponding options, are listed be‐
908 low. (Note: Command line options will always override these settings.)
909 SACCT_FEDERATION Same as --federation
912 SACCT_LOCAL Same as --local
914 SLURM_BITSTR_LEN Specifies the string length to be used for holding
916 a job array's task ID expression. The default value
917 is 64 bytes. A value of 0 will print the full ex‐
918 pression with any length required. Larger values
919 may adversely impact the application performance.
920 SLURM_CONF The location of the Slurm configuration file.
922 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
924 value of standard, the default value, generates
925 output in the form
926 "year-month-dateThour:minute:second". A value of
927 relative returns only "hour:minute:second" if the
928 current day. For other dates in the current year
929 it prints the "hour:minute" preceded by "Tomorr"
930 (tomorrow), "Ystday" (yesterday), the name of the
931 day for the coming week (e.g. "Mon", "Tue", etc.),
932 otherwise the date (e.g. "25 Apr"). For other
933 years it returns a date month and year without a
934 time (e.g. "6 Jun 2012"). All of the time stamps
935 use a 24 hour format.
936 A valid strftime() format can also be specified.
938 For example, a value of "%a %T" will report the day
939 of the week and a time stamp (e.g. "Mon 12:34:56").
940
942 This example illustrates the default invocation of the sacct command:
943 # sacct
945 Jobid Jobname Partition Account AllocCPUS State ExitCode
946 ---------- ---------- ---------- ---------- ---------- ---------- --------
947 2 script01 srun acct1 1 RUNNING 0
948 3 script02 srun acct1 1 RUNNING 0
949 4 endscript srun acct1 1 RUNNING 0
950 4.0 srun acct1 1 COMPLETED 0
951 This example shows the same job accounting information with the brief
954 option.
955 # sacct --brief
957 Jobid State ExitCode
958 ---------- ---------- --------
959 2 RUNNING 0
960 3 RUNNING 0
961 4 RUNNING 0
962 4.0 COMPLETED 0
963 # sacct --allocations
965 Jobid Jobname Partition Account AllocCPUS State ExitCode
966 ---------- ---------- ---------- ---------- ------- ---------- --------
967 3 sja_init andy acct1 1 COMPLETED 0
968 4 sjaload andy acct1 2 COMPLETED 0
969 5 sja_scr1 andy acct1 1 COMPLETED 0
970 6 sja_scr2 andy acct1 18 COMPLETED 2
971 7 sja_scr3 andy acct1 18 COMPLETED 0
972 8 sja_scr5 andy acct1 2 COMPLETED 0
973 9 sja_scr7 andy acct1 90 COMPLETED 1
974 10 endscript andy acct1 186 COMPLETED 0
975 This example demonstrates the ability to customize the output of the
978 sacct command. The fields are displayed in the order designated on the
979 command line.
980 # sacct --format=jobid,elapsed,ncpus,ntasks,state
982 Jobid Elapsed Ncpus Ntasks State
983 ---------- ---------- ---------- -------- ----------
984 3 00:01:30 2 1 COMPLETED
985 3.0 00:01:30 2 1 COMPLETED
986 4 00:00:00 2 2 COMPLETED
987 4.0 00:00:01 2 2 COMPLETED
988 5 00:01:23 2 1 COMPLETED
989 5.0 00:01:31 2 1 COMPLETED
990 This example demonstrates the use of the -T (--truncate) option when
993 used with -S (--starttime) and -E (--endtime). When the -T option is
994 used, the start time of the job will be the specified -S value if the
995 job was started before the specified time, otherwise the time will be
996 the job's start time. The end time will be the specified -E option if
997 the job ends after the specified time, otherwise it will be the jobs
998 end time.
999 Without -T (normal operation) sacct output would be like this.
1001 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1003 JobID Start End State
1004 --------- --------------------- -------------------- ------------
1005 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1006 3 2014-07-03T11:35:21 Unknown RUNNING
1007 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1008 5 2014-07-03T11:41:01 Unknown RUNNING
1009 By adding the -T option the job's start and end times are truncated to
1011 reflect only the time requested. If a job started after the start time
1012 requested or finished before the end time requested those times are not
1013 altered. The -T option is useful when determining exact run times dur‐
1014 ing any given period.
1015 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1017 JobID Start End State
1018 --------- --------------------- -------------------- ------------
1019 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1020 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1021 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1022 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1023 NOTE: If no -s (--state) option is given sacct will display eligible
1026 jobs during the specified period of time, otherwise it will return jobs
1027 that were in the state requested during that period of time.
1028 This example demonstrates the differences running sacct with and with‐
1030 out the --state flag for the same time period. Without the --state op‐
1031 tion, all eligible jobs in that time period are shown.
1032 # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1034 JobID Start End State
1035 ------------ ------------------- ------------------- ----------
1036 2955 11:15:12 11:20:12 COMPLETED
1037 2956 11:20:13 11:25:13 COMPLETED
1038 With the --state=pending option, only job 2956 will be shown because it
1040 had a dependency on 2955 and was still PENDING from 11:20:00 until it
1041 started at 11:21:13. Note that even though we requested PENDING jobs,
1042 the State shows as COMPLETED because that is the current State of the
1043 job.
1044 # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1046 JobID Start End State
1047 ------------ ------------------- ------------------- ----------
1048 2956 11:20:13 11:25:13 COMPLETED
1049
1052 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1053 L.P.
1054 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1055 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1056 Copyright (C) 2010-2022 SchedMD LLC.
1057 This file is part of Slurm, a resource management program. For de‐
1059 tails, see <https://slurm.schedmd.com/>.
1060 Slurm is free software; you can redistribute it and/or modify it under
1062 the terms of the GNU General Public License as published by the Free
1063 Software Foundation; either version 2 of the License, or (at your op‐
1064 tion) any later version.
1065 Slurm is distributed in the hope that it will be useful, but WITHOUT
1067 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1068 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1069 for more details.
1070
1073 /etc/slurm.conf
1074 Entries to this file enable job accounting and designate the
1075 job accounting log file that collects system job accounting.
1076 /var/log/slurm_accounting.log
1078 The default job accounting log file. By default, this file
1079 is set to read and write permission for root only.
1080
1083 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1084February 2022 Slurm Commands sacct(1)