1sacct(1) Slurm Commands sacct(1)
2
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
11 sacct [OPTIONS...]
12
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18 The sacct command displays job accounting data stored in the job ac‐
20 counting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24 Job records consist of a primary entry for the job as a whole as well
26 as entries for job steps. The Job Launch page has a more detailed de‐
27 scription of each type of job step.
28 <https://slurm.schedmd.com/job_launch.html#job_record>
29 For the root user, the sacct command displays job accounting data for
31 all users, although there are options to filter the output to report
32 only the jobs from a specified user or group.
33 For the non-root user, the sacct command limits the display of job ac‐
35 counting data to jobs that were launched with their own user identifier
36 (UID) by default. Data for other users can be displayed with the --al‐
37 lusers, --user, or --uid options.
38 Elapsed time fields are presented as [days-]hours:minutes:seconds[.mi‐
40 croseconds]. Only 'CPU' fields will ever have microseconds.
41 The default input file is the file named in the AccountingStorageLoc
43 parameter in slurm.conf.
44 NOTE: If designated, the slurmdbd.conf option PrivateData may further
46 restrict the accounting data visible to users which are not SlurmUser,
47 root, or a user with AdminLevel=Admin. See the slurmdbd.conf man page
48 for additional details on restricting access to accounting data.
49 NOTE: The contents of Slurm's database are maintained in lower case.
51 This may result in some sacct output differing from that of other Slurm
52 commands.
53 NOTE: Much of the data reported by sacct has been generated by the
55 wait3() and getrusage() system calls. Some systems gather and report
56 incomplete information for these calls; sacct reports values of 0 for
57 this missing data. See your systems getrusage [4m(3) man page for informa‐
58 tion about which data are actually available on your system.
59
62 -A, --accounts=<account_list>
63 Displays jobs when a comma separated list of accounts are
64 given as the argument.
65 -L, --allclusters
67 Display jobs ran on all clusters. By default, only jobs ran
68 on the cluster from where sacct is called are displayed.
69 -X, --allocations
71 Only show statistics relevant to the job allocation itself,
72 not taking steps into consideration.
73 NOTE: Without including steps, utilization statistics for job
75 allocation(s) will be reported as zero.
76 -a, --allusers
78 Displays all users' jobs when run by user root or if Private‐
79 Data is not configured to jobs. Otherwise display the cur‐
80 rent user's jobs
81 -x, --associations=<assoc_list>
83 Displays the statistics only for the jobs running under the
84 association ids specified by the assoc_list operand, which is
85 a comma-separated list of association ids. Space characters
86 are not allowed in the assoc_list. Default is all associa‐
87 tions.
88 -B, --batch-script
90 This option will print the batch script of job if the job
91 used one. If the job didn't have a script 'NONE' is output.
92 NOTE: AccountingStoreFlags=job_script is required for this.
93 NOTE: Requesting specific job(s) with '-j' is required for
94 this.
95 -b, --brief
97 Displays a brief listing consisting of JobID, State, and Ex‐
98 itCode.
99 -M, --clusters=<cluster_list>
101 Displays the statistics only for the jobs started on the
102 clusters specified by the cluster_list operand, which is a
103 comma-separated list of clusters. Space characters are not
104 allowed in the cluster_list. A value of 'all' will query to
105 run on all clusters. The default is current cluster you are
106 executing the sacct command on or all clusters in the federa‐
107 tion when executed on a federated cluster. This option im‐
108 plicitly sets the --local option.
109 -c, --completion
111 Use job completion data instead of job accounting. The Job‐
112 CompType parameter in the slurm.conf file must be defined to
113 a non-none option. Does not support federated cluster infor‐
114 mation (local data only).
115 -C, --constraints=<constraint_list>
117 Comma separated list to filter jobs based on what con‐
118 straints/features the job requested. Multiple options will
119 be treated as 'and' not 'or', so the job would need all con‐
120 straints specified to be returned not one or the other.
121 --delimiter=<characters>
123 ASCII characters used to separate the fields when specifying
124 the -p or -P options. The default delimiter is a '|'. This
125 option is ignored if -p or -P options are not specified.
126 -D, --duplicates
128 If Slurm job ids are reset, some job numbers will probably
129 appear more than once in the accounting log file but refer to
130 different jobs. Such jobs can be distinguished by the "sub‐
131 mit" time stamp in the data records.
132 When data for specific jobs are requested with the --jobs op‐
134 tion, sacct returns the most recent job with that number.
135 This behavior can be overridden by specifying --duplicates,
136 in which case all records that match the selection criteria
137 will be returned.
138 NOTE: Revoked federated sibling jobs are hidden unless the
140 --duplicates option is specified.
141 -E, --endtime=<end_time>
143 Select jobs in any state before the specified time. If
144 states are given with the -s option return jobs in this state
145 before this period. See the DEFAULT TIME WINDOW for more de‐
146 tails.
147 Valid time formats are:
149 HH:MM[:SS][AM|PM]
150 MMDD[YY][-HH:MM[:SS]]
151 MM.DD[.YY][-HH:MM[:SS]]
152 MM/DD[/YY][-HH:MM[:SS]]
153 YYYY-MM-DD[THH:MM[:SS]]
154 today, midnight, noon, fika (3 PM), teatime (4 PM)
155 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
156 --env-vars
158 This option will print the running environment of a batch
159 job, otherwise 'NONE' is output.
160 NOTE: AccountingStoreFlags=job_env is required for this.
161 NOTE: Requesting specific job(s) with '-j' is required for
162 this.
163 --federation
165 Show jobs from the federation if a member of one.
166 -f, --file=<file>
168 Causes the sacct command to read job accounting data from the
169 named file instead of the current Slurm job accounting log
170 file. Only applicable when running the jobcomp/filetxt
171 plugin. Setting this flag implicitly enables the -c flag.
172 -F, --flags=<flag_list>
174 Comma separated list to filter jobs based on what various
175 ways the jobs were handled. Current flags are SchedSubmit,
176 SchedMain, SchedBackfill. These particular options describe
177 the scheduler that started the job.
178 -o, --format
180 Comma separated list of fields. (use "--helpformat" for a
181 list of available fields).
182 NOTE: When using the format option for listing various fields
184 you can put a %NUMBER afterwards to specify how many charac‐
185 ters should be printed.
186 e.g. format=name%30 will print 30 characters of field name
188 right justified. A %-30 will print 30 characters left justi‐
189 fied.
190 When set, the SACCT_FORMAT environment variable will override
192 the default format. For example:
193 SACCT_FORMAT="jobid,user,account,cluster"
195 -g, --gid=, --group=<gid_or_group_list>
197 Displays the statistics only for the jobs started with the
198 GID or the GROUP specified by the gid_list or the group_list
199 operand, which is a comma-separated list. Space characters
200 are not allowed. Default is no restrictions.
201 -h, --help
203 Displays a general help message.
204 -e, --helpformat
206 Print a list of fields that can be specified with the --for‐
207 mat option.
208 Fields available:
210 Account AdminComment AllocCPUS AllocNodes
212 AllocTRES AssocID AveCPU AveCPUFreq
213 AveDiskRead AveDiskWrite AvePages AveRSS
214 AveVMSize BlockID Cluster Comment
215 Constraints ConsumedEnergy ConsumedEnergyRaw Container
216 CPUTime CPUTimeRAW DBIndex DerivedExitCode
217 Elapsed ElapsedRaw Eligible End
218 ExitCode Flags GID Group
219 JobID JobIDRaw JobName Layout
220 MaxDiskRead MaxDiskReadNode MaxDiskReadTask MaxDiskWrite
221 MaxDiskWriteNode MaxDiskWriteTask MaxPages MaxPagesNode
222 MaxPagesTask MaxRSS MaxRSSNode MaxRSSTask
223 MaxVMSize MaxVMSizeNode MaxVMSizeTask McsLabel
224 MinCPU MinCPUNode MinCPUTask NCPUS
225 NNodes NodeList NTasks Partition
226 Priority QOS QOSRAW Reason
227 ReqCPUFreq ReqCPUFreqGov ReqCPUFreqMax ReqCPUFreqMin
228 ReqCPUS ReqMem ReqNodes ReqTRES
229 Reservation ReservationId Reserved ResvCPU
230 ResvCPURAW Start State Submit
231 SubmitLine Suspended SystemComment SystemCPU
232 Timelimit TimelimitRaw TotalCPU TRESUsageInAve
233 TRESUsageInMax TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin
234 TRESUsageInMinNode TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve
235 TRESUsageOutMax TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
236 TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot UID
237 User UserCPU WCKey WCKeyID
238 WorkDir
239 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
241 SUsageIn[Ave|Tot]. They represent the average/total of the highest wa‐
242 termarks over all ranks in the step. When using sstat they represent
243 the average/total at the moment the command was run.
244 NOTE: TRESUsage*Min* values represent the lowest highwater mark in the
246 step.
247 The section titled "Job Accounting Fields" describes these fields.
249 -j, --jobs=<job[.step]>
251 Displays information about the specified job[.step] or list
252 of job[.step]s.
253 The job[.step] parameter is a comma-separated list of jobs.
255 Space characters are not permitted in this list.
256 NOTE: A step id of 'batch' will display the information about
257 the batch step.
258 By default sacct shows only jobs with Eligible time, but with
259 this option the non-eligible will be also shown.
260 NOTE: If --state is also specified, as non-eligible are not
261 PD, then non-eligible jobs will not be displayed. See the
262 DEFAULT TIME WINDOW for details about how this option changes
263 the default -S and -E options.
264 --json Dump job information as JSON. All other formatting arguments
266 will be ignored.
267 --local Show only jobs local to this cluster. Ignore other clusters
269 in this federation (if any). Overrides --federation.
270 -l, --long
272 Equivalent to specifying:
273 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
275 sizenode, maxvmsizetask,avevm‐
276 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
277 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
278 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
279 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
280 gov,reqmem,consumeden‐
281 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
282 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
283 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
284 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
285 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
286 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
287 outave,tresusageouttot
288 --name=<jobname_list>
290 Display jobs that have any of these name(s).
291 -i, --nnodes=<min[-max]>
293 Return jobs that ran on the specified number of nodes.
294 -I, --ncpus=<min[-max]>
296 Return jobs that ran on the specified number of cpus.
297 --noconvert
299 Don't convert units from their original type (e.g. 2048M
300 won't be converted to 2G).
301 -N, --nodelist=<node_list>
303 Display jobs that ran on any of these node(s). node_list can
304 be a ranged string.
305 -n, --noheader
307 No heading will be added to the output. The default action is
308 to display a header.
309 -p, --parsable
311 Output will be '|' delimited with a '|' at the end. See also
312 the --delimiter option.
313 -P, --parsable2
315 Output will be '|' delimited without a '|' at the end. See
316 also the --delimiter option.
317 -r, --partition
319 Comma separated list of partitions to select jobs and job
320 steps from. The default is all partitions.
321 -q, --qos Only send data about jobs using these qos. Default is all.
323 -R, --reason=<reason_list>
325 Comma separated list to filter jobs based on what reason the
326 job wasn't scheduled outside resources/priority.
327 -S, --starttime
329 Select jobs in any state after the specified time. Default is
330 00:00:00 of the current day, unless the '-s' or '-j' options
331 are used. If the '-s' option is used, then the default is
332 'now'. If states are given with the '-s' option then only
333 jobs in this state at this time will be returned. If the '-j'
334 option is used, then the default time is Unix Epoch 0. See
335 the DEFAULT TIME WINDOW for more details.
336 Valid time formats are:
338 HH:MM[:SS][AM|PM]
339 MMDD[YY][-HH:MM[:SS]]
340 MM.DD[.YY][-HH:MM[:SS]]
341 MM/DD[/YY][-HH:MM[:SS]]
342 YYYY-MM-DD[THH:MM[:SS]]
343 today, midnight, noon, fika (3 PM), teatime (4 PM)
344 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
345 -s, --state=<state_list>
347 Selects jobs based on their state during the time period
348 given. Unless otherwise specified, the start and end time
349 will be the current time when the --state option is specified
350 and only currently running jobs can be displayed. A start
351 and/or end time must be specified to view information about
352 jobs not currently running. See the JOB STATE CODES section
353 below for a list of state designators. Multiple state names
354 may be specified using comma separators. Either the short or
355 long form of the state name may be used (e.g. CA or CAN‐
356 CELLED) and the name is case insensitive (i.e. ca and CA both
357 work).
358 NOTE: Note for a job to be selected in the PENDING state it
360 must have "EligibleTime" in the requested time interval or
361 different from "Unknown". The "EligibleTime" is displayed by
362 the "scontrol show job" command. For example jobs submitted
363 with the "--hold" option will have "EligibleTime=Unknown" as
364 they are pending indefinitely.
365 NOTE: When specifying states and no start time is given the
367 default start time is 'now'. This is only when -j is not
368 used. If -j is used the start time will default to 'Epoch'.
369 In both cases if no end time is given it will default to
370 'now'. See the DEFAULT TIME WINDOW for more details.
371 -K, --timelimit-max
373 Ignored by itself, but if timelimit_min is set this will be
374 the maximum timelimit of the range. Default is no restric‐
375 tion.
376 -k, --timelimit-min
378 Only send data about jobs with this timelimit. If used with
379 timelimit_max this will be the minimum timelimit of the
380 range. Default is no restriction.
381 -T, --truncate
383 Truncate time. So if a job started before --starttime the
384 start time would be truncated to --starttime. The same for
385 end time and --endtime.
386 -u, --uid=, --user=<uid_or_user_list>
388 Use this comma separated list of UIDs or user names to select
389 jobs to display. By default, the running user's UID is used.
390 --units=[KMGTP]
392 Display values in specified unit type. Takes precedence over
393 --noconvert option.
394 --usage Display a command usage summary.
396 --use-local-uid
398 When displaying UID, sacct uses the UID stored in Slurm's ac‐
399 counting database by default. Use this command to make Slurm
400 use a system call to get the UID from the username. This op‐
401 tion may be useful in an environment with multiple clusters
402 and one database where the UID's aren't the same on all clus‐
403 ters.
404 -v, --verbose
406 Primarily for debugging purposes, report the state of various
407 variables during processing.
408 -V, --version
410 Print version.
411 -W, --wckeys=<wckey_list>
413 Displays the statistics only for the jobs started on the wck‐
414 eys specified by the wckey_list operand, which is a
415 comma-separated list of wckey names. Space characters are
416 not allowed in the wckey_list. Default is all wckeys.
417 --whole-hetjob[=yes|no]
419 When querying and filtering heterogeneous jobs with --jobs,
420 Slurm will default to retrieving information about all the
421 components of the job if the het_job_id (leader id) is se‐
422 lected. If a non-leader heterogeneous job component id is se‐
423 lected then only that component is retrieved by default. This
424 behavior can be changed by using this option. If set to 'yes'
425 (or no argument), then information about all the components
426 will be retrieved no matter which component is selected in
427 the job filter. If set to 'no' then only the selected hetero‐
428 geneous job component(s) will be retrieved, even when select‐
429 ing the leader.
430 --yaml Dump job information as YAML. All other formatting arguments
432 will be ignored.
433 Job Accounting Fields
435 Descriptions of each job accounting field can be found below. Note
436 that the Ave*, Max* and Min* accounting fields look at the values for
437 all the tasks of each step in a job and return the average, maximum or
438 minimum values for the job step.
439 ALL Print all fields listed below.
441 Account Account the job ran under.
443 AdminComment
445 A comment string on a job that must be set by an ad‐
446 ministrator, the SlurmUser or root.
447 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
449 AllocNodes
451 Number of nodes allocated to the job/step. 0 if the
452 job is pending.
453 AllocTres Trackable resources. These are the resources allocated
455 to the job/step after the job started running. For
456 pending jobs this should be blank. For more details
457 see AccountingStorageTRES in slurm.conf.
458 NOTE: When a generic resource is configured with the
460 no_consume flag, the allocation will be printed with a
461 zero.
462 AssocID Reference to the association of user, account and
464 cluster.
465 AveCPU Average (system + user) CPU time of all tasks in job.
467 AveCPUFreq
469 Average weighted CPU frequency of all tasks in job, in
470 kHz.
471 AveDiskRead
473 Average number of bytes read by all tasks in job.
474 AveDiskWrite
476 Average number of bytes written by all tasks in job.
477 AvePages Average number of page faults of all tasks in job.
479 AveRSS Average resident set size of all tasks in job.
481 AveVMSize Average Virtual Memory size of all tasks in job.
483 BlockID The name of the block to be used (used with Blue Gene
485 systems).
486 Cluster Cluster name.
488 Comment The job's comment string when the AccountingStoreFlags
490 parameter in the slurm.conf file contains 'job_com‐
491 ment'. The Comment string can be modified by invoking
492 sacctmgr modify job or the specialized sjobexitmod
493 command.
494 Constraints
496 Feature(s) the job requested as a constraint.
497 ConsumedEnergy
499 Total energy consumed by all tasks in a job, in
500 joules. Value may include a unit prefix (K,M,G,T,P).
501 Note: Only in the case of an exclusive job allocation
502 does this value reflect the job's real energy consump‐
503 tion.
504 ConsumedEnergyRaw
506 Total energy consumed by all tasks in a job, in
507 joules. Note: Only in the case of an exclusive job
508 allocation does this value reflect the job's real en‐
509 ergy consumption.
510 Container Path to OCI Container Bundle requested.
512 CPUTime Time used (Elapsed time * CPU count) by a job or step
514 in HH:MM:SS format.
515 CPUTimeRAW
517 Time used (Elapsed time * CPU count) by a job or step
518 in cpu-seconds.
519 DBIndex Unique database index for entries in the job table.
521 DerivedExitCode
523 The highest exit code returned by the job's job steps
524 (srun invocations). Following the colon is the signal
525 that caused the process to terminate if it was termi‐
526 nated by a signal. The DerivedExitCode can be modi‐
527 fied by invoking sacctmgr modify job or the special‐
528 ized sjobexitmod command.
529 Elapsed The job's elapsed time.
531 The format of this field's output is as follows:
533 [DD-[HH:]]MM:SS
535 as defined by the following:
537 DD days
539 hh hours
541 mm minutes
543 ss seconds
545 ElapsedRaw
547 The job's elapsed time in seconds.
548 Eligible When the job became eligible to run. In the same for‐
550 mat as End.
551 End Termination time of the job. The output is of the for‐
553 mat YYYY-MM-DDTHH:MM:SS, unless changed through the
554 SLURM_TIME_FORMAT environment variable.
555 ExitCode The exit code returned by the job script or salloc,
557 typically as set by the exit() function. Following
558 the colon is the signal that caused the process to
559 terminate if it was terminated by a signal.
560 Flags Job flags. Current flags are SchedSubmit, SchedMain,
562 SchedBackfill.
563 GID The group identifier of the user who ran the job.
565 Group The group name of the user who ran the job.
567 JobID The identification number of the job or job step.
569 Regular jobs are in the form:
571 JobID[.JobStep]
573 Array jobs are in the form:
575 ArrayJobID_ArrayTaskID
577 Heterogeneous jobs are in the form:
579 HetJobID+HetJobOffset
581 When printing job arrays, performance of the command
583 can be measurably improved for systems with large num‐
584 bers of jobs when a single job ID is specified. By de‐
585 fault, this field size will be limited to 64 bytes.
586 Use the environment variable SLURM_BITSTR_LEN to spec‐
587 ify larger field sizes.
588 JobIDRaw The identification number of the job or job step.
590 Prints the JobID in the form JobID[.JobStep] for regu‐
591 lar, heterogeneous and array jobs.
592 JobName The name of the job or job step. The slurm_account‐
594 ing.log file is a space delimited file. Because of
595 this if a space is used in the jobname an underscore
596 is substituted for the space before the record is
597 written to the accounting file. So when the jobname is
598 displayed by sacct the jobname that had a space in it
599 will now have an underscore in place of the space.
600 Layout What the layout of a step was when it was running.
602 This can be used to give you an idea of which node ran
603 which rank in your job.
604 MaxDiskRead
606 Maximum number of bytes read by all tasks in job.
607 MaxDiskReadNode
609 The node on which the maxdiskread occurred.
610 MaxDiskReadTask
612 The task ID where the maxdiskread occurred.
613 MaxDiskWrite
615 Maximum number of bytes written by all tasks in job.
616 MaxDiskWriteNode
618 The node on which the maxdiskwrite occurred.
619 MaxDiskWriteTask
621 The task ID where the maxdiskwrite occurred.
622 MaxPages Maximum number of page faults of all tasks in job.
624 MaxPagesNode
626 The node on which the maxpages occurred.
627 MaxPagesTask
629 The task ID where the maxpages occurred.
630 MaxRSS Maximum resident set size of all tasks in job.
632 MaxRSSNode
634 The node on which the maxrss occurred.
635 MaxRSSTask
637 The task ID where the maxrss occurred.
638 MaxVMSize Maximum Virtual Memory size of all tasks in job.
640 MaxVMSizeNode
642 The node on which the maxvmsize occurred.
643 MaxVMSizeTask
645 The task ID where the maxvmsize occurred.
646 MCSLabel Multi-Category Security (MCS) label associated with
648 the job. Added to a job when the MCSPlugin is enabled
649 in the slurm.conf.
650 MinCPU Minimum (system + user) CPU time of all tasks in job.
652 MinCPUNode
654 The node on which the mincpu occurred.
655 MinCPUTask
657 The task ID where the mincpu occurred.
658 NCPUS Total number of CPUs allocated to the job. Equivalent
660 to AllocCPUS.
661 NNodes Number of nodes in a job or step. If the job is run‐
663 ning, or ran, this count will be the number allocated,
664 else the number will be the number requested.
665 NodeList List of nodes in job/step.
667 NTasks Total number of tasks in a job or step.
669 Partition Identifies the partition on which the job ran.
671 Priority Slurm priority.
673 QOS Name of Quality of Service.
675 QOSRAW Numeric id of Quality of Service.
677 Reason The last reason a job was blocked from running for
679 something other than Priority or Resources. This will
680 be saved in the database even if the job ran to com‐
681 pletion.
682 ReqCPUFreq
684 Requested CPU frequency for the step, in kHz. Note:
685 This value applies only to a job step. No value is re‐
686 ported for the job.
687 ReqCPUFreqGov
689 Requested CPU frequency governor for the step, in kHz.
690 Note: This value applies only to a job step. No value
691 is reported for the job.
692 ReqCPUFreqMax
694 Maximum requested CPU frequency for the step, in kHz.
695 Note: This value applies only to a job step. No value
696 is reported for the job.
697 ReqCPUFreqMin
699 Minimum requested CPU frequency for the step, in kHz.
700 Note: This value applies only to a job step. No value
701 is reported for the job.
702 ReqCPUS Number of requested CPUs.
704 ReqMem Minimum required memory for the job. It may have a
706 letter appended to it indicating units (M for
707 megabytes, G for gigabytes, etc.). Note: This value
708 is only from the job allocation, not the step.
709 ReqNodes Requested minimum Node count for the job/step.
711 ReqTres Trackable resources. These are the minimum resource
713 counts requested by the job/step at submission time.
714 For more details see AccountingStorageTRES in
715 slurm.conf.
716 Reservation
718 Reservation Name.
719 ReservationId
721 Reservation Id.
722 Reserved How much wall clock time was used as reserved time for
724 this job. This is derived from how long a job was
725 waiting from eligible time to when it actually
726 started. Format is the same as Elapsed.
727 ResvCPU How many CPU seconds were used as reserved time for
729 this job. Format is the same as Elapsed.
730 ResvCPURAW
732 How many CPU seconds were used as reserved time for
733 this job. Format is in processor seconds.
734 Start Initiation time of the job. In the same format as End.
736 State Displays the job status, or state. See the JOB STATE
738 CODES section below for a list of possible states.
739 If more information is available on the job state than
741 will fit into the current field width (for example,
742 the UID that CANCELLED a job) the state will be fol‐
743 lowed by a "+". You can increase the size of the dis‐
744 played state using the "%NUMBER" format modifier de‐
745 scribed earlier.
746 NOTE: The RUNNING state will return suspended jobs as
748 well. In order to print suspended jobs you must re‐
749 quest SUSPENDED at a different call from RUNNING.
750 NOTE: The RUNNING state will return any jobs completed
752 (cancelled or otherwise) in the time period requested
753 as the job was also RUNNING during that time. If you
754 are only looking for jobs that finished, please choose
755 the appropriate state(s) without the RUNNING state.
756 Submit The time the job was submitted. In the same format as
758 End.
759 NOTE: If a job is requeued, the submit time is reset.
761 To obtain the original submit time it is necessary to
762 use the -D or --duplicate option to display all dupli‐
763 cate entries for a job.
764 SubmitLine
766 The full command issued to submit the job.
767 Suspended The amount of time a job or job step was suspended.
769 Format is the same as Elapsed.
770 SystemComment
772 The job's comment string that is typically set by a
773 plugin. Can only be modified by a Slurm administra‐
774 tor.
775 SystemCPU The amount of system CPU time used by the job or job
777 step. Format is the same as Elapsed.
778 NOTE: SystemCPU provides a measure of the task's par‐
780 ent process and does not include CPU time of child
781 processes.
782 Timelimit What the timelimit was/is for the job. Format is the
784 same as Elapsed.
785 TimelimitRaw
787 What the timelimit was/is for the job. Format is in
788 number of minutes.
789 TotalCPU The sum of the SystemCPU and UserCPU time used by the
791 job or job step. The total CPU time of the job may
792 exceed the job's elapsed time for jobs that include
793 multiple job steps. Format is the same as Elapsed.
794 NOTE: TotalCPU provides a measure of the task's parent
796 process and does not include CPU time of child pro‐
797 cesses.
798 TresUsageInAve
800 Tres average usage in by all tasks in job. NOTE: If
801 corresponding TresUsageInMaxTask is -1 the metric is
802 node centric instead of task.
803 TresUsageInMax
805 Tres maximum usage in by all tasks in job. NOTE: If
806 corresponding TresUsageInMaxTask is -1 the metric is
807 node centric instead of task.
808 TresUsageInMaxNode
810 Node for which each maximum TRES usage out occurred.
811 TresUsageInMaxTask
813 Task for which each maximum TRES usage out occurred.
814 TresUsageInMin
816 Tres minimum usage in by all tasks in job. NOTE: If
817 corresponding TresUsageInMinTask is -1 the metric is
818 node centric instead of task.
819 TresUsageInMinNode
821 Node for which each minimum TRES usage out occurred.
822 TresUsageInMinTask
824 Task for which each minimum TRES usage out occurred.
825 TresUsageInTot
827 Tres total usage in by all tasks in job.
828 TresUsageOutAve
830 Tres average usage out by all tasks in job. NOTE: If
831 corresponding TresUsageOutMaxTask is -1 the metric is
832 node centric instead of task.
833 TresUsageOutMax
835 Tres maximum usage out by all tasks in job. NOTE: If
836 corresponding TresUsageOutMaxTask is -1 the metric is
837 node centric instead of task.
838 TresUsageOutMaxNode
840 Node for which each maximum TRES usage out occurred.
841 TresUsageOutMaxTask
843 Task for which each maximum TRES usage out occurred.
844 TresUsageOutMin
846 Tres minimum usage out by all tasks in job.
847 TresUsageOutMinNode
849 Node for which each minimum TRES usage out occurred.
850 TresUsageOutMinTask
852 Task for which each minimum TRES usage out occurred.
853 TresUsageOutTot
855 Tres total usage out by all tasks in job.
856 UID The user identifier of the user who ran the job.
858 User The user name of the user who ran the job.
860 UserCPU The amount of user CPU time used by the job or job
862 step. Format is the same as Elapsed.
863 NOTE: UserCPU provides a measure of the task's parent
865 process and does not include CPU time of child pro‐
866 cesses.
867 WCKey Workload Characterization Key. Arbitrary string
869 for grouping orthogonal accounts together.
870 WCKeyID Reference to the wckey.
872 WorkDir The directory used by the job to execute commands.
874
876 BF BOOT_FAIL Job terminated due to launch failure, typically due
877 to a hardware failure (e.g. unable to boot the node
878 or block and the job can not be requeued).
879 CA CANCELLED Job was explicitly cancelled by the user or system
881 administrator. The job may or may not have been
882 initiated.
883 CD COMPLETED Job has terminated all processes on all nodes with
885 an exit code of zero.
886 DL DEADLINE Job terminated on deadline.
888 F FAILED Job terminated with non-zero exit code or other
890 failure condition.
891 NF NODE_FAIL Job terminated due to failure of one or more allo‐
893 cated nodes.
894 OOM OUT_OF_MEMORY Job experienced out of memory error.
896 PD PENDING Job is awaiting resource allocation.
898 PR PREEMPTED Job terminated due to preemption.
900 R RUNNING Job currently has an allocation.
902 RQ REQUEUED Job was requeued.
904 RS RESIZING Job is about to change size.
906 RV REVOKED Sibling was removed from cluster due to other clus‐
908 ter starting the job.
909 S SUSPENDED Job has an allocation, but execution has been sus‐
911 pended and CPUs have been released for other jobs.
912 TO TIMEOUT Job terminated upon reaching its time limit.
914
916 The options --starttime and --endtime define the time window between
917 which sacct is going to search. For historical and practical reasons
918 their default values (i.e. the default time window) depends on other
919 options: --jobs and --state.
920 Depending on if --jobs and/or --state are specified, the default values
922 of --starttime and --endtime options are:
923 WITHOUT EITHER --jobs NOR --state specified:
926 --starttime defaults to Midnight.
927 --endtime defaults to Now.
928 WITH --jobs AND WITHOUT --state specified:
931 --starttime defaults to Epoch 0.
932 --endtime defaults to Now.
933 WITHOUT --jobs AND WITH --state specified:
936 --starttime defaults to Now.
937 --endtime defaults to --starttime and to Now if --starttime is not
938 specified.
939 WITH BOTH --jobs AND --state specified:
942 --starttime defaults to Epoch 0.
943 --endtime defaults to --starttime or to Now if --starttime is not spec‐
944 ified.
945 NOTE: With -v/--verbose a message about the actual time window in use
948 is shown.
949
952 Executing sacct sends a remote procedure call to slurmdbd. If enough
953 calls from sacct or other Slurm client commands that send remote proce‐
954 dure calls to the slurmdbd daemon come in at once, it can result in a
955 degradation of performance of the slurmdbd daemon, possibly resulting
956 in a denial of service.
957 Do not run sacct or other Slurm client commands that send remote proce‐
959 dure calls to slurmdbd from loops in shell scripts or other programs.
960 Ensure that programs limit calls to sacct to the minimum necessary for
961 the information you are trying to gather.
962
965 Some sacct options may be set via environment variables. These environ‐
966 ment variables, along with their corresponding options, are listed be‐
967 low. (Note: Command line options will always override these settings.)
968 SACCT_FEDERATION Same as --federation
971 SACCT_LOCAL Same as --local
973 SLURM_BITSTR_LEN Specifies the string length to be used for holding
975 a job array's task ID expression. The default value
976 is 64 bytes. A value of 0 will print the full ex‐
977 pression with any length required. Larger values
978 may adversely impact the application performance.
979 SLURM_CONF The location of the Slurm configuration file.
981 SLURM_DEBUG_FLAGS Specify debug flags for sacct to use. See De‐
983 bugFlags in the slurm.conf(5) man page for a full
984 list of flags. The environment variable takes
985 precedence over the setting in the slurm.conf.
986 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
988 value of standard, the default value, generates
989 output in the form
990 "year-month-dateThour:minute:second". A value of
991 relative returns only "hour:minute:second" if the
992 current day. For other dates in the current year
993 it prints the "hour:minute" preceded by "Tomorr"
994 (tomorrow), "Ystday" (yesterday), the name of the
995 day for the coming week (e.g. "Mon", "Tue", etc.),
996 otherwise the date (e.g. "25 Apr"). For other
997 years it returns a date month and year without a
998 time (e.g. "6 Jun 2012"). All of the time stamps
999 use a 24 hour format.
1000 A valid strftime() format can also be specified.
1002 For example, a value of "%a %T" will report the day
1003 of the week and a time stamp (e.g. "Mon 12:34:56").
1004
1006 This example illustrates the default invocation of the sacct command:
1007 # sacct
1009 Jobid Jobname Partition Account AllocCPUS State ExitCode
1010 ---------- ---------- ---------- ---------- ---------- ---------- --------
1011 2 script01 srun acct1 1 RUNNING 0
1012 3 script02 srun acct1 1 RUNNING 0
1013 4 endscript srun acct1 1 RUNNING 0
1014 4.0 srun acct1 1 COMPLETED 0
1015 This example shows the same job accounting information with the brief
1018 option.
1019 # sacct --brief
1021 Jobid State ExitCode
1022 ---------- ---------- --------
1023 2 RUNNING 0
1024 3 RUNNING 0
1025 4 RUNNING 0
1026 4.0 COMPLETED 0
1027 # sacct --allocations
1029 Jobid Jobname Partition Account AllocCPUS State ExitCode
1030 ---------- ---------- ---------- ---------- ------- ---------- --------
1031 3 sja_init andy acct1 1 COMPLETED 0
1032 4 sjaload andy acct1 2 COMPLETED 0
1033 5 sja_scr1 andy acct1 1 COMPLETED 0
1034 6 sja_scr2 andy acct1 18 COMPLETED 2
1035 7 sja_scr3 andy acct1 18 COMPLETED 0
1036 8 sja_scr5 andy acct1 2 COMPLETED 0
1037 9 sja_scr7 andy acct1 90 COMPLETED 1
1038 10 endscript andy acct1 186 COMPLETED 0
1039 This example demonstrates the ability to customize the output of the
1042 sacct command. The fields are displayed in the order designated on the
1043 command line.
1044 # sacct --format=jobid,elapsed,ncpus,ntasks,state
1046 Jobid Elapsed Ncpus Ntasks State
1047 ---------- ---------- ---------- -------- ----------
1048 3 00:01:30 2 1 COMPLETED
1049 3.0 00:01:30 2 1 COMPLETED
1050 4 00:00:00 2 2 COMPLETED
1051 4.0 00:00:01 2 2 COMPLETED
1052 5 00:01:23 2 1 COMPLETED
1053 5.0 00:01:31 2 1 COMPLETED
1054 This example demonstrates the use of the -T (--truncate) option when
1057 used with -S (--starttime) and -E (--endtime). When the -T option is
1058 used, the start time of the job will be the specified -S value if the
1059 job was started before the specified time, otherwise the time will be
1060 the job's start time. The end time will be the specified -E option if
1061 the job ends after the specified time, otherwise it will be the jobs
1062 end time.
1063 Without -T (normal operation) sacct output would be like this.
1065 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1067 JobID Start End State
1068 --------- --------------------- -------------------- ------------
1069 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1070 3 2014-07-03T11:35:21 Unknown RUNNING
1071 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1072 5 2014-07-03T11:41:01 Unknown RUNNING
1073 By adding the -T option the job's start and end times are truncated to
1075 reflect only the time requested. If a job started after the start time
1076 requested or finished before the end time requested those times are not
1077 altered. The -T option is useful when determining exact run times dur‐
1078 ing any given period.
1079 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1081 JobID Start End State
1082 --------- --------------------- -------------------- ------------
1083 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1084 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1085 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1086 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1087 NOTE: If no -s (--state) option is given sacct will display eligible
1090 jobs during the specified period of time, otherwise it will return jobs
1091 that were in the state requested during that period of time.
1092 This example demonstrates the differences running sacct with and with‐
1094 out the --state flag for the same time period. Without the --state op‐
1095 tion, all eligible jobs in that time period are shown.
1096 # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1098 JobID Start End State
1099 ------------ ------------------- ------------------- ----------
1100 2955 11:15:12 11:20:12 COMPLETED
1101 2956 11:20:13 11:25:13 COMPLETED
1102 With the --state=pending option, only job 2956 will be shown because it
1104 had a dependency on 2955 and was still PENDING from 11:20:00 until it
1105 started at 11:21:13. Note that even though we requested PENDING jobs,
1106 the State shows as COMPLETED because that is the current State of the
1107 job.
1108 # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1110 JobID Start End State
1111 ------------ ------------------- ------------------- ----------
1112 2956 11:20:13 11:25:13 COMPLETED
1113
1116 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1117 L.P.
1118 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1119 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1120 Copyright (C) 2010-2022 SchedMD LLC.
1121 This file is part of Slurm, a resource management program. For de‐
1123 tails, see <https://slurm.schedmd.com/>.
1124 Slurm is free software; you can redistribute it and/or modify it under
1126 the terms of the GNU General Public License as published by the Free
1127 Software Foundation; either version 2 of the License, or (at your op‐
1128 tion) any later version.
1129 Slurm is distributed in the hope that it will be useful, but WITHOUT
1131 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1132 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1133 for more details.
1134
1137 /etc/slurm.conf
1138 Entries to this file enable job accounting and designate the
1139 job accounting log file that collects system job accounting.
1140 /var/log/slurm_accounting.log
1142 The default job accounting log file. By default, this file
1143 is set to read and write permission for root only.
1144
1147 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1148August 2022 Slurm Commands sacct(1)