1sacct(1) Slurm Commands sacct(1)
2
3
4
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
9
11 sacct [OPTIONS...]
12
13
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18
19 The sacct command displays job accounting data stored in the job
20 accounting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24
25 For the root user, the sacct command displays job accounting data for
26 all users, although there are options to filter the output to report
27 only the jobs from a specified user or group.
28
29 For the non-root user, the sacct command limits the display of job
30 accounting data to jobs that were launched with their own user identi‐
31 fier (UID) by default. Data for other users can be displayed with the
32 --allusers, --user, or --uid options.
33
34 Elapsed time fields are presented as [days-]hours:minutes:sec‐
35 onds[.microseconds]. Only 'CPU' fields will ever have microseconds.
36
37 The default input file is the file named in the AccountingStorageLoc
38 parameter in slurm.conf.
39
40 Note: If designated, the slurmdbd.conf option PrivateData may further
41 restrict the accounting data visible to users which are not
42 SlurmUser, root, or a user with AdminLevel=Admin. See the slur‐
43 mdbd.conf man page for additional details on restricting access
44 to accounting data.
45
46 Note: The contents of Slurm's database are maintained in lower case.
47 This may result in some sacct output differing from that of
48 other Slurm commands.
49
50 Note: Much of the data reported by sacct has been generated by the
51 wait3() and getrusage() system calls. Some systems gather and
52 report incomplete information for these calls; sacct reports
53 values of 0 for this missing data. See your systems getrusage
54 [4m(3) man page for information about which data are actually
55 available on your system.
56
57
59 -a, --allusers
60 Displays all users jobs when run by user root or if Private‐
61 Data is not configured to jobs. Otherwise display the cur‐
62 rent user's jobs
63
64 -A account_list , --accounts=account_list
65 Displays jobs when a comma separated list of accounts are
66 given as the argument.
67
68 -b, --brief
69 Displays a brief listing, which includes the following data:
70
71 jobid
72
73 status
74
75 exitcode
76
77 -c, --completion
78 Use job completion data instead of job accounting. The Job‐
79 CompType parameter in the slurm.conf file must be defined to
80 a non-none option. Does not support federated cluster infor‐
81 mation (local data only).
82
83 -C constraint_list, --constraints=constraint_list
84 Comma separated list to filter jobs based on what con‐
85 straints/features the job requested. Multiple options will
86 be treated as 'and' not 'or', so the job would need all con‐
87 straints specified to be returned not one or the other.
88
89
90 --delimiter=characters
91 ASCII characters used to separate the fields when specifying
92 the -p or -P options. The default delimiter is a '|'. This
93 option is ignored if -p or -P options are not specified.
94
95
96 -D, --duplicates
97 If Slurm job ids are reset, some job numbers will probably
98 appear more than once in the accounting log file but refer to
99 different jobs. Such jobs can be distinguished by the "sub‐
100 mit" time stamp in the data records.
101
102
103 When data for specific jobs are requested with the --jobs
104 option, sacct returns the most recent job with that number.
105 This behavior can be overridden by specifying --duplicates,
106 in which case all records that match the selection criteria
107 will be returned.
108
109
110 NOTE: Revoked federated sibling jobs are hidden unless the
111 --duplicates option is specified.
112
113
114 -e, --helpformat
115
116 Print a list of fields that can be specified with the --for‐
117 mat option.
118
119
120 Fields available:
121
122 Account AdminComment AllocCPUS AllocNodes
123 AllocTRES AssocID AveCPU AveCPUFreq
124 AveDiskRead AveDiskWrite AvePages AveRSS
125 AveVMSize BlockID Cluster Comment
126 Constraints ConsumedEnergy ConsumedEnergyRaw CPUTime
127 CPUTimeRAW DBIndex DerivedExitCode Elapsed
128 ElapsedRaw Eligible End ExitCode
129 Flags GID Group JobID
130 JobIDRaw JobName Layout MaxDiskRead
131 MaxDiskReadNode MaxDiskReadTask MaxDiskWrite MaxDiskWriteNode
132 MaxDiskWriteTask MaxPages MaxPagesNode MaxPagesTask
133 MaxRSS MaxRSSNode MaxRSSTask MaxVMSize
134 MaxVMSizeNode MaxVMSizeTask McsLabel MinCPU
135 MinCPUNode MinCPUTask NCPUS NNodes
136 NodeList NTasks Priority Partition
137 QOS QOSRAW Reason ReqCPUFreq
138 ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ReqCPUS
139 ReqMem ReqNodes ReqTRES Reservation
140 ReservationId Reserved ResvCPU ResvCPURAW
141 Start State Submit Suspended
142 SystemCPU SystemComment Timelimit TimelimitRaw
143 TotalCPU TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode
144 TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask
145 TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode
146 TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask
147 TRESUsageOutTot UID User UserCPU
148 WCKey WCKeyID WorkDir
149
150 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
151 SUsageIn[Ave|Tot]. They represent the average/total of the
152 highest watermarks over all ranks in the step. When using
153 sstat they represent the average/total at the moment the com‐
154 mand was run.
155
156 NOTE: TRESUsage*Min* values represent the lowest highwater
157 mark in the step.
158
159 The section titled "Job Accounting Fields" describes these
160 fields.
161
162
163 -E end_time, --endtime=end_time
164
165 Select jobs in any state before the specified time. If
166 states are given with the -s option return jobs in this state
167 before this period. See the DEFAULT TIME WINDOW for more
168 details.
169
170 Valid time formats are:
171 HH:MM[:SS][AM|PM]
172 MMDD[YY][-HH:MM[:SS]]
173 MM.DD[.YY][-HH:MM[:SS]]
174 MM/DD[/YY][-HH:MM[:SS]]
175 YYYY-MM-DD[THH:MM[:SS]]
176 today, midnight, noon, fika (3 PM), teatime (4 PM)
177 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
178
179 --federation
180 Show jobs from the federation if a member of one.
181
182
183 -f file, --file=file
184 Causes the sacct command to read job accounting data from the
185 named file instead of the current Slurm job accounting log
186 file. Only applicable when running the jobcomp/filetxt plug‐
187 in.
188
189
190 -F flag_list, --flags=flag_list
191 Comma separated list to filter jobs based on what various
192 ways the jobs were handled. Current flags are SchedSubmit,
193 SchedMain, SchedBackfill. These particular options describe
194 the scheduler that started the job.
195
196
197 -g gid_list, --gid=gid_list --group=group_list
198 Displays the statistics only for the jobs started with the
199 GID or the GROUP specified by the gid_list or the group_list
200 operand, which is a comma-separated list. Space characters
201 are not allowed. Default is no restrictions.
202
203
204 -h, --help
205 Displays a general help message.
206
207
208 -i, --nnodes=N
209 Return jobs which ran on this many nodes (N = min[-max])
210
211
212 -j job(.step) , --jobs=job(.step)
213 Displays information about the specified job(.step) or list
214 of job(.step)s.
215
216 The job(.step) parameter is a comma-separated list of jobs.
217 Space characters are not permitted in this list. NOTE: A
218 step id of 'batch' will display the information about the
219 batch step.
220
221 By default sacct shows only jobs with Eligible time, but with
222 this option the non-eligible will be also shown. NOTE: If
223 --state is also specified, as non-eligible are not PD, then
224 non-eligible jobs will not be displayed. See the DEFAULT
225 TIME WINDOW for details about how this option changes the
226 default -S and -E options.
227
228
229 -k, --timelimit-min
230 Only send data about jobs with this timelimit. If used with
231 timelimit_max this will be the minimum timelimit of the
232 range. Default is no restriction.
233
234
235 -K, --timelimit-max
236 Ignored by itself, but if timelimit_min is set this will be
237 the maximum timelimit of the range. Default is no restric‐
238 tion.
239
240
241 --local Show only jobs local to this cluster. Ignore other clusters
242 in this federation (if any). Overrides --federation.
243
244
245 -l, --long
246 Equivalent to specifying:
247
248 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
249 sizenode, maxvmsizetask,avevm‐
250 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
251 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
252 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
253 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
254 gov,reqmem,consumeden‐
255 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
256 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
257 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
258 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
259 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
260 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
261 outave,tresusageouttot
262
263
264 -L, --allclusters
265 Display jobs ran on all clusters. By default, only jobs ran
266 on the cluster from where sacct is called are displayed.
267
268
269 -M cluster_list, --clusters=cluster_list
270 Displays the statistics only for the jobs started on the
271 clusters specified by the cluster_list operand, which is a
272 comma-separated list of clusters. Space characters are not
273 allowed in the cluster_list. A value of 'all' will query to
274 run on all clusters. The default is current cluster you are
275 executing the sacct command on or all clusters in the federa‐
276 tion when executed on a federated cluster. This option
277 implicitly sets the --local option.
278
279
280 -n, --noheader
281 No heading will be added to the output. The default action is
282 to display a header.
283
284 --noconvert
285 Don't convert units from their original type (e.g. 2048M
286 won't be converted to 2G).
287
288 -N node_list, --nodelist=node_list
289 Display jobs that ran on any of these node(s). node_list can
290 be a ranged string.
291
292 --name=jobname_list
293 Display jobs that have any of these name(s).
294
295 -o, --format
296 Comma separated list of fields. (use "--helpformat" for a
297 list of available fields).
298
299 NOTE: When using the format option for listing various fields
300 you can put a %NUMBER afterwards to specify how many charac‐
301 ters should be printed.
302
303 e.g. format=name%30 will print 30 characters of field name
304 right justified. A %-30 will print 30 characters left justi‐
305 fied.
306
307 When set, the SACCT_FORMAT environment variable will override
308 the default format. For example:
309
310 SACCT_FORMAT="jobid,user,account,cluster"
311
312
313 -p, --parsable
314 output will be '|' delimited with a '|' at the end
315
316
317 -P, --parsable2
318 output will be '|' delimited without a '|' at the end
319
320
321 -q, --qos Only send data about jobs using these qos. Default is all.
322
323
324 -r, --partition
325
326 Comma separated list of partitions to select jobs and job
327 steps from. The default is all partitions.
328
329
330 -R reason_list, --reason=reason_list
331 Comma separated list to filter jobs based on what reason the
332 job wasn't scheduled outside resources/priority.
333
334
335 -s state_list, --state=state_list
336 Selects jobs based on their state during the time period
337 given. Unless otherwise specified, the start and end time
338 will be the current time when the --state option is specified
339 and only currently running jobs can be displayed. A start
340 and/or end time must be specified to view information about
341 jobs not currently running. See the JOB STATE CODES section
342 below for a list of state designators. Multiple state names
343 may be specified using comma separators. Either the short or
344 long form of the state name may be used (e.g. CA or CAN‐
345 CELLED) and the name is case insensitive (i.e. ca and CA both
346 work).
347
348 NOTE: Note for a job to be selected in the PENDING state it
349 must have "EligibleTime" in the requested time interval or
350 different from "Unknown". The "EligibleTime" is displayed by
351 the "scontrol show job" command. For example jobs submitted
352 with the "--hold" option will have "EligibleTime=Unknown" as
353 they are pending indefinitely.
354
355 NOTE: When specifying states and no start time is given the
356 default start time is 'now'. This is only when -j is not
357 used. If -j is used the start time will default to 'Epoch'.
358 In both cases if no end time is given it will default to
359 'now'. See the DEFAULT TIME WINDOW for more details.
360
361
362 -S, --starttime
363 Select jobs in any state after the specified time. Default is
364 00:00:00 of the current day, unless the '-s' or '-j' options
365 are used. If the '-s' option is used, then the default is
366 'now'. If states are given with the '-s' option then only
367 jobs in this state at this time will be returned. If the '-j'
368 option is used, then the default time is Unix Epoch 0. See
369 the DEFAULT TIME WINDOW for more details.
370
371 Valid time formats are:
372 HH:MM[:SS][AM|PM]
373 MMDD[YY][-HH:MM[:SS]]
374 MM.DD[.YY][-HH:MM[:SS]]
375 MM/DD[/YY][-HH:MM[:SS]]
376 YYYY-MM-DD[THH:MM[:SS]]
377 today, midnight, noon, fika (3 PM), teatime (4 PM)
378 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
379
380
381 -T, --truncate
382 Truncate time. So if a job started before --starttime the
383 start time would be truncated to --starttime. The same for
384 end time and --endtime.
385
386
387 -u uid_list, --uid=uid_list, --user=user_list
388 Use this comma separated list of uids or user names to select
389 jobs to display. By default, the running user's uid is used.
390
391
392 --use-local-uid
393 When displaying UID, sacct uses the UID stored in Slurm's
394 accounting database by default. Use this command to make
395 Slurm use a system call to get the UID from the username.
396 This option may be useful in an environment with multiple
397 clusters and one database where the UID's aren't the same on
398 all clusters.
399
400
401 --units=[KMGTP]
402 Display values in specified unit type. Takes precedence over
403 --noconvert option.
404
405
406 --usage Display a command usage summary.
407
408
409 -v, --verbose
410 Primarily for debugging purposes, report the state of various
411 variables during processing.
412
413
414 -V, --version
415 Print version.
416
417
418 --whole-hetjob=[yes|no]
419 When querying heterogeneous jobs, Slurm by default retrieves
420 the information about all the components of the job if the
421 het_job_id (leader id) is selected. If a non-leader hetero‐
422 geneous job component id is selected only that component is
423 retrieved by default. This behavior can be changed by using
424 this option. If set to 'yes' or no value is set, then infor‐
425 mation about all the components will be retrieved no matter
426 which component is selected in the job filter. Otherwise, if
427 set to 'no' then only the selected heterogeneous job compo‐
428 nents will be retrieved, even when selecting the leader.
429
430
431 -W wckey_list, --wckeys=wckey_list
432 Displays the statistics only for the jobs started on the wck‐
433 eys specified by the wckey_list operand, which is a
434 comma-separated list of wckey names. Space characters are
435 not allowed in the wckey_list. Default is all wckeys.
436
437
438 -x associd_list, --associations=assoc_list
439 Displays the statistics only for the jobs running under the
440 association ids specified by the assoc_list operand, which is
441 a comma-separated list of association ids. Space characters
442 are not allowed in the assoc_list. Default is all associa‐
443 tions.
444
445
446 -X, --allocations
447 Only show statistics relevant to the job allocation itself,
448 not taking steps into consideration.
449
450 NOTE: Without including steps, utilization statistics for job
451 allocation(s) will be reported as zero.
452
453
454 Job Accounting Fields
455 The following describes each job accounting field:
456
457 ALL Print all fields listed below.
458
459
460 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
461
462
463 AllocNodes
464 Number of nodes allocated to the job/step. 0 if the
465 job is pending.
466
467
468 AllocTres Trackable resources. These are the resources allocated
469 to the job/step after the job started running. For
470 pending jobs this should be blank. For more details
471 see AccountingStorageTRES in slurm.conf.
472
473 NOTE: When a generic resource is configured with the
474 no_consume flag, the allocation will be printed with a
475 zero.
476
477
478 Account Account the job ran under.
479
480
481 AssocID Reference to the association of user, account and
482 cluster.
483
484
485 AveCPU Average (system + user) CPU time of all tasks in job.
486
487
488 AveCPUFreq
489 Average weighted CPU frequency of all tasks in job, in
490 kHz.
491
492
493 AveDiskRead
494 Average number of bytes read by all tasks in job.
495
496
497 AveDiskWrite
498 Average number of bytes written by all tasks in job.
499
500
501 AvePages Average number of page faults of all tasks in job.
502
503
504 AveRSS Average resident set size of all tasks in job.
505
506
507 AveVMSize Average Virtual Memory size of all tasks in job.
508
509
510 Cluster Cluster name.
511
512
513 Comment The job's comment string when the AccountingStoreJob‐
514 Comment parameter in the slurm.conf file is set (or
515 defaults) to YES. The Comment string can be modified
516 by invoking sacctmgr modify job or the specialized
517 sjobexitmod command.
518
519
520 ConsumedEnergy
521 Total energy consumed by all tasks in job, in joules.
522 Note: Only in case of exclusive job allocation this
523 value reflects the jobs' real energy consumption.
524
525
526 CPUTime Time used (Elapsed time * CPU count) by a job or step
527 in HH:MM:SS format.
528
529
530 CPUTimeRAW
531 Time used (Elapsed time * CPU count) by a job or step
532 in cpu-seconds.
533
534
535 DerivedExitCode
536 The highest exit code returned by the job's job steps
537 (srun invocations). Following the colon is the signal
538 that caused the process to terminate if it was termi‐
539 nated by a signal. The DerivedExitCode can be modi‐
540 fied by invoking sacctmgr modify job or the special‐
541 ized sjobexitmod command.
542
543
544 Elapsed The jobs elapsed time.
545
546 The format of this fields output is as follows:
547 [DD-[HH:]]MM:SS
548
549 as defined by the following:
550
551 DD days
552
553 hh hours
554
555 mm minutes
556
557 ss seconds
558
559
560 Eligible When the job became eligible to run in the same format
561 as End.
562
563
564 End Termination time of the job. Format output is,
565 YYYY-MM-DDTHH:MM:SS, unless changed through the
566 SLURM_TIME_FORMAT environment variable.
567
568
569 ExitCode The exit code returned by the job script or salloc,
570 typically as set by the exit() function. Following
571 the colon is the signal that caused the process to
572 terminate if it was terminated by a signal.
573
574
575 GID The group identifier of the user who ran the job.
576
577
578 Group The group name of the user who ran the job.
579
580
581 JobID The identification number of the job or job step.
582
583 Regular jobs are in the form:
584
585 JobID[.JobStep]
586
587 Array jobs are in the form:
588
589 ArrayJobID_ArrayTaskID
590
591 Heterogeneous jobs are in the form:
592
593 HetJobID+HetJobOffset.
594
595
596 JobIDRaw The identification number of the job or job step.
597 Prints the JobID in the form JobID[.JobStep].
598
599
600 JobName The name of the job or job step. The slurm_account‐
601 ing.log file is a space delimited file. Because of
602 this if a space is used in the jobname an underscore
603 is substituted for the space before the record is
604 written to the accounting file. So when the jobname is
605 displayed by sacct the jobname that had a space in it
606 will now have an underscore in place of the space.
607
608
609 Layout What the layout of a step was when it was running.
610 This can be used to give you an idea of which node ran
611 which rank in your job.
612
613
614 MaxDiskRead
615 Maximum number of bytes read by all tasks in job.
616
617
618 MaxDiskReadNode
619 The node on which the maxdiskread occurred.
620
621
622 MaxDiskReadTask
623 The task ID where the maxdiskread occurred.
624
625
626 MaxDiskWrite
627 Maximum number of bytes written by all tasks in job.
628
629
630 MaxDiskWriteNode
631 The node on which the maxdiskwrite occurred.
632
633
634 MaxDiskWriteTask
635 The task ID where the maxdiskwrite occurred.
636
637
638 MaxPages Maximum number of page faults of all tasks in job.
639
640
641 MaxPagesNode
642 The node on which the maxpages occurred.
643
644
645 MaxPagesTask
646 The task ID where the maxpages occurred.
647
648
649 MaxRSS Maximum resident set size of all tasks in job.
650
651
652 MaxRSSNode
653 The node on which the maxrss occurred.
654
655
656 MaxRSSTask
657 The task ID where the maxrss occurred.
658
659
660 MaxVMSize Maximum Virtual Memory size of all tasks in job.
661
662
663 MaxVMSizeNode
664 The node on which the maxvmsize occurred.
665
666
667 MaxVMSizeTask
668 The task ID where the maxvmsize occurred.
669
670
671 MinCPU Minimum (system + user) CPU time of all tasks in job.
672
673
674 MinCPUNode
675 The node on which the mincpu occurred.
676
677
678 MinCPUTask
679 The task ID where the mincpu occurred.
680
681
682 NCPUS Total number of CPUs allocated to the job. Equivalent
683 to AllocCPUS.
684
685
686 NodeList List of nodes in job/step.
687
688
689 NNodes Number of nodes in a job or step. If the job is run‐
690 ning, or ran, this count will be the number allocated,
691 else the number will be the number requested.
692
693
694 NTasks Total number of tasks in a job or step.
695
696
697 Priority Slurm priority.
698
699
700 Partition Identifies the partition on which the job ran.
701
702
703 QOS Name of Quality of Service.
704
705
706 QOSRAW Numeric id of Quality of Service.
707
708
709 ReqCPUFreq
710 Requested CPU frequency for the step, in kHz. Note:
711 This value applies only to a job step. No value is
712 reported for the job.
713
714
715 ReqCPUFreqGov
716 Requested CPU frequency governor for the step, in kHz.
717 Note: This value applies only to a job step. No value
718 is reported for the job.
719
720
721 ReqCPUFreqMax
722 Maximum requested CPU frequency for the step, in kHz.
723 Note: This value applies only to a job step. No value
724 is reported for the job.
725
726
727 ReqCPUFreqMin
728 Minimum requested CPU frequency for the step, in kHz.
729 Note: This value applies only to a job step. No value
730 is reported for the job.
731
732
733 ReqCPUS Number of requested CPUs.
734
735
736 ReqMem Minimum required memory for the job, in MB. A 'c' at
737 the end of number represents Memory Per CPU, a 'n'
738 represents Memory Per Node. Note: This value is only
739 from the job allocation, not the step.
740
741
742 ReqNodes Requested minimum Node count for the job/step.
743
744
745 ReqTres Trackable resources. These are the minimum resource
746 counts requested by the job/step at submission time.
747 For more details see AccountingStorageTRES in
748 slurm.conf.
749
750
751 Reservation
752 Reservation Name.
753
754
755 ReservationId
756 Reservation Id.
757
758
759 Reserved How much wall clock time was used as reserved time for
760 this job. This is derived from how long a job was
761 waiting from eligible time to when it actually
762 started. Format is the same as Elapsed.
763
764
765 ResvCPU How many CPU seconds were used as reserved time for
766 this job. Format is the same as Elapsed.
767
768
769 ResvCPURAW
770 How many CPU seconds were used as reserved time for
771 this job. Format is in processor seconds.
772
773
774 Start Initiation time of the job in the same format as End.
775
776
777 State Displays the job status, or state. See the JOB STATE
778 CODES section below for a list of possible states.
779
780 If more information is available on the job state than
781 will fit into the current field width (for example,
782 the uid that CANCELLED a job) the state will be fol‐
783 lowed by a "+". You can increase the size of the dis‐
784 played state using the "%NUMBER" format modifier
785 described earlier.
786
787 NOTE: The RUNNING state will return suspended jobs as
788 well. In order to print suspended jobs you must
789 request SUSPENDED at a different call from RUNNING.
790
791 NOTE: The RUNNING state will return any jobs completed
792 (cancelled or otherwise) in the time period requested
793 as the job was also RUNNING during that time. If you
794 are only looking for jobs that finished, please choose
795 the appropriate state(s) without the RUNNING state.
796
797
798 Submit The time the job was submitted in the same format as
799 End.
800
801 NOTE: If a job is requeued, the submit time is reset.
802 To obtain the original submit time it is necessary to
803 use the -D or --duplicate option to display all dupli‐
804 cate entries for a job.
805
806
807 Suspended The amount of time a job or job step was suspended.
808 Format is the same as Elapsed.
809
810
811 SystemCPU The amount of system CPU time used by the job or job
812 step. Format is the same as Elapsed.
813
814 NOTE: SystemCPU provides a measure of the task's par‐
815 ent process and does not include CPU time of child
816 processes.
817
818
819 Timelimit What the timelimit was/is for the job. Format is the
820 same as Elapsed.
821
822
823 TimelimitRaw
824 What the timelimit was/is for the job. Format is in
825 number of minutes.
826
827
828 TotalCPU The sum of the SystemCPU and UserCPU time used by the
829 job or job step. The total CPU time of the job may
830 exceed the job's elapsed time for jobs that include
831 multiple job steps. Format is the same as Elapsed.
832
833 NOTE: TotalCPU provides a measure of the task's parent
834 process and does not include CPU time of child pro‐
835 cesses.
836
837
838 TresUsageInAve
839 Tres average usage in by all tasks in job. NOTE: If
840 corresponding TresUsageInMaxTask is -1 the metric is
841 node centric instead of task.
842
843
844 TresUsageInMax
845 Tres maximum usage in by all tasks in job. NOTE: If
846 corresponding TresUsageInMaxTask is -1 the metric is
847 node centric instead of task.
848
849
850 TresUsageInMaxNode
851 Node for which each maximum TRES usage out occurred.
852
853
854 TresUsageInMaxTask
855 Task for which each maximum TRES usage out occurred.
856
857
858 TresUsageInMin
859 Tres minimum usage in by all tasks in job. NOTE: If
860 corresponding TresUsageInMinTask is -1 the metric is
861 node centric instead of task.
862
863
864 TresUsageInMinNode
865 Node for which each minimum TRES usage out occurred.
866
867
868 TresUsageInMinTask
869 Task for which each minimum TRES usage out occurred.
870
871
872 TresUsageInTot
873 Tres total usage in by all tasks in job.
874
875
876 TresUsageOutAve
877 Tres average usage out by all tasks in job. NOTE: If
878 corresponding TresUsageOutMaxTask is -1 the metric is
879 node centric instead of task.
880
881
882 TresUsageOutMax
883 Tres maximum usage out by all tasks in job. NOTE: If
884 corresponding TresUsageOutMaxTask is -1 the metric is
885 node centric instead of task.
886
887
888 TresUsageOutMaxNode
889 Node for which each maximum TRES usage out occurred.
890
891
892 TresUsageOutMaxTask
893 Task for which each maximum TRES usage out occurred.
894
895
896 TresUsageOutTot
897 Tres total usage out by all tasks in job.
898
899
900 UID The user identifier of the user who ran the job.
901
902
903 User The user name of the user who ran the job.
904
905
906 UserCPU The amount of user CPU time used by the job or job
907 step. Format is the same as Elapsed.
908
909 NOTE: UserCPU provides a measure of the task's parent
910 process and does not include CPU time of child pro‐
911 cesses.
912
913
914 WCKey Workload Characterization Key. Arbitrary string
915 for grouping orthogonal accounts together.
916
917
918 WCKeyID Reference to the wckey.
919
920
921
923 BF BOOT_FAIL Job terminated due to launch failure, typically due
924 to a hardware failure (e.g. unable to boot the node
925 or block and the job can not be requeued).
926
927 CA CANCELLED Job was explicitly cancelled by the user or system
928 administrator. The job may or may not have been
929 initiated.
930
931 CD COMPLETED Job has terminated all processes on all nodes with
932 an exit code of zero.
933
934 DL DEADLINE Job terminated on deadline.
935
936 F FAILED Job terminated with non-zero exit code or other
937 failure condition.
938
939 NF NODE_FAIL Job terminated due to failure of one or more allo‐
940 cated nodes.
941
942 OOM OUT_OF_MEMORY Job experienced out of memory error.
943
944 PD PENDING Job is awaiting resource allocation.
945
946 PR PREEMPTED Job terminated due to preemption.
947
948 R RUNNING Job currently has an allocation.
949
950 RQ REQUEUED Job was requeued.
951
952 RS RESIZING Job is about to change size.
953
954 RV REVOKED Sibling was removed from cluster due to other clus‐
955 ter starting the job.
956
957 S SUSPENDED Job has an allocation, but execution has been sus‐
958 pended and CPUs have been released for other jobs.
959
960 TO TIMEOUT Job terminated upon reaching its time limit.
961
962
964 The options --starttime and --endtime define the time window between
965 which sacct is going to search. For historical and practical reasons
966 their default values (i.e. the default time window) depends on other
967 options: --jobs and --state.
968
969 Depending on if --jobs and/or --state are specified, the default values
970 of --starttime and --endtime options are:
971
972
973 WITHOUT EITHER --jobs NOR --state specified:
974
975 ---starttime
976 Defaults to Midnight.
977
978 ---endtime
979 Defaults to Now.
980
981
982 WITH --jobs AND WITHOUT --state specified:
983
984 ---starttime
985 Defaults to Epoch 0.
986
987 ---endtime
988 Defaults to Now.
989
990
991 WITHOUT --jobs AND WITH --state specified:
992
993 ---starttime
994 Defaults to Now.
995
996 ---endtime
997 Defaults to --starttime and to Now if --starttime is not
998 specified.
999
1000
1001 WITH BOTH --jobs AND --state specified:
1002
1003 ---starttime
1004 Defaults to Epoch 0.
1005
1006 ---endtime
1007 Defaults to --starttime or to Now if --starttime is not
1008 specified.
1009
1010
1011 NOTE: With -v/--verbose a message about the actual time window in use
1012 is shown.
1013
1014
1016 Executing sacct sends a remote procedure call to slurmdbd. If enough
1017 calls from sacct or other Slurm client commands that send remote proce‐
1018 dure calls to the slurmdbd daemon come in at once, it can result in a
1019 degradation of performance of the slurmdbd daemon, possibly resulting
1020 in a denial of service.
1021
1022 Do not run sacct or other Slurm client commands that send remote proce‐
1023 dure calls to slurmdbd from loops in shell scripts or other programs.
1024 Ensure that programs limit calls to sacct to the minimum necessary for
1025 the information you are trying to gather.
1026
1027
1029 Some sacct options may be set via environment variables. These environ‐
1030 ment variables, along with their corresponding options, are listed
1031 below. (Note: Commandline options will always override these settings.)
1032
1033 SACCT_FEDERATION Same as --federation
1034
1035 SACCT_LOCAL Same as --local
1036
1037 SLURM_CONF The location of the Slurm configuration file.
1038
1039 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
1040 value of standard, the default value, generates
1041 output in the form
1042 "year-month-dateThour:minute:second". A value of
1043 relative returns only "hour:minute:second" if the
1044 current day. For other dates in the current year
1045 it prints the "hour:minute" preceded by "Tomorr"
1046 (tomorrow), "Ystday" (yesterday), the name of the
1047 day for the coming week (e.g. "Mon", "Tue", etc.),
1048 otherwise the date (e.g. "25 Apr"). For other
1049 years it returns a date month and year without a
1050 time (e.g. "6 Jun 2012"). All of the time stamps
1051 use a 24 hour format.
1052
1053 A valid strftime() format can also be specified.
1054 For example, a value of "%a %T" will report the day
1055 of the week and a time stamp (e.g. "Mon 12:34:56").
1056
1057
1059 This example illustrates the default invocation of the sacct command:
1060
1061 # sacct
1062 Jobid Jobname Partition Account AllocCPUS State ExitCode
1063 ---------- ---------- ---------- ---------- ---------- ---------- --------
1064 2 script01 srun acct1 1 RUNNING 0
1065 3 script02 srun acct1 1 RUNNING 0
1066 4 endscript srun acct1 1 RUNNING 0
1067 4.0 srun acct1 1 COMPLETED 0
1068
1069
1070 This example shows the same job accounting information with the brief
1071 option.
1072
1073 # sacct --brief
1074 Jobid State ExitCode
1075 ---------- ---------- --------
1076 2 RUNNING 0
1077 3 RUNNING 0
1078 4 RUNNING 0
1079 4.0 COMPLETED 0
1080
1081 # sacct --allocations
1082 Jobid Jobname Partition Account AllocCPUS State ExitCode
1083 ---------- ---------- ---------- ---------- ------- ---------- --------
1084 3 sja_init andy acct1 1 COMPLETED 0
1085 4 sjaload andy acct1 2 COMPLETED 0
1086 5 sja_scr1 andy acct1 1 COMPLETED 0
1087 6 sja_scr2 andy acct1 18 COMPLETED 2
1088 7 sja_scr3 andy acct1 18 COMPLETED 0
1089 8 sja_scr5 andy acct1 2 COMPLETED 0
1090 9 sja_scr7 andy acct1 90 COMPLETED 1
1091 10 endscript andy acct1 186 COMPLETED 0
1092
1093
1094 This example demonstrates the ability to customize the output of the
1095 sacct command. The fields are displayed in the order designated on the
1096 command line.
1097
1098 # sacct --format=jobid,elapsed,ncpus,ntasks,state
1099 Jobid Elapsed Ncpus Ntasks State
1100 ---------- ---------- ---------- -------- ----------
1101 3 00:01:30 2 1 COMPLETED
1102 3.0 00:01:30 2 1 COMPLETED
1103 4 00:00:00 2 2 COMPLETED
1104 4.0 00:00:01 2 2 COMPLETED
1105 5 00:01:23 2 1 COMPLETED
1106 5.0 00:01:31 2 1 COMPLETED
1107
1108
1109 This example demonstrates the use of the -T (--truncate) option when
1110 used with -S (--starttime) and -E (--endtime). When the -T option is
1111 used, the start time of the job will be the specified -S value if the
1112 job was started before the specified time, otherwise the time will be
1113 the job's start time. The end time will be the specified -E option if
1114 the job ends after the specified time, otherwise it will be the jobs
1115 end time.
1116
1117 Without -T (normal operation) sacct output would be like this.
1118
1119 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1120 JobID Start End State
1121 --------- --------------------- -------------------- ------------
1122 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1123 3 2014-07-03T11:35:21 Unknown RUNNING
1124 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1125 5 2014-07-03T11:41:01 Unknown RUNNING
1126
1127 By adding the -T option the job's start and end times are truncated to
1128 reflect only the time requested. If a job started after the start time
1129 requested or finished before the end time requested those times are not
1130 altered. The -T option is useful when determining exact run times dur‐
1131 ing any given period.
1132
1133 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1134 JobID Start End State
1135 --------- --------------------- -------------------- ------------
1136 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1137 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1138 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1139 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1140
1141
1142 NOTE: If no -s (--state) option is given sacct will display eligible
1143 jobs during the specified period of time, otherwise it will return jobs
1144 that were in the state requested during that period of time.
1145
1146 This example demonstrates the differences running sacct with and with‐
1147 out the --state flag for the same time period. Without the --state
1148 option, all eligible jobs in that time period are shown.
1149
1150 # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1151 JobID Start End State
1152 ------------ ------------------- ------------------- ----------
1153 2955 11:15:12 11:20:12 COMPLETED
1154 2956 11:20:13 11:25:13 COMPLETED
1155
1156 With the --state=pending option, only job 2956 will be shown because it
1157 had a dependency on 2955 and was still PENDING from 11:20:00 until it
1158 started at 11:21:13. Note that even though we requested PENDING jobs,
1159 the State shows as COMPLETED because that is the current State of the
1160 job.
1161
1162 # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1163 JobID Start End State
1164 ------------ ------------------- ------------------- ----------
1165 2956 11:20:13 11:25:13 COMPLETED
1166
1168 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1169 L.P.
1170 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1171 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1172 Copyright (C) 2010-2014 SchedMD LLC.
1173
1174 This file is part of Slurm, a resource management program. For
1175 details, see <https://slurm.schedmd.com/>.
1176
1177 Slurm is free software; you can redistribute it and/or modify it under
1178 the terms of the GNU General Public License as published by the Free
1179 Software Foundation; either version 2 of the License, or (at your
1180 option) any later version.
1181
1182 Slurm is distributed in the hope that it will be useful, but WITHOUT
1183 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1184 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1185 for more details.
1186
1188 /etc/slurm.conf
1189 Entries to this file enable job accounting and designate the
1190 job accounting log file that collects system job accounting.
1191
1192 /var/log/slurm_accounting.log
1193 The default job accounting log file. By default, this file
1194 is set to read and write permission for root only.
1195
1197 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1198
1199
1200
1201October 2020 Slurm Commands sacct(1)