1sacct(1) Slurm Commands sacct(1)
2
3
4
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
9
11 sacct [OPTIONS...]
12
13
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18
19 The sacct command displays job accounting data stored in the job ac‐
20 counting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24
25 For the root user, the sacct command displays job accounting data for
26 all users, although there are options to filter the output to report
27 only the jobs from a specified user or group.
28
29 For the non-root user, the sacct command limits the display of job ac‐
30 counting data to jobs that were launched with their own user identifier
31 (UID) by default. Data for other users can be displayed with the --al‐
32 lusers, --user, or --uid options.
33
34 Elapsed time fields are presented as [days-]hours:minutes:seconds[.mi‐
35 croseconds]. Only 'CPU' fields will ever have microseconds.
36
37 The default input file is the file named in the AccountingStorageLoc
38 parameter in slurm.conf.
39
40 NOTE: If designated, the slurmdbd.conf option PrivateData may further
41 restrict the accounting data visible to users which are not SlurmUser,
42 root, or a user with AdminLevel=Admin. See the slurmdbd.conf man page
43 for additional details on restricting access to accounting data.
44
45 NOTE: The contents of Slurm's database are maintained in lower case.
46 This may result in some sacct output differing from that of other Slurm
47 commands.
48
49 NOTE: Much of the data reported by sacct has been generated by the
50 wait3() and getrusage() system calls. Some systems gather and report
51 incomplete information for these calls; sacct reports values of 0 for
52 this missing data. See your systems getrusage [4m(3) man page for informa‐
53 tion about which data are actually available on your system.
54
55
57 -a, --allusers
58 Displays all users' jobs when run by user root or if Private‐
59 Data is not configured to jobs. Otherwise display the cur‐
60 rent user's jobs
61
62 -A account_list , --accounts=account_list
63 Displays jobs when a comma separated list of accounts are
64 given as the argument.
65
66 -b, --brief
67 Displays a brief listing consisting of JobID, State, and Ex‐
68 itCode.
69
70 -c, --completion
71 Use job completion data instead of job accounting. The Job‐
72 CompType parameter in the slurm.conf file must be defined to
73 a non-none option. Does not support federated cluster infor‐
74 mation (local data only).
75
76 -C constraint_list, --constraints=constraint_list
77 Comma separated list to filter jobs based on what con‐
78 straints/features the job requested. Multiple options will
79 be treated as 'and' not 'or', so the job would need all con‐
80 straints specified to be returned not one or the other.
81
82
83 --delimiter=characters
84 ASCII characters used to separate the fields when specifying
85 the -p or -P options. The default delimiter is a '|'. This
86 option is ignored if -p or -P options are not specified.
87
88
89 -D, --duplicates
90 If Slurm job ids are reset, some job numbers will probably
91 appear more than once in the accounting log file but refer to
92 different jobs. Such jobs can be distinguished by the "sub‐
93 mit" time stamp in the data records.
94
95
96 When data for specific jobs are requested with the --jobs op‐
97 tion, sacct returns the most recent job with that number.
98 This behavior can be overridden by specifying --duplicates,
99 in which case all records that match the selection criteria
100 will be returned.
101
102
103 NOTE: Revoked federated sibling jobs are hidden unless the
104 --duplicates option is specified.
105
106
107 -e, --helpformat
108
109 Print a list of fields that can be specified with the --for‐
110 mat option.
111
112
113 Fields available:
114
115 Account AdminComment AllocCPUS AllocNodes
116 AllocTRES AssocID AveCPU AveCPUFreq
117 AveDiskRead AveDiskWrite AvePages AveRSS
118 AveVMSize BlockID Cluster Comment
119 Constraints ConsumedEnergy ConsumedEnergyRaw CPUTime
120 CPUTimeRAW DBIndex DerivedExitCode Elapsed
121 ElapsedRaw Eligible End ExitCode
122 Flags GID Group JobID
123 JobIDRaw JobName Layout MaxDiskRead
124 MaxDiskReadNode MaxDiskReadTask MaxDiskWrite MaxDiskWriteNode
125 MaxDiskWriteTask MaxPages MaxPagesNode MaxPagesTask
126 MaxRSS MaxRSSNode MaxRSSTask MaxVMSize
127 MaxVMSizeNode MaxVMSizeTask McsLabel MinCPU
128 MinCPUNode MinCPUTask NCPUS NNodes
129 NodeList NTasks Priority Partition
130 QOS QOSRAW Reason ReqCPUFreq
131 ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ReqCPUS
132 ReqMem ReqNodes ReqTRES Reservation
133 ReservationId Reserved ResvCPU ResvCPURAW
134 Start State Submit Suspended
135 SystemCPU SystemComment Timelimit TimelimitRaw
136 TotalCPU TRESUsageInAve TRESUsageInMax TRESUsageInMaxNode
137 TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode TRESUsageInMinTask
138 TRESUsageInTot TRESUsageOutAve TRESUsageOutMax TRESUsageOutMaxNode
139 TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode TRESUsageOutMinTask
140 TRESUsageOutTot UID User UserCPU
141 WCKey WCKeyID WorkDir
142
143 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
144 SUsageIn[Ave|Tot]. They represent the average/total of the
145 highest watermarks over all ranks in the step. When using
146 sstat they represent the average/total at the moment the com‐
147 mand was run.
148
149 NOTE: TRESUsage*Min* values represent the lowest highwater
150 mark in the step.
151
152 The section titled "Job Accounting Fields" describes these
153 fields.
154
155
156 -E end_time, --endtime=end_time
157
158 Select jobs in any state before the specified time. If
159 states are given with the -s option return jobs in this state
160 before this period. See the DEFAULT TIME WINDOW for more de‐
161 tails.
162
163 Valid time formats are:
164 HH:MM[:SS][AM|PM]
165 MMDD[YY][-HH:MM[:SS]]
166 MM.DD[.YY][-HH:MM[:SS]]
167 MM/DD[/YY][-HH:MM[:SS]]
168 YYYY-MM-DD[THH:MM[:SS]]
169 today, midnight, noon, fika (3 PM), teatime (4 PM)
170 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
171
172 --federation
173 Show jobs from the federation if a member of one.
174
175
176 -f file, --file=file
177 Causes the sacct command to read job accounting data from the
178 named file instead of the current Slurm job accounting log
179 file. Only applicable when running the jobcomp/filetxt
180 plugin.
181
182
183 -F flag_list, --flags=flag_list
184 Comma separated list to filter jobs based on what various
185 ways the jobs were handled. Current flags are SchedSubmit,
186 SchedMain, SchedBackfill. These particular options describe
187 the scheduler that started the job.
188
189
190 -g gid_list, --gid=gid_list --group=group_list
191 Displays the statistics only for the jobs started with the
192 GID or the GROUP specified by the gid_list or the group_list
193 operand, which is a comma-separated list. Space characters
194 are not allowed. Default is no restrictions.
195
196
197 -h, --help
198 Displays a general help message.
199
200
201 -i, --nnodes=N
202 Return jobs which ran on this many nodes (N = min[-max])
203
204
205 -j job(.step) , --jobs=job(.step)
206 Displays information about the specified job(.step) or list
207 of job(.step)s.
208
209 The job(.step) parameter is a comma-separated list of jobs.
210 Space characters are not permitted in this list. NOTE: A
211 step id of 'batch' will display the information about the
212 batch step.
213
214 By default sacct shows only jobs with Eligible time, but with
215 this option the non-eligible will be also shown. NOTE: If
216 --state is also specified, as non-eligible are not PD, then
217 non-eligible jobs will not be displayed. See the DEFAULT
218 TIME WINDOW for details about how this option changes the de‐
219 fault -S and -E options.
220
221
222 -k, --timelimit-min
223 Only send data about jobs with this timelimit. If used with
224 timelimit_max this will be the minimum timelimit of the
225 range. Default is no restriction.
226
227
228 -K, --timelimit-max
229 Ignored by itself, but if timelimit_min is set this will be
230 the maximum timelimit of the range. Default is no restric‐
231 tion.
232
233
234 --local Show only jobs local to this cluster. Ignore other clusters
235 in this federation (if any). Overrides --federation.
236
237
238 -l, --long
239 Equivalent to specifying:
240
241 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
242 sizenode, maxvmsizetask,avevm‐
243 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
244 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
245 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
246 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
247 gov,reqmem,consumeden‐
248 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
249 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
250 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
251 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
252 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
253 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
254 outave,tresusageouttot
255
256
257 -L, --allclusters
258 Display jobs ran on all clusters. By default, only jobs ran
259 on the cluster from where sacct is called are displayed.
260
261
262 -M cluster_list, --clusters=cluster_list
263 Displays the statistics only for the jobs started on the
264 clusters specified by the cluster_list operand, which is a
265 comma-separated list of clusters. Space characters are not
266 allowed in the cluster_list. A value of 'all' will query to
267 run on all clusters. The default is current cluster you are
268 executing the sacct command on or all clusters in the federa‐
269 tion when executed on a federated cluster. This option im‐
270 plicitly sets the --local option.
271
272
273 -n, --noheader
274 No heading will be added to the output. The default action is
275 to display a header.
276
277 --noconvert
278 Don't convert units from their original type (e.g. 2048M
279 won't be converted to 2G).
280
281 -N node_list, --nodelist=node_list
282 Display jobs that ran on any of these node(s). node_list can
283 be a ranged string.
284
285 --name=jobname_list
286 Display jobs that have any of these name(s).
287
288 -o, --format
289 Comma separated list of fields. (use "--helpformat" for a
290 list of available fields).
291
292 NOTE: When using the format option for listing various fields
293 you can put a %NUMBER afterwards to specify how many charac‐
294 ters should be printed.
295
296 e.g. format=name%30 will print 30 characters of field name
297 right justified. A %-30 will print 30 characters left justi‐
298 fied.
299
300 When set, the SACCT_FORMAT environment variable will override
301 the default format. For example:
302
303 SACCT_FORMAT="jobid,user,account,cluster"
304
305
306 -p, --parsable
307 Output will be '|' delimited with a '|' at the end. See also
308 the --delimiter option.
309
310
311 -P, --parsable2
312 Output will be '|' delimited without a '|' at the end. See
313 also the --delimiter option.
314
315
316 -q, --qos Only send data about jobs using these qos. Default is all.
317
318
319 -r, --partition
320
321 Comma separated list of partitions to select jobs and job
322 steps from. The default is all partitions.
323
324
325 -R reason_list, --reason=reason_list
326 Comma separated list to filter jobs based on what reason the
327 job wasn't scheduled outside resources/priority.
328
329
330 -s state_list, --state=state_list
331 Selects jobs based on their state during the time period
332 given. Unless otherwise specified, the start and end time
333 will be the current time when the --state option is specified
334 and only currently running jobs can be displayed. A start
335 and/or end time must be specified to view information about
336 jobs not currently running. See the JOB STATE CODES section
337 below for a list of state designators. Multiple state names
338 may be specified using comma separators. Either the short or
339 long form of the state name may be used (e.g. CA or CAN‐
340 CELLED) and the name is case insensitive (i.e. ca and CA both
341 work).
342
343 NOTE: Note for a job to be selected in the PENDING state it
344 must have "EligibleTime" in the requested time interval or
345 different from "Unknown". The "EligibleTime" is displayed by
346 the "scontrol show job" command. For example jobs submitted
347 with the "--hold" option will have "EligibleTime=Unknown" as
348 they are pending indefinitely.
349
350 NOTE: When specifying states and no start time is given the
351 default start time is 'now'. This is only when -j is not
352 used. If -j is used the start time will default to 'Epoch'.
353 In both cases if no end time is given it will default to
354 'now'. See the DEFAULT TIME WINDOW for more details.
355
356
357 -S, --starttime
358 Select jobs in any state after the specified time. Default is
359 00:00:00 of the current day, unless the '-s' or '-j' options
360 are used. If the '-s' option is used, then the default is
361 'now'. If states are given with the '-s' option then only
362 jobs in this state at this time will be returned. If the '-j'
363 option is used, then the default time is Unix Epoch 0. See
364 the DEFAULT TIME WINDOW for more details.
365
366 Valid time formats are:
367 HH:MM[:SS][AM|PM]
368 MMDD[YY][-HH:MM[:SS]]
369 MM.DD[.YY][-HH:MM[:SS]]
370 MM/DD[/YY][-HH:MM[:SS]]
371 YYYY-MM-DD[THH:MM[:SS]]
372 today, midnight, noon, fika (3 PM), teatime (4 PM)
373 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
374
375
376 -T, --truncate
377 Truncate time. So if a job started before --starttime the
378 start time would be truncated to --starttime. The same for
379 end time and --endtime.
380
381
382 -u uid_list, --uid=uid_list, --user=user_list
383 Use this comma separated list of uids or user names to select
384 jobs to display. By default, the running user's uid is used.
385
386
387 --use-local-uid
388 When displaying UID, sacct uses the UID stored in Slurm's ac‐
389 counting database by default. Use this command to make Slurm
390 use a system call to get the UID from the username. This op‐
391 tion may be useful in an environment with multiple clusters
392 and one database where the UID's aren't the same on all clus‐
393 ters.
394
395
396 --units=[KMGTP]
397 Display values in specified unit type. Takes precedence over
398 --noconvert option.
399
400
401 --usage Display a command usage summary.
402
403
404 -v, --verbose
405 Primarily for debugging purposes, report the state of various
406 variables during processing.
407
408
409 -V, --version
410 Print version.
411
412
413 --whole-hetjob=[yes|no]
414 When querying heterogeneous jobs, Slurm by default retrieves
415 the information about all the components of the job if the
416 het_job_id (leader id) is selected. If a non-leader hetero‐
417 geneous job component id is selected only that component is
418 retrieved by default. This behavior can be changed by using
419 this option. If set to 'yes' or no value is set, then infor‐
420 mation about all the components will be retrieved no matter
421 which component is selected in the job filter. Otherwise, if
422 set to 'no' then only the selected heterogeneous job compo‐
423 nents will be retrieved, even when selecting the leader.
424
425
426 -W wckey_list, --wckeys=wckey_list
427 Displays the statistics only for the jobs started on the wck‐
428 eys specified by the wckey_list operand, which is a
429 comma-separated list of wckey names. Space characters are
430 not allowed in the wckey_list. Default is all wckeys.
431
432
433 -x associd_list, --associations=assoc_list
434 Displays the statistics only for the jobs running under the
435 association ids specified by the assoc_list operand, which is
436 a comma-separated list of association ids. Space characters
437 are not allowed in the assoc_list. Default is all associa‐
438 tions.
439
440
441 -X, --allocations
442 Only show statistics relevant to the job allocation itself,
443 not taking steps into consideration.
444
445 NOTE: Without including steps, utilization statistics for job
446 allocation(s) will be reported as zero.
447
448
449 Job Accounting Fields
450 The following describes each job accounting field:
451
452 ALL Print all fields listed below.
453
454
455 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
456
457
458 AllocNodes
459 Number of nodes allocated to the job/step. 0 if the
460 job is pending.
461
462
463 AllocTres Trackable resources. These are the resources allocated
464 to the job/step after the job started running. For
465 pending jobs this should be blank. For more details
466 see AccountingStorageTRES in slurm.conf.
467
468 NOTE: When a generic resource is configured with the
469 no_consume flag, the allocation will be printed with a
470 zero.
471
472
473 Account Account the job ran under.
474
475
476 AssocID Reference to the association of user, account and
477 cluster.
478
479
480 AveCPU Average (system + user) CPU time of all tasks in job.
481
482
483 AveCPUFreq
484 Average weighted CPU frequency of all tasks in job, in
485 kHz.
486
487
488 AveDiskRead
489 Average number of bytes read by all tasks in job.
490
491
492 AveDiskWrite
493 Average number of bytes written by all tasks in job.
494
495
496 AvePages Average number of page faults of all tasks in job.
497
498
499 AveRSS Average resident set size of all tasks in job.
500
501
502 AveVMSize Average Virtual Memory size of all tasks in job.
503
504
505 Cluster Cluster name.
506
507
508 Comment The job's comment string when the AccountingStoreJob‐
509 Comment parameter in the slurm.conf file is set (or
510 defaults) to YES. The Comment string can be modified
511 by invoking sacctmgr modify job or the specialized
512 sjobexitmod command.
513
514
515 ConsumedEnergy
516 Total energy consumed by all tasks in job, in joules.
517 Note: Only in case of exclusive job allocation this
518 value reflects the jobs' real energy consumption.
519
520
521 CPUTime Time used (Elapsed time * CPU count) by a job or step
522 in HH:MM:SS format.
523
524
525 CPUTimeRAW
526 Time used (Elapsed time * CPU count) by a job or step
527 in cpu-seconds.
528
529
530 DerivedExitCode
531 The highest exit code returned by the job's job steps
532 (srun invocations). Following the colon is the signal
533 that caused the process to terminate if it was termi‐
534 nated by a signal. The DerivedExitCode can be modi‐
535 fied by invoking sacctmgr modify job or the special‐
536 ized sjobexitmod command.
537
538
539 Elapsed The jobs elapsed time.
540
541 The format of this fields output is as follows:
542 [DD-[HH:]]MM:SS
543
544 as defined by the following:
545
546 DD days
547
548 hh hours
549
550 mm minutes
551
552 ss seconds
553
554
555 Eligible When the job became eligible to run in the same format
556 as End.
557
558
559 End Termination time of the job. Format output is,
560 YYYY-MM-DDTHH:MM:SS, unless changed through the
561 SLURM_TIME_FORMAT environment variable.
562
563
564 ExitCode The exit code returned by the job script or salloc,
565 typically as set by the exit() function. Following
566 the colon is the signal that caused the process to
567 terminate if it was terminated by a signal.
568
569
570 GID The group identifier of the user who ran the job.
571
572
573 Group The group name of the user who ran the job.
574
575
576 JobID The identification number of the job or job step.
577
578 Regular jobs are in the form:
579
580 JobID[.JobStep]
581
582 Array jobs are in the form:
583
584 ArrayJobID_ArrayTaskID
585
586 Heterogeneous jobs are in the form:
587
588 HetJobID+HetJobOffset .
589
590
591 JobIDRaw The identification number of the job or job step.
592 Prints the JobID in the form JobID[.JobStep] for regu‐
593 lar, heterogeneous and array jobs. .
594
595
596 JobName The name of the job or job step. The slurm_account‐
597 ing.log file is a space delimited file. Because of
598 this if a space is used in the jobname an underscore
599 is substituted for the space before the record is
600 written to the accounting file. So when the jobname is
601 displayed by sacct the jobname that had a space in it
602 will now have an underscore in place of the space.
603
604
605 Layout What the layout of a step was when it was running.
606 This can be used to give you an idea of which node ran
607 which rank in your job.
608
609
610 MaxDiskRead
611 Maximum number of bytes read by all tasks in job.
612
613
614 MaxDiskReadNode
615 The node on which the maxdiskread occurred.
616
617
618 MaxDiskReadTask
619 The task ID where the maxdiskread occurred.
620
621
622 MaxDiskWrite
623 Maximum number of bytes written by all tasks in job.
624
625
626 MaxDiskWriteNode
627 The node on which the maxdiskwrite occurred.
628
629
630 MaxDiskWriteTask
631 The task ID where the maxdiskwrite occurred.
632
633
634 MaxPages Maximum number of page faults of all tasks in job.
635
636
637 MaxPagesNode
638 The node on which the maxpages occurred.
639
640
641 MaxPagesTask
642 The task ID where the maxpages occurred.
643
644
645 MaxRSS Maximum resident set size of all tasks in job.
646
647
648 MaxRSSNode
649 The node on which the maxrss occurred.
650
651
652 MaxRSSTask
653 The task ID where the maxrss occurred.
654
655
656 MaxVMSize Maximum Virtual Memory size of all tasks in job.
657
658
659 MaxVMSizeNode
660 The node on which the maxvmsize occurred.
661
662
663 MaxVMSizeTask
664 The task ID where the maxvmsize occurred.
665
666
667 MinCPU Minimum (system + user) CPU time of all tasks in job.
668
669
670 MinCPUNode
671 The node on which the mincpu occurred.
672
673
674 MinCPUTask
675 The task ID where the mincpu occurred.
676
677
678 NCPUS Total number of CPUs allocated to the job. Equivalent
679 to AllocCPUS.
680
681
682 NodeList List of nodes in job/step.
683
684
685 NNodes Number of nodes in a job or step. If the job is run‐
686 ning, or ran, this count will be the number allocated,
687 else the number will be the number requested.
688
689
690 NTasks Total number of tasks in a job or step.
691
692
693 Priority Slurm priority.
694
695
696 Partition Identifies the partition on which the job ran.
697
698
699 QOS Name of Quality of Service.
700
701
702 QOSRAW Numeric id of Quality of Service.
703
704
705 ReqCPUFreq
706 Requested CPU frequency for the step, in kHz. Note:
707 This value applies only to a job step. No value is re‐
708 ported for the job.
709
710
711 ReqCPUFreqGov
712 Requested CPU frequency governor for the step, in kHz.
713 Note: This value applies only to a job step. No value
714 is reported for the job.
715
716
717 ReqCPUFreqMax
718 Maximum requested CPU frequency for the step, in kHz.
719 Note: This value applies only to a job step. No value
720 is reported for the job.
721
722
723 ReqCPUFreqMin
724 Minimum requested CPU frequency for the step, in kHz.
725 Note: This value applies only to a job step. No value
726 is reported for the job.
727
728
729 ReqCPUS Number of requested CPUs.
730
731
732 ReqMem Minimum required memory for the job, in MB. A 'c' at
733 the end of number represents Memory Per CPU, a 'n'
734 represents Memory Per Node. Note: This value is only
735 from the job allocation, not the step.
736
737
738 ReqNodes Requested minimum Node count for the job/step.
739
740
741 ReqTres Trackable resources. These are the minimum resource
742 counts requested by the job/step at submission time.
743 For more details see AccountingStorageTRES in
744 slurm.conf.
745
746
747 Reservation
748 Reservation Name.
749
750
751 ReservationId
752 Reservation Id.
753
754
755 Reserved How much wall clock time was used as reserved time for
756 this job. This is derived from how long a job was
757 waiting from eligible time to when it actually
758 started. Format is the same as Elapsed.
759
760
761 ResvCPU How many CPU seconds were used as reserved time for
762 this job. Format is the same as Elapsed.
763
764
765 ResvCPURAW
766 How many CPU seconds were used as reserved time for
767 this job. Format is in processor seconds.
768
769
770 Start Initiation time of the job in the same format as End.
771
772
773 State Displays the job status, or state. See the JOB STATE
774 CODES section below for a list of possible states.
775
776 If more information is available on the job state than
777 will fit into the current field width (for example,
778 the uid that CANCELLED a job) the state will be fol‐
779 lowed by a "+". You can increase the size of the dis‐
780 played state using the "%NUMBER" format modifier de‐
781 scribed earlier.
782
783 NOTE: The RUNNING state will return suspended jobs as
784 well. In order to print suspended jobs you must re‐
785 quest SUSPENDED at a different call from RUNNING.
786
787 NOTE: The RUNNING state will return any jobs completed
788 (cancelled or otherwise) in the time period requested
789 as the job was also RUNNING during that time. If you
790 are only looking for jobs that finished, please choose
791 the appropriate state(s) without the RUNNING state.
792
793
794 Submit The time the job was submitted in the same format as
795 End.
796
797 NOTE: If a job is requeued, the submit time is reset.
798 To obtain the original submit time it is necessary to
799 use the -D or --duplicate option to display all dupli‐
800 cate entries for a job.
801
802
803 Suspended The amount of time a job or job step was suspended.
804 Format is the same as Elapsed.
805
806
807 SystemCPU The amount of system CPU time used by the job or job
808 step. Format is the same as Elapsed.
809
810 NOTE: SystemCPU provides a measure of the task's par‐
811 ent process and does not include CPU time of child
812 processes.
813
814
815 Timelimit What the timelimit was/is for the job. Format is the
816 same as Elapsed.
817
818
819 TimelimitRaw
820 What the timelimit was/is for the job. Format is in
821 number of minutes.
822
823
824 TotalCPU The sum of the SystemCPU and UserCPU time used by the
825 job or job step. The total CPU time of the job may
826 exceed the job's elapsed time for jobs that include
827 multiple job steps. Format is the same as Elapsed.
828
829 NOTE: TotalCPU provides a measure of the task's parent
830 process and does not include CPU time of child pro‐
831 cesses.
832
833
834 TresUsageInAve
835 Tres average usage in by all tasks in job. NOTE: If
836 corresponding TresUsageInMaxTask is -1 the metric is
837 node centric instead of task.
838
839
840 TresUsageInMax
841 Tres maximum usage in by all tasks in job. NOTE: If
842 corresponding TresUsageInMaxTask is -1 the metric is
843 node centric instead of task.
844
845
846 TresUsageInMaxNode
847 Node for which each maximum TRES usage out occurred.
848
849
850 TresUsageInMaxTask
851 Task for which each maximum TRES usage out occurred.
852
853
854 TresUsageInMin
855 Tres minimum usage in by all tasks in job. NOTE: If
856 corresponding TresUsageInMinTask is -1 the metric is
857 node centric instead of task.
858
859
860 TresUsageInMinNode
861 Node for which each minimum TRES usage out occurred.
862
863
864 TresUsageInMinTask
865 Task for which each minimum TRES usage out occurred.
866
867
868 TresUsageInTot
869 Tres total usage in by all tasks in job.
870
871
872 TresUsageOutAve
873 Tres average usage out by all tasks in job. NOTE: If
874 corresponding TresUsageOutMaxTask is -1 the metric is
875 node centric instead of task.
876
877
878 TresUsageOutMax
879 Tres maximum usage out by all tasks in job. NOTE: If
880 corresponding TresUsageOutMaxTask is -1 the metric is
881 node centric instead of task.
882
883
884 TresUsageOutMaxNode
885 Node for which each maximum TRES usage out occurred.
886
887
888 TresUsageOutMaxTask
889 Task for which each maximum TRES usage out occurred.
890
891
892 TresUsageOutTot
893 Tres total usage out by all tasks in job.
894
895
896 UID The user identifier of the user who ran the job.
897
898
899 User The user name of the user who ran the job.
900
901
902 UserCPU The amount of user CPU time used by the job or job
903 step. Format is the same as Elapsed.
904
905 NOTE: UserCPU provides a measure of the task's parent
906 process and does not include CPU time of child pro‐
907 cesses.
908
909
910 WCKey Workload Characterization Key. Arbitrary string
911 for grouping orthogonal accounts together.
912
913
914 WCKeyID Reference to the wckey.
915
916
917
919 BF BOOT_FAIL Job terminated due to launch failure, typically due
920 to a hardware failure (e.g. unable to boot the node
921 or block and the job can not be requeued).
922
923 CA CANCELLED Job was explicitly cancelled by the user or system
924 administrator. The job may or may not have been
925 initiated.
926
927 CD COMPLETED Job has terminated all processes on all nodes with
928 an exit code of zero.
929
930 DL DEADLINE Job terminated on deadline.
931
932 F FAILED Job terminated with non-zero exit code or other
933 failure condition.
934
935 NF NODE_FAIL Job terminated due to failure of one or more allo‐
936 cated nodes.
937
938 OOM OUT_OF_MEMORY Job experienced out of memory error.
939
940 PD PENDING Job is awaiting resource allocation.
941
942 PR PREEMPTED Job terminated due to preemption.
943
944 R RUNNING Job currently has an allocation.
945
946 RQ REQUEUED Job was requeued.
947
948 RS RESIZING Job is about to change size.
949
950 RV REVOKED Sibling was removed from cluster due to other clus‐
951 ter starting the job.
952
953 S SUSPENDED Job has an allocation, but execution has been sus‐
954 pended and CPUs have been released for other jobs.
955
956 TO TIMEOUT Job terminated upon reaching its time limit.
957
958
960 The options --starttime and --endtime define the time window between
961 which sacct is going to search. For historical and practical reasons
962 their default values (i.e. the default time window) depends on other
963 options: --jobs and --state.
964
965 Depending on if --jobs and/or --state are specified, the default values
966 of --starttime and --endtime options are:
967
968
969 WITHOUT EITHER --jobs NOR --state specified:
970 --starttime defaults to Midnight.
971 --endtime defaults to Now.
972
973
974 WITH --jobs AND WITHOUT --state specified:
975 --starttime defaults to Epoch 0.
976 --endtime defaults to Now.
977
978
979 WITHOUT --jobs AND WITH --state specified:
980 --starttime defaults to Now.
981 --endtime defaults to --starttime and to Now if --starttime is
982 not specified.
983
984
985 WITH BOTH --jobs AND --state specified:
986 --starttime defaults to Epoch 0.
987 --endtime defaults to --starttime or to Now if --starttime is
988 not specified.
989
990
991 NOTE: With -v/--verbose a message about the actual time window in use
992 is shown.
993
994
996 Executing sacct sends a remote procedure call to slurmdbd. If enough
997 calls from sacct or other Slurm client commands that send remote proce‐
998 dure calls to the slurmdbd daemon come in at once, it can result in a
999 degradation of performance of the slurmdbd daemon, possibly resulting
1000 in a denial of service.
1001
1002 Do not run sacct or other Slurm client commands that send remote proce‐
1003 dure calls to slurmdbd from loops in shell scripts or other programs.
1004 Ensure that programs limit calls to sacct to the minimum necessary for
1005 the information you are trying to gather.
1006
1007
1009 Some sacct options may be set via environment variables. These environ‐
1010 ment variables, along with their corresponding options, are listed be‐
1011 low. (Note: Commandline options will always override these settings.)
1012
1013 SACCT_FEDERATION Same as --federation
1014
1015 SACCT_LOCAL Same as --local
1016
1017 SLURM_CONF The location of the Slurm configuration file.
1018
1019 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
1020 value of standard, the default value, generates
1021 output in the form
1022 "year-month-dateThour:minute:second". A value of
1023 relative returns only "hour:minute:second" if the
1024 current day. For other dates in the current year
1025 it prints the "hour:minute" preceded by "Tomorr"
1026 (tomorrow), "Ystday" (yesterday), the name of the
1027 day for the coming week (e.g. "Mon", "Tue", etc.),
1028 otherwise the date (e.g. "25 Apr"). For other
1029 years it returns a date month and year without a
1030 time (e.g. "6 Jun 2012"). All of the time stamps
1031 use a 24 hour format.
1032
1033 A valid strftime() format can also be specified.
1034 For example, a value of "%a %T" will report the day
1035 of the week and a time stamp (e.g. "Mon 12:34:56").
1036
1037
1039 This example illustrates the default invocation of the sacct command:
1040
1041 # sacct
1042 Jobid Jobname Partition Account AllocCPUS State ExitCode
1043 ---------- ---------- ---------- ---------- ---------- ---------- --------
1044 2 script01 srun acct1 1 RUNNING 0
1045 3 script02 srun acct1 1 RUNNING 0
1046 4 endscript srun acct1 1 RUNNING 0
1047 4.0 srun acct1 1 COMPLETED 0
1048
1049
1050 This example shows the same job accounting information with the brief
1051 option.
1052
1053 # sacct --brief
1054 Jobid State ExitCode
1055 ---------- ---------- --------
1056 2 RUNNING 0
1057 3 RUNNING 0
1058 4 RUNNING 0
1059 4.0 COMPLETED 0
1060
1061 # sacct --allocations
1062 Jobid Jobname Partition Account AllocCPUS State ExitCode
1063 ---------- ---------- ---------- ---------- ------- ---------- --------
1064 3 sja_init andy acct1 1 COMPLETED 0
1065 4 sjaload andy acct1 2 COMPLETED 0
1066 5 sja_scr1 andy acct1 1 COMPLETED 0
1067 6 sja_scr2 andy acct1 18 COMPLETED 2
1068 7 sja_scr3 andy acct1 18 COMPLETED 0
1069 8 sja_scr5 andy acct1 2 COMPLETED 0
1070 9 sja_scr7 andy acct1 90 COMPLETED 1
1071 10 endscript andy acct1 186 COMPLETED 0
1072
1073
1074 This example demonstrates the ability to customize the output of the
1075 sacct command. The fields are displayed in the order designated on the
1076 command line.
1077
1078 # sacct --format=jobid,elapsed,ncpus,ntasks,state
1079 Jobid Elapsed Ncpus Ntasks State
1080 ---------- ---------- ---------- -------- ----------
1081 3 00:01:30 2 1 COMPLETED
1082 3.0 00:01:30 2 1 COMPLETED
1083 4 00:00:00 2 2 COMPLETED
1084 4.0 00:00:01 2 2 COMPLETED
1085 5 00:01:23 2 1 COMPLETED
1086 5.0 00:01:31 2 1 COMPLETED
1087
1088
1089 This example demonstrates the use of the -T (--truncate) option when
1090 used with -S (--starttime) and -E (--endtime). When the -T option is
1091 used, the start time of the job will be the specified -S value if the
1092 job was started before the specified time, otherwise the time will be
1093 the job's start time. The end time will be the specified -E option if
1094 the job ends after the specified time, otherwise it will be the jobs
1095 end time.
1096
1097 Without -T (normal operation) sacct output would be like this.
1098
1099 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1100 JobID Start End State
1101 --------- --------------------- -------------------- ------------
1102 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1103 3 2014-07-03T11:35:21 Unknown RUNNING
1104 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1105 5 2014-07-03T11:41:01 Unknown RUNNING
1106
1107 By adding the -T option the job's start and end times are truncated to
1108 reflect only the time requested. If a job started after the start time
1109 requested or finished before the end time requested those times are not
1110 altered. The -T option is useful when determining exact run times dur‐
1111 ing any given period.
1112
1113 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1114 JobID Start End State
1115 --------- --------------------- -------------------- ------------
1116 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1117 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1118 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1119 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1120
1121
1122 NOTE: If no -s (--state) option is given sacct will display eligible
1123 jobs during the specified period of time, otherwise it will return jobs
1124 that were in the state requested during that period of time.
1125
1126 This example demonstrates the differences running sacct with and with‐
1127 out the --state flag for the same time period. Without the --state op‐
1128 tion, all eligible jobs in that time period are shown.
1129
1130 # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1131 JobID Start End State
1132 ------------ ------------------- ------------------- ----------
1133 2955 11:15:12 11:20:12 COMPLETED
1134 2956 11:20:13 11:25:13 COMPLETED
1135
1136 With the --state=pending option, only job 2956 will be shown because it
1137 had a dependency on 2955 and was still PENDING from 11:20:00 until it
1138 started at 11:21:13. Note that even though we requested PENDING jobs,
1139 the State shows as COMPLETED because that is the current State of the
1140 job.
1141
1142 # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1143 JobID Start End State
1144 ------------ ------------------- ------------------- ----------
1145 2956 11:20:13 11:25:13 COMPLETED
1146
1148 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1149 L.P.
1150 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1151 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1152 Copyright (C) 2010-2014 SchedMD LLC.
1153
1154 This file is part of Slurm, a resource management program. For de‐
1155 tails, see <https://slurm.schedmd.com/>.
1156
1157 Slurm is free software; you can redistribute it and/or modify it under
1158 the terms of the GNU General Public License as published by the Free
1159 Software Foundation; either version 2 of the License, or (at your op‐
1160 tion) any later version.
1161
1162 Slurm is distributed in the hope that it will be useful, but WITHOUT
1163 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1164 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1165 for more details.
1166
1168 /etc/slurm.conf
1169 Entries to this file enable job accounting and designate the
1170 job accounting log file that collects system job accounting.
1171
1172 /var/log/slurm_accounting.log
1173 The default job accounting log file. By default, this file
1174 is set to read and write permission for root only.
1175
1177 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1178
1179
1180
1181April 2021 Slurm Commands sacct(1)