1sacct(1) Slurm Commands sacct(1)
2
3
4
6 sacct - displays accounting data for all jobs and job steps in the
7 Slurm job accounting log or Slurm database
8
9
11 sacct [OPTIONS...]
12
13
15 Accounting information for jobs invoked with Slurm are either logged in
16 the job accounting log file or saved to the Slurm database, as config‐
17 ured with the AccountingStorageType parameter.
18
19 The sacct command displays job accounting data stored in the job
20 accounting log file or Slurm database in a variety of forms for your
21 analysis. The sacct command displays information on jobs, job steps,
22 status, and exitcodes by default. You can tailor the output with the
23 use of the --format= option to specify the fields to be shown.
24
25 For the root user, the sacct command displays job accounting data for
26 all users, although there are options to filter the output to report
27 only the jobs from a specified user or group.
28
29 For the non-root user, the sacct command limits the display of job
30 accounting data to jobs that were launched with their own user identi‐
31 fier (UID) by default. Data for other users can be displayed with the
32 --allusers, --user, or --uid options.
33
34 Elapsed time fields are presented as [days-]hours:minutes:sec‐
35 onds[.microseconds]. Only 'CPU' fields will ever have microseconds.
36
37 The default input file is the file named in the AccountingStorageLoc
38 parameter in slurm.conf.
39
40 Note: If designated, the slurmdbd.conf option PrivateData may further
41 restrict the accounting data visible to users which are not
42 SlurmUser, root, or a user with AdminLevel=Admin. See the slur‐
43 mdbd.conf man page for additional details on restricting access
44 to accounting data.
45
46 Note: If the AccountingStorageType is set to "accounting_stor‐
47 age/filetxt", space characters embedded within account names,
48 job names, and step names will be replaced by underscores. If
49 account names with embedded spaces are needed, it is recommended
50 that a database type of accounting storage be configured.
51
52 Note: The contents of Slurm's database are maintained in lower case.
53 This may result in some sacct output differing from that of
54 other Slurm commands.
55
56 Note: Much of the data reported by sacct has been generated by the
57 wait3() and getrusage() system calls. Some systems gather and
58 report incomplete information for these calls; sacct reports
59 values of 0 for this missing data. See your systems getrusage
60 [4m(3) man page for information about which data are actually
61 available on your system.
62
63
65 -a, --allusers
66 Displays all users jobs when run by user root or if Private‐
67 Data is not configured to jobs. Otherwise display the cur‐
68 rent user's jobs
69
70 -A account_list , --accounts=account_list
71 Displays jobs when a comma separated list of accounts are
72 given as the argument.
73
74 -b, --brief
75 Displays a brief listing, which includes the following data:
76
77 jobid
78
79 status
80
81 exitcode
82
83 -c, --completion
84 Use job completion data instead of job accounting. The Job‐
85 CompType parameter in the slurm.conf file must be defined to
86 a non-none option. Does not support federated cluster infor‐
87 mation (local data only).
88
89 -C constraint_list, --constraints=constraint_list
90 Comma separated list to filter jobs based on what con‐
91 straints/features the job requested. Multiple options will
92 be treated as 'and' not 'or', so the job would need all con‐
93 straints specified to be returned not one or the other.
94
95
96 --delimiter=characters
97 ASCII characters used to separate the fields when specifying
98 the -p or -P options. The default delimiter is a '|'. This
99 option is ignored if -p or -P options are not specified.
100
101
102 -D, --duplicates
103 If Slurm job ids are reset, some job numbers will probably
104 appear more than once in the accounting log file but refer to
105 different jobs. Such jobs can be distinguished by the "sub‐
106 mit" time stamp in the data records.
107
108
109 When data for specific jobs are requested with the --jobs
110 option, sacct returns the most recent job with that number.
111 This behavior can be overridden by specifying --duplicates,
112 in which case all records that match the selection criteria
113 will be returned.
114
115
116 NOTE: Revoked federated sibling jobs are hidden unless the
117 --duplicates option is specified.
118
119
120 -e, --helpformat
121
122 Print a list of fields that can be specified with the --for‐
123 mat option.
124
125
126 Fields available:
127
128 Account AdminComment AllocCPUS AllocGRES
129 AllocNodes AllocTRES AssocID AveCPU
130 AveCPUFreq AveDiskRead AveDiskWrite AvePages
131 AveRSS AveVMSize BlockID Cluster
132 Comment Constraints ConsumedEnergy ConsumedEnergyRaw
133 CPUTime CPUTimeRAW DerivedExitCode Elapsed
134 ElapsedRaw Eligible End ExitCode
135 Flags GID Group JobID
136 JobIDRaw JobName Layout MaxDiskRead
137 MaxDiskReadNode MaxDiskReadTask MaxDiskWrite MaxDiskWriteNode
138 MaxDiskWriteTask MaxPages MaxPagesNode MaxPagesTask
139 MaxRSS MaxRSSNode MaxRSSTask MaxVMSize
140 MaxVMSizeNode MaxVMSizeTask McsLabel MinCPU
141 MinCPUNode MinCPUTask NCPUS NNodes
142 NodeList NTasks Priority Partition
143 QOS QOSRAW Reason ReqCPUFreq
144 ReqCPUFreqMin ReqCPUFreqMax ReqCPUFreqGov ReqCPUS
145 ReqGRES ReqMem ReqNodes ReqTRES
146 Reservation ReservationId Reserved ResvCPU
147 ResvCPURAW Start State Submit
148 Suspended SystemCPU SystemComment Timelimit
149 TimelimitRaw TotalCPU TRESUsageInAve TRESUsageInMax
150 TRESUsageInMaxNode TRESUsageInMaxTask TRESUsageInMin TRESUsageInMinNode
151 TRESUsageInMinTask TRESUsageInTot TRESUsageOutAve TRESUsageOutMax
152 TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin TRESUsageOutMinNode
153 TRESUsageOutMinTask TRESUsageOutTot UID User
154 UserCPU WCKey WCKeyID WorkDir
155
156 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
157 SUsageIn[Ave|Tot]. They represent the average/total of the
158 highest watermarks over all ranks in the step. When using
159 sstat they represent the average/total at the moment the com‐
160 mand was run.
161
162 NOTE: TRESUsage*Min* values represent the lowest highwater
163 mark in the step.
164
165 The section titled "Job Accounting Fields" describes these
166 fields.
167
168
169 -E end_time, --endtime=end_time
170
171 Select jobs in any state before the specified time. If
172 states are given with the -s option return jobs in this state
173 before this period. See the DEFAULT TIME WINDOW for more
174 details.
175
176 Valid time formats are...
177
178 HH:MM[:SS] [AM|PM]
179 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
180 MM/DD[/YY]-HH:MM[:SS]
181 YYYY-MM-DD[THH:MM[:SS]]
182
183 --federation
184 Show jobs from the federation if a member of one.
185
186
187 -f file, --file=file
188 Causes the sacct command to read job accounting data from the
189 named file instead of the current Slurm job accounting log
190 file. Only applicable when running the filetxt plugin.
191
192
193 -F flag_list, --flags=flag_list
194 Comma separated list to filter jobs based on what various
195 ways the jobs were handled. Current flags are SchedSubmit,
196 SchedMain, SchedBackfill. These particular options describe
197 the scheduler that started the job.
198
199
200 -g gid_list, --gid=gid_list --group=group_list
201 Displays the statistics only for the jobs started with the
202 GID or the GROUP specified by the gid_list or the group_list
203 operand, which is a comma-separated list. Space characters
204 are not allowed. Default is no restrictions.
205
206
207 -h, --help
208 Displays a general help message.
209
210
211 -i, --nnodes=N
212 Return jobs which ran on this many nodes (N = min[-max])
213
214
215 -j job(.step) , --jobs=job(.step)
216 Displays information about the specified job(.step) or list
217 of job(.step)s.
218
219 The job(.step) parameter is a comma-separated list of jobs.
220 Space characters are not permitted in this list. NOTE: A
221 step id of 'batch' will display the information about the
222 batch step.
223
224 By default sacct shows only jobs with Eligible time, but with
225 this option the non-eligible will be also shown. NOTE: If
226 --state is also specified, as non-eligible are not PD, then
227 non-eligible jobs will not be displayed. See the DEFAULT
228 TIME WINDOW for details about how this option changes the
229 default -S and -E options.
230
231
232 -k, --timelimit-min
233 Only send data about jobs with this timelimit. If used with
234 timelimit_max this will be the minimum timelimit of the
235 range. Default is no restriction.
236
237
238 -K, --timelimit-max
239 Ignored by itself, but if timelimit_min is set this will be
240 the maximum timelimit of the range. Default is no restric‐
241 tion.
242
243
244 --local Show only jobs local to this cluster. Ignore other clusters
245 in this federation (if any). Overrides --federation.
246
247
248 -l, --long
249 Equivalent to specifying:
250
251 --format=jobid,jobname,partition,maxvmsize,maxvmsizen‐
252 ode,maxvmsizetask, avevm‐
253 size,maxrss,maxrssnode,maxrsstask,averss,maxpages,max‐
254 pagesnode, maxpagestask,avepages,mincpu,mincpun‐
255 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
256 code,maxdiskread,maxdiskreadnode,maxdiskreadtask,
257 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
258 task,avediskwrite, allocgres,reqgres,avecpufreq,reqcpufre‐
259 qmin,reqcpufreqmax,reqcpufreqgov, tresusageinmax,tresusagein‐
260 maxn,tresusageinmaxt,tresusageoutmax,tresusageoutmaxn, tre‐
261 susageoutmaxt,tresusageoutave,tresusageinave
262
263
264 -L, --allclusters
265 Display jobs ran on all clusters. By default, only jobs ran
266 on the cluster from where sacct is called are displayed.
267
268
269 -M cluster_list, --clusters=cluster_list
270 Displays the statistics only for the jobs started on the
271 clusters specified by the cluster_list operand, which is a
272 comma-separated list of clusters. Space characters are not
273 allowed in the cluster_list. A value of 'all' will query to
274 run on all clusters. The default is current cluster you are
275 executing the sacct command on or all clusters in the federa‐
276 tion when executed on a federated cluster. This option
277 implicitly sets the --local option.
278
279
280 -n, --noheader
281 No heading will be added to the output. The default action is
282 to display a header.
283
284 --noconvert
285 Don't convert units from their original type (e.g. 2048M
286 won't be converted to 2G).
287
288 -N node_list, --nodelist=node_list
289 Display jobs that ran on any of these node(s). node_list can
290 be a ranged string.
291
292 --name=jobname_list
293 Display jobs that have any of these name(s).
294
295 -o, --format
296 Comma separated list of fields. (use "--helpformat" for a
297 list of available fields).
298
299 NOTE: When using the format option for listing various fields
300 you can put a %NUMBER afterwards to specify how many charac‐
301 ters should be printed.
302
303 e.g. format=name%30 will print 30 characters of field name
304 right justified. A %-30 will print 30 characters left justi‐
305 fied.
306
307 When set, the SACCT_FORMAT environment variable will override
308 the default format. For example:
309
310 SACCT_FORMAT="jobid,user,account,cluster"
311
312
313 -p, --parsable
314 output will be '|' delimited with a '|' at the end
315
316
317 -P, --parsable2
318 output will be '|' delimited without a '|' at the end
319
320
321 -q, --qos Only send data about jobs using these qos. Default is all.
322
323
324 -r, --partition
325
326 Comma separated list of partitions to select jobs and job
327 steps from. The default is all partitions.
328
329
330 -R reason_list, --reason=reason_list
331 Comma separated list to filter jobs based on what reason the
332 job wasn't scheduled outside resources/priority.
333
334
335 -s state_list, --state=state_list
336 Selects jobs based on their state during the time period
337 given. Unless otherwise specified, the start and end time
338 will be the current time when the --state option is specified
339 and only currently running jobs can be displayed. A start
340 and/or end time must be specified to view information about
341 jobs not currently running. See the JOB STATE CODES section
342 below for a list of state designators. Multiple state names
343 may be specified using comma separators. Either the short or
344 long form of the state name may be used (e.g. CA or CAN‐
345 CELLED) and the name is case insensitive (i.e. ca and CA both
346 work).
347
348 NOTE: Note for a job to be selected in the PENDING state it
349 must have "EligibleTime" in the requested time interval or
350 different from "Unknown". The "EligibleTime" is displayed by
351 the "scontrol show job" command. For example jobs submitted
352 with the "--hold" option will have "EligibleTime=Unknown" as
353 they are pending indefinitely.
354
355 NOTE: When specifying states and no start time is given the
356 default start time is 'now'. This is only when -j is not
357 used. If -j is used the start time will default to 'Epoch'.
358 In both cases if no end time is given it will default to
359 'now'. See the DEFAULT TIME WINDOW for more details.
360
361
362 -S, --starttime
363 Select jobs in any state after the specified time. Default is
364 00:00:00 of the current day, unless the '-s' or '-j' options
365 are used. If the '-s' option is used, then the default is
366 'now'. If states are given with the '-s' option then only
367 jobs in this state at this time will be returned. If the '-j'
368 option is used, then the default time is Unix Epoch 0. See
369 the DEFAULT TIME WINDOW for more details.
370
371 Valid time formats are...
372
373 HH:MM[:SS] [AM|PM]
374 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
375 MM/DD[/YY]-HH:MM[:SS]
376 YYYY-MM-DD[THH:MM[:SS]]
377
378
379 -T, --truncate
380 Truncate time. So if a job started before --starttime the
381 start time would be truncated to --starttime. The same for
382 end time and --endtime.
383
384
385 -u uid_list, --uid=uid_list, --user=user_list
386 Use this comma separated list of uids or user names to select
387 jobs to display. By default, the running user's uid is used.
388
389
390 --units=[KMGTP]
391 Display values in specified unit type. Takes precedence over
392 --noconvert option.
393
394
395 --usage Display a command usage summary.
396
397
398 -v, --verbose
399 Primarily for debugging purposes, report the state of various
400 variables during processing.
401
402
403 -V, --version
404 Print version.
405
406
407 --whole-hetjob=[yes|no]
408 When querying heterogeneous jobs, Slurm by default retrieves
409 the information about all the components of the job if the
410 pack_job_id (leader) is selected. If a non-leader heteroge‐
411 neous job component is selected only that component is
412 retrieved by default. This behavior can be changed by using
413 this option. If set to 'yes' or no value is set, then infor‐
414 mation about all the components will be retrieved no matter
415 which component is selected in the job filter. Otherwise, if
416 set to 'no' then only the selected heterogeneous job compo‐
417 nents will be retrieved, even when selecting the leader.
418
419
420 -W wckey_list, --wckeys=wckey_list
421 Displays the statistics only for the jobs started on the wck‐
422 eys specified by the wckey_list operand, which is a
423 comma-separated list of wckey names. Space characters are
424 not allowed in the wckey_list. Default is all wckeys.
425
426
427 -x associd_list, --associations=assoc_list
428 Displays the statistics only for the jobs running under the
429 association ids specified by the assoc_list operand, which is
430 a comma-separated list of association ids. Space characters
431 are not allowed in the assoc_list. Default is all associa‐
432 tions.
433
434
435 -X, --allocations
436 Only show statistics relevant to the job allocation itself,
437 not taking steps into consideration.
438
439
440 Job Accounting Fields
441 The following describes each job accounting field:
442
443 ALL Print all fields listed below.
444
445
446 AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
447
448
449 AllocGRES Names and counts of generic resources allocated.
450
451
452 AllocNodes
453 Number of nodes allocated to the job/step. 0 if the
454 job is pending.
455
456
457 AllocTres Trackable resources. These are the resources allocated
458 to the job/step after the job started running. For
459 pending jobs this should be blank. For more details
460 see AccountingStorageTRES in slurm.conf.
461
462
463 Account Account the job ran under.
464
465
466 AssocID Reference to the association of user, account and
467 cluster.
468
469
470 AveCPU Average (system + user) CPU time of all tasks in job.
471
472
473 AveCPUFreq
474 Average weighted CPU frequency of all tasks in job, in
475 kHz.
476
477
478 AveDiskRead
479 Average number of bytes read by all tasks in job.
480
481
482 AveDiskWrite
483 Average number of bytes written by all tasks in job.
484
485
486 AvePages Average number of page faults of all tasks in job.
487
488
489 AveRSS Average resident set size of all tasks in job.
490
491
492 AveVMSize Average Virtual Memory size of all tasks in job.
493
494
495 Cluster Cluster name.
496
497
498 Comment The job's comment string when the AccountingStoreJob‐
499 Comment parameter in the slurm.conf file is set (or
500 defaults) to YES. The Comment string can be modified
501 by invoking sacctmgr modify job or the specialized
502 sjobexitmod command.
503
504
505 ConsumedEnergy
506 Total energy consumed by all tasks in job, in joules.
507 Note: Only in case of exclusive job allocation this
508 value reflects the jobs' real energy consumption.
509
510
511 CPUTime Time used (Elapsed time * CPU count) by a job or step
512 in HH:MM:SS format.
513
514
515 CPUTimeRAW
516 Time used (Elapsed time * CPU count) by a job or step
517 in cpu-seconds.
518
519
520 DerivedExitCode
521 The highest exit code returned by the job's job steps
522 (srun invocations). Following the colon is the signal
523 that caused the process to terminate if it was termi‐
524 nated by a signal. The DerivedExitCode can be modi‐
525 fied by invoking sacctmgr modify job or the special‐
526 ized sjobexitmod command.
527
528
529 Elapsed The jobs elapsed time.
530
531 The format of this fields output is as follows:
532 [DD-[HH:]]MM:SS
533
534 as defined by the following:
535
536 DD days
537
538 hh hours
539
540 mm minutes
541
542 ss seconds
543
544
545 Eligible When the job became eligible to run in the same format
546 as End.
547
548
549 End Termination time of the job. Format output is,
550 YYYY-MM-DDTHH:MM:SS, unless changed through the
551 SLURM_TIME_FORMAT environment variable.
552
553
554 ExitCode The exit code returned by the job script or salloc,
555 typically as set by the exit() function. Following
556 the colon is the signal that caused the process to
557 terminate if it was terminated by a signal.
558
559
560 GID The group identifier of the user who ran the job.
561
562
563 Group The group name of the user who ran the job.
564
565
566 JobID The identification number of the job or job step.
567
568 Regular jobs are in the form:
569
570 JobID[.JobStep]
571
572 Array jobs are in the form:
573
574 ArrayJobID_ArrayTaskID
575
576 Heterogeneous jobs are in the form:
577
578 PackJobID+PackJobOffset.
579
580
581 JobIDRaw The identification number of the job or job step.
582 Prints the JobID in the form JobID[.JobStep].
583
584
585 JobName The name of the job or job step. The slurm_account‐
586 ing.log file is a space delimited file. Because of
587 this if a space is used in the jobname an underscore
588 is substituted for the space before the record is
589 written to the accounting file. So when the jobname is
590 displayed by sacct the jobname that had a space in it
591 will now have an underscore in place of the space.
592
593
594 Layout What the layout of a step was when it was running.
595 This can be used to give you an idea of which node ran
596 which rank in your job.
597
598
599 MaxDiskRead
600 Maximum number of bytes read by all tasks in job.
601
602
603 MaxDiskReadNode
604 The node on which the maxdiskread occurred.
605
606
607 MaxDiskReadTask
608 The task ID where the maxdiskread occurred.
609
610
611 MaxDiskWrite
612 Maximum number of bytes written by all tasks in job.
613
614
615 MaxDiskWriteNode
616 The node on which the maxdiskwrite occurred.
617
618
619 MaxDiskWriteTask
620 The task ID where the maxdiskwrite occurred.
621
622
623 MaxPages Maximum number of page faults of all tasks in job.
624
625
626 MaxPagesNode
627 The node on which the maxpages occurred.
628
629
630 MaxPagesTask
631 The task ID where the maxpages occurred.
632
633
634 MaxRSS Maximum resident set size of all tasks in job.
635
636
637 MaxRSSNode
638 The node on which the maxrss occurred.
639
640
641 MaxRSSTask
642 The task ID where the maxrss occurred.
643
644
645 MaxVMSize Maximum Virtual Memory size of all tasks in job.
646
647
648 MaxVMSizeNode
649 The node on which the maxvmsize occurred.
650
651
652 MaxVMSizeTask
653 The task ID where the maxvmsize occurred.
654
655
656 MinCPU Minimum (system + user) CPU time of all tasks in job.
657
658
659 MinCPUNode
660 The node on which the mincpu occurred.
661
662
663 MinCPUTask
664 The task ID where the mincpu occurred.
665
666
667 NCPUS Total number of CPUs allocated to the job. Equivalent
668 to AllocCPUS.
669
670
671 NodeList List of nodes in job/step.
672
673
674 NNodes Number of nodes in a job or step. If the job is run‐
675 ning, or ran, this count will be the number allocated,
676 else the number will be the number requested.
677
678
679 NTasks Total number of tasks in a job or step.
680
681
682 Priority Slurm priority.
683
684
685 Partition Identifies the partition on which the job ran.
686
687
688 QOS Name of Quality of Service.
689
690
691 QOSRAW Numeric id of Quality of Service.
692
693
694 ReqCPUFreq
695 Requested CPU frequency for the step, in kHz. Note:
696 This value applies only to a job step. No value is
697 reported for the job.
698
699
700 ReqCPUS Number of requested CPUs.
701
702
703 ReqGRES Names and counts of generic resources requested.
704
705
706 ReqMem Minimum required memory for the job, in MB. A 'c' at
707 the end of number represents Memory Per CPU, a 'n'
708 represents Memory Per Node. Note: This value is only
709 from the job allocation, not the step.
710
711
712 ReqNodes Requested minimum Node count for the job/step.
713
714
715 ReqTres Trackable resources. These are the minimum resource
716 counts requested by the job/step at submission time.
717 For more details see AccountingStorageTRES in
718 slurm.conf.
719
720
721 Reservation
722 Reservation Name.
723
724
725 ReservationId
726 Reservation Id.
727
728
729 Reserved How much wall clock time was used as reserved time for
730 this job. This is derived from how long a job was
731 waiting from eligible time to when it actually
732 started. Format is the same as Elapsed.
733
734
735 ResvCPU How many CPU seconds were used as reserved time for
736 this job. Format is the same as Elapsed.
737
738
739 ResvCPURAW
740 How many CPU seconds were used as reserved time for
741 this job. Format is in processor seconds.
742
743
744 Start Initiation time of the job in the same format as End.
745
746
747 State Displays the job status, or state. See the JOB STATE
748 CODES section below for a list of possible states.
749
750 If more information is available on the job state than
751 will fit into the current field width (for example,
752 the uid that CANCELLED a job) the state will be fol‐
753 lowed by a "+". You can increase the size of the dis‐
754 played state using the "%NUMBER" format modifier
755 described earlier.
756
757 NOTE: The RUNNING state will return suspended jobs as
758 well. In order to print suspended jobs you must
759 request SUSPENDED at a different call from RUNNING.
760
761 NOTE: The RUNNING state will return any jobs completed
762 (cancelled or otherwise) in the time period requested
763 as the job was also RUNNING during that time. If you
764 are only looking for jobs that finished, please choose
765 the appropriate state(s) without the RUNNING state.
766
767
768 Submit The time the job was submitted in the same format as
769 End.
770
771 NOTE: If a job is requeued, the submit time is reset.
772 To obtain the original submit time it is necessary to
773 use the -D or --duplicate option to display all dupli‐
774 cate entries for a job.
775
776
777 Suspended The amount of time a job or job step was suspended.
778 Format is the same as Elapsed.
779
780
781 SystemCPU The amount of system CPU time used by the job or job
782 step. Format is the same as Elapsed.
783
784 NOTE: SystemCPU provides a measure of the task's par‐
785 ent process and does not include CPU time of child
786 processes.
787
788
789 Timelimit What the timelimit was/is for the job. Format is the
790 same as Elapsed.
791
792
793 TimelimitRaw
794 What the timelimit was/is for the job. Format is in
795 number of minutes.
796
797
798 TotalCPU The sum of the SystemCPU and UserCPU time used by the
799 job or job step. The total CPU time of the job may
800 exceed the job's elapsed time for jobs that include
801 multiple job steps. Format is the same as Elapsed.
802
803 NOTE: TotalCPU provides a measure of the task's parent
804 process and does not include CPU time of child pro‐
805 cesses.
806
807
808 TresUsageInAve
809 Tres average usage in by all tasks in job. NOTE: If
810 corresponding TresUsageInMaxTask is -1 the metric is
811 node centric instead of task.
812
813
814 TresUsageInMax
815 Tres maximum usage in by all tasks in job. NOTE: If
816 corresponding TresUsageInMaxTask is -1 the metric is
817 node centric instead of task.
818
819
820 TresUsageInMaxNode
821 Node for which each maximum TRES usage out occurred.
822
823
824 TresUsageInMaxTask
825 Task for which each maximum TRES usage out occurred.
826
827
828 TresUsageOutAve
829 Tres average usage out by all tasks in job. NOTE: If
830 corresponding TresUsageOutMaxTask is -1 the metric is
831 node centric instead of task.
832
833
834 TresUsageOutMax
835 Tres maximum usage out by all tasks in job. NOTE: If
836 corresponding TresUsageOutMaxTask is -1 the metric is
837 node centric instead of task.
838
839
840 TresUsageOutMaxNode
841 Node for which each maximum TRES usage out occurred.
842
843
844 TresUsageOutMaxTask
845 Task for which each maximum TRES usage out occurred.
846
847
848 UID The user identifier of the user who ran the job.
849
850
851 User The user name of the user who ran the job.
852
853
854 UserCPU The amount of user CPU time used by the job or job
855 step. Format is the same as Elapsed.
856
857 NOTE: UserCPU provides a measure of the task's parent
858 process and does not include CPU time of child pro‐
859 cesses.
860
861
862 WCKey Workload Characterization Key. Arbitrary string
863 for grouping orthogonal accounts together.
864
865
866 WCKeyID Reference to the wckey.
867
868
869
871 BF BOOT_FAIL Job terminated due to launch failure, typically due
872 to a hardware failure (e.g. unable to boot the node
873 or block and the job can not be requeued).
874
875 CA CANCELLED Job was explicitly cancelled by the user or system
876 administrator. The job may or may not have been
877 initiated.
878
879 CD COMPLETED Job has terminated all processes on all nodes with
880 an exit code of zero.
881
882 DL DEADLINE Job terminated on deadline.
883
884 F FAILED Job terminated with non-zero exit code or other
885 failure condition.
886
887 NF NODE_FAIL Job terminated due to failure of one or more allo‐
888 cated nodes.
889
890 OOM OUT_OF_MEMORY Job experienced out of memory error.
891
892 PD PENDING Job is awaiting resource allocation.
893
894 PR PREEMPTED Job terminated due to preemption.
895
896 R RUNNING Job currently has an allocation.
897
898 RQ REQUEUED Job was requeued.
899
900 RS RESIZING Job is about to change size.
901
902 RV REVOKED Sibling was removed from cluster due to other clus‐
903 ter starting the job.
904
905 S SUSPENDED Job has an allocation, but execution has been sus‐
906 pended and CPUs have been released for other jobs.
907
908 TO TIMEOUT Job terminated upon reaching its time limit.
909
910
912 The options --starttime and --endtime define the time window between
913 which sacct is going to search. For historical and practical reasons
914 their default values (i.e. the default time window) depends on other
915 options: --jobs and --state.
916
917 Depending on if --jobs and/or --state are specified, the default values
918 of --starttime and --endtime options are:
919
920
921 WITHOUT EITHER --jobs NOR --state specified:
922
923 ---starttime
924 Defaults to Midnight.
925
926 ---endtime
927 Defaults to Now.
928
929
930 WITH --jobs AND WITHOUT --state specified:
931
932 ---starttime
933 Defaults to Epoch 0.
934
935 ---endtime
936 Defaults to Now.
937
938
939 WITHOUT --jobs AND WITH --state specified:
940
941 ---starttime
942 Defaults to Now.
943
944 ---endtime
945 Defaults to --starttime and to Now if --starttime is not
946 specified.
947
948
949 WITH BOTH --jobs AND --state specified:
950
951 ---starttime
952 Defaults to Epoch 0.
953
954 ---endtime
955 Defaults to --starttime or to Now if --starttime is not
956 specified.
957
958
959 NOTE: With -v/--verbose a message about the actual time window in use
960 is shown.
961
962
964 Some sacct options may be set via environment variables. These environ‐
965 ment variables, along with their corresponding options, are listed
966 below. (Note: Commandline options will always override these settings.)
967
968 SACCT_FEDERATION Same as --federation
969
970 SACCT_LOCAL Same as --local
971
972 SLURM_CONF The location of the Slurm configuration file.
973
974 SLURM_TIME_FORMAT Specify the format used to report time stamps. A
975 value of standard, the default value, generates
976 output in the form
977 "year-month-dateThour:minute:second". A value of
978 relative returns only "hour:minute:second" if the
979 current day. For other dates in the current year
980 it prints the "hour:minute" preceded by "Tomorr"
981 (tomorrow), "Ystday" (yesterday), the name of the
982 day for the coming week (e.g. "Mon", "Tue", etc.),
983 otherwise the date (e.g. "25 Apr"). For other
984 years it returns a date month and year without a
985 time (e.g. "6 Jun 2012"). All of the time stamps
986 use a 24 hour format.
987
988 A valid strftime() format can also be specified.
989 For example, a value of "%a %T" will report the day
990 of the week and a time stamp (e.g. "Mon 12:34:56").
991
992
994 This example illustrates the default invocation of the sacct command:
995
996 # sacct
997 Jobid Jobname Partition Account AllocCPUS State ExitCode
998 ---------- ---------- ---------- ---------- ---------- ---------- --------
999 2 script01 srun acct1 1 RUNNING 0
1000 3 script02 srun acct1 1 RUNNING 0
1001 4 endscript srun acct1 1 RUNNING 0
1002 4.0 srun acct1 1 COMPLETED 0
1003
1004
1005 This example shows the same job accounting information with the brief
1006 option.
1007
1008 # sacct --brief
1009 Jobid State ExitCode
1010 ---------- ---------- --------
1011 2 RUNNING 0
1012 3 RUNNING 0
1013 4 RUNNING 0
1014 4.0 COMPLETED 0
1015
1016 # sacct --allocations
1017 Jobid Jobname Partition Account AllocCPUS State ExitCode
1018 ---------- ---------- ---------- ---------- ------- ---------- --------
1019 3 sja_init andy acct1 1 COMPLETED 0
1020 4 sjaload andy acct1 2 COMPLETED 0
1021 5 sja_scr1 andy acct1 1 COMPLETED 0
1022 6 sja_scr2 andy acct1 18 COMPLETED 2
1023 7 sja_scr3 andy acct1 18 COMPLETED 0
1024 8 sja_scr5 andy acct1 2 COMPLETED 0
1025 9 sja_scr7 andy acct1 90 COMPLETED 1
1026 10 endscript andy acct1 186 COMPLETED 0
1027
1028
1029 This example demonstrates the ability to customize the output of the
1030 sacct command. The fields are displayed in the order designated on the
1031 command line.
1032
1033 # sacct --format=jobid,elapsed,ncpus,ntasks,state
1034 Jobid Elapsed Ncpus Ntasks State
1035 ---------- ---------- ---------- -------- ----------
1036 3 00:01:30 2 1 COMPLETED
1037 3.0 00:01:30 2 1 COMPLETED
1038 4 00:00:00 2 2 COMPLETED
1039 4.0 00:00:01 2 2 COMPLETED
1040 5 00:01:23 2 1 COMPLETED
1041 5.0 00:01:31 2 1 COMPLETED
1042
1043
1044 This example demonstrates the use of the -T (--truncate) option when
1045 used with -S (--starttime) and -E (--endtime). When the -T option is
1046 used, the start time of the job will be the specified -S value if the
1047 job was started before the specified time, otherwise the time will be
1048 the job's start time. The end time will be the specified -E option if
1049 the job ends after the specified time, otherwise it will be the jobs
1050 end time.
1051
1052 NOTE: If no -s (--state) option is given sacct will display jobs that
1053 ran during the specified time, otherwise it returns jobs that were in
1054 the state requested during that period of time.
1055
1056 Without -T (normal operation) sacct output would be like this.
1057
1058 # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1059 JobID Start End State
1060 --------- --------------------- -------------------- ------------
1061 2 2014-07-03T11:33:16 2014-07-03T11:59:01 COMPLETED
1062 3 2014-07-03T11:35:21 Unknown RUNNING
1063 4 2014-07-03T11:35:21 2014-07-03T11:45:21 COMPLETED
1064 5 2014-07-03T11:41:01 Unknown RUNNING
1065
1066 By adding the -T option the job's start and end times are truncated to
1067 reflect only the time requested. If a job started after the start time
1068 requested or finished before the end time requested those times are not
1069 altered. The -T option is useful when determining exact run times dur‐
1070 ing any given period.
1071
1072 # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1073 JobID Start End State
1074 --------- --------------------- -------------------- ------------
1075 2 2014-07-03T11:40:00 2014-07-03T11:59:01 COMPLETED
1076 3 2014-07-03T11:40:00 2014-07-03T12:00:00 RUNNING
1077 4 2014-07-03T11:40:00 2014-07-03T11:45:21 COMPLETED
1078 5 2014-07-03T11:41:01 2014-07-03T12:00:00 RUNNING
1079
1080
1081
1083 Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company
1084 L.P.
1085 Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
1086 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1087 Copyright (C) 2010-2014 SchedMD LLC.
1088
1089 This file is part of Slurm, a resource management program. For
1090 details, see <https://slurm.schedmd.com/>.
1091
1092 Slurm is free software; you can redistribute it and/or modify it under
1093 the terms of the GNU General Public License as published by the Free
1094 Software Foundation; either version 2 of the License, or (at your
1095 option) any later version.
1096
1097 Slurm is distributed in the hope that it will be useful, but WITHOUT
1098 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1099 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
1100 for more details.
1101
1103 /etc/slurm.conf
1104 Entries to this file enable job accounting and designate the
1105 job accounting log file that collects system job accounting.
1106
1107 /var/log/slurm_accounting.log
1108 The default job accounting log file. By default, this file
1109 is set to read and write permission for root only.
1110
1112 sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1113
1114
1115
1116May 2019 Slurm Commands sacct(1)