1sacct(1)                        Slurm Commands                        sacct(1)
2
3
4

NAME

6       sacct  -  displays  accounting  data  for all jobs and job steps in the
7       Slurm job accounting log or Slurm database
8
9

SYNOPSIS

11       sacct [OPTIONS...]
12
13

DESCRIPTION

15       Accounting information for jobs invoked with Slurm are either logged in
16       the  job accounting log file or saved to the Slurm database, as config‐
17       ured with the AccountingStorageType parameter.
18
19       The sacct command displays job accounting data stored in  the  job  ac‐
20       counting  log  file  or  Slurm  database in a variety of forms for your
21       analysis.  The sacct command displays information on jobs,  job  steps,
22       status,  and  exitcodes by default.  You can tailor the output with the
23       use of the --format= option to specify the fields to be shown.
24
25       For the root user, the sacct command displays job accounting  data  for
26       all  users,  although  there are options to filter the output to report
27       only the jobs from a specified user or group.
28
29       For the non-root user, the sacct command limits the display of job  ac‐
30       counting data to jobs that were launched with their own user identifier
31       (UID) by default.  Data for other users can be displayed with the --al‐
32       lusers, --user, or --uid options.
33
34       Elapsed  time fields are presented as [days-]hours:minutes:seconds[.mi‐
35       croseconds]. Only 'CPU' fields will ever have microseconds.
36
37       The default input file is the file named  in  the  AccountingStorageLoc
38       parameter in slurm.conf.
39
40       NOTE:  If  designated, the slurmdbd.conf option PrivateData may further
41       restrict the accounting data visible to users which are not  SlurmUser,
42       root,  or  a user with AdminLevel=Admin. See the slurmdbd.conf man page
43       for additional details on restricting access to accounting data.
44
45       NOTE: The contents of Slurm's database are maintained  in  lower  case.
46       This may result in some sacct output differing from that of other Slurm
47       commands.
48
49       NOTE: Much of the data reported by sacct  has  been  generated  by  the
50       wait3()  and  getrusage()  system calls. Some systems gather and report
51       incomplete information for these calls; sacct reports values of  0  for
52       this missing data. See your systems getrusage (3) man page for informa‐
53       tion about which data are actually available on your system.
54
55

OPTIONS

57       -A, --accounts=<account_list>
58                 Displays jobs when a comma separated  list  of  accounts  are
59                 given as the argument.
60
61       -L, --allclusters
62                 Display  jobs  ran on all clusters. By default, only jobs ran
63                 on the cluster from where sacct is called are displayed.
64
65
66       -X, --allocations
67                 Only show statistics relevant to the job  allocation  itself,
68                 not taking steps into consideration.
69
70                 NOTE: Without including steps, utilization statistics for job
71                 allocation(s) will be reported as zero.
72
73
74       -a, --allusers
75                 Displays all users' jobs when run by user root or if Private‐
76                 Data  is  not configured to jobs.  Otherwise display the cur‐
77                 rent user's jobs
78
79       -x, --associations=<assoc_list>
80                 Displays the statistics only for the jobs running  under  the
81                 association ids specified by the assoc_list operand, which is
82                 a comma-separated list of association ids.  Space  characters
83                 are  not  allowed  in the assoc_list. Default is all associa‐
84                 tions.
85
86
87       -B, --batch-script
88                 This option will print the batch script of  job  if  the  job
89                 used one. If the job didn't have a script 'NONE' is output.
90                 NOTE: AccountingStoreFlags=job_script is required for this.
91                 NOTE:  Requesting  specific  job(s) with '-j' is required for
92                 this.
93
94       -b, --brief
95                 Displays a brief listing consisting of JobID, State, and  Ex‐
96                 itCode.
97
98       -M, --clusters=<cluster_list>
99                 Displays  the  statistics  only  for  the jobs started on the
100                 clusters specified by the cluster_list operand,  which  is  a
101                 comma-separated  list  of clusters.  Space characters are not
102                 allowed in the cluster_list.  A value of 'all' will query  to
103                 run  on all clusters.  The default is current cluster you are
104                 executing the sacct command on or all clusters in the federa‐
105                 tion  when  executed on a federated cluster.  This option im‐
106                 plicitly sets the --local option.
107
108
109       -c, --completion
110                 Use job completion data instead of job accounting.  The  Job‐
111                 CompType  parameter in the slurm.conf file must be defined to
112                 a non-none option.  Does not support federated cluster infor‐
113                 mation (local data only).
114
115       -C, --constraints=<constraint_list>
116                 Comma  separated  list  to  filter  jobs  based  on what con‐
117                 straints/features the job requested.  Multiple  options  will
118                 be  treated as 'and' not 'or', so the job would need all con‐
119                 straints specified to be returned not one or the other.
120
121
122       --delimiter=<characters>
123                 ASCII characters used to separate the fields when  specifying
124                 the  -p  or  -P options. The default delimiter is a '|'. This
125                 option is ignored if -p or -P options are not specified.
126
127
128       -D, --duplicates
129                 If Slurm job ids are reset, some job  numbers  will  probably
130                 appear more than once in the accounting log file but refer to
131                 different jobs.  Such jobs can be distinguished by the  "sub‐
132                 mit" time stamp in the data records.
133
134
135                 When data for specific jobs are requested with the --jobs op‐
136                 tion, sacct returns the most recent  job  with  that  number.
137                 This  behavior  can be overridden by specifying --duplicates,
138                 in which case all records that match the  selection  criteria
139                 will be returned.
140
141
142                 NOTE:  Revoked  federated  sibling jobs are hidden unless the
143                 --duplicates option is specified.
144
145
146       -E, --endtime=<end_time>
147
148                 Select jobs in any  state  before  the  specified  time.   If
149                 states are given with the -s option return jobs in this state
150                 before this period.  See the DEFAULT TIME WINDOW for more de‐
151                 tails.
152
153                 Valid time formats are:
154                 HH:MM[:SS][AM|PM]
155                 MMDD[YY][-HH:MM[:SS]]
156                 MM.DD[.YY][-HH:MM[:SS]]
157                 MM/DD[/YY][-HH:MM[:SS]]
158                 YYYY-MM-DD[THH:MM[:SS]]
159                 today, midnight, noon, fika (3 PM), teatime (4 PM)
160                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
161
162       --env-vars
163                 This  option  will  print  the running environment of a batch
164                 job, otherwise 'NONE' is output.
165                 NOTE: AccountingStoreFlags=job_env is required for this.
166                 NOTE: Requesting specific job(s) with '-j'  is  required  for
167                 this.
168
169       --federation
170                 Show jobs from the federation if a member of one.
171
172
173       -f, --file=<file>
174                 Causes the sacct command to read job accounting data from the
175                 named file instead of the current Slurm  job  accounting  log
176                 file.   Only  applicable  when  running  the  jobcomp/filetxt
177                 plugin.
178
179
180       -F, --flags=<flag_list>
181                 Comma separated list to filter jobs  based  on  what  various
182                 ways  the  jobs were handled.  Current flags are SchedSubmit,
183                 SchedMain, SchedBackfill.  These particular options  describe
184                 the scheduler that started the job.
185
186
187       -o, --format
188                 Comma  separated  list  of  fields. (use "--helpformat" for a
189                 list of available fields).
190
191                 NOTE: When using the format option for listing various fields
192                 you  can put a %NUMBER afterwards to specify how many charac‐
193                 ters should be printed.
194
195                 e.g. format=name%30 will print 30 characters  of  field  name
196                 right justified.  A %-30 will print 30 characters left justi‐
197                 fied.
198
199                 When set, the SACCT_FORMAT environment variable will override
200                 the default format.  For example:
201
202                 SACCT_FORMAT="jobid,user,account,cluster"
203
204
205       -g, --gid=, --group=<gid_or_group_list>
206                 Displays  the  statistics  only for the jobs started with the
207                 GID or the GROUP specified by the gid_list or the  group_list
208                 operand,  which  is a comma-separated list.  Space characters
209                 are not allowed.  Default is no restrictions.
210
211
212       -h, --help
213                 Displays a general help message.
214
215
216       -e, --helpformat
217
218                 Print a list of fields that can be specified with the  --for‐
219                 mat option.
220
221
222                 Fields available:
223
224                 Account             AdminComment        AllocCPUS           AllocNodes
225                 AllocTRES           AssocID             AveCPU              AveCPUFreq
226                 AveDiskRead         AveDiskWrite        AvePages            AveRSS
227                 AveVMSize           BlockID             Cluster             Comment
228                 Constraints         Container           ConsumedEnergy      ConsumedEnergyRaw
229                 CPUTime             CPUTimeRAW          DBIndex             DerivedExitCode
230                 Elapsed             ElapsedRaw          Eligible            End
231                 ExitCode            Flags               GID                 Group
232                 JobID               JobIDRaw            JobName             Layout
233                 MaxDiskRead         MaxDiskReadNode     MaxDiskReadTask     MaxDiskWrite
234                 MaxDiskWriteNode    MaxDiskWriteTask    MaxPages            MaxPagesNode
235                 MaxPagesTask        MaxRSS              MaxRSSNode          MaxRSSTask
236                 MaxVMSize           MaxVMSizeNode       MaxVMSizeTask       McsLabel
237                 MinCPU              MinCPUNode          MinCPUTask          NCPUS
238                 NNodes              NodeList            NTasks              Priority
239                 Partition           QOS                 QOSRAW              Reason
240                 ReqCPUFreq          ReqCPUFreqMin       ReqCPUFreqMax       ReqCPUFreqGov
241                 ReqCPUS             ReqMem              ReqNodes            ReqTRES
242                 Reservation         ReservationId       Reserved            ResvCPU
243                 ResvCPURAW          Start               State               Submit
244                 SubmitLine          Suspended           SystemCPU           SystemComment
245                 Timelimit           TimelimitRaw        TotalCPU            TRESUsageInAve
246                 TRESUsageInMax      TRESUsageInMaxNode  TRESUsageInMaxTask  TRESUsageInMin
247                 TRESUsageInMinNode  TRESUsageInMinTask  TRESUsageInTot      TRESUsageOutAve
248                 TRESUsageOutMax     TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
249                 TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot     UID
250                 User                UserCPU             WCKey               WCKeyID
251                 WorkDir
252
253                 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
254                 SUsageIn[Ave|Tot].  They represent the average/total  of  the
255                 highest  watermarks  over  all ranks in the step.  When using
256                 sstat they represent the average/total at the moment the com‐
257                 mand was run.
258
259                 NOTE:  TRESUsage*Min*  values  represent the lowest highwater
260                 mark in the step.
261
262                 The section titled "Job Accounting  Fields"  describes  these
263                 fields.
264
265
266       -j, --jobs=<job[.step]>
267                 Displays  information  about the specified job[.step] or list
268                 of job[.step]s.
269
270                 The job[.step] parameter is a comma-separated list  of  jobs.
271                 Space characters are not permitted in this list.
272                 NOTE: A step id of 'batch' will display the information about
273                 the batch step.
274
275                 By default sacct shows only jobs with Eligible time, but with
276                 this option the non-eligible will be also shown.
277                 NOTE:  If  --state is also specified, as non-eligible are not
278                 PD, then non-eligible jobs will not be  displayed.   See  the
279                 DEFAULT TIME WINDOW for details about how this option changes
280                 the default -S and -E options.
281
282
283       --json    Dump job information as JSON. All other  formating  arugments
284                 will be ignored.
285
286
287       --local   Show  only  jobs local to this cluster. Ignore other clusters
288                 in this federation (if any). Overrides --federation.
289
290
291       -l, --long
292                 Equivalent to specifying:
293
294                 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
295                 sizenode, maxvmsizetask,avevm‐
296                 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
297                 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
298                 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
299                 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
300                 gov,reqmem,consumeden‐
301                 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
302                 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
303                 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
304                 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
305                 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
306                 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
307                 outave,tresusageouttot
308
309
310       --name=<jobname_list>
311                 Display jobs that have any of these name(s).
312
313       -i, --nnodes=<min[-max]>
314                 Return jobs that ran on the specified number of nodes.
315
316
317       --noconvert
318                 Don't  convert  units  from  their  original type (e.g. 2048M
319                 won't be converted to 2G).
320
321       -N, --nodelist=<node_list>
322                 Display jobs that ran on any of these node(s). node_list  can
323                 be a ranged string.
324
325       -n, --noheader
326                 No heading will be added to the output. The default action is
327                 to display a header.
328
329       -p, --parsable
330                 Output will be '|' delimited with a '|' at the end. See  also
331                 the --delimiter option.
332
333
334       -P, --parsable2
335                 Output  will  be  '|' delimited without a '|' at the end. See
336                 also the --delimiter option.
337
338
339       -r, --partition
340                 Comma separated list of partitions to  select  jobs  and  job
341                 steps from. The default is all partitions.
342
343
344       -q, --qos Only send data about jobs using these qos.  Default is all.
345
346
347       -R, --reason=<reason_list>
348                 Comma  separated list to filter jobs based on what reason the
349                 job wasn't scheduled outside resources/priority.
350
351
352       -S, --starttime
353                 Select jobs in any state after the specified time. Default is
354                 00:00:00  of the current day, unless the '-s' or '-j' options
355                 are used. If the '-s' option is used,  then  the  default  is
356                 'now'.  If  states  are  given with the '-s' option then only
357                 jobs in this state at this time will be returned. If the '-j'
358                 option  is  used,  then the default time is Unix Epoch 0. See
359                 the DEFAULT TIME WINDOW for more details.
360
361                 Valid time formats are:
362                 HH:MM[:SS][AM|PM]
363                 MMDD[YY][-HH:MM[:SS]]
364                 MM.DD[.YY][-HH:MM[:SS]]
365                 MM/DD[/YY][-HH:MM[:SS]]
366                 YYYY-MM-DD[THH:MM[:SS]]
367                 today, midnight, noon, fika (3 PM), teatime (4 PM)
368                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
369
370
371       -s, --state=<state_list>
372                 Selects jobs based on their  state  during  the  time  period
373                 given.   Unless  otherwise  specified, the start and end time
374                 will be the current time when the --state option is specified
375                 and  only  currently  running jobs can be displayed.  A start
376                 and/or end time must be specified to view  information  about
377                 jobs  not currently running.  See the JOB STATE CODES section
378                 below for a list of state designators.  Multiple state  names
379                 may  be specified using comma separators. Either the short or
380                 long form of the state name may be  used  (e.g.  CA  or  CAN‐
381                 CELLED) and the name is case insensitive (i.e. ca and CA both
382                 work).
383
384                 NOTE: Note for a job to be selected in the PENDING  state  it
385                 must  have  "EligibleTime"  in the requested time interval or
386                 different from "Unknown". The "EligibleTime" is displayed  by
387                 the  "scontrol show job" command.  For example jobs submitted
388                 with the "--hold" option will have "EligibleTime=Unknown"  as
389                 they are pending indefinitely.
390
391                 NOTE:  When  specifying states and no start time is given the
392                 default start time is 'now'.  This is only  when  -j  is  not
393                 used.   If -j is used the start time will default to 'Epoch'.
394                 In both cases if no end time is  given  it  will  default  to
395                 'now'. See the DEFAULT TIME WINDOW for more details.
396
397
398       -K, --timelimit-max
399                 Ignored  by  itself, but if timelimit_min is set this will be
400                 the maximum timelimit of the range.  Default is  no  restric‐
401                 tion.
402
403
404       -k, --timelimit-min
405                 Only  send data about jobs with this timelimit.  If used with
406                 timelimit_max this will  be  the  minimum  timelimit  of  the
407                 range.  Default is no restriction.
408
409
410       -T, --truncate
411                 Truncate  time.   So  if a job started before --starttime the
412                 start time would be truncated to --starttime.  The  same  for
413                 end time and --endtime.
414
415
416       -u, --uid=, --user=<uid_or_user_list>
417                 Use this comma separated list of UIDs or user names to select
418                 jobs to display.  By default, the running user's UID is used.
419
420
421       --units=[KMGTP]
422                 Display values in specified unit type. Takes precedence  over
423                 --noconvert option.
424
425
426       --usage   Display a command usage summary.
427
428
429       --use-local-uid
430                 When displaying UID, sacct uses the UID stored in Slurm's ac‐
431                 counting database by default. Use this command to make  Slurm
432                 use  a system call to get the UID from the username. This op‐
433                 tion may be useful in an environment with  multiple  clusters
434                 and one database where the UID's aren't the same on all clus‐
435                 ters.
436
437
438       -v, --verbose
439                 Primarily for debugging purposes, report the state of various
440                 variables during processing.
441
442
443       -V, --version
444                 Print version.
445
446
447       -W, --wckeys=<wckey_list>
448                 Displays the statistics only for the jobs started on the wck‐
449                 eys  specified  by  the  wckey_list  operand,  which   is   a
450                 comma-separated  list  of  wckey names.  Space characters are
451                 not allowed in the wckey_list. Default is all wckeys.
452
453
454       --whole-hetjob[=yes|no]
455                 When querying and filtering heterogeneous jobs  with  --jobs,
456                 Slurm  will  default  to retrieving information about all the
457                 components of the job if the het_job_id (leader  id)  is  se‐
458                 lected. If a non-leader heterogeneous job component id is se‐
459                 lected then only that component is retrieved by default. This
460                 behavior can be changed by using this option. If set to 'yes'
461                 (or no argument), then information about all  the  components
462                 will  be  retrieved  no matter which component is selected in
463                 the job filter. If set to 'no' then only the selected hetero‐
464                 geneous job component(s) will be retrieved, even when select‐
465                 ing the leader.
466
467
468       --yaml    Dump job information as YAML. All other  formating  arugments
469                 will be ignored.
470
471
472   Job Accounting Fields
473       The following describes each job accounting field:
474
475              ALL       Print all fields listed below.
476
477
478              AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
479
480
481              AllocNodes
482                        Number  of  nodes allocated to the job/step.  0 if the
483                        job is pending.
484
485
486              AllocTres Trackable resources. These are the resources allocated
487                        to  the  job/step  after the job started running.  For
488                        pending jobs this should be blank.  For  more  details
489                        see AccountingStorageTRES in slurm.conf.
490
491                        NOTE:  When  a generic resource is configured with the
492                        no_consume flag, the allocation will be printed with a
493                        zero.
494
495
496              Account   Account the job ran under.
497
498
499              AssocID   Reference  to  the  association  of  user, account and
500                        cluster.
501
502
503              AveCPU    Average (system + user) CPU time of all tasks in job.
504
505
506              AveCPUFreq
507                        Average weighted CPU frequency of all tasks in job, in
508                        kHz.
509
510
511              AveDiskRead
512                        Average number of bytes read by all tasks in job.
513
514
515              AveDiskWrite
516                        Average number of bytes written by all tasks in job.
517
518
519              AvePages  Average number of page faults of all tasks in job.
520
521
522              AveRSS    Average resident set size of all tasks in job.
523
524
525              AveVMSize Average Virtual Memory size of all tasks in job.
526
527
528              Cluster   Cluster name.
529
530
531              Comment   The job's comment string when the AccountingStoreFlags
532                        parameter in the slurm.conf  file  contains  'job_com‐
533                        ment'.  The Comment string can be modified by invoking
534                        sacctmgr modify job  or  the  specialized  sjobexitmod
535                        command.
536
537
538              ConsumedEnergy
539                        Total  energy consumed by all tasks in job, in joules.
540                        Note: Only in case of exclusive  job  allocation  this
541                        value reflects the jobs' real energy consumption.
542
543
544              Container Path to OCI Container Bundle requested.
545
546
547              CPUTime   Time  used (Elapsed time * CPU count) by a job or step
548                        in HH:MM:SS format.
549
550
551              CPUTimeRAW
552                        Time used (Elapsed time * CPU count) by a job or  step
553                        in cpu-seconds.
554
555
556              DerivedExitCode
557                        The  highest exit code returned by the job's job steps
558                        (srun invocations).  Following the colon is the signal
559                        that  caused the process to terminate if it was termi‐
560                        nated by a signal.  The DerivedExitCode can  be  modi‐
561                        fied  by  invoking sacctmgr modify job or the special‐
562                        ized sjobexitmod command.
563
564
565              Elapsed   The job's elapsed time.
566
567                        The format of this field's output is as follows:
568                        [DD-[HH:]]MM:SS
569
570                        as defined by the following:
571
572                        DD        days
573
574                        hh        hours
575
576                        mm        minutes
577
578                        ss        seconds
579
580
581              Eligible  When the job became eligible to run in the same format
582                        as End.
583
584
585              End       Termination   time  of  the  job.  Format  output  is,
586                        YYYY-MM-DDTHH:MM:SS,  unless   changed   through   the
587                        SLURM_TIME_FORMAT environment variable.
588
589
590              ExitCode  The  exit  code  returned by the job script or salloc,
591                        typically as set by the  exit()  function.   Following
592                        the  colon  is  the  signal that caused the process to
593                        terminate if it was terminated by a signal.
594
595
596              GID       The group identifier of the user who ran the job.
597
598
599              Group     The group name of the user who ran the job.
600
601
602              JobID     The identification number of the job or job step.
603
604                        Regular jobs are in the form:
605
606                        JobID[.JobStep]
607
608                        Array jobs are in the form:
609
610                        ArrayJobID_ArrayTaskID
611
612                        Heterogeneous jobs are in the form:
613
614                        HetJobID+HetJobOffset
615
616                        When printing job arrays, performance of  the  command
617                        can be measurably improved for systems with large num‐
618                        bers of jobs when a single job ID is specified. By de‐
619                        fault,  this  field  size will be limited to 64 bytes.
620                        Use the environment variable SLURM_BITSTR_LEN to spec‐
621                        ify larger field sizes.
622
623
624              JobIDRaw  The  identification  number  of  the  job or job step.
625                        Prints the JobID in the form JobID[.JobStep] for regu‐
626                        lar, heterogeneous and array jobs.
627
628
629              JobName   The  name  of  the job or job step. The slurm_account‐
630                        ing.log file is a space  delimited  file.  Because  of
631                        this  if  a space is used in the jobname an underscore
632                        is substituted for the  space  before  the  record  is
633                        written to the accounting file. So when the jobname is
634                        displayed by sacct the jobname that had a space in  it
635                        will now have an underscore in place of the space.
636
637
638              Layout    What  the  layout  of  a step was when it was running.
639                        This can be used to give you an idea of which node ran
640                        which rank in your job.
641
642
643              MaxDiskRead
644                        Maximum number of bytes read by all tasks in job.
645
646
647              MaxDiskReadNode
648                        The node on which the maxdiskread occurred.
649
650
651              MaxDiskReadTask
652                        The task ID where the maxdiskread occurred.
653
654
655              MaxDiskWrite
656                        Maximum number of bytes written by all tasks in job.
657
658
659              MaxDiskWriteNode
660                        The node on which the maxdiskwrite occurred.
661
662
663              MaxDiskWriteTask
664                        The task ID where the maxdiskwrite occurred.
665
666
667              MaxPages  Maximum number of page faults of all tasks in job.
668
669
670              MaxPagesNode
671                        The node on which the maxpages occurred.
672
673
674              MaxPagesTask
675                        The task ID where the maxpages occurred.
676
677
678              MaxRSS    Maximum resident set size of all tasks in job.
679
680
681              MaxRSSNode
682                        The node on which the maxrss occurred.
683
684
685              MaxRSSTask
686                        The task ID where the maxrss occurred.
687
688
689              MaxVMSize Maximum Virtual Memory size of all tasks in job.
690
691
692              MaxVMSizeNode
693                        The node on which the maxvmsize occurred.
694
695
696              MaxVMSizeTask
697                        The task ID where the maxvmsize occurred.
698
699
700              MinCPU    Minimum (system + user) CPU time of all tasks in job.
701
702
703              MinCPUNode
704                        The node on which the mincpu occurred.
705
706
707              MinCPUTask
708                        The task ID where the mincpu occurred.
709
710
711              NCPUS     Total number of CPUs allocated to the job.  Equivalent
712                        to AllocCPUS.
713
714
715              NodeList  List of nodes in job/step.
716
717
718              NNodes    Number of nodes in a job or step.  If the job is  run‐
719                        ning, or ran, this count will be the number allocated,
720                        else the number will be the number requested.
721
722
723              NTasks    Total number of tasks in a job or step.
724
725
726              Priority  Slurm priority.
727
728
729              Partition Identifies the partition on which the job ran.
730
731
732              QOS       Name of Quality of Service.
733
734
735              QOSRAW    Numeric id of Quality of Service.
736
737
738              ReqCPUFreq
739                        Requested CPU frequency for the step, in  kHz.   Note:
740                        This value applies only to a job step. No value is re‐
741                        ported for the job.
742
743
744              ReqCPUFreqGov
745                        Requested CPU frequency governor for the step, in kHz.
746                        Note:  This value applies only to a job step. No value
747                        is reported for the job.
748
749
750              ReqCPUFreqMax
751                        Maximum requested CPU frequency for the step, in  kHz.
752                        Note:  This value applies only to a job step. No value
753                        is reported for the job.
754
755
756              ReqCPUFreqMin
757                        Minimum requested CPU frequency for the step, in  kHz.
758                        Note:  This value applies only to a job step. No value
759                        is reported for the job.
760
761
762              ReqCPUS   Number of requested CPUs.
763
764
765              ReqMem    Minimum required memory for the job.  It  may  have  a
766                        letter   appended   to  it  indicating  units  (M  for
767                        megabytes, G for gigabytes, etc.).  Note:  This  value
768                        is only from the job allocation, not the step.
769
770
771              ReqNodes  Requested minimum Node count for the job/step.
772
773
774              ReqTres   Trackable  resources.  These  are the minimum resource
775                        counts requested by the job/step at  submission  time.
776                        For   more   details   see   AccountingStorageTRES  in
777                        slurm.conf.
778
779
780              Reservation
781                        Reservation Name.
782
783
784              ReservationId
785                        Reservation Id.
786
787
788              Reserved  How much wall clock time was used as reserved time for
789                        this  job.   This  is  derived from how long a job was
790                        waiting  from  eligible  time  to  when  it   actually
791                        started.  Format is the same as Elapsed.
792
793
794              ResvCPU   How  many  CPU  seconds were used as reserved time for
795                        this job.  Format is the same as Elapsed.
796
797
798              ResvCPURAW
799                        How many CPU seconds were used as  reserved  time  for
800                        this job.  Format is in processor seconds.
801
802
803              Start     Initiation time of the job in the same format as End.
804
805
806              State     Displays  the job status, or state.  See the JOB STATE
807                        CODES section below for a list of possible states.
808
809                        If more information is available on the job state than
810                        will  fit  into  the current field width (for example,
811                        the UID that CANCELLED a job) the state will  be  fol‐
812                        lowed by a "+".  You can increase the size of the dis‐
813                        played state using the "%NUMBER" format  modifier  de‐
814                        scribed earlier.
815
816                        NOTE:  The RUNNING state will return suspended jobs as
817                        well.  In order to print suspended jobs you  must  re‐
818                        quest SUSPENDED at a different call from RUNNING.
819
820                        NOTE: The RUNNING state will return any jobs completed
821                        (cancelled or otherwise) in the time period  requested
822                        as  the job was also RUNNING during that time.  If you
823                        are only looking for jobs that finished, please choose
824                        the appropriate state(s) without the RUNNING state.
825
826
827              Submit    The  time  the job was submitted in the same format as
828                        End.
829
830                        NOTE: If a job is requeued, the submit time is  reset.
831                        To  obtain the original submit time it is necessary to
832                        use the -D or --duplicate option to display all dupli‐
833                        cate entries for a job.
834
835
836              SubmitLine
837                        The full command issued to submit the job.
838
839
840              Suspended The  amount  of  time a job or job step was suspended.
841                        Format is the same as Elapsed.
842
843
844              SystemCPU The amount of system CPU time used by the job  or  job
845                        step. Format is the same as Elapsed.
846
847                        NOTE:  SystemCPU provides a measure of the task's par‐
848                        ent process and does not include  CPU  time  of  child
849                        processes.
850
851
852              Timelimit What  the  timelimit was/is for the job. Format is the
853                        same as Elapsed.
854
855
856              TimelimitRaw
857                        What the timelimit was/is for the job.  Format  is  in
858                        number of minutes.
859
860
861              TotalCPU  The  sum of the SystemCPU and UserCPU time used by the
862                        job or job step.  The total CPU time of  the  job  may
863                        exceed  the  job's  elapsed time for jobs that include
864                        multiple job steps. Format is the same as Elapsed.
865
866                        NOTE: TotalCPU provides a measure of the task's parent
867                        process  and  does  not include CPU time of child pro‐
868                        cesses.
869
870
871              TresUsageInAve
872                        Tres average usage in by all tasks in job.   NOTE:  If
873                        corresponding  TresUsageInMaxTask  is -1 the metric is
874                        node centric instead of task.
875
876
877              TresUsageInMax
878                        Tres maximum usage in by all tasks in job.   NOTE:  If
879                        corresponding  TresUsageInMaxTask  is -1 the metric is
880                        node centric instead of task.
881
882
883              TresUsageInMaxNode
884                        Node for which each maximum TRES usage out occurred.
885
886
887              TresUsageInMaxTask
888                        Task for which each maximum TRES usage out occurred.
889
890
891              TresUsageInMin
892                        Tres minimum usage in by all tasks in job.   NOTE:  If
893                        corresponding  TresUsageInMinTask  is -1 the metric is
894                        node centric instead of task.
895
896
897              TresUsageInMinNode
898                        Node for which each minimum TRES usage out occurred.
899
900
901              TresUsageInMinTask
902                        Task for which each minimum TRES usage out occurred.
903
904
905              TresUsageInTot
906                        Tres total usage in by all tasks in job.
907
908
909              TresUsageOutAve
910                        Tres average usage out by all tasks in job.  NOTE:  If
911                        corresponding  TresUsageOutMaxTask is -1 the metric is
912                        node centric instead of task.
913
914
915              TresUsageOutMax
916                        Tres maximum usage out by all tasks in job.  NOTE:  If
917                        corresponding  TresUsageOutMaxTask is -1 the metric is
918                        node centric instead of task.
919
920
921              TresUsageOutMaxNode
922                        Node for which each maximum TRES usage out occurred.
923
924
925              TresUsageOutMaxTask
926                        Task for which each maximum TRES usage out occurred.
927
928
929              TresUsageOutTot
930                        Tres total usage out by all tasks in job.
931
932
933              UID       The user identifier of the user who ran the job.
934
935
936              User      The user name of the user who ran the job.
937
938
939              UserCPU   The amount of user CPU time used by  the  job  or  job
940                        step. Format is the same as Elapsed.
941
942                        NOTE:  UserCPU provides a measure of the task's parent
943                        process and does not include CPU time  of  child  pro‐
944                        cesses.
945
946
947              WCKey     Workload   Characterization   Key.   Arbitrary  string
948                        for grouping orthogonal accounts together.
949
950
951              WCKeyID   Reference to the wckey.
952
953
954

JOB STATE CODES

956       BF  BOOT_FAIL       Job terminated due to launch failure, typically due
957                           to a hardware failure (e.g. unable to boot the node
958                           or block and the job can not be requeued).
959
960       CA  CANCELLED       Job was explicitly cancelled by the user or  system
961                           administrator.   The  job  may or may not have been
962                           initiated.
963
964       CD  COMPLETED       Job has terminated all processes on all nodes  with
965                           an exit code of zero.
966
967       DL  DEADLINE        Job terminated on deadline.
968
969       F   FAILED          Job  terminated  with  non-zero  exit code or other
970                           failure condition.
971
972       NF  NODE_FAIL       Job terminated due to failure of one or more  allo‐
973                           cated nodes.
974
975       OOM OUT_OF_MEMORY   Job experienced out of memory error.
976
977       PD  PENDING         Job is awaiting resource allocation.
978
979       PR  PREEMPTED       Job terminated due to preemption.
980
981       R   RUNNING         Job currently has an allocation.
982
983       RQ  REQUEUED        Job was requeued.
984
985       RS  RESIZING        Job is about to change size.
986
987       RV  REVOKED         Sibling was removed from cluster due to other clus‐
988                           ter starting the job.
989
990       S   SUSPENDED       Job has an allocation, but execution has been  sus‐
991                           pended and CPUs have been released for other jobs.
992
993       TO  TIMEOUT         Job terminated upon reaching its time limit.
994
995

DEFAULT TIME WINDOW

997       The  options  --starttime  and --endtime define the time window between
998       which sacct is going to search. For historical  and  practical  reasons
999       their  default  values  (i.e. the default time window) depends on other
1000       options: --jobs and --state.
1001
1002       Depending on if --jobs and/or --state are specified, the default values
1003       of --starttime  and --endtime options are:
1004
1005
1006       WITHOUT EITHER --jobs NOR --state specified:
1007              --starttime defaults to Midnight.
1008              --endtime defaults to Now.
1009
1010
1011       WITH --jobs AND WITHOUT --state specified:
1012              --starttime defaults to Epoch 0.
1013              --endtime defaults to Now.
1014
1015
1016       WITHOUT --jobs AND WITH --state specified:
1017              --starttime defaults to Now.
1018              --endtime  defaults  to --starttime and to Now if --starttime is
1019              not specified.
1020
1021
1022       WITH BOTH --jobs AND --state specified:
1023              --starttime defaults to Epoch 0.
1024              --endtime defaults to --starttime or to Now  if  --starttime  is
1025              not specified.
1026
1027
1028       NOTE:  With  -v/--verbose a message about the actual time window in use
1029       is shown.
1030
1031

PERFORMANCE

1033       Executing sacct sends a remote procedure call to  slurmdbd.  If  enough
1034       calls from sacct or other Slurm client commands that send remote proce‐
1035       dure calls to the slurmdbd daemon come in at once, it can result  in  a
1036       degradation  of  performance of the slurmdbd daemon, possibly resulting
1037       in a denial of service.
1038
1039       Do not run sacct or other Slurm client commands that send remote proce‐
1040       dure  calls  to slurmdbd from loops in shell scripts or other programs.
1041       Ensure that programs limit calls to sacct to the minimum necessary  for
1042       the information you are trying to gather.
1043
1044

ENVIRONMENT VARIABLES

1046       Some sacct options may be set via environment variables. These environ‐
1047       ment variables, along with their corresponding options, are listed  be‐
1048       low. (Note: Command line options will always override these settings.)
1049
1050       SACCT_FEDERATION    Same as --federation
1051
1052       SACCT_LOCAL         Same as --local
1053
1054
1055       SLURM_BITSTR_LEN    Specifies  the string length to be used for holding
1056                           a job array's task ID expression. The default value
1057                           is  64  bytes. A value of 0 will print the full ex‐
1058                           pression with any length  required.  Larger  values
1059                           may adversely impact the application performance.
1060
1061
1062       SLURM_CONF          The location of the Slurm configuration file.
1063
1064       SLURM_TIME_FORMAT   Specify  the  format  used to report time stamps. A
1065                           value of standard,  the  default  value,  generates
1066                           output            in            the            form
1067                           "year-month-dateThour:minute:second".  A  value  of
1068                           relative  returns  only "hour:minute:second" if the
1069                           current day.  For other dates in the  current  year
1070                           it  prints  the  "hour:minute" preceded by "Tomorr"
1071                           (tomorrow), "Ystday" (yesterday), the name  of  the
1072                           day  for the coming week (e.g. "Mon", "Tue", etc.),
1073                           otherwise the date  (e.g.  "25  Apr").   For  other
1074                           years  it  returns  a date month and year without a
1075                           time (e.g.  "6 Jun 2012"). All of the  time  stamps
1076                           use a 24 hour format.
1077
1078                           A  valid  strftime()  format can also be specified.
1079                           For example, a value of "%a %T" will report the day
1080                           of the week and a time stamp (e.g. "Mon 12:34:56").
1081
1082

EXAMPLES

1084       This example illustrates the default invocation of the sacct command:
1085
1086              # sacct
1087              Jobid      Jobname    Partition    Account AllocCPUS State     ExitCode
1088              ---------- ---------- ---------- ---------- ---------- ---------- --------
1089              2          script01   srun       acct1               1 RUNNING           0
1090              3          script02   srun       acct1               1 RUNNING           0
1091              4          endscript  srun       acct1               1 RUNNING           0
1092              4.0                   srun       acct1               1 COMPLETED         0
1093
1094
1095       This  example  shows the same job accounting information with the brief
1096       option.
1097
1098              # sacct --brief
1099                   Jobid     State  ExitCode
1100              ---------- ---------- --------
1101              2          RUNNING           0
1102              3          RUNNING           0
1103              4          RUNNING           0
1104              4.0        COMPLETED         0
1105
1106              # sacct --allocations
1107              Jobid      Jobname    Partition Account    AllocCPUS  State     ExitCode
1108              ---------- ---------- ---------- ---------- ------- ---------- --------
1109              3          sja_init   andy       acct1            1 COMPLETED         0
1110              4          sjaload    andy       acct1            2 COMPLETED         0
1111              5          sja_scr1   andy       acct1            1 COMPLETED         0
1112              6          sja_scr2   andy       acct1           18 COMPLETED         2
1113              7          sja_scr3   andy       acct1           18 COMPLETED         0
1114              8          sja_scr5   andy       acct1            2 COMPLETED         0
1115              9          sja_scr7   andy       acct1           90 COMPLETED         1
1116              10         endscript  andy       acct1          186 COMPLETED         0
1117
1118
1119       This example demonstrates the ability to customize the  output  of  the
1120       sacct command.  The fields are displayed in the order designated on the
1121       command line.
1122
1123              # sacct --format=jobid,elapsed,ncpus,ntasks,state
1124                   Jobid    Elapsed      Ncpus   Ntasks     State
1125              ---------- ---------- ---------- -------- ----------
1126              3            00:01:30          2        1 COMPLETED
1127              3.0          00:01:30          2        1 COMPLETED
1128              4            00:00:00          2        2 COMPLETED
1129              4.0          00:00:01          2        2 COMPLETED
1130              5            00:01:23          2        1 COMPLETED
1131              5.0          00:01:31          2        1 COMPLETED
1132
1133
1134       This example demonstrates the use of the -T  (--truncate)  option  when
1135       used  with  -S  (--starttime) and -E (--endtime). When the -T option is
1136       used, the start time of the job will be the specified -S value  if  the
1137       job  was  started before the specified time, otherwise the time will be
1138       the job's start time. The end time will be the specified -E  option  if
1139       the  job  ends  after the specified time, otherwise it will be the jobs
1140       end time.
1141
1142       Without -T (normal operation) sacct output would be like this.
1143
1144              # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1145                  JobID                 Start                  End        State
1146              --------- --------------------- -------------------- ------------
1147              2         2014-07-03T11:33:16   2014-07-03T11:59:01   COMPLETED
1148              3         2014-07-03T11:35:21   Unknown               RUNNING
1149              4         2014-07-03T11:35:21   2014-07-03T11:45:21   COMPLETED
1150              5         2014-07-03T11:41:01   Unknown               RUNNING
1151
1152       By adding the -T option the job's start and end times are truncated  to
1153       reflect only the time requested.  If a job started after the start time
1154       requested or finished before the end time requested those times are not
1155       altered.  The -T option is useful when determining exact run times dur‐
1156       ing any given period.
1157
1158              # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1159                  JobID                 Start                  End        State
1160              --------- --------------------- -------------------- ------------
1161              2         2014-07-03T11:40:00   2014-07-03T11:59:01   COMPLETED
1162              3         2014-07-03T11:40:00   2014-07-03T12:00:00   RUNNING
1163              4         2014-07-03T11:40:00   2014-07-03T11:45:21   COMPLETED
1164              5         2014-07-03T11:41:01   2014-07-03T12:00:00   RUNNING
1165
1166
1167       NOTE: If no -s (--state) option is given sacct  will  display  eligible
1168       jobs during the specified period of time, otherwise it will return jobs
1169       that were in the state requested during that period of time.
1170
1171       This example demonstrates the differences running sacct with and  with‐
1172       out  the --state flag for the same time period. Without the --state op‐
1173       tion, all eligible jobs in that time period are shown.
1174
1175              # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1176                     JobID               Start                 End      State
1177              ------------ ------------------- ------------------- ----------
1178              2955                    11:15:12            11:20:12  COMPLETED
1179              2956                    11:20:13            11:25:13  COMPLETED
1180
1181       With the --state=pending option, only job 2956 will be shown because it
1182       had  a  dependency on 2955 and was still PENDING from 11:20:00 until it
1183       started at 11:21:13. Note that even though we requested  PENDING  jobs,
1184       the  State  shows as COMPLETED because that is the current State of the
1185       job.
1186
1187              # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1188                     JobID               Start                 End      State
1189              ------------ ------------------- ------------------- ----------
1190              2956                    11:20:13            11:25:13  COMPLETED
1191

COPYING

1193       Copyright (C) 2005-2007 Copyright Hewlett-Packard  Development  Company
1194       L.P.
1195       Copyright (C) 2008-2010 Lawrence Livermore National Security.  Produced
1196       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1197       Copyright (C) 2010-2021 SchedMD LLC.
1198
1199       This file is part of Slurm, a resource  management  program.   For  de‐
1200       tails, see <https://slurm.schedmd.com/>.
1201
1202       Slurm  is free software; you can redistribute it and/or modify it under
1203       the terms of the GNU General Public License as published  by  the  Free
1204       Software  Foundation;  either version 2 of the License, or (at your op‐
1205       tion) any later version.
1206
1207       Slurm is distributed in the hope that it will be  useful,  but  WITHOUT
1208       ANY  WARRANTY;  without even the implied warranty of MERCHANTABILITY or
1209       FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General  Public  License
1210       for more details.
1211

FILES

1213       /etc/slurm.conf
1214                 Entries  to this file enable job accounting and designate the
1215                 job accounting log file that collects system job accounting.
1216
1217       /var/log/slurm_accounting.log
1218                 The default job accounting log file.  By default,  this  file
1219                 is set to read and write permission for root only.
1220

SEE ALSO

1222       sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1223
1224
1225
1226September 2021                  Slurm Commands                        sacct(1)
Impressum