1sacct(1)                        Slurm Commands                        sacct(1)
2
3
4

NAME

6       sacct  -  displays  accounting  data  for all jobs and job steps in the
7       Slurm job accounting log or Slurm database
8
9

SYNOPSIS

11       sacct [OPTIONS...]
12
13

DESCRIPTION

15       Accounting information for jobs invoked with Slurm are either logged in
16       the  job accounting log file or saved to the Slurm database, as config‐
17       ured with the AccountingStorageType parameter.
18
19       The sacct command displays  job  accounting  data  stored  in  the  job
20       accounting  log  file  or Slurm database in a variety of forms for your
21       analysis.  The sacct command displays information on jobs,  job  steps,
22       status,  and  exitcodes by default.  You can tailor the output with the
23       use of the --format= option to specify the fields to be shown.
24
25       For the root user, the sacct command displays job accounting  data  for
26       all  users,  although  there are options to filter the output to report
27       only the jobs from a specified user or group.
28
29       For the non-root user, the sacct command  limits  the  display  of  job
30       accounting  data to jobs that were launched with their own user identi‐
31       fier (UID) by default.  Data for other users can be displayed with  the
32       --allusers, --user, or --uid options.
33
34       Elapsed   time   fields   are  presented  as  [days-]hours:minutes:sec‐
35       onds[.microseconds]. Only 'CPU' fields will ever have microseconds.
36
37       The default input file is the file named  in  the  AccountingStorageLoc
38       parameter in slurm.conf.
39
40       Note:  If  designated, the slurmdbd.conf option PrivateData may further
41              restrict the accounting data visible  to  users  which  are  not
42              SlurmUser,  root, or a user with AdminLevel=Admin. See the slur‐
43              mdbd.conf man page for additional details on restricting  access
44              to accounting data.
45
46       Note:  If   the   AccountingStorageType  is  set  to  "accounting_stor‐
47              age/filetxt", space characters embedded  within  account  names,
48              job  names,  and  step names will be replaced by underscores. If
49              account names with embedded spaces are needed, it is recommended
50              that a database type of accounting storage be configured.
51
52       Note:  The  contents  of Slurm's database are maintained in lower case.
53              This may result in some sacct  output  differing  from  that  of
54              other Slurm commands.
55
56       Note:  Much  of  the  data  reported by sacct has been generated by the
57              wait3() and getrusage() system calls. Some  systems  gather  and
58              report  incomplete  information  for  these calls; sacct reports
59              values of 0 for this missing data. See  your  systems  getrusage
60              (3)  man  page  for  information  about  which data are actually
61              available on your system.
62
63

OPTIONS

65       -a, --allusers
66                 Displays all users jobs when run by user root or if  Private‐
67                 Data  is  not configured to jobs.  Otherwise display the cur‐
68                 rent user's jobs
69
70       -A account_list , --accounts=account_list
71                 Displays jobs when a comma separated  list  of  accounts  are
72                 given as the argument.
73
74       -b, --brief
75                 Displays a brief listing, which includes the following data:
76
77                 jobid
78
79                 status
80
81                 exitcode
82
83       -c, --completion
84                 Use  job completion data instead of job accounting.  The Job‐
85                 CompType parameter in the slurm.conf file must be defined  to
86                 a non-none option.  Does not support federated cluster infor‐
87                 mation (local data only).
88
89       -C constraint_list, --constraints=constraint_list
90                 Comma separated list  to  filter  jobs  based  on  what  con‐
91                 straints/features  the  job requested.  Multiple options will
92                 be treated as 'and' not 'or', so the job would need all  con‐
93                 straints specified to be returned not one or the other.
94
95
96       --delimiter=characters
97                 ASCII  characters used to separate the fields when specifying
98                 the -p or -P options. The default delimiter is  a  '|'.  This
99                 option is ignored if -p or -P options are not specified.
100
101
102       -D, --duplicates
103                 If  Slurm  job  ids are reset, some job numbers will probably
104                 appear more than once in the accounting log file but refer to
105                 different  jobs.  Such jobs can be distinguished by the "sub‐
106                 mit" time stamp in the data records.
107
108
109                 When data for specific jobs are  requested  with  the  --jobs
110                 option,  sacct  returns the most recent job with that number.
111                 This behavior can be overridden by  specifying  --duplicates,
112                 in  which  case all records that match the selection criteria
113                 will be returned.
114
115
116                 NOTE: Revoked federated sibling jobs are  hidden  unless  the
117                 --duplicates option is specified.
118
119
120       -e, --helpformat
121
122                 Print  a list of fields that can be specified with the --for‐
123                 mat option.
124
125
126                 Fields available:
127
128                 Account             AdminComment        AllocCPUS           AllocGRES
129                 AllocNodes          AllocTRES           AssocID             AveCPU
130                 AveCPUFreq          AveDiskRead         AveDiskWrite        AvePages
131                 AveRSS              AveVMSize           BlockID             Cluster
132                 Comment             Constraints         ConsumedEnergy      ConsumedEnergyRaw
133                 CPUTime             CPUTimeRAW          DerivedExitCode     Elapsed
134                 ElapsedRaw          Eligible            End                 ExitCode
135                 Flags               GID                 Group               JobID
136                 JobIDRaw            JobName             Layout              MaxDiskRead
137                 MaxDiskReadNode     MaxDiskReadTask     MaxDiskWrite        MaxDiskWriteNode
138                 MaxDiskWriteTask    MaxPages            MaxPagesNode        MaxPagesTask
139                 MaxRSS              MaxRSSNode          MaxRSSTask          MaxVMSize
140                 MaxVMSizeNode       MaxVMSizeTask       McsLabel            MinCPU
141                 MinCPUNode          MinCPUTask          NCPUS               NNodes
142                 NodeList            NTasks              Priority            Partition
143                 QOS                 QOSRAW              Reason              ReqCPUFreq
144                 ReqCPUFreqMin       ReqCPUFreqMax       ReqCPUFreqGov       ReqCPUS
145                 ReqGRES             ReqMem              ReqNodes            ReqTRES
146                 Reservation         ReservationId       Reserved            ResvCPU
147                 ResvCPURAW          Start               State               Submit
148                 Suspended           SystemCPU           SystemComment       Timelimit
149                 TimelimitRaw        TotalCPU            TRESUsageInAve      TRESUsageInMax
150                 TRESUsageInMaxNode  TRESUsageInMaxTask  TRESUsageInMin      TRESUsageInMinNode
151                 TRESUsageInMinTask  TRESUsageInTot      TRESUsageOutAve     TRESUsageOutMax
152                 TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin     TRESUsageOutMinNode
153                 TRESUsageOutMinTask TRESUsageOutTot     UID                 User
154                 UserCPU             WCKey               WCKeyID             WorkDir
155
156                 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
157                 SUsageIn[Ave|Tot].   They  represent the average/total of the
158                 highest watermarks over all ranks in the  step.   When  using
159                 sstat they represent the average/total at the moment the com‐
160                 mand was run.
161
162                 NOTE: TRESUsage*Min* values represent  the  lowest  highwater
163                 mark in the step.
164
165                 The  section  titled  "Job Accounting Fields" describes these
166                 fields.
167
168
169       -E end_time, --endtime=end_time
170
171                 Select jobs in any  state  before  the  specified  time.   If
172                 states are given with the -s option return jobs in this state
173                 before this period.  See the DEFAULT  TIME  WINDOW  for  more
174                 details.
175
176                 Valid time formats are...
177
178                 HH:MM[:SS] [AM|PM]
179                 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
180                 MM/DD[/YY]-HH:MM[:SS]
181                 YYYY-MM-DD[THH:MM[:SS]]
182
183       --federation
184                 Show jobs from the federation if a member of one.
185
186
187       -f file,  --file=file
188                 Causes the sacct command to read job accounting data from the
189                 named file instead of the current Slurm  job  accounting  log
190                 file. Only applicable when running the filetxt plugin.
191
192
193       -F flag_list,  --flags=flag_list
194                 Comma  separated  list  to  filter jobs based on what various
195                 ways the jobs were handled.  Current flags  are  SchedSubmit,
196                 SchedMain,  SchedBackfill.  These particular options describe
197                 the scheduler that started the job.
198
199
200       -g gid_list, --gid=gid_list --group=group_list
201                 Displays the statistics only for the jobs  started  with  the
202                 GID  or the GROUP specified by the gid_list or the group_list
203                 operand, which is a comma-separated list.   Space  characters
204                 are not allowed.  Default is no restrictions.
205
206
207       -h, --help
208                 Displays a general help message.
209
210
211       -i, --nnodes=N
212                 Return jobs which ran on this many nodes (N = min[-max])
213
214
215       -j job(.step) ,  --jobs=job(.step)
216                 Displays  information  about the specified job(.step) or list
217                 of job(.step)s.
218
219                 The job(.step) parameter is a comma-separated list  of  jobs.
220                 Space  characters  are  not  permitted in this list.  NOTE: A
221                 step id of 'batch' will display  the  information  about  the
222                 batch step.
223
224                 By default sacct shows only jobs with Eligible time, but with
225                 this option the non-eligible will be also  shown.   NOTE:  If
226                 --state  is  also specified, as non-eligible are not PD, then
227                 non-eligible jobs will not be  displayed.   See  the  DEFAULT
228                 TIME  WINDOW  for  details  about how this option changes the
229                 default -S and -E options.
230
231
232       -k, --timelimit-min
233                 Only send data about jobs with this timelimit.  If used  with
234                 timelimit_max  this  will  be  the  minimum  timelimit of the
235                 range.  Default is no restriction.
236
237
238       -K, --timelimit-max
239                 Ignored by itself, but if timelimit_min is set this  will  be
240                 the  maximum  timelimit of the range.  Default is no restric‐
241                 tion.
242
243
244       --local   Show only jobs local to this cluster. Ignore  other  clusters
245                 in this federation (if any). Overrides --federation.
246
247
248       -l, --long
249                 Equivalent to specifying:
250
251                 --format=jobid,jobname,partition,maxvmsize,maxvmsizen‐
252                 ode,maxvmsizetask, avevm‐
253                 size,maxrss,maxrssnode,maxrsstask,averss,maxpages,max‐
254                 pagesnode, maxpagestask,avepages,mincpu,mincpun‐
255                 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
256                 code,maxdiskread,maxdiskreadnode,maxdiskreadtask,
257                 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
258                 task,avediskwrite, allocgres,reqgres,avecpufreq,reqcpufre‐
259                 qmin,reqcpufreqmax,reqcpufreqgov, tresusageinmax,tresusagein‐
260                 maxn,tresusageinmaxt,tresusageoutmax,tresusageoutmaxn, tre‐
261                 susageoutmaxt,tresusageoutave,tresusageinave
262
263
264       -L, --allclusters
265                 Display  jobs  ran on all clusters. By default, only jobs ran
266                 on the cluster from where sacct is called are displayed.
267
268
269       -M cluster_list, --clusters=cluster_list
270                 Displays the statistics only for  the  jobs  started  on  the
271                 clusters  specified  by  the cluster_list operand, which is a
272                 comma-separated list of clusters.  Space characters  are  not
273                 allowed  in the cluster_list.  A value of 'all' will query to
274                 run on all clusters.  The default is current cluster you  are
275                 executing the sacct command on or all clusters in the federa‐
276                 tion when executed  on  a  federated  cluster.   This  option
277                 implicitly sets the --local option.
278
279
280       -n, --noheader
281                 No heading will be added to the output. The default action is
282                 to display a header.
283
284       --noconvert
285                 Don't convert units from  their  original  type  (e.g.  2048M
286                 won't be converted to 2G).
287
288       -N node_list, --nodelist=node_list
289                 Display jobs that ran on any of these node(s).  node_list can
290                 be a ranged string.
291
292       --name=jobname_list
293                 Display jobs that have any of these name(s).
294
295       -o, --format
296                 Comma separated list of fields.  (use  "--helpformat"  for  a
297                 list of available fields).
298
299                 NOTE: When using the format option for listing various fields
300                 you can put a %NUMBER afterwards to specify how many  charac‐
301                 ters should be printed.
302
303                 e.g.  format=name%30  will  print 30 characters of field name
304                 right justified.  A %-30 will print 30 characters left justi‐
305                 fied.
306
307                 When set, the SACCT_FORMAT environment variable will override
308                 the default format.  For example:
309
310                 SACCT_FORMAT="jobid,user,account,cluster"
311
312
313       -p, --parsable
314                 output will be '|' delimited with a '|' at the end
315
316
317       -P, --parsable2
318                 output will be '|' delimited without a '|' at the end
319
320
321       -q, --qos Only send data about jobs using these qos.  Default is all.
322
323
324       -r, --partition
325
326                 Comma separated list of partitions to  select  jobs  and  job
327                 steps from. The default is all partitions.
328
329
330       -R reason_list,  --reason=reason_list
331                 Comma  separated list to filter jobs based on what reason the
332                 job wasn't scheduled outside resources/priority.
333
334
335       -s state_list, --state=state_list
336                 Selects jobs based on their  state  during  the  time  period
337                 given.   Unless  otherwise  specified, the start and end time
338                 will be the current time when the --state option is specified
339                 and  only  currently  running jobs can be displayed.  A start
340                 and/or end time must be specified to view  information  about
341                 jobs  not currently running.  See the JOB STATE CODES section
342                 below for a list of state designators.  Multiple state  names
343                 may  be specified using comma separators. Either the short or
344                 long form of the state name may be  used  (e.g.  CA  or  CAN‐
345                 CELLED) and the name is case insensitive (i.e. ca and CA both
346                 work).
347
348                 NOTE: Note for a job to be selected in the PENDING  state  it
349                 must  have  "EligibleTime"  in the requested time interval or
350                 different from "Unknown". The "EligibleTime" is displayed  by
351                 the  "scontrol show job" command.  For example jobs submitted
352                 with the "--hold" option will have "EligibleTime=Unknown"  as
353                 they are pending indefinitely.
354
355                 NOTE:  When  specifying states and no start time is given the
356                 default start time is 'now'.  This is only  when  -j  is  not
357                 used.   If -j is used the start time will default to 'Epoch'.
358                 In both cases if no end time is  given  it  will  default  to
359                 'now'. See the DEFAULT TIME WINDOW for more details.
360
361
362       -S, --starttime
363                 Select jobs in any state after the specified time. Default is
364                 00:00:00 of the current day, unless the '-s' or '-j'  options
365                 are  used.  If  the  '-s' option is used, then the default is
366                 'now'. If states are given with the  '-s'  option  then  only
367                 jobs in this state at this time will be returned. If the '-j'
368                 option is used, then the default time is Unix  Epoch  0.  See
369                 the DEFAULT TIME WINDOW for more details.
370
371                 Valid time formats are...
372
373                 HH:MM[:SS] [AM|PM]
374                 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
375                 MM/DD[/YY]-HH:MM[:SS]
376                 YYYY-MM-DD[THH:MM[:SS]]
377
378
379       -T, --truncate
380                 Truncate  time.   So  if a job started before --starttime the
381                 start time would be truncated to --starttime.  The  same  for
382                 end time and --endtime.
383
384
385       -u uid_list, --uid=uid_list, --user=user_list
386                 Use this comma separated list of uids or user names to select
387                 jobs to display.  By default, the running user's uid is used.
388
389
390       --units=[KMGTP]
391                 Display values in specified unit type. Takes precedence  over
392                 --noconvert option.
393
394
395       --usage   Display a command usage summary.
396
397
398       -v, --verbose
399                 Primarily for debugging purposes, report the state of various
400                 variables during processing.
401
402
403       -V, --version
404                 Print version.
405
406
407       --whole-hetjob=[yes|no]
408                 When querying heterogeneous jobs, Slurm by default  retrieves
409                 the  information  about  all the components of the job if the
410                 pack_job_id (leader) is selected. If a  non-leader  heteroge‐
411                 neous  job  component  is  selected  only  that  component is
412                 retrieved by default. This behavior can be changed  by  using
413                 this  option. If set to 'yes' or no value is set, then infor‐
414                 mation about all the components will be retrieved  no  matter
415                 which component is selected in the job filter.  Otherwise, if
416                 set to 'no' then only the selected heterogeneous  job  compo‐
417                 nents will be retrieved, even when selecting the leader.
418
419
420       -W wckey_list, --wckeys=wckey_list
421                 Displays the statistics only for the jobs started on the wck‐
422                 eys  specified  by  the  wckey_list  operand,  which   is   a
423                 comma-separated  list  of  wckey names.  Space characters are
424                 not allowed in the wckey_list. Default is all wckeys.
425
426
427       -x associd_list, --associations=assoc_list
428                 Displays the statistics only for the jobs running  under  the
429                 association ids specified by the assoc_list operand, which is
430                 a comma-separated list of association ids.  Space  characters
431                 are  not  allowed  in the assoc_list. Default is all associa‐
432                 tions.
433
434
435       -X, --allocations
436                 Only show statistics relevant to the job  allocation  itself,
437                 not taking steps into consideration.
438
439
440   Job Accounting Fields
441       The following describes each job accounting field:
442
443              ALL       Print all fields listed below.
444
445
446              AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
447
448
449              AllocGRES Names and counts of generic resources allocated.
450
451
452              AllocNodes
453                        Number  of  nodes allocated to the job/step.  0 if the
454                        job is pending.
455
456
457              AllocTres Trackable resources. These are the resources allocated
458                        to  the  job/step  after the job started running.  For
459                        pending jobs this should be blank.  For  more  details
460                        see AccountingStorageTRES in slurm.conf.
461
462
463              Account   Account the job ran under.
464
465
466              AssocID   Reference  to  the  association  of  user, account and
467                        cluster.
468
469
470              AveCPU    Average (system + user) CPU time of all tasks in job.
471
472
473              AveCPUFreq
474                        Average weighted CPU frequency of all tasks in job, in
475                        kHz.
476
477
478              AveDiskRead
479                        Average number of bytes read by all tasks in job.
480
481
482              AveDiskWrite
483                        Average number of bytes written by all tasks in job.
484
485
486              AvePages  Average number of page faults of all tasks in job.
487
488
489              AveRSS    Average resident set size of all tasks in job.
490
491
492              AveVMSize Average Virtual Memory size of all tasks in job.
493
494
495              Cluster   Cluster name.
496
497
498              Comment   The  job's comment string when the AccountingStoreJob‐
499                        Comment parameter in the slurm.conf file  is  set  (or
500                        defaults)  to YES.  The Comment string can be modified
501                        by invoking sacctmgr modify  job  or  the  specialized
502                        sjobexitmod command.
503
504
505              ConsumedEnergy
506                        Total  energy consumed by all tasks in job, in joules.
507                        Note: Only in case of exclusive  job  allocation  this
508                        value reflects the jobs' real energy consumption.
509
510
511              CPUTime   Time  used (Elapsed time * CPU count) by a job or step
512                        in HH:MM:SS format.
513
514
515              CPUTimeRAW
516                        Time used (Elapsed time * CPU count) by a job or  step
517                        in cpu-seconds.
518
519
520              DerivedExitCode
521                        The  highest exit code returned by the job's job steps
522                        (srun invocations).  Following the colon is the signal
523                        that  caused the process to terminate if it was termi‐
524                        nated by a signal.  The DerivedExitCode can  be  modi‐
525                        fied  by  invoking sacctmgr modify job or the special‐
526                        ized sjobexitmod command.
527
528
529              Elapsed   The jobs elapsed time.
530
531                        The format of this fields output is as follows:
532                        [DD-[HH:]]MM:SS
533
534                        as defined by the following:
535
536                        DD        days
537
538                        hh        hours
539
540                        mm        minutes
541
542                        ss        seconds
543
544
545              Eligible  When the job became eligible to run in the same format
546                        as End.
547
548
549              End       Termination   time  of  the  job.  Format  output  is,
550                        YYYY-MM-DDTHH:MM:SS,  unless   changed   through   the
551                        SLURM_TIME_FORMAT environment variable.
552
553
554              ExitCode  The  exit  code  returned by the job script or salloc,
555                        typically as set by the  exit()  function.   Following
556                        the  colon  is  the  signal that caused the process to
557                        terminate if it was terminated by a signal.
558
559
560              GID       The group identifier of the user who ran the job.
561
562
563              Group     The group name of the user who ran the job.
564
565
566              JobID     The identification number of the job or job step.
567
568                        Regular jobs are in the form:
569
570                        JobID[.JobStep]
571
572                        Array jobs are in the form:
573
574                        ArrayJobID_ArrayTaskID
575
576                        Heterogeneous jobs are in the form:
577
578                        PackJobID+PackJobOffset.
579
580
581              JobIDRaw  The identification number of  the  job  or  job  step.
582                        Prints the JobID in the form JobID[.JobStep].
583
584
585              JobName   The  name  of  the job or job step. The slurm_account‐
586                        ing.log file is a space  delimited  file.  Because  of
587                        this  if  a space is used in the jobname an underscore
588                        is substituted for the  space  before  the  record  is
589                        written to the accounting file. So when the jobname is
590                        displayed by sacct the jobname that had a space in  it
591                        will now have an underscore in place of the space.
592
593
594              Layout    What  the  layout  of  a step was when it was running.
595                        This can be used to give you an idea of which node ran
596                        which rank in your job.
597
598
599              MaxDiskRead
600                        Maximum number of bytes read by all tasks in job.
601
602
603              MaxDiskReadNode
604                        The node on which the maxdiskread occurred.
605
606
607              MaxDiskReadTask
608                        The task ID where the maxdiskread occurred.
609
610
611              MaxDiskWrite
612                        Maximum number of bytes written by all tasks in job.
613
614
615              MaxDiskWriteNode
616                        The node on which the maxdiskwrite occurred.
617
618
619              MaxDiskWriteTask
620                        The task ID where the maxdiskwrite occurred.
621
622
623              MaxPages  Maximum number of page faults of all tasks in job.
624
625
626              MaxPagesNode
627                        The node on which the maxpages occurred.
628
629
630              MaxPagesTask
631                        The task ID where the maxpages occurred.
632
633
634              MaxRSS    Maximum resident set size of all tasks in job.
635
636
637              MaxRSSNode
638                        The node on which the maxrss occurred.
639
640
641              MaxRSSTask
642                        The task ID where the maxrss occurred.
643
644
645              MaxVMSize Maximum Virtual Memory size of all tasks in job.
646
647
648              MaxVMSizeNode
649                        The node on which the maxvmsize occurred.
650
651
652              MaxVMSizeTask
653                        The task ID where the maxvmsize occurred.
654
655
656              MinCPU    Minimum (system + user) CPU time of all tasks in job.
657
658
659              MinCPUNode
660                        The node on which the mincpu occurred.
661
662
663              MinCPUTask
664                        The task ID where the mincpu occurred.
665
666
667              NCPUS     Total number of CPUs allocated to the job.  Equivalent
668                        to AllocCPUS.
669
670
671              NodeList  List of nodes in job/step.
672
673
674              NNodes    Number of nodes in a job or step.  If the job is  run‐
675                        ning, or ran, this count will be the number allocated,
676                        else the number will be the number requested.
677
678
679              NTasks    Total number of tasks in a job or step.
680
681
682              Priority  Slurm priority.
683
684
685              Partition Identifies the partition on which the job ran.
686
687
688              QOS       Name of Quality of Service.
689
690
691              QOSRAW    Numeric id of Quality of Service.
692
693
694              ReqCPUFreq
695                        Requested CPU frequency for the step, in  kHz.   Note:
696                        This  value  applies  only  to a job step. No value is
697                        reported for the job.
698
699
700              ReqCPUS   Number of requested CPUs.
701
702
703              ReqGRES   Names and counts of generic resources requested.
704
705
706              ReqMem    Minimum required memory for the job, in MB. A  'c'  at
707                        the  end  of  number  represents Memory Per CPU, a 'n'
708                        represents Memory Per Node.  Note: This value is  only
709                        from the job allocation, not the step.
710
711
712              ReqNodes  Requested minimum Node count for the job/step.
713
714
715              ReqTres   Trackable  resources.  These  are the minimum resource
716                        counts requested by the job/step at  submission  time.
717                        For   more   details   see   AccountingStorageTRES  in
718                        slurm.conf.
719
720
721              Reservation
722                        Reservation Name.
723
724
725              ReservationId
726                        Reservation Id.
727
728
729              Reserved  How much wall clock time was used as reserved time for
730                        this  job.   This  is  derived from how long a job was
731                        waiting  from  eligible  time  to  when  it   actually
732                        started.  Format is the same as Elapsed.
733
734
735              ResvCPU   How  many  CPU  seconds were used as reserved time for
736                        this job.  Format is the same as Elapsed.
737
738
739              ResvCPURAW
740                        How many CPU seconds were used as  reserved  time  for
741                        this job.  Format is in processor seconds.
742
743
744              Start     Initiation time of the job in the same format as End.
745
746
747              State     Displays  the job status, or state.  See the JOB STATE
748                        CODES section below for a list of possible states.
749
750                        If more information is available on the job state than
751                        will  fit  into  the current field width (for example,
752                        the uid that CANCELLED a job) the state will  be  fol‐
753                        lowed by a "+".  You can increase the size of the dis‐
754                        played  state  using  the  "%NUMBER"  format  modifier
755                        described earlier.
756
757                        NOTE:  The RUNNING state will return suspended jobs as
758                        well.  In order  to  print  suspended  jobs  you  must
759                        request SUSPENDED at a different call from RUNNING.
760
761                        NOTE: The RUNNING state will return any jobs completed
762                        (cancelled or otherwise) in the time period  requested
763                        as  the job was also RUNNING during that time.  If you
764                        are only looking for jobs that finished, please choose
765                        the appropriate state(s) without the RUNNING state.
766
767
768              Submit    The  time  the job was submitted in the same format as
769                        End.
770
771                        NOTE: If a job is requeued, the submit time is  reset.
772                        To  obtain the original submit time it is necessary to
773                        use the -D or --duplicate option to display all dupli‐
774                        cate entries for a job.
775
776
777              Suspended The  amount  of  time a job or job step was suspended.
778                        Format is the same as Elapsed.
779
780
781              SystemCPU The amount of system CPU time used by the job  or  job
782                        step. Format is the same as Elapsed.
783
784                        NOTE:  SystemCPU provides a measure of the task's par‐
785                        ent process and does not include  CPU  time  of  child
786                        processes.
787
788
789              Timelimit What  the  timelimit was/is for the job. Format is the
790                        same as Elapsed.
791
792
793              TimelimitRaw
794                        What the timelimit was/is for the job.  Format  is  in
795                        number of minutes.
796
797
798              TotalCPU  The  sum of the SystemCPU and UserCPU time used by the
799                        job or job step.  The total CPU time of  the  job  may
800                        exceed  the  job's  elapsed time for jobs that include
801                        multiple job steps. Format is the same as Elapsed.
802
803                        NOTE: TotalCPU provides a measure of the task's parent
804                        process  and  does  not include CPU time of child pro‐
805                        cesses.
806
807
808              TresUsageInAve
809                        Tres average usage in by all tasks in job.   NOTE:  If
810                        corresponding  TresUsageInMaxTask  is -1 the metric is
811                        node centric instead of task.
812
813
814              TresUsageInMax
815                        Tres maximum usage in by all tasks in job.   NOTE:  If
816                        corresponding  TresUsageInMaxTask  is -1 the metric is
817                        node centric instead of task.
818
819
820              TresUsageInMaxNode
821                        Node for which each maximum TRES usage out occurred.
822
823
824              TresUsageInMaxTask
825                        Task for which each maximum TRES usage out occurred.
826
827
828              TresUsageOutAve
829                        Tres average usage out by all tasks in job.  NOTE:  If
830                        corresponding  TresUsageOutMaxTask is -1 the metric is
831                        node centric instead of task.
832
833
834              TresUsageOutMax
835                        Tres maximum usage out by all tasks in job.  NOTE:  If
836                        corresponding  TresUsageOutMaxTask is -1 the metric is
837                        node centric instead of task.
838
839
840              TresUsageOutMaxNode
841                        Node for which each maximum TRES usage out occurred.
842
843
844              TresUsageOutMaxTask
845                        Task for which each maximum TRES usage out occurred.
846
847
848              UID       The user identifier of the user who ran the job.
849
850
851              User      The user name of the user who ran the job.
852
853
854              UserCPU   The amount of user CPU time used by  the  job  or  job
855                        step. Format is the same as Elapsed.
856
857                        NOTE:  UserCPU provides a measure of the task's parent
858                        process and does not include CPU time  of  child  pro‐
859                        cesses.
860
861
862              WCKey     Workload   Characterization   Key.   Arbitrary  string
863                        for grouping orthogonal accounts together.
864
865
866              WCKeyID   Reference to the wckey.
867
868
869

JOB STATE CODES

871       BF  BOOT_FAIL       Job terminated due to launch failure, typically due
872                           to a hardware failure (e.g. unable to boot the node
873                           or block and the job can not be requeued).
874
875       CA  CANCELLED       Job was explicitly cancelled by the user or  system
876                           administrator.   The  job  may or may not have been
877                           initiated.
878
879       CD  COMPLETED       Job has terminated all processes on all nodes  with
880                           an exit code of zero.
881
882       DL  DEADLINE        Job terminated on deadline.
883
884       F   FAILED          Job  terminated  with  non-zero  exit code or other
885                           failure condition.
886
887       NF  NODE_FAIL       Job terminated due to failure of one or more  allo‐
888                           cated nodes.
889
890       OOM OUT_OF_MEMORY   Job experienced out of memory error.
891
892       PD  PENDING         Job is awaiting resource allocation.
893
894       PR  PREEMPTED       Job terminated due to preemption.
895
896       R   RUNNING         Job currently has an allocation.
897
898       RQ  REQUEUED        Job was requeued.
899
900       RS  RESIZING        Job is about to change size.
901
902       RV  REVOKED         Sibling was removed from cluster due to other clus‐
903                           ter starting the job.
904
905       S   SUSPENDED       Job has an allocation, but execution has been  sus‐
906                           pended and CPUs have been released for other jobs.
907
908       TO  TIMEOUT         Job terminated upon reaching its time limit.
909
910

DEFAULT TIME WINDOW

912       The  options  --starttime  and --endtime define the time window between
913       which sacct is going to search. For historical  and  practical  reasons
914       their  default  values  (i.e. the default time window) depends on other
915       options: --jobs and --state.
916
917       Depending on if --jobs and/or --state are specified, the default values
918       of --starttime  and --endtime options are:
919
920
921       WITHOUT EITHER --jobs NOR --state specified:
922
923              ---starttime
924                     Defaults to Midnight.
925
926              ---endtime
927                     Defaults to Now.
928
929
930       WITH --jobs AND WITHOUT --state specified:
931
932              ---starttime
933                     Defaults to Epoch 0.
934
935              ---endtime
936                     Defaults to Now.
937
938
939       WITHOUT --jobs AND WITH --state specified:
940
941              ---starttime
942                     Defaults to Now.
943
944              ---endtime
945                     Defaults  to --starttime and to Now if --starttime is not
946                     specified.
947
948
949       WITH BOTH --jobs AND --state specified:
950
951              ---starttime
952                     Defaults to Epoch 0.
953
954              ---endtime
955                     Defaults to --starttime or to Now if --starttime  is  not
956                     specified.
957
958
959       NOTE:  With  -v/--verbose a message about the actual time window in use
960       is shown.
961
962

ENVIRONMENT VARIABLES

964       Some sacct options may be set via environment variables. These environ‐
965       ment  variables,  along  with  their  corresponding options, are listed
966       below. (Note: Commandline options will always override these settings.)
967
968       SACCT_FEDERATION    Same as --federation
969
970       SACCT_LOCAL         Same as --local
971
972       SLURM_CONF          The location of the Slurm configuration file.
973
974       SLURM_TIME_FORMAT   Specify the format used to report  time  stamps.  A
975                           value  of  standard,  the  default value, generates
976                           output            in            the            form
977                           "year-month-dateThour:minute:second".   A  value of
978                           relative returns only "hour:minute:second"  if  the
979                           current  day.   For other dates in the current year
980                           it prints the "hour:minute"  preceded  by  "Tomorr"
981                           (tomorrow),  "Ystday"  (yesterday), the name of the
982                           day for the coming week (e.g. "Mon", "Tue",  etc.),
983                           otherwise  the  date  (e.g.  "25  Apr").  For other
984                           years it returns a date month and  year  without  a
985                           time  (e.g.   "6 Jun 2012"). All of the time stamps
986                           use a 24 hour format.
987
988                           A valid strftime() format can  also  be  specified.
989                           For example, a value of "%a %T" will report the day
990                           of the week and a time stamp (e.g. "Mon 12:34:56").
991
992

EXAMPLES

994       This example illustrates the default invocation of the sacct command:
995
996              # sacct
997              Jobid      Jobname    Partition    Account AllocCPUS State     ExitCode
998              ---------- ---------- ---------- ---------- ---------- ---------- --------
999              2          script01   srun       acct1               1 RUNNING           0
1000              3          script02   srun       acct1               1 RUNNING           0
1001              4          endscript  srun       acct1               1 RUNNING           0
1002              4.0                   srun       acct1               1 COMPLETED         0
1003
1004
1005       This example shows the same job accounting information with  the  brief
1006       option.
1007
1008              # sacct --brief
1009                   Jobid     State  ExitCode
1010              ---------- ---------- --------
1011              2          RUNNING           0
1012              3          RUNNING           0
1013              4          RUNNING           0
1014              4.0        COMPLETED         0
1015
1016              # sacct --allocations
1017              Jobid      Jobname    Partition Account    AllocCPUS  State     ExitCode
1018              ---------- ---------- ---------- ---------- ------- ---------- --------
1019              3          sja_init   andy       acct1            1 COMPLETED         0
1020              4          sjaload    andy       acct1            2 COMPLETED         0
1021              5          sja_scr1   andy       acct1            1 COMPLETED         0
1022              6          sja_scr2   andy       acct1           18 COMPLETED         2
1023              7          sja_scr3   andy       acct1           18 COMPLETED         0
1024              8          sja_scr5   andy       acct1            2 COMPLETED         0
1025              9          sja_scr7   andy       acct1           90 COMPLETED         1
1026              10         endscript  andy       acct1          186 COMPLETED         0
1027
1028
1029       This  example  demonstrates  the ability to customize the output of the
1030       sacct command.  The fields are displayed in the order designated on the
1031       command line.
1032
1033              # sacct --format=jobid,elapsed,ncpus,ntasks,state
1034                   Jobid    Elapsed      Ncpus   Ntasks     State
1035              ---------- ---------- ---------- -------- ----------
1036              3            00:01:30          2        1 COMPLETED
1037              3.0          00:01:30          2        1 COMPLETED
1038              4            00:00:00          2        2 COMPLETED
1039              4.0          00:00:01          2        2 COMPLETED
1040              5            00:01:23          2        1 COMPLETED
1041              5.0          00:01:31          2        1 COMPLETED
1042
1043
1044       This  example  demonstrates  the use of the -T (--truncate) option when
1045       used with -S (--starttime) and -E (--endtime). When the  -T  option  is
1046       used,  the  start time of the job will be the specified -S value if the
1047       job was started before the specified time, otherwise the time  will  be
1048       the  job's  start time. The end time will be the specified -E option if
1049       the job ends after the specified time, otherwise it will  be  the  jobs
1050       end time.
1051
1052       NOTE:  If  no -s (--state) option is given sacct will display jobs that
1053       ran during the specified time, otherwise it returns jobs that  were  in
1054       the state requested during that period of time.
1055
1056       Without -T (normal operation) sacct output would be like this.
1057
1058              # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1059                  JobID                 Start                  End        State
1060              --------- --------------------- -------------------- ------------
1061              2         2014-07-03T11:33:16   2014-07-03T11:59:01   COMPLETED
1062              3         2014-07-03T11:35:21   Unknown               RUNNING
1063              4         2014-07-03T11:35:21   2014-07-03T11:45:21   COMPLETED
1064              5         2014-07-03T11:41:01   Unknown               RUNNING
1065
1066       By  adding the -T option the job's start and end times are truncated to
1067       reflect only the time requested.  If a job started after the start time
1068       requested or finished before the end time requested those times are not
1069       altered.  The -T option is useful when determining exact run times dur‐
1070       ing any given period.
1071
1072              # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1073                  JobID                 Start                  End        State
1074              --------- --------------------- -------------------- ------------
1075              2         2014-07-03T11:40:00   2014-07-03T11:59:01   COMPLETED
1076              3         2014-07-03T11:40:00   2014-07-03T12:00:00   RUNNING
1077              4         2014-07-03T11:40:00   2014-07-03T11:45:21   COMPLETED
1078              5         2014-07-03T11:41:01   2014-07-03T12:00:00   RUNNING
1079
1080
1081

COPYING

1083       Copyright  (C)  2005-2007 Copyright Hewlett-Packard Development Company
1084       L.P.
1085       Copyright (C) 2008-2010 Lawrence Livermore National Security.  Produced
1086       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1087       Copyright (C) 2010-2014 SchedMD LLC.
1088
1089       This  file  is  part  of  Slurm,  a  resource  management program.  For
1090       details, see <https://slurm.schedmd.com/>.
1091
1092       Slurm is free software; you can redistribute it and/or modify it  under
1093       the  terms  of  the GNU General Public License as published by the Free
1094       Software Foundation; either version 2  of  the  License,  or  (at  your
1095       option) any later version.
1096
1097       Slurm  is  distributed  in the hope that it will be useful, but WITHOUT
1098       ANY WARRANTY; without even the implied warranty of  MERCHANTABILITY  or
1099       FITNESS  FOR  A PARTICULAR PURPOSE.  See the GNU General Public License
1100       for more details.
1101

FILES

1103       /etc/slurm.conf
1104                 Entries to this file enable job accounting and designate  the
1105                 job accounting log file that collects system job accounting.
1106
1107       /var/log/slurm_accounting.log
1108                 The  default  job accounting log file.  By default, this file
1109                 is set to read and write permission for root only.
1110

SEE ALSO

1112       sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1113
1114
1115
1116May 2019                        Slurm Commands                        sacct(1)
Impressum