1sacct(1)                        Slurm Commands                        sacct(1)
2
3
4

NAME

6       sacct  -  displays  accounting  data  for all jobs and job steps in the
7       Slurm job accounting log or Slurm database
8
9

SYNOPSIS

11       sacct [OPTIONS...]
12
13

DESCRIPTION

15       Accounting information for jobs invoked with Slurm are either logged in
16       the  job accounting log file or saved to the Slurm database, as config‐
17       ured with the AccountingStorageType parameter.
18
19       The sacct command displays  job  accounting  data  stored  in  the  job
20       accounting  log  file  or Slurm database in a variety of forms for your
21       analysis.  The sacct command displays information on jobs,  job  steps,
22       status,  and  exitcodes by default.  You can tailor the output with the
23       use of the --format= option to specify the fields to be shown.
24
25       For the root user, the sacct command displays job accounting  data  for
26       all  users,  although  there are options to filter the output to report
27       only the jobs from a specified user or group.
28
29       For the non-root user, the sacct command  limits  the  display  of  job
30       accounting  data to jobs that were launched with their own user identi‐
31       fier (UID) by default.  Data for other users can be displayed with  the
32       --allusers, --user, or --uid options.
33
34       Elapsed   time   fields   are  presented  as  [days-]hours:minutes:sec‐
35       onds[.microseconds]. Only 'CPU' fields will ever have microseconds.
36
37       The default input file is the file named  in  the  AccountingStorageLoc
38       parameter in slurm.conf.
39
40       Note:  If  designated, the slurmdbd.conf option PrivateData may further
41              restrict the accounting data visible  to  users  which  are  not
42              SlurmUser,  root, or a user with AdminLevel=Admin. See the slur‐
43              mdbd.conf man page for additional details on restricting  access
44              to accounting data.
45
46       Note:  The  contents  of Slurm's database are maintained in lower case.
47              This may result in some sacct  output  differing  from  that  of
48              other Slurm commands.
49
50       Note:  Much  of  the  data  reported by sacct has been generated by the
51              wait3() and getrusage() system calls. Some  systems  gather  and
52              report  incomplete  information  for  these calls; sacct reports
53              values of 0 for this missing data. See  your  systems  getrusage
54              (3)  man  page  for  information  about  which data are actually
55              available on your system.
56
57

OPTIONS

59       -a, --allusers
60                 Displays all users jobs when run by user root or if  Private‐
61                 Data  is  not configured to jobs.  Otherwise display the cur‐
62                 rent user's jobs
63
64       -A account_list , --accounts=account_list
65                 Displays jobs when a comma separated  list  of  accounts  are
66                 given as the argument.
67
68       -b, --brief
69                 Displays a brief listing, which includes the following data:
70
71                 jobid
72
73                 status
74
75                 exitcode
76
77       -c, --completion
78                 Use  job completion data instead of job accounting.  The Job‐
79                 CompType parameter in the slurm.conf file must be defined  to
80                 a non-none option.  Does not support federated cluster infor‐
81                 mation (local data only).
82
83       -C constraint_list, --constraints=constraint_list
84                 Comma separated list  to  filter  jobs  based  on  what  con‐
85                 straints/features  the  job requested.  Multiple options will
86                 be treated as 'and' not 'or', so the job would need all  con‐
87                 straints specified to be returned not one or the other.
88
89
90       --delimiter=characters
91                 ASCII  characters used to separate the fields when specifying
92                 the -p or -P options. The default delimiter is  a  '|'.  This
93                 option is ignored if -p or -P options are not specified.
94
95
96       -D, --duplicates
97                 If  Slurm  job  ids are reset, some job numbers will probably
98                 appear more than once in the accounting log file but refer to
99                 different  jobs.  Such jobs can be distinguished by the "sub‐
100                 mit" time stamp in the data records.
101
102
103                 When data for specific jobs are  requested  with  the  --jobs
104                 option,  sacct  returns the most recent job with that number.
105                 This behavior can be overridden by  specifying  --duplicates,
106                 in  which  case all records that match the selection criteria
107                 will be returned.
108
109
110                 NOTE: Revoked federated sibling jobs are  hidden  unless  the
111                 --duplicates option is specified.
112
113
114       -e, --helpformat
115
116                 Print  a list of fields that can be specified with the --for‐
117                 mat option.
118
119
120                 Fields available:
121
122                 Account             AdminComment        AllocCPUS           AllocNodes
123                 AllocTRES           AssocID             AveCPU              AveCPUFreq
124                 AveDiskRead         AveDiskWrite        AvePages            AveRSS
125                 AveVMSize           BlockID             Cluster             Comment
126                 Constraints         ConsumedEnergy      ConsumedEnergyRaw   CPUTime
127                 CPUTimeRAW          DBIndex             DerivedExitCode     Elapsed
128                 ElapsedRaw          Eligible            End                 ExitCode
129                 Flags               GID                 Group               JobID
130                 JobIDRaw            JobName             Layout              MaxDiskRead
131                 MaxDiskReadNode     MaxDiskReadTask     MaxDiskWrite        MaxDiskWriteNode
132                 MaxDiskWriteTask    MaxPages            MaxPagesNode        MaxPagesTask
133                 MaxRSS              MaxRSSNode          MaxRSSTask          MaxVMSize
134                 MaxVMSizeNode       MaxVMSizeTask       McsLabel            MinCPU
135                 MinCPUNode          MinCPUTask          NCPUS               NNodes
136                 NodeList            NTasks              Priority            Partition
137                 QOS                 QOSRAW              Reason              ReqCPUFreq
138                 ReqCPUFreqMin       ReqCPUFreqMax       ReqCPUFreqGov       ReqCPUS
139                 ReqMem              ReqNodes            ReqTRES             Reservation
140                 ReservationId       Reserved            ResvCPU             ResvCPURAW
141                 Start               State               Submit              Suspended
142                 SystemCPU           SystemComment       Timelimit           TimelimitRaw
143                 TotalCPU            TRESUsageInAve      TRESUsageInMax      TRESUsageInMaxNode
144                 TRESUsageInMaxTask  TRESUsageInMin      TRESUsageInMinNode  TRESUsageInMinTask
145                 TRESUsageInTot      TRESUsageOutAve     TRESUsageOutMax     TRESUsageOutMaxNode
146                 TRESUsageOutMaxTask TRESUsageOutMin     TRESUsageOutMinNode TRESUsageOutMinTask
147                 TRESUsageOutTot     UID                 User                UserCPU
148                 WCKey               WCKeyID             WorkDir
149
150                 NOTE: When using with Ave[RSS|VM]Size or their values in TRE‐
151                 SUsageIn[Ave|Tot].   They  represent the average/total of the
152                 highest watermarks over all ranks in the  step.   When  using
153                 sstat they represent the average/total at the moment the com‐
154                 mand was run.
155
156                 NOTE: TRESUsage*Min* values represent  the  lowest  highwater
157                 mark in the step.
158
159                 The  section  titled  "Job Accounting Fields" describes these
160                 fields.
161
162
163       -E end_time, --endtime=end_time
164
165                 Select jobs in any  state  before  the  specified  time.   If
166                 states are given with the -s option return jobs in this state
167                 before this period.  See the DEFAULT  TIME  WINDOW  for  more
168                 details.
169
170                 Valid time formats are:
171                 HH:MM[:SS][AM|PM]
172                 MMDD[YY][-HH:MM[:SS]]
173                 MM.DD[.YY][-HH:MM[:SS]]
174                 MM/DD[/YY][-HH:MM[:SS]]
175                 YYYY-MM-DD[THH:MM[:SS]]
176                 today, midnight, noon, fika (3 PM), teatime (4 PM)
177                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
178
179       --federation
180                 Show jobs from the federation if a member of one.
181
182
183       -f file,  --file=file
184                 Causes the sacct command to read job accounting data from the
185                 named file instead of the current Slurm  job  accounting  log
186                 file.  Only applicable when running the jobcomp/filetxt plug‐
187                 in.
188
189
190       -F flag_list,  --flags=flag_list
191                 Comma separated list to filter jobs  based  on  what  various
192                 ways  the  jobs were handled.  Current flags are SchedSubmit,
193                 SchedMain, SchedBackfill.  These particular options  describe
194                 the scheduler that started the job.
195
196
197       -g gid_list, --gid=gid_list --group=group_list
198                 Displays  the  statistics  only for the jobs started with the
199                 GID or the GROUP specified by the gid_list or the  group_list
200                 operand,  which  is a comma-separated list.  Space characters
201                 are not allowed.  Default is no restrictions.
202
203
204       -h, --help
205                 Displays a general help message.
206
207
208       -i, --nnodes=N
209                 Return jobs which ran on this many nodes (N = min[-max])
210
211
212       -j job(.step) ,  --jobs=job(.step)
213                 Displays information about the specified job(.step)  or  list
214                 of job(.step)s.
215
216                 The  job(.step)  parameter is a comma-separated list of jobs.
217                 Space characters are not permitted in  this  list.   NOTE:  A
218                 step  id  of  'batch'  will display the information about the
219                 batch step.
220
221                 By default sacct shows only jobs with Eligible time, but with
222                 this  option  the  non-eligible will be also shown.  NOTE: If
223                 --state is also specified, as non-eligible are not  PD,  then
224                 non-eligible  jobs  will  not  be displayed.  See the DEFAULT
225                 TIME WINDOW for details about how  this  option  changes  the
226                 default -S and -E options.
227
228
229       -k, --timelimit-min
230                 Only  send data about jobs with this timelimit.  If used with
231                 timelimit_max this will  be  the  minimum  timelimit  of  the
232                 range.  Default is no restriction.
233
234
235       -K, --timelimit-max
236                 Ignored  by  itself, but if timelimit_min is set this will be
237                 the maximum timelimit of the range.  Default is  no  restric‐
238                 tion.
239
240
241       --local   Show  only  jobs local to this cluster. Ignore other clusters
242                 in this federation (if any). Overrides --federation.
243
244
245       -l, --long
246                 Equivalent to specifying:
247
248                 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
249                 sizenode, maxvmsizetask,avevm‐
250                 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
251                 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
252                 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
253                 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
254                 gov,reqmem,consumeden‐
255                 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
256                 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
257                 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
258                 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
259                 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
260                 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
261                 outave,tresusageouttot
262
263
264       -L, --allclusters
265                 Display  jobs  ran on all clusters. By default, only jobs ran
266                 on the cluster from where sacct is called are displayed.
267
268
269       -M cluster_list, --clusters=cluster_list
270                 Displays the statistics only for  the  jobs  started  on  the
271                 clusters  specified  by  the cluster_list operand, which is a
272                 comma-separated list of clusters.  Space characters  are  not
273                 allowed  in the cluster_list.  A value of 'all' will query to
274                 run on all clusters.  The default is current cluster you  are
275                 executing the sacct command on or all clusters in the federa‐
276                 tion when executed  on  a  federated  cluster.   This  option
277                 implicitly sets the --local option.
278
279
280       -n, --noheader
281                 No heading will be added to the output. The default action is
282                 to display a header.
283
284       --noconvert
285                 Don't convert units from  their  original  type  (e.g.  2048M
286                 won't be converted to 2G).
287
288       -N node_list, --nodelist=node_list
289                 Display jobs that ran on any of these node(s).  node_list can
290                 be a ranged string.
291
292       --name=jobname_list
293                 Display jobs that have any of these name(s).
294
295       -o, --format
296                 Comma separated list of fields.  (use  "--helpformat"  for  a
297                 list of available fields).
298
299                 NOTE: When using the format option for listing various fields
300                 you can put a %NUMBER afterwards to specify how many  charac‐
301                 ters should be printed.
302
303                 e.g.  format=name%30  will  print 30 characters of field name
304                 right justified.  A %-30 will print 30 characters left justi‐
305                 fied.
306
307                 When set, the SACCT_FORMAT environment variable will override
308                 the default format.  For example:
309
310                 SACCT_FORMAT="jobid,user,account,cluster"
311
312
313       -p, --parsable
314                 output will be '|' delimited with a '|' at the end
315
316
317       -P, --parsable2
318                 output will be '|' delimited without a '|' at the end
319
320
321       -q, --qos Only send data about jobs using these qos.  Default is all.
322
323
324       -r, --partition
325
326                 Comma separated list of partitions to  select  jobs  and  job
327                 steps from. The default is all partitions.
328
329
330       -R reason_list,  --reason=reason_list
331                 Comma  separated list to filter jobs based on what reason the
332                 job wasn't scheduled outside resources/priority.
333
334
335       -s state_list, --state=state_list
336                 Selects jobs based on their  state  during  the  time  period
337                 given.   Unless  otherwise  specified, the start and end time
338                 will be the current time when the --state option is specified
339                 and  only  currently  running jobs can be displayed.  A start
340                 and/or end time must be specified to view  information  about
341                 jobs  not currently running.  See the JOB STATE CODES section
342                 below for a list of state designators.  Multiple state  names
343                 may  be specified using comma separators. Either the short or
344                 long form of the state name may be  used  (e.g.  CA  or  CAN‐
345                 CELLED) and the name is case insensitive (i.e. ca and CA both
346                 work).
347
348                 NOTE: Note for a job to be selected in the PENDING  state  it
349                 must  have  "EligibleTime"  in the requested time interval or
350                 different from "Unknown". The "EligibleTime" is displayed  by
351                 the  "scontrol show job" command.  For example jobs submitted
352                 with the "--hold" option will have "EligibleTime=Unknown"  as
353                 they are pending indefinitely.
354
355                 NOTE:  When  specifying states and no start time is given the
356                 default start time is 'now'.  This is only  when  -j  is  not
357                 used.   If -j is used the start time will default to 'Epoch'.
358                 In both cases if no end time is  given  it  will  default  to
359                 'now'. See the DEFAULT TIME WINDOW for more details.
360
361
362       -S, --starttime
363                 Select jobs in any state after the specified time. Default is
364                 00:00:00 of the current day, unless the '-s' or '-j'  options
365                 are  used.  If  the  '-s' option is used, then the default is
366                 'now'. If states are given with the  '-s'  option  then  only
367                 jobs in this state at this time will be returned. If the '-j'
368                 option is used, then the default time is Unix  Epoch  0.  See
369                 the DEFAULT TIME WINDOW for more details.
370
371                 Valid time formats are:
372                 HH:MM[:SS][AM|PM]
373                 MMDD[YY][-HH:MM[:SS]]
374                 MM.DD[.YY][-HH:MM[:SS]]
375                 MM/DD[/YY][-HH:MM[:SS]]
376                 YYYY-MM-DD[THH:MM[:SS]]
377                 today, midnight, noon, fika (3 PM), teatime (4 PM)
378                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
379
380
381       -T, --truncate
382                 Truncate  time.   So  if a job started before --starttime the
383                 start time would be truncated to --starttime.  The  same  for
384                 end time and --endtime.
385
386
387       -u uid_list, --uid=uid_list, --user=user_list
388                 Use this comma separated list of uids or user names to select
389                 jobs to display.  By default, the running user's uid is used.
390
391
392       --use-local-uid
393                 When displaying UID, sacct uses the  UID  stored  in  Slurm's
394                 accounting  database  by  default.  Use  this command to make
395                 Slurm use a system call to get the  UID  from  the  username.
396                 This  option  may  be  useful in an environment with multiple
397                 clusters and one database where the UID's aren't the same  on
398                 all clusters.
399
400
401       --units=[KMGTP]
402                 Display  values in specified unit type. Takes precedence over
403                 --noconvert option.
404
405
406       --usage   Display a command usage summary.
407
408
409       -v, --verbose
410                 Primarily for debugging purposes, report the state of various
411                 variables during processing.
412
413
414       -V, --version
415                 Print version.
416
417
418       --whole-hetjob=[yes|no]
419                 When  querying heterogeneous jobs, Slurm by default retrieves
420                 the information about all the components of the  job  if  the
421                 het_job_id  (leader id) is selected.  If a non-leader hetero‐
422                 geneous job component id is selected only that  component  is
423                 retrieved  by  default. This behavior can be changed by using
424                 this option. If set to 'yes' or no value is set, then  infor‐
425                 mation  about  all the components will be retrieved no matter
426                 which component is selected in the job filter.  Otherwise, if
427                 set  to  'no' then only the selected heterogeneous job compo‐
428                 nents will be retrieved, even when selecting the leader.
429
430
431       -W wckey_list, --wckeys=wckey_list
432                 Displays the statistics only for the jobs started on the wck‐
433                 eys   specified   by  the  wckey_list  operand,  which  is  a
434                 comma-separated list of wckey names.   Space  characters  are
435                 not allowed in the wckey_list. Default is all wckeys.
436
437
438       -x associd_list, --associations=assoc_list
439                 Displays  the  statistics only for the jobs running under the
440                 association ids specified by the assoc_list operand, which is
441                 a  comma-separated list of association ids.  Space characters
442                 are not allowed in the assoc_list. Default  is  all  associa‐
443                 tions.
444
445
446       -X, --allocations
447                 Only  show  statistics relevant to the job allocation itself,
448                 not taking steps into consideration.
449
450                 NOTE: Without including steps, utilization statistics for job
451                 allocation(s) will be reported as zero.
452
453
454   Job Accounting Fields
455       The following describes each job accounting field:
456
457              ALL       Print all fields listed below.
458
459
460              AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
461
462
463              AllocNodes
464                        Number  of  nodes allocated to the job/step.  0 if the
465                        job is pending.
466
467
468              AllocTres Trackable resources. These are the resources allocated
469                        to  the  job/step  after the job started running.  For
470                        pending jobs this should be blank.  For  more  details
471                        see AccountingStorageTRES in slurm.conf.
472
473                        NOTE:  When  a generic resource is configured with the
474                        no_consume flag, the allocation will be printed with a
475                        zero.
476
477
478              Account   Account the job ran under.
479
480
481              AssocID   Reference  to  the  association  of  user, account and
482                        cluster.
483
484
485              AveCPU    Average (system + user) CPU time of all tasks in job.
486
487
488              AveCPUFreq
489                        Average weighted CPU frequency of all tasks in job, in
490                        kHz.
491
492
493              AveDiskRead
494                        Average number of bytes read by all tasks in job.
495
496
497              AveDiskWrite
498                        Average number of bytes written by all tasks in job.
499
500
501              AvePages  Average number of page faults of all tasks in job.
502
503
504              AveRSS    Average resident set size of all tasks in job.
505
506
507              AveVMSize Average Virtual Memory size of all tasks in job.
508
509
510              Cluster   Cluster name.
511
512
513              Comment   The  job's comment string when the AccountingStoreJob‐
514                        Comment parameter in the slurm.conf file  is  set  (or
515                        defaults)  to YES.  The Comment string can be modified
516                        by invoking sacctmgr modify  job  or  the  specialized
517                        sjobexitmod command.
518
519
520              ConsumedEnergy
521                        Total  energy consumed by all tasks in job, in joules.
522                        Note: Only in case of exclusive  job  allocation  this
523                        value reflects the jobs' real energy consumption.
524
525
526              CPUTime   Time  used (Elapsed time * CPU count) by a job or step
527                        in HH:MM:SS format.
528
529
530              CPUTimeRAW
531                        Time used (Elapsed time * CPU count) by a job or  step
532                        in cpu-seconds.
533
534
535              DerivedExitCode
536                        The  highest exit code returned by the job's job steps
537                        (srun invocations).  Following the colon is the signal
538                        that  caused the process to terminate if it was termi‐
539                        nated by a signal.  The DerivedExitCode can  be  modi‐
540                        fied  by  invoking sacctmgr modify job or the special‐
541                        ized sjobexitmod command.
542
543
544              Elapsed   The jobs elapsed time.
545
546                        The format of this fields output is as follows:
547                        [DD-[HH:]]MM:SS
548
549                        as defined by the following:
550
551                        DD        days
552
553                        hh        hours
554
555                        mm        minutes
556
557                        ss        seconds
558
559
560              Eligible  When the job became eligible to run in the same format
561                        as End.
562
563
564              End       Termination   time  of  the  job.  Format  output  is,
565                        YYYY-MM-DDTHH:MM:SS,  unless   changed   through   the
566                        SLURM_TIME_FORMAT environment variable.
567
568
569              ExitCode  The  exit  code  returned by the job script or salloc,
570                        typically as set by the  exit()  function.   Following
571                        the  colon  is  the  signal that caused the process to
572                        terminate if it was terminated by a signal.
573
574
575              GID       The group identifier of the user who ran the job.
576
577
578              Group     The group name of the user who ran the job.
579
580
581              JobID     The identification number of the job or job step.
582
583                        Regular jobs are in the form:
584
585                        JobID[.JobStep]
586
587                        Array jobs are in the form:
588
589                        ArrayJobID_ArrayTaskID
590
591                        Heterogeneous jobs are in the form:
592
593                        HetJobID+HetJobOffset.
594
595
596              JobIDRaw  The identification number of  the  job  or  job  step.
597                        Prints the JobID in the form JobID[.JobStep].
598
599
600              JobName   The  name  of  the job or job step. The slurm_account‐
601                        ing.log file is a space  delimited  file.  Because  of
602                        this  if  a space is used in the jobname an underscore
603                        is substituted for the  space  before  the  record  is
604                        written to the accounting file. So when the jobname is
605                        displayed by sacct the jobname that had a space in  it
606                        will now have an underscore in place of the space.
607
608
609              Layout    What  the  layout  of  a step was when it was running.
610                        This can be used to give you an idea of which node ran
611                        which rank in your job.
612
613
614              MaxDiskRead
615                        Maximum number of bytes read by all tasks in job.
616
617
618              MaxDiskReadNode
619                        The node on which the maxdiskread occurred.
620
621
622              MaxDiskReadTask
623                        The task ID where the maxdiskread occurred.
624
625
626              MaxDiskWrite
627                        Maximum number of bytes written by all tasks in job.
628
629
630              MaxDiskWriteNode
631                        The node on which the maxdiskwrite occurred.
632
633
634              MaxDiskWriteTask
635                        The task ID where the maxdiskwrite occurred.
636
637
638              MaxPages  Maximum number of page faults of all tasks in job.
639
640
641              MaxPagesNode
642                        The node on which the maxpages occurred.
643
644
645              MaxPagesTask
646                        The task ID where the maxpages occurred.
647
648
649              MaxRSS    Maximum resident set size of all tasks in job.
650
651
652              MaxRSSNode
653                        The node on which the maxrss occurred.
654
655
656              MaxRSSTask
657                        The task ID where the maxrss occurred.
658
659
660              MaxVMSize Maximum Virtual Memory size of all tasks in job.
661
662
663              MaxVMSizeNode
664                        The node on which the maxvmsize occurred.
665
666
667              MaxVMSizeTask
668                        The task ID where the maxvmsize occurred.
669
670
671              MinCPU    Minimum (system + user) CPU time of all tasks in job.
672
673
674              MinCPUNode
675                        The node on which the mincpu occurred.
676
677
678              MinCPUTask
679                        The task ID where the mincpu occurred.
680
681
682              NCPUS     Total number of CPUs allocated to the job.  Equivalent
683                        to AllocCPUS.
684
685
686              NodeList  List of nodes in job/step.
687
688
689              NNodes    Number of nodes in a job or step.  If the job is  run‐
690                        ning, or ran, this count will be the number allocated,
691                        else the number will be the number requested.
692
693
694              NTasks    Total number of tasks in a job or step.
695
696
697              Priority  Slurm priority.
698
699
700              Partition Identifies the partition on which the job ran.
701
702
703              QOS       Name of Quality of Service.
704
705
706              QOSRAW    Numeric id of Quality of Service.
707
708
709              ReqCPUFreq
710                        Requested CPU frequency for the step, in  kHz.   Note:
711                        This  value  applies  only  to a job step. No value is
712                        reported for the job.
713
714
715              ReqCPUFreqGov
716                        Requested CPU frequency governor for the step, in kHz.
717                        Note:  This value applies only to a job step. No value
718                        is reported for the job.
719
720
721              ReqCPUFreqMax
722                        Maximum requested CPU frequency for the step, in  kHz.
723                        Note:  This value applies only to a job step. No value
724                        is reported for the job.
725
726
727              ReqCPUFreqMin
728                        Minimum requested CPU frequency for the step, in  kHz.
729                        Note:  This value applies only to a job step. No value
730                        is reported for the job.
731
732
733              ReqCPUS   Number of requested CPUs.
734
735
736              ReqMem    Minimum required memory for the job, in MB. A  'c'  at
737                        the  end  of  number  represents Memory Per CPU, a 'n'
738                        represents Memory Per Node.  Note: This value is  only
739                        from the job allocation, not the step.
740
741
742              ReqNodes  Requested minimum Node count for the job/step.
743
744
745              ReqTres   Trackable  resources.  These  are the minimum resource
746                        counts requested by the job/step at  submission  time.
747                        For   more   details   see   AccountingStorageTRES  in
748                        slurm.conf.
749
750
751              Reservation
752                        Reservation Name.
753
754
755              ReservationId
756                        Reservation Id.
757
758
759              Reserved  How much wall clock time was used as reserved time for
760                        this  job.   This  is  derived from how long a job was
761                        waiting  from  eligible  time  to  when  it   actually
762                        started.  Format is the same as Elapsed.
763
764
765              ResvCPU   How  many  CPU  seconds were used as reserved time for
766                        this job.  Format is the same as Elapsed.
767
768
769              ResvCPURAW
770                        How many CPU seconds were used as  reserved  time  for
771                        this job.  Format is in processor seconds.
772
773
774              Start     Initiation time of the job in the same format as End.
775
776
777              State     Displays  the job status, or state.  See the JOB STATE
778                        CODES section below for a list of possible states.
779
780                        If more information is available on the job state than
781                        will  fit  into  the current field width (for example,
782                        the uid that CANCELLED a job) the state will  be  fol‐
783                        lowed by a "+".  You can increase the size of the dis‐
784                        played  state  using  the  "%NUMBER"  format  modifier
785                        described earlier.
786
787                        NOTE:  The RUNNING state will return suspended jobs as
788                        well.  In order  to  print  suspended  jobs  you  must
789                        request SUSPENDED at a different call from RUNNING.
790
791                        NOTE: The RUNNING state will return any jobs completed
792                        (cancelled or otherwise) in the time period  requested
793                        as  the job was also RUNNING during that time.  If you
794                        are only looking for jobs that finished, please choose
795                        the appropriate state(s) without the RUNNING state.
796
797
798              Submit    The  time  the job was submitted in the same format as
799                        End.
800
801                        NOTE: If a job is requeued, the submit time is  reset.
802                        To  obtain the original submit time it is necessary to
803                        use the -D or --duplicate option to display all dupli‐
804                        cate entries for a job.
805
806
807              Suspended The  amount  of  time a job or job step was suspended.
808                        Format is the same as Elapsed.
809
810
811              SystemCPU The amount of system CPU time used by the job  or  job
812                        step. Format is the same as Elapsed.
813
814                        NOTE:  SystemCPU provides a measure of the task's par‐
815                        ent process and does not include  CPU  time  of  child
816                        processes.
817
818
819              Timelimit What  the  timelimit was/is for the job. Format is the
820                        same as Elapsed.
821
822
823              TimelimitRaw
824                        What the timelimit was/is for the job.  Format  is  in
825                        number of minutes.
826
827
828              TotalCPU  The  sum of the SystemCPU and UserCPU time used by the
829                        job or job step.  The total CPU time of  the  job  may
830                        exceed  the  job's  elapsed time for jobs that include
831                        multiple job steps. Format is the same as Elapsed.
832
833                        NOTE: TotalCPU provides a measure of the task's parent
834                        process  and  does  not include CPU time of child pro‐
835                        cesses.
836
837
838              TresUsageInAve
839                        Tres average usage in by all tasks in job.   NOTE:  If
840                        corresponding  TresUsageInMaxTask  is -1 the metric is
841                        node centric instead of task.
842
843
844              TresUsageInMax
845                        Tres maximum usage in by all tasks in job.   NOTE:  If
846                        corresponding  TresUsageInMaxTask  is -1 the metric is
847                        node centric instead of task.
848
849
850              TresUsageInMaxNode
851                        Node for which each maximum TRES usage out occurred.
852
853
854              TresUsageInMaxTask
855                        Task for which each maximum TRES usage out occurred.
856
857
858              TresUsageInMin
859                        Tres minimum usage in by all tasks in job.   NOTE:  If
860                        corresponding  TresUsageInMinTask  is -1 the metric is
861                        node centric instead of task.
862
863
864              TresUsageInMinNode
865                        Node for which each minimum TRES usage out occurred.
866
867
868              TresUsageInMinTask
869                        Task for which each minimum TRES usage out occurred.
870
871
872              TresUsageInTot
873                        Tres total usage in by all tasks in job.
874
875
876              TresUsageOutAve
877                        Tres average usage out by all tasks in job.  NOTE:  If
878                        corresponding  TresUsageOutMaxTask is -1 the metric is
879                        node centric instead of task.
880
881
882              TresUsageOutMax
883                        Tres maximum usage out by all tasks in job.  NOTE:  If
884                        corresponding  TresUsageOutMaxTask is -1 the metric is
885                        node centric instead of task.
886
887
888              TresUsageOutMaxNode
889                        Node for which each maximum TRES usage out occurred.
890
891
892              TresUsageOutMaxTask
893                        Task for which each maximum TRES usage out occurred.
894
895
896              TresUsageOutTot
897                        Tres total usage out by all tasks in job.
898
899
900              UID       The user identifier of the user who ran the job.
901
902
903              User      The user name of the user who ran the job.
904
905
906              UserCPU   The amount of user CPU time used by  the  job  or  job
907                        step. Format is the same as Elapsed.
908
909                        NOTE:  UserCPU provides a measure of the task's parent
910                        process and does not include CPU time  of  child  pro‐
911                        cesses.
912
913
914              WCKey     Workload   Characterization   Key.   Arbitrary  string
915                        for grouping orthogonal accounts together.
916
917
918              WCKeyID   Reference to the wckey.
919
920
921

JOB STATE CODES

923       BF  BOOT_FAIL       Job terminated due to launch failure, typically due
924                           to a hardware failure (e.g. unable to boot the node
925                           or block and the job can not be requeued).
926
927       CA  CANCELLED       Job was explicitly cancelled by the user or  system
928                           administrator.   The  job  may or may not have been
929                           initiated.
930
931       CD  COMPLETED       Job has terminated all processes on all nodes  with
932                           an exit code of zero.
933
934       DL  DEADLINE        Job terminated on deadline.
935
936       F   FAILED          Job  terminated  with  non-zero  exit code or other
937                           failure condition.
938
939       NF  NODE_FAIL       Job terminated due to failure of one or more  allo‐
940                           cated nodes.
941
942       OOM OUT_OF_MEMORY   Job experienced out of memory error.
943
944       PD  PENDING         Job is awaiting resource allocation.
945
946       PR  PREEMPTED       Job terminated due to preemption.
947
948       R   RUNNING         Job currently has an allocation.
949
950       RQ  REQUEUED        Job was requeued.
951
952       RS  RESIZING        Job is about to change size.
953
954       RV  REVOKED         Sibling was removed from cluster due to other clus‐
955                           ter starting the job.
956
957       S   SUSPENDED       Job has an allocation, but execution has been  sus‐
958                           pended and CPUs have been released for other jobs.
959
960       TO  TIMEOUT         Job terminated upon reaching its time limit.
961
962

DEFAULT TIME WINDOW

964       The  options  --starttime  and --endtime define the time window between
965       which sacct is going to search. For historical  and  practical  reasons
966       their  default  values  (i.e. the default time window) depends on other
967       options: --jobs and --state.
968
969       Depending on if --jobs and/or --state are specified, the default values
970       of --starttime  and --endtime options are:
971
972
973       WITHOUT EITHER --jobs NOR --state specified:
974
975              ---starttime
976                     Defaults to Midnight.
977
978              ---endtime
979                     Defaults to Now.
980
981
982       WITH --jobs AND WITHOUT --state specified:
983
984              ---starttime
985                     Defaults to Epoch 0.
986
987              ---endtime
988                     Defaults to Now.
989
990
991       WITHOUT --jobs AND WITH --state specified:
992
993              ---starttime
994                     Defaults to Now.
995
996              ---endtime
997                     Defaults  to --starttime and to Now if --starttime is not
998                     specified.
999
1000
1001       WITH BOTH --jobs AND --state specified:
1002
1003              ---starttime
1004                     Defaults to Epoch 0.
1005
1006              ---endtime
1007                     Defaults to --starttime or to Now if --starttime  is  not
1008                     specified.
1009
1010
1011       NOTE:  With  -v/--verbose a message about the actual time window in use
1012       is shown.
1013
1014

PERFORMANCE

1016       Executing sacct sends a remote procedure call to  slurmdbd.  If  enough
1017       calls from sacct or other Slurm client commands that send remote proce‐
1018       dure calls to the slurmdbd daemon come in at once, it can result  in  a
1019       degradation  of  performance of the slurmdbd daemon, possibly resulting
1020       in a denial of service.
1021
1022       Do not run sacct or other Slurm client commands that send remote proce‐
1023       dure  calls  to slurmdbd from loops in shell scripts or other programs.
1024       Ensure that programs limit calls to sacct to the minimum necessary  for
1025       the information you are trying to gather.
1026
1027

ENVIRONMENT VARIABLES

1029       Some sacct options may be set via environment variables. These environ‐
1030       ment variables, along with  their  corresponding  options,  are  listed
1031       below. (Note: Commandline options will always override these settings.)
1032
1033       SACCT_FEDERATION    Same as --federation
1034
1035       SACCT_LOCAL         Same as --local
1036
1037       SLURM_CONF          The location of the Slurm configuration file.
1038
1039       SLURM_TIME_FORMAT   Specify  the  format  used to report time stamps. A
1040                           value of standard,  the  default  value,  generates
1041                           output            in            the            form
1042                           "year-month-dateThour:minute:second".  A  value  of
1043                           relative  returns  only "hour:minute:second" if the
1044                           current day.  For other dates in the  current  year
1045                           it  prints  the  "hour:minute" preceded by "Tomorr"
1046                           (tomorrow), "Ystday" (yesterday), the name  of  the
1047                           day  for the coming week (e.g. "Mon", "Tue", etc.),
1048                           otherwise the date  (e.g.  "25  Apr").   For  other
1049                           years  it  returns  a date month and year without a
1050                           time (e.g.  "6 Jun 2012"). All of the  time  stamps
1051                           use a 24 hour format.
1052
1053                           A  valid  strftime()  format can also be specified.
1054                           For example, a value of "%a %T" will report the day
1055                           of the week and a time stamp (e.g. "Mon 12:34:56").
1056
1057

EXAMPLES

1059       This example illustrates the default invocation of the sacct command:
1060
1061              # sacct
1062              Jobid      Jobname    Partition    Account AllocCPUS State     ExitCode
1063              ---------- ---------- ---------- ---------- ---------- ---------- --------
1064              2          script01   srun       acct1               1 RUNNING           0
1065              3          script02   srun       acct1               1 RUNNING           0
1066              4          endscript  srun       acct1               1 RUNNING           0
1067              4.0                   srun       acct1               1 COMPLETED         0
1068
1069
1070       This  example  shows the same job accounting information with the brief
1071       option.
1072
1073              # sacct --brief
1074                   Jobid     State  ExitCode
1075              ---------- ---------- --------
1076              2          RUNNING           0
1077              3          RUNNING           0
1078              4          RUNNING           0
1079              4.0        COMPLETED         0
1080
1081              # sacct --allocations
1082              Jobid      Jobname    Partition Account    AllocCPUS  State     ExitCode
1083              ---------- ---------- ---------- ---------- ------- ---------- --------
1084              3          sja_init   andy       acct1            1 COMPLETED         0
1085              4          sjaload    andy       acct1            2 COMPLETED         0
1086              5          sja_scr1   andy       acct1            1 COMPLETED         0
1087              6          sja_scr2   andy       acct1           18 COMPLETED         2
1088              7          sja_scr3   andy       acct1           18 COMPLETED         0
1089              8          sja_scr5   andy       acct1            2 COMPLETED         0
1090              9          sja_scr7   andy       acct1           90 COMPLETED         1
1091              10         endscript  andy       acct1          186 COMPLETED         0
1092
1093
1094       This example demonstrates the ability to customize the  output  of  the
1095       sacct command.  The fields are displayed in the order designated on the
1096       command line.
1097
1098              # sacct --format=jobid,elapsed,ncpus,ntasks,state
1099                   Jobid    Elapsed      Ncpus   Ntasks     State
1100              ---------- ---------- ---------- -------- ----------
1101              3            00:01:30          2        1 COMPLETED
1102              3.0          00:01:30          2        1 COMPLETED
1103              4            00:00:00          2        2 COMPLETED
1104              4.0          00:00:01          2        2 COMPLETED
1105              5            00:01:23          2        1 COMPLETED
1106              5.0          00:01:31          2        1 COMPLETED
1107
1108
1109       This example demonstrates the use of the -T  (--truncate)  option  when
1110       used  with  -S  (--starttime) and -E (--endtime). When the -T option is
1111       used, the start time of the job will be the specified -S value  if  the
1112       job  was  started before the specified time, otherwise the time will be
1113       the job's start time. The end time will be the specified -E  option  if
1114       the  job  ends  after the specified time, otherwise it will be the jobs
1115       end time.
1116
1117       Without -T (normal operation) sacct output would be like this.
1118
1119              # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1120                  JobID                 Start                  End        State
1121              --------- --------------------- -------------------- ------------
1122              2         2014-07-03T11:33:16   2014-07-03T11:59:01   COMPLETED
1123              3         2014-07-03T11:35:21   Unknown               RUNNING
1124              4         2014-07-03T11:35:21   2014-07-03T11:45:21   COMPLETED
1125              5         2014-07-03T11:41:01   Unknown               RUNNING
1126
1127       By adding the -T option the job's start and end times are truncated  to
1128       reflect only the time requested.  If a job started after the start time
1129       requested or finished before the end time requested those times are not
1130       altered.  The -T option is useful when determining exact run times dur‐
1131       ing any given period.
1132
1133              # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1134                  JobID                 Start                  End        State
1135              --------- --------------------- -------------------- ------------
1136              2         2014-07-03T11:40:00   2014-07-03T11:59:01   COMPLETED
1137              3         2014-07-03T11:40:00   2014-07-03T12:00:00   RUNNING
1138              4         2014-07-03T11:40:00   2014-07-03T11:45:21   COMPLETED
1139              5         2014-07-03T11:41:01   2014-07-03T12:00:00   RUNNING
1140
1141
1142       NOTE: If no -s (--state) option is given sacct  will  display  eligible
1143       jobs during the specified period of time, otherwise it will return jobs
1144       that were in the state requested during that period of time.
1145
1146       This example demonstrates the differences running sacct with and  with‐
1147       out  the  --state  flag  for  the same time period. Without the --state
1148       option, all eligible jobs in that time period are shown.
1149
1150              # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1151                     JobID               Start                 End      State
1152              ------------ ------------------- ------------------- ----------
1153              2955                    11:15:12            11:20:12  COMPLETED
1154              2956                    11:20:13            11:25:13  COMPLETED
1155
1156       With the --state=pending option, only job 2956 will be shown because it
1157       had  a  dependency on 2955 and was still PENDING from 11:20:00 until it
1158       started at 11:21:13. Note that even though we requested  PENDING  jobs,
1159       the  State  shows as COMPLETED because that is the current State of the
1160       job.
1161
1162              # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1163                     JobID               Start                 End      State
1164              ------------ ------------------- ------------------- ----------
1165              2956                    11:20:13            11:25:13  COMPLETED
1166

COPYING

1168       Copyright (C) 2005-2007 Copyright Hewlett-Packard  Development  Company
1169       L.P.
1170       Copyright (C) 2008-2010 Lawrence Livermore National Security.  Produced
1171       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1172       Copyright (C) 2010-2014 SchedMD LLC.
1173
1174       This file is  part  of  Slurm,  a  resource  management  program.   For
1175       details, see <https://slurm.schedmd.com/>.
1176
1177       Slurm  is free software; you can redistribute it and/or modify it under
1178       the terms of the GNU General Public License as published  by  the  Free
1179       Software  Foundation;  either  version  2  of  the License, or (at your
1180       option) any later version.
1181
1182       Slurm is distributed in the hope that it will be  useful,  but  WITHOUT
1183       ANY  WARRANTY;  without even the implied warranty of MERCHANTABILITY or
1184       FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General  Public  License
1185       for more details.
1186

FILES

1188       /etc/slurm.conf
1189                 Entries  to this file enable job accounting and designate the
1190                 job accounting log file that collects system job accounting.
1191
1192       /var/log/slurm_accounting.log
1193                 The default job accounting log file.  By default,  this  file
1194                 is set to read and write permission for root only.
1195

SEE ALSO

1197       sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1198
1199
1200
1201October 2020                    Slurm Commands                        sacct(1)
Impressum