1sacct(1)                        Slurm Commands                        sacct(1)
2
3
4

NAME

6       sacct  -  displays  accounting  data  for all jobs and job steps in the
7       Slurm job accounting log or Slurm database
8
9

SYNOPSIS

11       sacct [OPTIONS...]
12
13

DESCRIPTION

15       Accounting information for jobs invoked with Slurm are either logged in
16       the  job accounting log file or saved to the Slurm database, as config‐
17       ured with the AccountingStorageType parameter.
18
19       The sacct command displays job accounting data stored in  the  job  ac‐
20       counting  log  file  or  Slurm  database in a variety of forms for your
21       analysis.  The sacct command displays information on jobs,  job  steps,
22       status,  and  exitcodes by default.  You can tailor the output with the
23       use of the --format= option to specify the fields to be shown.
24
25       Job records consist of a primary entry for the job as a whole  as  well
26       as  entries  for job steps. The Job Launch page has a more detailed de‐
27       scription       of       each       type       of       job       step.
28       <https://slurm.schedmd.com/job_launch.html#job_record>
29
30       For  the  root user, the sacct command displays job accounting data for
31       all users, although there are options to filter the  output  to  report
32       only the jobs from a specified user or group.
33
34       For  the non-root user, the sacct command limits the display of job ac‐
35       counting data to jobs that were launched with their own user identifier
36       (UID) by default.  Data for other users can be displayed with the --al‐
37       lusers, --user, or --uid options.
38
39       Elapsed time fields are presented as  [days-]hours:minutes:seconds[.mi‐
40       croseconds]. Only 'CPU' fields will ever have microseconds.
41
42       The  default  input  file is the file named in the AccountingStorageLoc
43       parameter in slurm.conf.
44
45       NOTE: If designated, the slurmdbd.conf option PrivateData  may  further
46       restrict  the accounting data visible to users which are not SlurmUser,
47       root, or a user with AdminLevel=Admin. See the slurmdbd.conf  man  page
48       for additional details on restricting access to accounting data.
49
50       NOTE:  The  contents  of Slurm's database are maintained in lower case.
51       This may result in some sacct output differing from that of other Slurm
52       commands.
53
54       NOTE:  Much  of  the  data  reported by sacct has been generated by the
55       wait3() and getrusage() system calls. Some systems  gather  and  report
56       incomplete  information  for these calls; sacct reports values of 0 for
57       this missing data. See your systems getrusage (3) man page for informa‐
58       tion about which data are actually available on your system.
59
60

OPTIONS

62       -A, --accounts=<account_list>
63                 Displays  jobs  when  a  comma separated list of accounts are
64                 given as the argument.
65
66       -L, --allclusters
67                 Display jobs ran on all clusters. By default, only  jobs  ran
68                 on the cluster from where sacct is called are displayed.
69
70       -X, --allocations
71                 Only  show  statistics relevant to the job allocation itself,
72                 not taking steps into consideration.
73
74                 NOTE: Without including steps, utilization statistics for job
75                 allocation(s) will be reported as zero.
76
77       -a, --allusers
78                 Displays all users' jobs when run by user root or if Private‐
79                 Data is not configured to jobs.  Otherwise display  the  cur‐
80                 rent user's jobs
81
82       -x, --associations=<assoc_list>
83                 Displays  the  statistics only for the jobs running under the
84                 association ids specified by the assoc_list operand, which is
85                 a  comma-separated list of association ids.  Space characters
86                 are not allowed in the assoc_list. Default  is  all  associa‐
87                 tions.
88
89       -B, --batch-script
90                 This  option  will  print  the batch script of job if the job
91                 used one. If the job didn't have a script 'NONE' is output.
92                 NOTE: AccountingStoreFlags=job_script is required for this.
93                 NOTE: Requesting specific job(s) with '-j'  is  required  for
94                 this.
95
96       -b, --brief
97                 Displays  a brief listing consisting of JobID, State, and Ex‐
98                 itCode.
99
100       -M, --clusters=<cluster_list>
101                 Displays the statistics only for  the  jobs  started  on  the
102                 clusters  specified  by  the cluster_list operand, which is a
103                 comma-separated list of clusters.  Space characters  are  not
104                 allowed  in the cluster_list.  A value of 'all' will query to
105                 run on all clusters.  The default is current cluster you  are
106                 executing the sacct command on or all clusters in the federa‐
107                 tion when executed on a federated cluster.  This  option  im‐
108                 plicitly sets the --local option.
109
110       -c, --completion
111                 Use  job completion data instead of job accounting.  The Job‐
112                 CompType parameter in the slurm.conf file must be defined  to
113                 a non-none option.  Does not support federated cluster infor‐
114                 mation (local data only).
115
116       -C, --constraints=<constraint_list>
117                 Comma separated list  to  filter  jobs  based  on  what  con‐
118                 straints/features  the  job requested.  Multiple options will
119                 be treated as 'and' not 'or', so the job would need all  con‐
120                 straints specified to be returned not one or the other.
121
122       --delimiter=<characters>
123                 ASCII  characters used to separate the fields when specifying
124                 the -p or -P options. The default delimiter is  a  '|'.  This
125                 option is ignored if -p or -P options are not specified.
126
127       -D, --duplicates
128                 If  Slurm  job  ids are reset, some job numbers will probably
129                 appear more than once in the accounting log file but refer to
130                 different  jobs.  Such jobs can be distinguished by the "sub‐
131                 mit" time stamp in the data records.
132
133                 When data for specific jobs are requested with the --jobs op‐
134                 tion,  sacct  returns  the  most recent job with that number.
135                 This behavior can be overridden by  specifying  --duplicates,
136                 in  which  case all records that match the selection criteria
137                 will be returned.
138
139                 NOTE: Revoked federated sibling jobs are  hidden  unless  the
140                 --duplicates option is specified.
141
142       -E, --endtime=<end_time>
143                 Select  jobs  in  any  state  before  the specified time.  If
144                 states are given with the -s option return jobs in this state
145                 before this period.  See the DEFAULT TIME WINDOW for more de‐
146                 tails.
147
148                 Valid time formats are:
149                 HH:MM[:SS][AM|PM]
150                 MMDD[YY][-HH:MM[:SS]]
151                 MM.DD[.YY][-HH:MM[:SS]]
152                 MM/DD[/YY][-HH:MM[:SS]]
153                 YYYY-MM-DD[THH:MM[:SS]]
154                 today, midnight, noon, fika (3 PM), teatime (4 PM)
155                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
156
157       --env-vars
158                 This option will print the running  environment  of  a  batch
159                 job, otherwise 'NONE' is output.
160                 NOTE: AccountingStoreFlags=job_env is required for this.
161                 NOTE:  Requesting  specific  job(s) with '-j' is required for
162                 this.
163
164       --federation
165                 Show jobs from the federation if a member of one.
166
167       -f, --file=<file>
168                 Causes the sacct command to read job accounting data from the
169                 named  file  instead  of the current Slurm job accounting log
170                 file.  Only  applicable  when  running  the   jobcomp/filetxt
171                 plugin. Setting this flag implicitly enables the -c flag.
172
173       -F, --flags=<flag_list>
174                 Comma  separated  list  to  filter jobs based on what various
175                 ways the jobs were handled.  Current flags  are  SchedSubmit,
176                 SchedMain,  SchedBackfill.  These particular options describe
177                 the scheduler that started the job.
178
179       -o, --format
180                 Comma separated list of fields.  (use  "--helpformat"  for  a
181                 list of available fields).
182
183                 NOTE: When using the format option for listing various fields
184                 you can put a %NUMBER afterwards to specify how many  charac‐
185                 ters should be printed.
186
187                 e.g.  format=name%30  will  print 30 characters of field name
188                 right justified.  A %-30 will print 30 characters left justi‐
189                 fied.
190
191                 When set, the SACCT_FORMAT environment variable will override
192                 the default format.  For example:
193
194                 SACCT_FORMAT="jobid,user,account,cluster"
195
196       -g, --gid=, --group=<gid_or_group_list>
197                 Displays the statistics only for the jobs  started  with  the
198                 GID  or the GROUP specified by the gid_list or the group_list
199                 operand, which is a comma-separated list.   Space  characters
200                 are not allowed.  Default is no restrictions.
201
202       -h, --help
203                 Displays a general help message.
204
205       -e, --helpformat
206                 Print  a list of fields that can be specified with the --for‐
207                 mat option.
208
209                 Fields available:
210
211                 Account             AdminComment        AllocCPUS           AllocNodes
212                 AllocTRES           AssocID             AveCPU              AveCPUFreq
213                 AveDiskRead         AveDiskWrite        AvePages            AveRSS
214                 AveVMSize           BlockID             Cluster             Comment
215                 Constraints         ConsumedEnergy      ConsumedEnergyRaw   Container
216                 CPUTime             CPUTimeRAW          DBIndex             DerivedExitCode
217                 Elapsed             ElapsedRaw          Eligible            End
218                 ExitCode            Flags               GID                 Group
219                 JobID               JobIDRaw            JobName             Layout
220                 MaxDiskRead         MaxDiskReadNode     MaxDiskReadTask     MaxDiskWrite
221                 MaxDiskWriteNode    MaxDiskWriteTask    MaxPages            MaxPagesNode
222                 MaxPagesTask        MaxRSS              MaxRSSNode          MaxRSSTask
223                 MaxVMSize           MaxVMSizeNode       MaxVMSizeTask       McsLabel
224                 MinCPU              MinCPUNode          MinCPUTask          NCPUS
225                 NNodes              NodeList            NTasks              Partition
226                 Priority            QOS                 QOSRAW              Reason
227                 ReqCPUFreq          ReqCPUFreqGov       ReqCPUFreqMax       ReqCPUFreqMin
228                 ReqCPUS             ReqMem              ReqNodes            ReqTRES
229                 Reservation         ReservationId       Reserved            ResvCPU
230                 ResvCPURAW          Start               State               Submit
231                 SubmitLine          Suspended           SystemComment       SystemCPU
232                 Timelimit           TimelimitRaw        TotalCPU            TRESUsageInAve
233                 TRESUsageInMax      TRESUsageInMaxNode  TRESUsageInMaxTask  TRESUsageInMin
234                 TRESUsageInMinNode  TRESUsageInMinTask  TRESUsageInTot      TRESUsageOutAve
235                 TRESUsageOutMax     TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
236                 TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot     UID
237                 User                UserCPU             WCKey               WCKeyID
238                 WorkDir
239
240       NOTE:  When  using  with  Ave[RSS|VM]Size  or  their  values  in   TRE‐
241       SUsageIn[Ave|Tot].  They represent the average/total of the highest wa‐
242       termarks over all ranks in the step.  When using sstat  they  represent
243       the average/total at the moment the command was run.
244
245       NOTE:  TRESUsage*Min* values represent the lowest highwater mark in the
246       step.
247
248       The section titled "Job Accounting Fields" describes these fields.
249
250       -j, --jobs=<job[.step]>
251                 Displays information about the specified job[.step]  or  list
252                 of job[.step]s.
253
254                 The  job[.step]  parameter is a comma-separated list of jobs.
255                 Space characters are not permitted in this list.
256                 NOTE: A step id of 'batch' will display the information about
257                 the batch step.
258                 By default sacct shows only jobs with Eligible time, but with
259                 this option the non-eligible will be also shown.
260                 NOTE: If --state is also specified, as non-eligible  are  not
261                 PD,  then  non-eligible  jobs will not be displayed.  See the
262                 DEFAULT TIME WINDOW for details about how this option changes
263                 the default -S and -E options.
264
265       --json    Dump  job information as JSON. All other formatting arguments
266                 will be ignored.
267
268       --local   Show only jobs local to this cluster. Ignore  other  clusters
269                 in this federation (if any). Overrides --federation.
270
271       -l, --long
272                 Equivalent to specifying:
273
274                 --format=jobid,jodidraw,jobname,partition,maxvmsize,maxvm‐
275                 sizenode, maxvmsizetask,avevm‐
276                 size,maxrss,maxrssnode,maxrsstask,averss,maxpages, max‐
277                 pagesnode,maxpagestask,avepages,mincpu,mincpun‐
278                 ode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exit‐
279                 code,avecpufreq,reqcpufreqmin,reqcpufreqmax, reqcpufreq‐
280                 gov,reqmem,consumeden‐
281                 ergy,maxdiskread,maxdiskreadnode,maxdiskreadtask,
282                 avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwrite‐
283                 task,avediskwrite, reqtres,alloctres,tresusageinave,tre‐
284                 susageinmax, tresusageinmaxn,tresusageinmaxt,tresusagein‐
285                 min,tresusageinminn,tresusageinmint, tresusageintot,tre‐
286                 susageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusage‐
287                 outave,tresusageouttot
288
289       --name=<jobname_list>
290                 Display jobs that have any of these name(s).
291
292       -i, --nnodes=<min[-max]>
293                 Return jobs that ran on the specified number of nodes.
294
295       -I, --ncpus=<min[-max]>
296                 Return jobs that ran on the specified number of cpus.
297
298       --noconvert
299                 Don't convert units from  their  original  type  (e.g.  2048M
300                 won't be converted to 2G).
301
302       -N, --nodelist=<node_list>
303                 Display  jobs that ran on any of these node(s). node_list can
304                 be a ranged string.
305
306       -n, --noheader
307                 No heading will be added to the output. The default action is
308                 to display a header.
309
310       -p, --parsable
311                 Output  will be '|' delimited with a '|' at the end. See also
312                 the --delimiter option.
313
314       -P, --parsable2
315                 Output will be '|' delimited without a '|' at  the  end.  See
316                 also the --delimiter option.
317
318       -r, --partition
319                 Comma  separated  list  of  partitions to select jobs and job
320                 steps from. The default is all partitions.
321
322       -q, --qos Only send data about jobs using these qos.  Default is all.
323
324       -R, --reason=<reason_list>
325                 Comma separated list to filter jobs based on what reason  the
326                 job wasn't scheduled outside resources/priority.
327
328       -S, --starttime
329                 Select jobs in any state after the specified time. Default is
330                 00:00:00 of the current day, unless the '-s' or '-j'  options
331                 are  used.  If  the  '-s' option is used, then the default is
332                 'now'. If states are given with the  '-s'  option  then  only
333                 jobs in this state at this time will be returned. If the '-j'
334                 option is used, then the default time is Unix  Epoch  0.  See
335                 the DEFAULT TIME WINDOW for more details.
336
337                 Valid time formats are:
338                 HH:MM[:SS][AM|PM]
339                 MMDD[YY][-HH:MM[:SS]]
340                 MM.DD[.YY][-HH:MM[:SS]]
341                 MM/DD[/YY][-HH:MM[:SS]]
342                 YYYY-MM-DD[THH:MM[:SS]]
343                 today, midnight, noon, fika (3 PM), teatime (4 PM)
344                 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
345
346       -s, --state=<state_list>
347                 Selects  jobs  based  on  their  state during the time period
348                 given.  Unless otherwise specified, the start  and  end  time
349                 will be the current time when the --state option is specified
350                 and only currently running jobs can be  displayed.   A  start
351                 and/or  end  time must be specified to view information about
352                 jobs not currently running.  See the JOB STATE CODES  section
353                 below  for a list of state designators.  Multiple state names
354                 may be specified using comma separators. Either the short  or
355                 long  form  of  the  state  name may be used (e.g. CA or CAN‐
356                 CELLED) and the name is case insensitive (i.e. ca and CA both
357                 work).
358
359                 NOTE:  Note  for a job to be selected in the PENDING state it
360                 must have "EligibleTime" in the requested  time  interval  or
361                 different  from "Unknown". The "EligibleTime" is displayed by
362                 the "scontrol show job" command.  For example jobs  submitted
363                 with  the "--hold" option will have "EligibleTime=Unknown" as
364                 they are pending indefinitely.
365
366                 NOTE: When specifying states and no start time is  given  the
367                 default  start  time  is  'now'.  This is only when -j is not
368                 used.  If -j is used the start time will default to  'Epoch'.
369                 In  both  cases  if  no  end time is given it will default to
370                 'now'. See the DEFAULT TIME WINDOW for more details.
371
372       -K, --timelimit-max
373                 Ignored by itself, but if timelimit_min is set this  will  be
374                 the  maximum  timelimit of the range.  Default is no restric‐
375                 tion.
376
377       -k, --timelimit-min
378                 Only send data about jobs with this timelimit.  If used  with
379                 timelimit_max  this  will  be  the  minimum  timelimit of the
380                 range.  Default is no restriction.
381
382       -T, --truncate
383                 Truncate time.  So if a job started  before  --starttime  the
384                 start  time  would be truncated to --starttime.  The same for
385                 end time and --endtime.
386
387       -u, --uid=, --user=<uid_or_user_list>
388                 Use this comma separated list of UIDs or user names to select
389                 jobs to display.  By default, the running user's UID is used.
390
391       --units=[KMGTP]
392                 Display  values in specified unit type. Takes precedence over
393                 --noconvert option.
394
395       --usage   Display a command usage summary.
396
397       --use-local-uid
398                 When displaying UID, sacct uses the UID stored in Slurm's ac‐
399                 counting  database by default. Use this command to make Slurm
400                 use a system call to get the UID from the username. This  op‐
401                 tion  may  be useful in an environment with multiple clusters
402                 and one database where the UID's aren't the same on all clus‐
403                 ters.
404
405       -v, --verbose
406                 Primarily for debugging purposes, report the state of various
407                 variables during processing.
408
409       -V, --version
410                 Print version.
411
412       -W, --wckeys=<wckey_list>
413                 Displays the statistics only for the jobs started on the wck‐
414                 eys   specified   by  the  wckey_list  operand,  which  is  a
415                 comma-separated list of wckey names.   Space  characters  are
416                 not allowed in the wckey_list. Default is all wckeys.
417
418       --whole-hetjob[=yes|no]
419                 When  querying  and filtering heterogeneous jobs with --jobs,
420                 Slurm will default to retrieving information  about  all  the
421                 components  of  the  job if the het_job_id (leader id) is se‐
422                 lected. If a non-leader heterogeneous job component id is se‐
423                 lected then only that component is retrieved by default. This
424                 behavior can be changed by using this option. If set to 'yes'
425                 (or  no  argument), then information about all the components
426                 will be retrieved no matter which component  is  selected  in
427                 the job filter. If set to 'no' then only the selected hetero‐
428                 geneous job component(s) will be retrieved, even when select‐
429                 ing the leader.
430
431       --yaml    Dump  job information as YAML. All other formatting arguments
432                 will be ignored.
433
434   Job Accounting Fields
435       Descriptions of each job accounting field can  be  found  below.   Note
436       that  the  Ave*, Max* and Min* accounting fields look at the values for
437       all the tasks of each step in a job and return the average, maximum  or
438       minimum values for the job step.
439
440              ALL       Print all fields listed below.
441
442              Account   Account the job ran under.
443
444              AdminComment
445                        A  comment  string on a job that must be set by an ad‐
446                        ministrator, the SlurmUser or root.
447
448              AllocCPUs Count of allocated CPUs. Equivalent to NCPUS.
449
450              AllocNodes
451                        Number of nodes allocated to the job/step.  0  if  the
452                        job is pending.
453
454              AllocTres Trackable resources. These are the resources allocated
455                        to the job/step after the job  started  running.   For
456                        pending  jobs  this should be blank.  For more details
457                        see AccountingStorageTRES in slurm.conf.
458
459                        NOTE: When a generic resource is configured  with  the
460                        no_consume flag, the allocation will be printed with a
461                        zero.
462
463              AssocID   Reference to the  association  of  user,  account  and
464                        cluster.
465
466              AveCPU    Average (system + user) CPU time of all tasks in job.
467
468              AveCPUFreq
469                        Average weighted CPU frequency of all tasks in job, in
470                        kHz.
471
472              AveDiskRead
473                        Average number of bytes read by all tasks in job.
474
475              AveDiskWrite
476                        Average number of bytes written by all tasks in job.
477
478              AvePages  Average number of page faults of all tasks in job.
479
480              AveRSS    Average resident set size of all tasks in job.
481
482              AveVMSize Average Virtual Memory size of all tasks in job.
483
484              BlockID   The name of the block to be used (used with Blue  Gene
485                        systems).
486
487              Cluster   Cluster name.
488
489              Comment   The job's comment string when the AccountingStoreFlags
490                        parameter in the slurm.conf  file  contains  'job_com‐
491                        ment'.  The Comment string can be modified by invoking
492                        sacctmgr modify job  or  the  specialized  sjobexitmod
493                        command.
494
495              Constraints
496                        Feature(s) the job requested as a constraint.
497
498              ConsumedEnergy
499                        Total  energy  consumed  by  all  tasks  in  a job, in
500                        joules.  Value may include a unit prefix  (K,M,G,T,P).
501                        Note:  Only in the case of an exclusive job allocation
502                        does this value reflect the job's real energy consump‐
503                        tion.
504
505              ConsumedEnergyRaw
506                        Total  energy  consumed  by  all  tasks  in  a job, in
507                        joules.  Note: Only in the case of  an  exclusive  job
508                        allocation  does this value reflect the job's real en‐
509                        ergy consumption.
510
511              Container Path to OCI Container Bundle requested.
512
513              CPUTime   Time used (Elapsed time * CPU count) by a job or  step
514                        in HH:MM:SS format.
515
516              CPUTimeRAW
517                        Time  used (Elapsed time * CPU count) by a job or step
518                        in cpu-seconds.
519
520              DBIndex   Unique database index for entries in the job table.
521
522              DerivedExitCode
523                        The highest exit code returned by the job's job  steps
524                        (srun invocations).  Following the colon is the signal
525                        that caused the process to terminate if it was  termi‐
526                        nated  by  a signal.  The DerivedExitCode can be modi‐
527                        fied by invoking sacctmgr modify job or  the  special‐
528                        ized sjobexitmod command.
529
530              Elapsed   The job's elapsed time.
531
532                        The format of this field's output is as follows:
533
534                        [DD-[HH:]]MM:SS
535
536                        as defined by the following:
537
538                        DD        days
539
540                        hh        hours
541
542                        mm        minutes
543
544                        ss        seconds
545
546              ElapsedRaw
547                        The job's elapsed time in seconds.
548
549              Eligible  When  the job became eligible to run. In the same for‐
550                        mat as End.
551
552              End       Termination time of the job. The output is of the for‐
553                        mat  YYYY-MM-DDTHH:MM:SS,  unless  changed through the
554                        SLURM_TIME_FORMAT environment variable.
555
556              ExitCode  The exit code returned by the job  script  or  salloc,
557                        typically  as  set  by the exit() function.  Following
558                        the colon is the signal that  caused  the  process  to
559                        terminate if it was terminated by a signal.
560
561              Flags     Job  flags.  Current flags are SchedSubmit, SchedMain,
562                        SchedBackfill.
563
564              GID       The group identifier of the user who ran the job.
565
566              Group     The group name of the user who ran the job.
567
568              JobID     The identification number of the job or job step.
569
570                        Regular jobs are in the form:
571
572                        JobID[.JobStep]
573
574                        Array jobs are in the form:
575
576                        ArrayJobID_ArrayTaskID
577
578                        Heterogeneous jobs are in the form:
579
580                        HetJobID+HetJobOffset
581
582                        When printing job arrays, performance of  the  command
583                        can be measurably improved for systems with large num‐
584                        bers of jobs when a single job ID is specified. By de‐
585                        fault,  this  field  size will be limited to 64 bytes.
586                        Use the environment variable SLURM_BITSTR_LEN to spec‐
587                        ify larger field sizes.
588
589              JobIDRaw  The  identification  number  of  the  job or job step.
590                        Prints the JobID in the form JobID[.JobStep] for regu‐
591                        lar, heterogeneous and array jobs.
592
593              JobName   The  name  of  the job or job step. The slurm_account‐
594                        ing.log file is a space  delimited  file.  Because  of
595                        this  if  a space is used in the jobname an underscore
596                        is substituted for the  space  before  the  record  is
597                        written to the accounting file. So when the jobname is
598                        displayed by sacct the jobname that had a space in  it
599                        will now have an underscore in place of the space.
600
601              Layout    What  the  layout  of  a step was when it was running.
602                        This can be used to give you an idea of which node ran
603                        which rank in your job.
604
605              MaxDiskRead
606                        Maximum number of bytes read by all tasks in job.
607
608              MaxDiskReadNode
609                        The node on which the maxdiskread occurred.
610
611              MaxDiskReadTask
612                        The task ID where the maxdiskread occurred.
613
614              MaxDiskWrite
615                        Maximum number of bytes written by all tasks in job.
616
617              MaxDiskWriteNode
618                        The node on which the maxdiskwrite occurred.
619
620              MaxDiskWriteTask
621                        The task ID where the maxdiskwrite occurred.
622
623              MaxPages  Maximum number of page faults of all tasks in job.
624
625              MaxPagesNode
626                        The node on which the maxpages occurred.
627
628              MaxPagesTask
629                        The task ID where the maxpages occurred.
630
631              MaxRSS    Maximum resident set size of all tasks in job.
632
633              MaxRSSNode
634                        The node on which the maxrss occurred.
635
636              MaxRSSTask
637                        The task ID where the maxrss occurred.
638
639              MaxVMSize Maximum Virtual Memory size of all tasks in job.
640
641              MaxVMSizeNode
642                        The node on which the maxvmsize occurred.
643
644              MaxVMSizeTask
645                        The task ID where the maxvmsize occurred.
646
647              MCSLabel  Multi-Category  Security  (MCS)  label associated with
648                        the job.  Added to a job when the MCSPlugin is enabled
649                        in the slurm.conf.
650
651              MinCPU    Minimum (system + user) CPU time of all tasks in job.
652
653              MinCPUNode
654                        The node on which the mincpu occurred.
655
656              MinCPUTask
657                        The task ID where the mincpu occurred.
658
659              NCPUS     Total number of CPUs allocated to the job.  Equivalent
660                        to AllocCPUS.
661
662              NNodes    Number of nodes in a job or step.  If the job is  run‐
663                        ning, or ran, this count will be the number allocated,
664                        else the number will be the number requested.
665
666              NodeList  List of nodes in job/step.
667
668              NTasks    Total number of tasks in a job or step.
669
670              Partition Identifies the partition on which the job ran.
671
672              Priority  Slurm priority.
673
674              QOS       Name of Quality of Service.
675
676              QOSRAW    Numeric id of Quality of Service.
677
678              Reason    The last reason a job was  blocked  from  running  for
679                        something  other than Priority or Resources. This will
680                        be saved in the database even if the job ran  to  com‐
681                        pletion.
682
683              ReqCPUFreq
684                        Requested  CPU  frequency for the step, in kHz.  Note:
685                        This value applies only to a job step. No value is re‐
686                        ported for the job.
687
688              ReqCPUFreqGov
689                        Requested CPU frequency governor for the step, in kHz.
690                        Note: This value applies only to a job step. No  value
691                        is reported for the job.
692
693              ReqCPUFreqMax
694                        Maximum  requested CPU frequency for the step, in kHz.
695                        Note: This value applies only to a job step. No  value
696                        is reported for the job.
697
698              ReqCPUFreqMin
699                        Minimum  requested CPU frequency for the step, in kHz.
700                        Note: This value applies only to a job step. No  value
701                        is reported for the job.
702
703              ReqCPUS   Number of requested CPUs.
704
705              ReqMem    Minimum  required  memory  for  the job. It may have a
706                        letter  appended  to  it  indicating  units   (M   for
707                        megabytes,  G  for gigabytes, etc.).  Note: This value
708                        is only from the job allocation, not the step.
709
710              ReqNodes  Requested minimum Node count for the job/step.
711
712              ReqTres   Trackable resources. These are  the  minimum  resource
713                        counts  requested  by the job/step at submission time.
714                        For  more   details   see   AccountingStorageTRES   in
715                        slurm.conf.
716
717              Reservation
718                        Reservation Name.
719
720              ReservationId
721                        Reservation Id.
722
723              Reserved  How much wall clock time was used as reserved time for
724                        this job.  This is derived from how  long  a  job  was
725                        waiting   from  eligible  time  to  when  it  actually
726                        started.  Format is the same as Elapsed.
727
728              ResvCPU   How many CPU seconds were used as  reserved  time  for
729                        this job.  Format is the same as Elapsed.
730
731              ResvCPURAW
732                        How  many  CPU  seconds were used as reserved time for
733                        this job.  Format is in processor seconds.
734
735              Start     Initiation time of the job. In the same format as End.
736
737              State     Displays the job status, or state.  See the JOB  STATE
738                        CODES section below for a list of possible states.
739
740                        If more information is available on the job state than
741                        will fit into the current field  width  (for  example,
742                        the  UID  that CANCELLED a job) the state will be fol‐
743                        lowed by a "+".  You can increase the size of the dis‐
744                        played  state  using the "%NUMBER" format modifier de‐
745                        scribed earlier.
746
747                        NOTE: The RUNNING state will return suspended jobs  as
748                        well.   In  order to print suspended jobs you must re‐
749                        quest SUSPENDED at a different call from RUNNING.
750
751                        NOTE: The RUNNING state will return any jobs completed
752                        (cancelled  or otherwise) in the time period requested
753                        as the job was also RUNNING during that time.  If  you
754                        are only looking for jobs that finished, please choose
755                        the appropriate state(s) without the RUNNING state.
756
757              Submit    The time the job was submitted. In the same format  as
758                        End.
759
760                        NOTE:  If a job is requeued, the submit time is reset.
761                        To obtain the original submit time it is necessary  to
762                        use the -D or --duplicate option to display all dupli‐
763                        cate entries for a job.
764
765              SubmitLine
766                        The full command issued to submit the job.
767
768              Suspended The amount of time a job or job  step  was  suspended.
769                        Format is the same as Elapsed.
770
771              SystemComment
772                        The  job's  comment  string that is typically set by a
773                        plugin.  Can only be modified by a  Slurm  administra‐
774                        tor.
775
776              SystemCPU The  amount  of system CPU time used by the job or job
777                        step. Format is the same as Elapsed.
778
779                        NOTE: SystemCPU provides a measure of the task's  par‐
780                        ent  process  and  does  not include CPU time of child
781                        processes.
782
783              Timelimit What the timelimit was/is for the job. Format  is  the
784                        same as Elapsed.
785
786              TimelimitRaw
787                        What  the  timelimit  was/is for the job. Format is in
788                        number of minutes.
789
790              TotalCPU  The sum of the SystemCPU and UserCPU time used by  the
791                        job  or  job  step.  The total CPU time of the job may
792                        exceed the job's elapsed time for  jobs  that  include
793                        multiple job steps. Format is the same as Elapsed.
794
795                        NOTE: TotalCPU provides a measure of the task's parent
796                        process and does not include CPU time  of  child  pro‐
797                        cesses.
798
799              TresUsageInAve
800                        Tres  average  usage in by all tasks in job.  NOTE: If
801                        corresponding TresUsageInMaxTask is -1 the  metric  is
802                        node centric instead of task.
803
804              TresUsageInMax
805                        Tres  maximum  usage in by all tasks in job.  NOTE: If
806                        corresponding TresUsageInMaxTask is -1 the  metric  is
807                        node centric instead of task.
808
809              TresUsageInMaxNode
810                        Node for which each maximum TRES usage out occurred.
811
812              TresUsageInMaxTask
813                        Task for which each maximum TRES usage out occurred.
814
815              TresUsageInMin
816                        Tres  minimum  usage in by all tasks in job.  NOTE: If
817                        corresponding TresUsageInMinTask is -1 the  metric  is
818                        node centric instead of task.
819
820              TresUsageInMinNode
821                        Node for which each minimum TRES usage out occurred.
822
823              TresUsageInMinTask
824                        Task for which each minimum TRES usage out occurred.
825
826              TresUsageInTot
827                        Tres total usage in by all tasks in job.
828
829              TresUsageOutAve
830                        Tres  average usage out by all tasks in job.  NOTE: If
831                        corresponding TresUsageOutMaxTask is -1 the metric  is
832                        node centric instead of task.
833
834              TresUsageOutMax
835                        Tres  maximum usage out by all tasks in job.  NOTE: If
836                        corresponding TresUsageOutMaxTask is -1 the metric  is
837                        node centric instead of task.
838
839              TresUsageOutMaxNode
840                        Node for which each maximum TRES usage out occurred.
841
842              TresUsageOutMaxTask
843                        Task for which each maximum TRES usage out occurred.
844
845              TresUsageOutMin
846                        Tres minimum usage out by all tasks in job.
847
848              TresUsageOutMinNode
849                        Node for which each minimum TRES usage out occurred.
850
851              TresUsageOutMinTask
852                        Task for which each minimum TRES usage out occurred.
853
854              TresUsageOutTot
855                        Tres total usage out by all tasks in job.
856
857              UID       The user identifier of the user who ran the job.
858
859              User      The user name of the user who ran the job.
860
861              UserCPU   The  amount  of  user  CPU time used by the job or job
862                        step. Format is the same as Elapsed.
863
864                        NOTE: UserCPU provides a measure of the task's  parent
865                        process  and  does  not include CPU time of child pro‐
866                        cesses.
867
868              WCKey     Workload  Characterization  Key.    Arbitrary   string
869                        for grouping orthogonal accounts together.
870
871              WCKeyID   Reference to the wckey.
872
873              WorkDir   The directory used by the job to execute commands.
874

JOB STATE CODES

876       BF  BOOT_FAIL       Job terminated due to launch failure, typically due
877                           to a hardware failure (e.g. unable to boot the node
878                           or block and the job can not be requeued).
879
880       CA  CANCELLED       Job  was explicitly cancelled by the user or system
881                           administrator.  The job may or may  not  have  been
882                           initiated.
883
884       CD  COMPLETED       Job  has terminated all processes on all nodes with
885                           an exit code of zero.
886
887       DL  DEADLINE        Job terminated on deadline.
888
889       F   FAILED          Job terminated with non-zero  exit  code  or  other
890                           failure condition.
891
892       NF  NODE_FAIL       Job  terminated due to failure of one or more allo‐
893                           cated nodes.
894
895       OOM OUT_OF_MEMORY   Job experienced out of memory error.
896
897       PD  PENDING         Job is awaiting resource allocation.
898
899       PR  PREEMPTED       Job terminated due to preemption.
900
901       R   RUNNING         Job currently has an allocation.
902
903       RQ  REQUEUED        Job was requeued.
904
905       RS  RESIZING        Job is about to change size.
906
907       RV  REVOKED         Sibling was removed from cluster due to other clus‐
908                           ter starting the job.
909
910       S   SUSPENDED       Job  has an allocation, but execution has been sus‐
911                           pended and CPUs have been released for other jobs.
912
913       TO  TIMEOUT         Job terminated upon reaching its time limit.
914

DEFAULT TIME WINDOW

916       The options --starttime and --endtime define the  time  window  between
917       which  sacct  is  going to search. For historical and practical reasons
918       their default values (i.e. the default time window)  depends  on  other
919       options: --jobs and --state.
920
921       Depending on if --jobs and/or --state are specified, the default values
922       of --starttime  and --endtime options are:
923
924
925       WITHOUT EITHER --jobs NOR --state specified:
926       --starttime defaults to Midnight.
927       --endtime defaults to Now.
928
929
930       WITH --jobs AND WITHOUT --state specified:
931       --starttime defaults to Epoch 0.
932       --endtime defaults to Now.
933
934
935       WITHOUT --jobs AND WITH --state specified:
936       --starttime defaults to Now.
937       --endtime defaults to --starttime and to  Now  if  --starttime  is  not
938       specified.
939
940
941       WITH BOTH --jobs AND --state specified:
942       --starttime defaults to Epoch 0.
943       --endtime defaults to --starttime or to Now if --starttime is not spec‐
944       ified.
945
946
947       NOTE: With -v/--verbose a message about the actual time window  in  use
948       is shown.
949
950

PERFORMANCE

952       Executing  sacct  sends  a remote procedure call to slurmdbd. If enough
953       calls from sacct or other Slurm client commands that send remote proce‐
954       dure  calls  to the slurmdbd daemon come in at once, it can result in a
955       degradation of performance of the slurmdbd daemon,  possibly  resulting
956       in a denial of service.
957
958       Do not run sacct or other Slurm client commands that send remote proce‐
959       dure calls to slurmdbd from loops in shell scripts or  other  programs.
960       Ensure  that programs limit calls to sacct to the minimum necessary for
961       the information you are trying to gather.
962
963

ENVIRONMENT VARIABLES

965       Some sacct options may be set via environment variables. These environ‐
966       ment  variables, along with their corresponding options, are listed be‐
967       low. (Note: Command line options will always override these settings.)
968
969
970       SACCT_FEDERATION    Same as --federation
971
972       SACCT_LOCAL         Same as --local
973
974       SLURM_BITSTR_LEN    Specifies the string length to be used for  holding
975                           a job array's task ID expression. The default value
976                           is 64 bytes. A value of 0 will print the  full  ex‐
977                           pression  with  any  length required. Larger values
978                           may adversely impact the application performance.
979
980       SLURM_CONF          The location of the Slurm configuration file.
981
982       SLURM_TIME_FORMAT   Specify the format used to report  time  stamps.  A
983                           value  of  standard,  the  default value, generates
984                           output            in            the            form
985                           "year-month-dateThour:minute:second".   A  value of
986                           relative returns only "hour:minute:second"  if  the
987                           current  day.   For other dates in the current year
988                           it prints the "hour:minute"  preceded  by  "Tomorr"
989                           (tomorrow),  "Ystday"  (yesterday), the name of the
990                           day for the coming week (e.g. "Mon", "Tue",  etc.),
991                           otherwise  the  date  (e.g.  "25  Apr").  For other
992                           years it returns a date month and  year  without  a
993                           time  (e.g.   "6 Jun 2012"). All of the time stamps
994                           use a 24 hour format.
995
996                           A valid strftime() format can  also  be  specified.
997                           For example, a value of "%a %T" will report the day
998                           of the week and a time stamp (e.g. "Mon 12:34:56").
999

EXAMPLES

1001       This example illustrates the default invocation of the sacct command:
1002
1003              # sacct
1004              Jobid      Jobname    Partition    Account AllocCPUS State     ExitCode
1005              ---------- ---------- ---------- ---------- ---------- ---------- --------
1006              2          script01   srun       acct1               1 RUNNING           0
1007              3          script02   srun       acct1               1 RUNNING           0
1008              4          endscript  srun       acct1               1 RUNNING           0
1009              4.0                   srun       acct1               1 COMPLETED         0
1010
1011
1012       This example shows the same job accounting information with  the  brief
1013       option.
1014
1015              # sacct --brief
1016                   Jobid     State  ExitCode
1017              ---------- ---------- --------
1018              2          RUNNING           0
1019              3          RUNNING           0
1020              4          RUNNING           0
1021              4.0        COMPLETED         0
1022
1023              # sacct --allocations
1024              Jobid      Jobname    Partition Account    AllocCPUS  State     ExitCode
1025              ---------- ---------- ---------- ---------- ------- ---------- --------
1026              3          sja_init   andy       acct1            1 COMPLETED         0
1027              4          sjaload    andy       acct1            2 COMPLETED         0
1028              5          sja_scr1   andy       acct1            1 COMPLETED         0
1029              6          sja_scr2   andy       acct1           18 COMPLETED         2
1030              7          sja_scr3   andy       acct1           18 COMPLETED         0
1031              8          sja_scr5   andy       acct1            2 COMPLETED         0
1032              9          sja_scr7   andy       acct1           90 COMPLETED         1
1033              10         endscript  andy       acct1          186 COMPLETED         0
1034
1035
1036       This  example  demonstrates  the ability to customize the output of the
1037       sacct command.  The fields are displayed in the order designated on the
1038       command line.
1039
1040              # sacct --format=jobid,elapsed,ncpus,ntasks,state
1041                   Jobid    Elapsed      Ncpus   Ntasks     State
1042              ---------- ---------- ---------- -------- ----------
1043              3            00:01:30          2        1 COMPLETED
1044              3.0          00:01:30          2        1 COMPLETED
1045              4            00:00:00          2        2 COMPLETED
1046              4.0          00:00:01          2        2 COMPLETED
1047              5            00:01:23          2        1 COMPLETED
1048              5.0          00:01:31          2        1 COMPLETED
1049
1050
1051       This  example  demonstrates  the use of the -T (--truncate) option when
1052       used with -S (--starttime) and -E (--endtime). When the  -T  option  is
1053       used,  the  start time of the job will be the specified -S value if the
1054       job was started before the specified time, otherwise the time  will  be
1055       the  job's  start time. The end time will be the specified -E option if
1056       the job ends after the specified time, otherwise it will  be  the  jobs
1057       end time.
1058
1059       Without -T (normal operation) sacct output would be like this.
1060
1061              # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
1062                  JobID                 Start                  End        State
1063              --------- --------------------- -------------------- ------------
1064              2         2014-07-03T11:33:16   2014-07-03T11:59:01   COMPLETED
1065              3         2014-07-03T11:35:21   Unknown               RUNNING
1066              4         2014-07-03T11:35:21   2014-07-03T11:45:21   COMPLETED
1067              5         2014-07-03T11:41:01   Unknown               RUNNING
1068
1069       By  adding the -T option the job's start and end times are truncated to
1070       reflect only the time requested.  If a job started after the start time
1071       requested or finished before the end time requested those times are not
1072       altered.  The -T option is useful when determining exact run times dur‐
1073       ing any given period.
1074
1075              # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
1076                  JobID                 Start                  End        State
1077              --------- --------------------- -------------------- ------------
1078              2         2014-07-03T11:40:00   2014-07-03T11:59:01   COMPLETED
1079              3         2014-07-03T11:40:00   2014-07-03T12:00:00   RUNNING
1080              4         2014-07-03T11:40:00   2014-07-03T11:45:21   COMPLETED
1081              5         2014-07-03T11:41:01   2014-07-03T12:00:00   RUNNING
1082
1083
1084       NOTE:  If  no  -s (--state) option is given sacct will display eligible
1085       jobs during the specified period of time, otherwise it will return jobs
1086       that were in the state requested during that period of time.
1087
1088       This  example demonstrates the differences running sacct with and with‐
1089       out the --state flag for the same time period. Without the --state  op‐
1090       tion, all eligible jobs in that time period are shown.
1091
1092              # sacct -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1093                     JobID               Start                 End      State
1094              ------------ ------------------- ------------------- ----------
1095              2955                    11:15:12            11:20:12  COMPLETED
1096              2956                    11:20:13            11:25:13  COMPLETED
1097
1098       With the --state=pending option, only job 2956 will be shown because it
1099       had a dependency on 2955 and was still PENDING from 11:20:00  until  it
1100       started  at  11:21:13. Note that even though we requested PENDING jobs,
1101       the State shows as COMPLETED because that is the current State  of  the
1102       job.
1103
1104              # sacct --state=pending -S11:20:00 -E11:25:00 -X -ojobid,start,end,state
1105                     JobID               Start                 End      State
1106              ------------ ------------------- ------------------- ----------
1107              2956                    11:20:13            11:25:13  COMPLETED
1108
1109

COPYING

1111       Copyright  (C)  2005-2007 Copyright Hewlett-Packard Development Company
1112       L.P.
1113       Copyright (C) 2008-2010 Lawrence Livermore National Security.  Produced
1114       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
1115       Copyright (C) 2010-2022 SchedMD LLC.
1116
1117       This  file  is  part  of Slurm, a resource management program.  For de‐
1118       tails, see <https://slurm.schedmd.com/>.
1119
1120       Slurm is free software; you can redistribute it and/or modify it  under
1121       the  terms  of  the GNU General Public License as published by the Free
1122       Software Foundation; either version 2 of the License, or (at  your  op‐
1123       tion) any later version.
1124
1125       Slurm  is  distributed  in the hope that it will be useful, but WITHOUT
1126       ANY WARRANTY; without even the implied warranty of  MERCHANTABILITY  or
1127       FITNESS  FOR  A PARTICULAR PURPOSE.  See the GNU General Public License
1128       for more details.
1129
1130

FILES

1132       /etc/slurm.conf
1133                 Entries to this file enable job accounting and designate  the
1134                 job accounting log file that collects system job accounting.
1135
1136       /var/log/slurm_accounting.log
1137                 The  default  job accounting log file.  By default, this file
1138                 is set to read and write permission for root only.
1139
1140

SEE ALSO

1142       sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
1143
1144
1145
1146August 2022                     Slurm Commands                        sacct(1)
Impressum