1sreport(1)                      Slurm Commands                      sreport(1)
2
3
4

NAME

6       sreport - Generate reports from the slurm accounting data.
7
8

SYNOPSIS

10       sreport [OPTIONS...] [COMMAND...]
11
12

DESCRIPTION

14       sreport  is  used to generate reports of job usage and cluster utiliza‐
15       tion for Slurm jobs saved to the Slurm Database, slurmdbd.
16
17

OPTIONS

19       -a, --all_clusters
20              Use all clusters instead of only the cluster from which the com‐
21              mand was executed.
22
23       --federation
24              Generate a report for the federation if a member of one.
25
26       -h, --help
27              Print a help message describing the usage of sreport.
28
29       --local
30              Generate  a report for the local cluster, even if part of a fed‐
31              eration.  Overrides --federation
32
33       -M, --cluster
34              The cluster(s) to generate reports for. Default is  local  clus‐
35              ter,  unless the local cluster is currently part of a federation
36              and in that case generate a report for all clusters in the  cur‐
37              rent  federation.  If the clusters included in a federation vary
38              through time, use this option to identify the clusters to be in‐
39              cluded in report. Implies --local.
40
41       -n, --noheader
42              Don't display header when listing results.
43
44       -p, --parsable
45              Output will be '|' delimited with a '|' at the end.
46
47       -P, --parsable2
48              Output will be '|' delimited without a '|' at the end.
49
50       -Q, --quiet
51              Print no warning or informational messages, only error messages.
52
53       -t <format>
54              Specify the output time format. Time format options are case in‐
55              sensitive and may be abbreviated. The default format is Minutes.
56              Supported  time  format  options  are listed in the time command
57              section below.
58
59       -T,  --tres <tres_names>
60              Trackable resource (TRES) to report values for.  By default  CPU
61              resource  use  is  reported (except for reservation reports. All
62              TRES types reserved by a reservation will be  shown  by  default
63              unless overridden with this option).  Multiple TRES names may be
64              separated using a comma separated list for  all  reports  except
65              the  job  reports, which can only support a single TRES name, or
66              "ALL" for all TRES.
67
68              The "Reported" Billing  TRES  is  calculated  from  the  largest
69              Billing  TRES of each node multiplied by the time frame. For ex‐
70              ample, if a node is part of multiple partitions and each  has  a
71              different  TRESBillingWeights  defined  the Billing TRES for the
72              node will be the highest of  the  partitions.   If  TRESBilling‐
73              Weights  is  not  defined  on  any partition for a node then the
74              Billing TRES will be equal to the number of CPUs on the node.
75
76              TRES node usage is no longer reported in percent  format  or  in
77              Cluster Utilization.  Please use TRES CPU instead.  The main is‐
78              sue with using node is in most configurations multiple jobs  are
79              able  to  run on the same node.  This makes TRES node accounting
80              count the same node multiple times in the same period.   In  ex‐
81              clusive node configurations, CPU accounting returns the same us‐
82              age node accounting would.
83
84
85       -v, --verbose
86              Print detailed event logging.
87
88       -V , --version
89              Print version information and exit.
90
91

COMMANDS

93       <keyword> may be omitted from the execute line and sreport will
94              execute in interactive mode. sreport will  process  commands  as
95              entered until explicitly terminated.
96
97
98       exit   Terminate  the execution of sreport.  Identical to the quit com‐
99              mand.
100
101
102       help   Display a description of sreport options and commands.
103
104
105       nonparsable
106              Return output to normal after parsable  or  parsable2  has  been
107              set.
108
109
110       parsable
111              Output will be | delimited with an ending '|'.
112
113
114       parsable2
115              Output will be | delimited without an ending '|'.
116
117
118       quiet  Print  no  warning  or  informational messages, only fatal error
119              messages.
120
121
122       quit   Terminate the execution of sreport.  Identical to the exit  com‐
123              mand.
124
125
126       time <time_format>
127              Specify the output time format. Time format options are case in‐
128              sensitive and may be abbreviated. The default format is Minutes.
129              Supported time format options include:
130
131
132              SecPer    Seconds/Percentage of Total
133
134              MinPer    Minutes/Percentage of Total
135
136              HourPer   Hours/Percentage of Total
137
138              Seconds   Seconds
139
140              Minutes   Minutes
141
142              Hours     Hours
143
144              Percent   Percentage of Total
145
146
147       verbose
148              Enable detailed event logging.
149
150
151       version
152              Display the sreport version number.
153
154
155       !!     Repeat the last command executed.
156
157

REPORT TYPES

159       Valid report types are:
160              cluster <REPORT> <OPTIONS>
161
162              job <REPORT> <OPTIONS>
163
164              reservation <REPORT> <OPTIONS>
165
166              user <REPORT> <OPTIONS>
167
168
169       <REPORT> options for each type include:
170
171              cluster  -  AccountUtilizationByUser,  UserUtilizationByAccount,
172              UserUtilizationByWckey, Utilization, WCKeyUtilizationByUser
173
174              job - SizesByAccount, SizesByAccountAndWcKey, SizesByWckey
175
176              reservation - Utilization
177
178              user - TopUsage
179
180
181       NOTE: If OverSubscribe is configured to FORCE or YES in your slurm.conf
182       and  the  system  is not configured to use preemption (PreemptMode=OFF)
183       accounting can easily grow to values greater than the  actual  utiliza‐
184       tion.  It  may  be  common on such systems to get error messages in the
185       slurmdbd log stating: "We have more allocated time than is possible."
186
187

REPORT DESCRIPTION

189       cluster AccountUtilizationByUser
190              This report will display account utilization as  it  appears  on
191              the  hierarchical  tree.  Starting with the specified account or
192              the root account by default this report will list the underlying
193              usage  with  a sum on each level.  Use the 'tree' option to span
194              the tree for better visibility.
195
196              NOTE: Idle reservation time  will  be  split  evenly  among  ac‐
197              counts/users  given access to it. When a reservation is assigned
198              to whole accounts, the time will be counted in  the  association
199              for  the accounts, not the user associations in the accounts. In
200              this case, the usage of a parent account can be larger than  the
201              sum of its children.
202
203       cluster UserUtilizationByAccount
204              This  report  will display users by account in order of utiliza‐
205              tion without grouping multiple accounts by user  into  one,  but
206              displaying them on separate lines.
207
208       cluster UserUtilizationByWCKey
209              This  report will display users by wckey in order of utilization
210              without grouping multiple wckey by user into one, but displaying
211              them on separate lines.
212
213       cluster Utilization
214              This  report will display total usage divided amongst Allocated,
215              Down, Planned Down, Idle, and Reserved time for  selected  clus‐
216              ters.
217
218              Allocated    Time  that nodes were in use with active jobs or an
219                           active reservation.  This does not include reserva‐
220                           tions created with the MAINT or IGNORE_JOBS flags.
221
222              Down         Time  that  nodes  were marked as Down or time that
223                           slurmctld was not responding (assuming  TrackSlurm‐
224                           ctldDown is set in slurmdbd.conf).
225
226              Idle         Time  where  nodes  had  no active jobs or reserva‐
227                           tions.
228
229              Overcommited Time of eligible jobs waiting in the queue over the
230                           Reserved  time. Unlike Reserved, this has no limit.
231                           It is typically useful to  determine  whether  your
232                           system is overloaded and by how much.
233
234              PlannedDown  Time  that  nodes were in use by a reservation cre‐
235                           ated with the MAINT flag but  not  the  IGNORE_JOBS
236                           flag.
237
238              Reserved     Time  that  a node spent idle with eligible jobs in
239                           the queue that were unable to start due to time  or
240                           size  constraints.  The  maximum time is limited to
241                           the amount of time possible on the system. If  this
242                           value  is not of importance for you then the number
243                           can be grouped with idle time.
244
245              TresCount    Configured number of TRES' on the  reported  nodes.
246                           See also the TresName field.
247
248              Note:  Reservations  created  with  the IGNORE_JOBS flag are not
249              tracked in the Cluster Utilization report due to the  fact  that
250              allowing  any  current/active  jobs  to  continue  to run in the
251              reservation introduces the possibility for them to be  accounted
252              for incorrectly.
253
254              Note:  The default view for the "Cluster Utilization" report in‐
255              cludes the following fields: Cluster, Allocated, Down,  Planned‐
256              Down,  Idle,  Reserved,  Reported.  You  can  include additional
257              fields like OverCommitted and TresCount fields with  the  Format
258              option.  The  TresName  will  also  be included if using the -T,
259              --tres <tres_names> option.
260
261       cluster WCKeyUtilizationByUser
262              This report will display wckey utilization sorted by WCKey  name
263              for each user on each cluster.
264
265
266       job SizesByAccount
267              This  report will display the amount of time used for job ranges
268              specified by the 'grouping=' option.  Only a single level in the
269              tree  is  displayed  defaulting to the root dir.  If you specify
270              other accounts with the 'account=' option sreport will use those
271              accounts as the root account and you will receive the aggregated
272              totals of each listed account plus their sub accounts.
273
274       job SizesByAccountAndWckey
275              This report is very similar to SizesByAccount with  the  differ‐
276              ence being each account is pair with wckeys so the identifier is
277              account:wckey instead of just account so there will most  likely
278              be  multiple  accounts  listed depending on the number of wckeys
279              used.
280
281       job SizesByWckey
282              This report will display the amount of time for each  wckey  for
283              job ranges specified by the 'grouping=' option.
284
285
286       reservation Utilization
287              This  report  will  display  total usage for reservations on the
288              systems.  Note: Time requests on this report will  not  truncate
289              the time the reservation used, only the reservations that ran at
290              any time during the period requested.
291
292
293       user TopUsage
294              Displays the top users on a cluster, i.e. users with the highest
295              CPUTime.   Use the group option to group accounts together.  The
296              default is to have a different line for each user account combi‐
297              nation.
298
299              NOTE: TopUsage is always based on CPUTime. The -T, --tres option
300              will show the TRES usage of the top users, but the users are al‐
301              ways sorted by their CPUTime, not by the specified TRES.
302
303
304       Each report type has various options...
305
306
307       OPTIONS FOR ALL REPORT TYPES
308
309
310       All_Clusters
311              Use all monitored clusters. Default is local cluster.
312
313       Clusters=<OPT>
314              List  of  clusters to include in report.  Default is local clus‐
315              ter.
316
317
318       End=<OPT>
319              Period ending for report. Default is 23:59:59 of  previous  day.
320              Valid time formats are...
321
322              HH:MM[:SS] [AM|PM]
323              MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
324              MM/DD[/YY]-HH:MM[:SS]
325              YYYY-MM-DD[THH:MM[:SS]]
326              now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
327
328
329       Format=<OPT>
330              Comma separated list of fields to display in report.
331
332              When  using the format option for listing various fields you can
333              put a %NUMBER afterwards to specify how many  characters  should
334              be printed.
335
336              e.g. format=name%30 will print 30 characters of field name right
337              justified.  A -30 will print 30 characters left justified.
338
339
340       Start=<OPT>
341              Period start for report.  Default is 00:00:00 of  previous  day.
342              Valid time formats are...
343
344              HH:MM[:SS] [AM|PM]
345              MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
346              MM/DD[/YY]-HH:MM[:SS]
347              YYYY-MM-DD[THH:MM[:SS]]
348              now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
349
350       OPTIONS SPECIFICALLY FOR CLUSTER REPORTS
351
352
353       Accounts=<OPT>
354              When  used with the UserUtilizationByAccount, or AccountUtiliza‐
355              tionByUser, List of accounts to include in report.   Default  is
356              all.
357
358       Tree   When used with the AccountUtilizationByUser report will span the
359              accounts as they are in the hierarchy.
360
361       Users=<OPT>
362              When used with any report other than Utilization, List of  users
363              to include in report.  Default is all.
364
365       Wckeys=<OPT>
366              When  used  with the UserUtilizationByWckey or WCKeyUtilization‐
367              ByUser, List of wckeys to include in report. Default is all.
368
369
370       OPTIONS SPECIFICALLY FOR JOB REPORTS
371
372
373       Accounts=<OPT>
374              List of accounts to use for the report.  Default  is  all  which
375              will  show  only one line corresponding to the totals of all ac‐
376              counts in the hierarchy.  This explanation does not  apply  when
377              ran with the FlatView or AcctAsParent options.
378
379       AcctAsParent
380              When  used  with  the  SizesbyAccount(*) will take specified ac‐
381              counts as parents and the next layer  of  accounts  under  those
382              specified  will be displayed. Default is root if no specific Ac‐
383              counts are requested.  When FlatView is used, this option is ig‐
384              nored.
385
386       FlatView
387              When  used with the SizesbyAccount(*) will not group accounts in
388              a hierarchical level, but print each account where jobs ran on a
389              separate line without any hierarchy.
390
391       GID=<OPT>
392              List of group ids to include in report.  Default is all.
393
394       Grouping=<OPT>
395              Comma separated list of size groupings.   (e.g. 50,100,150 would
396              group job cpu count 1-49, 50-99, 100-149, > 150).   grouping=in‐
397              dividual will result in a single column for each job size found.
398
399       Jobs=<OPT>
400              List of jobs/steps to include in report.  Default is all.
401
402       Nodes=<OPT>
403              Only show jobs that ran on these nodes. Default is all.
404
405       Partitions=<OPT>
406              List of partitions jobs ran on to include in report.  Default is
407              all.
408
409       PrintJobCount
410              When used with the Sizes report will print number  of  jobs  ran
411              instead of time used.
412
413       Users=<OPT>
414              List of users jobs to include in report.  Default is all.
415
416       Wckeys=<OPT>
417              List  of  wckeys  to  use  for the report.  Default is all.  The
418              SizesbyWckey report all users summed together.  If you want only
419              certain users specify them with the Users= option.
420
421
422       OPTIONS SPECIFICALLY FOR RESERVATION REPORTS
423
424       Names=<OPT>
425              List of reservations to use for the report. Default is all.
426
427       Nodes=<OPT>
428              Only show reservations that used these nodes. Default is all.
429
430
431
432       OPTIONS SPECIFICALLY FOR USER REPORTS
433
434
435       Accounts=<OPT>
436              List of accounts to use for the report. Default is all.
437
438       Group  Group  all  accounts together for each user.  Default is a sepa‐
439              rate entry for each user and account reference.
440
441       TopCount=<OPT>
442              Used in the TopUsage report.  Change the number  of  users  dis‐
443              played.  Default is 10.
444
445       Users=<OPT>
446              List of users jobs to include in report.  Default is all.
447
448

FORMAT OPTIONS FOR EACH REPORT

450       FORMAT OPTIONS FOR CLUSTER REPORTS
451          AccountUtilizationByUser:
452          UserUtilizationByAccount:
453              Accounts, Cluster, Login, Proper, TresCount, Used
454
455          UserUtilizationByWckey:
456          WCKeyUtilizationByUser:
457              Cluster, Login, Proper, TresCount, Used, Wckey
458
459          Utilization:
460              Allocated,  Cluster, Down, Idle, Overcommitted, PlannedDown, Re‐
461              ported, Reserved, TresCount, TresName
462
463
464       FORMAT OPTIONS FOR JOB REPORTS
465          SizesByAccount, SizesByAccountAndWckey:
466              Account, Cluster
467
468          SizesByWckey:
469              Wckey, Cluster
470
471
472       FORMAT OPTIONS FOR RESERVATION REPORTS
473          Utilization:
474              Allocated, Associations, Cluster, End, Flags, Idle, Name, Nodes,
475              ReservationId, Start, TotalTime, TresCount, TresName, TresTime
476
477
478       FORMAT OPTIONS FOR USER REPORTS
479          TopUsage:
480              Account, Cluster, Login, Proper, Used
481
482
483       All commands and options are case-insensitive.
484
485

PERFORMANCE

487       Executing  sreport sends a remote procedure call to slurmdbd. If enough
488       calls from sreport or other Slurm client commands that send remote pro‐
489       cedure calls to the slurmdbd daemon come in at once, it can result in a
490       degradation of performance of the slurmdbd daemon,  possibly  resulting
491       in a denial of service.
492
493       Do not run sreport or other Slurm client commands that send remote pro‐
494       cedure calls to slurmdbd from loops in shell scripts or other programs.
495       Ensure  that  programs  limit calls to sreport to the minimum necessary
496       for the information you are trying to gather.
497
498

ENVIRONMENT VARIABLES

500       Some sreport options may be set via environment variables. These  envi‐
501       ronment  variables,  along with their corresponding options, are listed
502       below.  (Note: commandline options will always override these settings)
503
504       SREPORT_CLUSTER     Same as -M, --cluster
505
506       SREPORT_FEDERATION  Same as --federation
507
508       SREPORT_LOCAL       Same as --local
509
510       SREPORT_TRES        Same as -t, --tres
511
512       SLURM_CONF          The location of the Slurm configuration file.
513
514

EXAMPLES

516       Report number of jobs per account according to different job size bins:
517
518              $ sreport job sizesbyaccount
519
520
521       Report cluster utilization:
522
523              $ sreport cluster utilization
524
525
526       Report top usage:
527
528              $ sreport user top
529
530
531       Report number of jobs by user gore1 within the environ account:
532
533              $ sreport job sizesbyaccount All_Clusters users=gore1 account=environ PrintJobCount
534
535
536       Report cluster account utilization with the specified fields during the
537       specified 24 hour day of February 23, 2009, by user gore1:
538
539              $ sreport cluster AccountUtilizationByUser cluster=zeus user=gore1 start=2/23/09 end=2/24/09 format=Accounts,Cluster,TresCount,Login,Proper,Used
540
541
542       Report cluster account utilization by user in the LC account on cluster
543       zeus:
544
545              $ sreport cluster AccountUtilizationByUser cluster=zeus accounts=lc start=2/23/09 end=2/24/09
546
547
548       Report top usage in percent of the  lc  account  during  the  specified
549       week:
550
551              $ sreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc
552
553

COPYING

555       Copyright (C) 2009-2010 Lawrence Livermore National Security.  Produced
556       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
557       Copyright (C) 2010-2015 SchedMD LLC.
558
559       This file is part of Slurm, a resource  management  program.   For  de‐
560       tails, see <https://slurm.schedmd.com/>.
561
562       Slurm  is free software; you can redistribute it and/or modify it under
563       the terms of the GNU General Public License as published  by  the  Free
564       Software  Foundation;  either version 2 of the License, or (at your op‐
565       tion) any later version.
566
567       Slurm is distributed in the hope that it will be  useful,  but  WITHOUT
568       ANY  WARRANTY;  without even the implied warranty of MERCHANTABILITY or
569       FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General  Public  License
570       for more details.
571
572

SEE ALSO

574       sacct(1), slurmdbd(8)
575
576
577
578February 2021                   Slurm Commands                      sreport(1)
Impressum