1sreport(1)                      Slurm Commands                      sreport(1)
2
3
4

NAME

6       sreport - Generate reports from the slurm accounting data.
7
8

SYNOPSIS

10       sreport [OPTIONS...] [COMMAND...]
11
12

DESCRIPTION

14       sreport  is  used to generate reports of job usage and cluster utiliza‐
15       tion for Slurm jobs saved to the Slurm Database, slurmdbd.
16
17

OPTIONS

19       -a, --all_clusters
20              Use all clusters instead of only the cluster from which the com‐
21              mand was executed.
22
23       --federation
24              Generate a report for the federation if a member of one.
25
26       -h, --help
27              Print a help message describing the usage of sreport.
28
29       --local
30              Generate  a report for the local cluster, even if part of a fed‐
31              eration.  Overrides --federation
32
33       -M, --cluster
34              The cluster(s) to generate reports for. Default is  local  clus‐
35              ter,  unless the local cluster is currently part of a federation
36              and in that case generate a report for all clusters in the  cur‐
37              rent  federation.  If the clusters included in a federation vary
38              through time, use this option to identify  the  clusters  to  be
39              included in report. Implies --local.
40
41       -n, --noheader
42              Don't display header when listing results.
43
44       -p, --parsable
45              Output will be '|' delimited with a '|' at the end.
46
47       -P, --parsable2
48              Output will be '|' delimited without a '|' at the end.
49
50       -Q, --quiet
51              Print no warning or informational messages, only error messages.
52
53       -t <format>
54              Specify  the  output  time  format. Time format options are case
55              insensitive and may be abbreviated. The default format  is  Min‐
56              utes.  Supported time format options are listed in the time com‐
57              mand section below.
58
59       -T,  --tres <tres_names>
60              Trackable resource (TRES) to report values for.  By default  CPU
61              resource  use  is  reported (except for reservation reports. All
62              TRES types reserved by a reservation will be  shown  by  default
63              unless overridden with this option).  Multiple TRES names may be
64              separated using a comma separated list for  all  reports  except
65              the  job  reports, which can only support a single TRES name, or
66              "ALL" for all TRES.
67
68              The "Reported" Billing  TRES  is  calculated  from  the  largest
69              Billing  TRES  of  each  node  multiplied by the time frame. For
70              example, if a node is part of multiple partitions and each has a
71              different  TRESBillingWeights  defined  the Billing TRES for the
72              node will be the highest of  the  partitions.   If  TRESBilling‐
73              Weights  is  not  defined  on  any partition for a node then the
74              Billing TRES will be equal to the number of CPUs on the node.
75
76              TRES node usage is no longer reported in percent  format  or  in
77              Cluster  Utilization.   Please  use  TRES CPU instead.  The main
78              issue with using node is in most  configurations  multiple  jobs
79              are able to run on the same node.  This makes TRES node account‐
80              ing count the same node multiple times in the same  period.   In
81              exclusive  node  configurations, CPU accounting returns the same
82              usage node accounting would.
83
84
85       -v, --verbose
86              Print detailed event logging.
87
88       -V , --version
89              Print version information and exit.
90
91

COMMANDS

93       <keyword> may be omitted from the execute line and sreport will
94              execute in interactive mode. sreport will  process  commands  as
95              entered until explicitly terminated.
96
97
98       exit   Terminate  the execution of sreport.  Identical to the quit com‐
99              mand.
100
101
102       help   Display a description of sreport options and commands.
103
104
105       nonparsable
106              Return output to normal after parsable  or  parsable2  has  been
107              set.
108
109
110       parsable
111              Output will be | delimited with an ending '|'.
112
113
114       parsable2
115              Output will be | delimited without an ending '|'.
116
117
118       quiet  Print  no  warning  or  informational messages, only fatal error
119              messages.
120
121
122       quit   Terminate the execution of sreport.  Identical to the exit  com‐
123              mand.
124
125
126       time <time_format>
127              Specify  the  output  time  format. Time format options are case
128              insensitive and may be abbreviated. The default format  is  Min‐
129              utes.  Supported time format options include:
130
131
132              SecPer    Seconds/Percentage of Total
133
134              MinPer    Minutes/Percentage of Total
135
136              HourPer   Hours/Percentage of Total
137
138              Seconds   Seconds
139
140              Minutes   Minutes
141
142              Hours     Hours
143
144              Percent   Percentage of Total
145
146
147       verbose
148              Enable detailed event logging.
149
150
151       version
152              Display the sreport version number.
153
154
155       !!     Repeat the last command executed.
156
157

REPORT TYPES

159       Valid report types are:
160              cluster <REPORT> <OPTIONS>
161
162              job <REPORT> <OPTIONS>
163
164              reservation <REPORT> <OPTIONS>
165
166              user <REPORT> <OPTIONS>
167
168
169       <REPORT> options for each type include:
170
171              cluster  -  AccountUtilizationByUser,  UserUtilizationByAccount,
172              UserUtilizationByWckey, Utilization, WCKeyUtilizationByUser
173
174              job - SizesByAccount, SizesByAccountAndWcKey, SizesByWckey
175
176              reservation - Utilization
177
178              user - TopUsage
179
180

REPORT DESCRIPTION

182       cluster AccountUtilizationByUser
183              This report will display account utilization as  it  appears  on
184              the  hierarchical  tree.  Starting with the specified account or
185              the root account by default this report will list the underlying
186              usage  with  a sum on each level.  Use the 'tree' option to span
187              the tree for better visibility.  NOTE: If  there  were  reserva‐
188              tions  allowing a whole account any idle time in the reservation
189              given to the association for the account, not the user  associa‐
190              tions in the account, so it can be possible a parent account can
191              be larger than the sum of it's children.
192
193       cluster UserUtilizationByAccount
194              This report will display users by account in order  of  utiliza‐
195              tion  without  grouping  multiple accounts by user into one, but
196              displaying them on separate lines.
197
198       cluster UserUtilizationByWCKey
199              This report will display users by wckey in order of  utilization
200              without grouping multiple wckey by user into one, but displaying
201              them on separate lines.
202
203       cluster Utilization
204              This report will display total usage divided amongst  Allocated,
205              Down,  Planned  Down, Idle, and Reserved time for selected clus‐
206              ters.
207
208              Allocated    Time that nodes were in use with active jobs or  an
209                           active reservation.  This does not include reserva‐
210                           tions created with the MAINT or IGNORE_JOBS flags.
211
212              Down         Time that nodes were marked as Down  or  time  that
213                           slurmctld  was not responding (assuming TrackSlurm‐
214                           ctldDown is set in slurmdbd.conf).
215
216              Idle         Time where nodes had no  active  jobs  or  reserva‐
217                           tions.
218
219              Overcommited Time  that  the  nodes  were over allocated, either
220                           with the -O, --overcommit flag at  submission  time
221                           or  OverSubscribe  set  to FORCE in the slurm.conf.
222                           This time is not counted against the total reported
223                           time.
224
225              PlannedDown  Time  that  nodes were in use by a reservation cre‐
226                           ated with the MAINT flag but  not  the  IGNORE_JOBS
227                           flag.
228
229              Reserved     Time  that  a node spent idle with eligible jobs in
230                           the queue that were unable to start due to time  or
231                           size  constraints.  If  this value is not of impor‐
232                           tance for you then the number can be  grouped  with
233                           idle time.
234
235              TresCount    Configured  number  of TRES' on the reported nodes.
236                           See also the TresName field.
237
238              Note: Reservations created with the  IGNORE_JOBS  flag  are  not
239              tracked  in  the Cluster Utilization report due to the fact that
240              allowing any current/active jobs  to  continue  to  run  in  the
241              reservation  introduces the possibility for them to be accounted
242              for incorrectly.
243
244              Note: The default view  for  the  "Cluster  Utilization"  report
245              includes   the   following  fields:  Cluster,  Allocated,  Down,
246              PlannedDown, Idle, Reserved, Reported.  You  can  include  addi‐
247              tional  fields  like OverCommitted and TresCount fields with the
248              Format option. The TresName will also be included if  using  the
249              -T,  --tres <tres_names> option.
250
251       cluster WCKeyUtilizationByUser
252              This  report will display wckey utilization sorted by WCKey name
253              for each user on each cluster.
254
255
256       job SizesByAccount
257              This report will display the amount of time used for job  ranges
258              specified by the 'grouping=' option.  Only a single level in the
259              tree is displayed defaulting to the root dir.   If  you  specify
260              other accounts with the 'account=' option sreport will use those
261              accounts as the root account and you will receive the aggregated
262              totals of each listed account plus their sub accounts.
263
264       job SizesByAccountAndWckey
265              This  report  is very similar to SizesByAccount with the differ‐
266              ence being each account is pair with wckeys so the identifier is
267              account:wckey  instead of just account so there will most likely
268              be multiple accounts listed depending on the  number  of  wckeys
269              used.
270
271       job SizesByWckey
272              This  report  will display the amount of time for each wckey for
273              job ranges specified by the 'grouping=' option.
274
275
276       reservation Utilization
277              This report will display total usage  for  reservations  on  the
278              systems.
279
280
281       user TopUsage
282              Displays  the  top  users on a cluster.  Use the group option to
283              group accounts together.  The default is  to  have  a  different
284              line for each user account combination.
285
286
287       Each report type has various options...
288
289
290       OPTIONS FOR ALL REPORT TYPES
291
292
293       All_Clusters
294              Use all monitored clusters. Default is local cluster.
295
296       Clusters=<OPT>
297              List  of  clusters to include in report.  Default is local clus‐
298              ter.
299
300
301       End=<OPT>
302              Period ending for report. Default is 23:59:59 of  previous  day.
303              Valid time formats are...
304
305              HH:MM[:SS] [AM|PM]
306              MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
307              MM/DD[/YY]-HH:MM[:SS]
308              YYYY-MM-DD[THH:MM[:SS]]
309
310
311       Format=<OPT>
312              Comma separated list of fields to display in report.
313
314              When  using the format option for listing various fields you can
315              put a %NUMBER afterwards to specify how many  characters  should
316              be printed.
317
318              e.g. format=name%30 will print 30 characters of field name right
319              justified.  A -30 will print 30 characters left justified.
320
321
322       Start=<OPT>
323              Period start for report.  Default is 00:00:00 of  previous  day.
324              Valid time formats are...
325
326              HH:MM[:SS] [AM|PM]
327              MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
328              MM/DD[/YY]-HH:MM[:SS]
329              YYYY-MM-DD[THH:MM[:SS]]
330
331       OPTIONS SPECIFICALLY FOR CLUSTER REPORTS
332
333
334       Accounts=<OPT>
335              When  used with the UserUtilizationByAccount, or AccountUtiliza‐
336              tionByUser, List of accounts to include in report.   Default  is
337              all.
338
339       Tree   When used with the AccountUtilizationByUser report will span the
340              accounts as they are in the hierarchy.
341
342       Users=<OPT>
343              When used with any report other than Utilization, List of  users
344              to include in report.  Default is all.
345
346       Wckeys=<OPT>
347              When  used  with the UserUtilizationByWckey or WCKeyUtilization‐
348              ByUser, List of wckeys to include in report. Default is all.
349
350
351       OPTIONS SPECIFICALLY FOR JOB REPORTS
352
353
354       Accounts=<OPT>
355              List of accounts to use for the report.  Default  is  all  which
356              will  show  only  one  line  corresponding  to the totals of all
357              accounts in the hierarchy.  This explanation does not apply when
358              ran with the FlatView or AcctAsParent options.
359
360       AcctAsParent
361              When   used  with  the  SizesbyAccount(*)  will  take  specified
362              accounts as parents and the next layer of accounts  under  those
363              specified  will  be  displayed.  Default  is root if no specific
364              Accounts are requested.  When FlatView is used, this  option  is
365              ignored.
366
367       FlatView
368              When  used with the SizesbyAccount(*) will not group accounts in
369              a hierarchical level, but print each account where jobs ran on a
370              separate line without any hierarchy.
371
372       GID=<OPT>
373              List of group ids to include in report.  Default is all.
374
375       Grouping=<OPT>
376              Comma separated list of size groupings.   (e.g. 50,100,150 would
377              group job cpu  count  1-49,  50-99,  100-149,  >  150).   group‐
378              ing=individual  will result in a single column for each job size
379              found.
380
381       Jobs=<OPT>
382              List of jobs/steps to include in report.  Default is all.
383
384       Nodes=<OPT>
385              Only show jobs that ran on these nodes. Default is all.
386
387       Partitions=<OPT>
388              List of partitions jobs ran on to include in report.  Default is
389              all.
390
391       PrintJobCount
392              When  used  with  the Sizes report will print number of jobs ran
393              instead of time used.
394
395       Users=<OPT>
396              List of users jobs to include in report.  Default is all.
397
398       Wckeys=<OPT>
399              List of wckeys to use for the  report.   Default  is  all.   The
400              SizesbyWckey report all users summed together.  If you want only
401              certain users specify them them with the Users= option.
402
403
404       OPTIONS SPECIFICALLY FOR RESERVATION REPORTS
405
406       Names=<OPT>
407              List of reservations to use for the report. Default is all.
408
409       Nodes=<OPT>
410              Only show reservations that used these nodes. Default is all.
411
412
413
414       OPTIONS SPECIFICALLY FOR USER REPORTS
415
416
417       Accounts=<OPT>
418              List of accounts to use for the report. Default is all.
419
420       Group  Group all accounts together for each user.  Default is  a  sepa‐
421              rate entry for each user and account reference.
422
423       TopCount=<OPT>
424              Used  in  the  TopUsage report.  Change the number of users dis‐
425              played.  Default is 10.
426
427       Users=<OPT>
428              List of users jobs to include in report.  Default is all.
429
430

FORMAT OPTIONS FOR EACH REPORT

432       FORMAT OPTIONS FOR CLUSTER REPORTS
433          AccountUtilizationByUser:
434          UserUtilizationByAccount:
435              Accounts, Cluster, Login, Proper, TresCount, Used
436
437          UserUtilizationByWckey:
438          WCKeyUtilizationByUser:
439              Cluster, Login, Proper, TresCount, Used, Wckey
440
441          Utilization:
442              Allocated,  Cluster,  Down,  Idle,  Overcommited,   PlannedDown,
443              Reported, Reserved, TresCount, TresName
444
445
446       FORMAT OPTIONS FOR JOB REPORTS
447          SizesByAccount, SizesByAccountAndWckey:
448              Account, Cluster
449
450          SizesByWckey:
451              Wckey, Cluster
452
453
454       FORMAT OPTIONS FOR RESERVATION REPORTS
455          Utilization:
456              Allocated, Associations, Cluster, End, Flags, Idle, Name, Nodes,
457              ReservationId, Start, TotalTime, TresCount, TresName, TresTime
458
459
460       FORMAT OPTIONS FOR USER REPORTS
461          TopUsage:
462              Account, Cluster, Login, Proper, Used
463
464
465       All commands and options are case-insensitive.
466
467

ENVIRONMENT VARIABLES

469       Some sreport options may be set via environment variables. These  envi‐
470       ronment  variables,  along with their corresponding options, are listed
471       below.  (Note: commandline options will always override these settings)
472
473       SREPORT_CLUSTER     Same as -M, --cluster
474
475       SREPORT_FEDERATION  Same as --federation
476
477       SREPORT_LOCAL       Same as --local
478
479       SREPORT_TRES        Same as -t, --tres
480
481       SLURM_CONF          The location of the Slurm configuration file.
482
483

EXAMPLES

485       sreport job sizesbyaccount
486
487       sreport cluster utilization
488
489       sreport user top
490
491       sreport job  sizesbyaccount  All_Clusters  users=gore1  account=environ
492       PrintJobCount
493              Report number of jobs by user gore1 within the environ account
494
495       sreport   cluster   AccountUtilizationByUser   cluster=zeus  user=gore1
496       start=2/23/08             end=2/24/09             format=Accounts,Clus‐
497       ter,TresCount,Login,Proper,Used
498              Report  cluster  account  utilization  with the specified fields
499              during the specified 24 hour day of February 23, 2009,  by  user
500              gore1
501
502       sreport   cluster   AccountUtilizationByUser  cluster=zeus  accounts=lc
503       start=2/23/08 end=2/24/09
504              Report cluster account utilization by user in the LC account  on
505              cluster zeus
506
507       sreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc
508              Report  top usage in percent of the lc account during the speci‐
509              fied week
510
511

COPYING

513       Copyright (C) 2009-2010 Lawrence Livermore National Security.  Produced
514       at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
515       Copyright (C) 2010-2015 SchedMD LLC.
516
517       This  file  is  part  of  Slurm,  a  resource  management program.  For
518       details, see <https://slurm.schedmd.com/>.
519
520       Slurm is free software; you can redistribute it and/or modify it  under
521       the  terms  of  the GNU General Public License as published by the Free
522       Software Foundation; either version 2  of  the  License,  or  (at  your
523       option) any later version.
524
525       Slurm  is  distributed  in the hope that it will be useful, but WITHOUT
526       ANY WARRANTY; without even the implied warranty of  MERCHANTABILITY  or
527       FITNESS  FOR  A PARTICULAR PURPOSE.  See the GNU General Public License
528       for more details.
529
530

SEE ALSO

532       sacct(1), slurmdbd(8)
533
534
535
536May 2019                        Slurm Commands                      sreport(1)
Impressum