1sreport(1) Slurm Commands sreport(1)
2
6 sreport - Generate reports from the slurm accounting data.
7
10 sreport [OPTIONS...] [COMMAND...]
11
14 sreport is used to generate reports of job usage and cluster utiliza‐
15 tion for Slurm jobs saved to the Slurm Database, slurmdbd.
16
19 -a, --all_clusters
20 Use all clusters instead of only the cluster from which the com‐
21 mand was executed.
22 -M, --cluster
24 The cluster(s) to generate reports for. Default is local clus‐
25 ter, unless the local cluster is currently part of a federation
26 and in that case generate a report for all clusters in the cur‐
27 rent federation. If the clusters included in a federation vary
28 through time, use this option to identify the clusters to be in‐
29 cluded in report. Implies --local.
30 --federation
32 Generate a report for the federation if a member of one.
33 -h, --help
35 Print a help message describing the usage of sreport.
36 --local
38 Generate a report for the local cluster, even if part of a fed‐
39 eration. Overrides --federation.
40 -n, --noheader
42 Don't display header when listing results.
43 -p, --parsable
45 Output will be '|' delimited with a '|' at the end.
46 -P, --parsable2
48 Output will be '|' delimited without a '|' at the end.
49 -Q, --quiet
51 Print no warning or informational messages, only error messages.
52 -t <time_format>
54 Specify the output time format. Time format options are case in‐
55 sensitive and may be abbreviated. The default format is Minutes.
56 Supported time format options are listed in the time command
57 section below.
58 -T, --tres=<tres_names>
60 Trackable resource (TRES) to report values for. By default CPU
61 resource use is reported (except for reservation reports. All
62 TRES types reserved by a reservation will be shown by default
63 unless overridden with this option). Multiple TRES names may be
64 separated using a comma separated list for all reports except
65 the job reports, which can only support a single TRES name, or
66 "ALL" for all TRES.
67 The "Reported" Billing TRES is calculated from the largest
69 Billing TRES of each node multiplied by the time frame. For ex‐
70 ample, if a node is part of multiple partitions and each has a
71 different TRESBillingWeights defined the Billing TRES for the
72 node will be the highest of the partitions. If TRESBilling‐
73 Weights is not defined on any partition for a node then the
74 Billing TRES will be equal to the number of CPUs on the node.
75 TRES node usage is no longer reported in percent format or in
77 Cluster Utilization. Please use TRES CPU instead. The main is‐
78 sue with using node is in most configurations multiple jobs are
79 able to run on the same node. This makes TRES node accounting
80 count the same node multiple times in the same period. In ex‐
81 clusive node configurations, CPU accounting returns the same us‐
82 age node accounting would.
83 -v, --verbose
86 Print detailed event logging.
87 -V , --version
89 Print version information and exit.
90
93 <keyword> may be omitted from the execute line and sreport will
94 execute in interactive mode. sreport will process commands as
95 entered until explicitly terminated.
96 exit Terminate the execution of sreport. Identical to the quit com‐
99 mand.
100 help Display a description of sreport options and commands.
103 nonparsable
106 Return output to normal after parsable or parsable2 has been
107 set.
108 parsable
111 Output will be | delimited with an ending '|'.
112 parsable2
115 Output will be | delimited without an ending '|'.
116 quiet Print no warning or informational messages, only fatal error
119 messages.
120 quit Terminate the execution of sreport. Identical to the exit com‐
123 mand.
124 time <time_format>
127 Specify the output time format. Time format options are case in‐
128 sensitive and may be abbreviated. The default format is Minutes.
129 Supported time format options include:
130 SecPer Seconds/Percentage of Total
133 MinPer Minutes/Percentage of Total
135 HourPer Hours/Percentage of Total
137 Seconds Seconds
139 Minutes Minutes
141 Hours Hours
143 Percent Percentage of Total
145 verbose
148 Enable detailed event logging.
149 version
152 Display the sreport version number.
153 !! Repeat the last command executed.
156
159 Valid report types are:
160 cluster <REPORT> <OPTIONS>
161 job <REPORT> <OPTIONS>
163 reservation <REPORT> <OPTIONS>
165 user <REPORT> <OPTIONS>
167 <REPORT> options for each type include:
170 cluster - AccountUtilizationByUser, UserUtilizationByAccount,
172 UserUtilizationByWckey, Utilization, WCKeyUtilizationByUser
173 job - SizesByAccount, SizesByAccountAndWcKey, SizesByWckey
175 reservation - Utilization
177 user - TopUsage
179 NOTE: If OverSubscribe is configured to FORCE or YES in your slurm.conf
182 and the system is not configured to use preemption (PreemptMode=OFF)
183 accounting can easily grow to values greater than the actual utiliza‐
184 tion. It may be common on such systems to get error messages in the
185 slurmdbd log stating: "We have more allocated time than is possible."
186
189 cluster AccountUtilizationByUser
190 This report will display account utilization as it appears on
191 the hierarchical tree. Starting with the specified account or
192 the root account by default this report will list the underlying
193 usage with a sum on each level. Use the 'tree' option to span
194 the tree for better visibility.
195 NOTE: Idle reservation time will be split evenly among ac‐
197 counts/users given access to it. When a reservation is assigned
198 to whole accounts, the time will be counted in the association
199 for the accounts, not the user associations in the accounts. In
200 this case, the usage of a parent account can be larger than the
201 sum of its children.
202 cluster UserUtilizationByAccount
204 This report will display users by account in order of utiliza‐
205 tion without grouping multiple accounts by user into one, but
206 displaying them on separate lines.
207 cluster UserUtilizationByWCKey
209 This report will display users by wckey in order of utilization
210 without grouping multiple wckey by user into one, but displaying
211 them on separate lines.
212 cluster Utilization
214 This report will display total usage divided amongst Allocated,
215 Down, Planned Down, Idle, and Reserved time for selected clus‐
216 ters.
217 Allocated Time that nodes were in use with active jobs or an
219 active reservation. This does not include reser‐
220 vations created with the MAINT or IGNORE_JOBS
221 flags.
222 Down Time that nodes were marked as Down or time that
224 slurmctld was not responding (assuming TrackSlurm‐
225 ctldDown is set in slurmdbd.conf).
226 Idle Time where nodes had no active jobs or reserva‐
228 tions.
229 OverCommitted Time of eligible jobs waiting in the queue over
231 the Reserved time. Unlike Reserved, this has no
232 limit. It is typically useful to determine whether
233 your system is overloaded and by how much.
234 PlannedDown Time that nodes were in use by a reservation cre‐
236 ated with the MAINT flag but not the IGNORE_JOBS
237 flag.
238 Reserved Time that a node spent idle with eligible jobs in
240 the queue that were unable to start due to time or
241 size constraints. The maximum time is limited to
242 the amount of time possible on the system. If this
243 value is not of importance for you then the number
244 can be grouped with idle time.
245 TresCount Configured number of TRES' on the reported nodes.
247 See also the TresName field.
248 Note: Reservations created with the IGNORE_JOBS flag are not
250 tracked in the Cluster Utilization report due to the fact that
251 allowing any current/active jobs to continue to run in the
252 reservation introduces the possibility for them to be accounted
253 for incorrectly.
254 Note: The default view for the "Cluster Utilization" report in‐
256 cludes the following fields: Cluster, Allocated, Down, Planned‐
257 Down, Idle, Reserved, Reported. You can include additional
258 fields like OverCommitted and TresCount fields with the Format
259 option. The TresName will also be included if using the -T,
260 --tres <tres_names> option.
261 cluster WCKeyUtilizationByUser
263 This report will display wckey utilization sorted by WCKey name
264 for each user on each cluster.
265 job SizesByAccount
268 This report will display the amount of time used for job ranges
269 specified by the 'grouping=' option. Only a single level in the
270 tree is displayed defaulting to the root dir. If you specify
271 other accounts with the 'account=' option sreport will use those
272 accounts as the root account and you will receive the aggregated
273 totals of each listed account plus their sub accounts.
274 job SizesByAccountAndWckey
276 This report is very similar to SizesByAccount with the differ‐
277 ence being each account is pair with wckeys so the identifier is
278 account:wckey instead of just account so there will most likely
279 be multiple accounts listed depending on the number of wckeys
280 used.
281 job SizesByWckey
283 This report will display the amount of time for each wckey for
284 job ranges specified by the 'grouping=' option.
285 reservation Utilization
288 This report will display total usage for reservations on the
289 systems. Note: Time requests on this report will not truncate
290 the time the reservation used, only the reservations that ran at
291 any time during the period requested.
292 user TopUsage
295 Displays the top users on a cluster, i.e. users with the highest
296 usage. By default users are sorted by CPUTime, but the -T,
297 --tres option will sort users by the first TRES specified.
298 Use the group option to group accounts together. The default is
300 to have a different line for each user account combination.
301 Each report type has various options...
304 OPTIONS FOR ALL REPORT TYPES
307 All_Clusters
310 Use all monitored clusters. Default is local cluster.
311 Clusters=<OPT>
313 List of clusters to include in report. Default is local clus‐
314 ter.
315 End=<OPT>
318 Period ending for report. Default is 23:59:59 of previous day.
319 Valid time formats are...
320 HH:MM[:SS] [AM|PM]
322 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
323 MM/DD[/YY]-HH:MM[:SS]
324 YYYY-MM-DD[THH:MM[:SS]]
325 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
326 Format=<OPT>
329 Comma separated list of fields to display in report.
330 When using the format option for listing various fields you can
332 put a %NUMBER afterwards to specify how many characters should
333 be printed.
334 e.g. format=name%30 will print 30 characters of field name right
336 justified. A -30 will print 30 characters left justified.
337 Start=<OPT>
340 Period start for report. Default is 00:00:00 of previous day.
341 Valid time formats are...
342 HH:MM[:SS] [AM|PM]
344 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
345 MM/DD[/YY]-HH:MM[:SS]
346 YYYY-MM-DD[THH:MM[:SS]]
347 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
348 OPTIONS SPECIFICALLY FOR CLUSTER REPORTS
350 Accounts=<OPT>
353 When used with the UserUtilizationByAccount, or AccountUtiliza‐
354 tionByUser, List of accounts to include in report. Default is
355 all.
356 Tree When used with the AccountUtilizationByUser report will span the
358 accounts as they are in the hierarchy.
359 Users=<OPT>
361 When used with any report other than Utilization, List of users
362 to include in report. Default is all.
363 Wckeys=<OPT>
365 When used with the UserUtilizationByWckey or WCKeyUtilization‐
366 ByUser, List of wckeys to include in report. Default is all.
367 OPTIONS SPECIFICALLY FOR JOB REPORTS
370 Accounts=<OPT>
373 List of accounts to use for the report. Default is all which
374 will show only one line corresponding to the totals of all ac‐
375 counts in the hierarchy. This explanation does not apply when
376 ran with the FlatView or AcctAsParent options.
377 AcctAsParent
379 When used with the SizesbyAccount(*) will take specified ac‐
380 counts as parents and the next layer of accounts under those
381 specified will be displayed. Default is root if no specific Ac‐
382 counts are requested. When FlatView is used, this option is ig‐
383 nored.
384 FlatView
386 When used with the SizesbyAccount(*) will not group accounts in
387 a hierarchical level, but print each account where jobs ran on a
388 separate line without any hierarchy.
389 GID=<OPT>
391 List of group ids to include in report. Default is all.
392 Grouping=<OPT>
394 Comma separated list of size groupings. (e.g. 50,100,150 would
395 group job cpu count 1-49, 50-99, 100-149, > 150). grouping=in‐
396 dividual will result in a single column for each job size found.
397 Jobs=<OPT>
399 List of jobs/steps to include in report. Default is all.
400 Nodes=<OPT>
402 Only show jobs that ran on these nodes. Default is all.
403 Partitions=<OPT>
405 List of partitions jobs ran on to include in report. Default is
406 all.
407 PrintJobCount
409 When used with the Sizes report will print number of jobs ran
410 instead of time used.
411 Users=<OPT>
413 List of users jobs to include in report. Default is all.
414 Wckeys=<OPT>
416 List of wckeys to use for the report. Default is all. The
417 SizesbyWckey report all users summed together. If you want only
418 certain users specify them with the Users= option.
419 OPTIONS SPECIFICALLY FOR RESERVATION REPORTS
422 Names=<OPT>
424 List of reservations to use for the report. Default is all.
425 Nodes=<OPT>
427 Only show reservations that used these nodes. Default is all.
428 OPTIONS SPECIFICALLY FOR USER REPORTS
432 Accounts=<OPT>
435 List of accounts to use for the report. Default is all.
436 Group Group all accounts together for each user. Default is a sepa‐
438 rate entry for each user and account reference.
439 TopCount=<OPT>
441 Used in the TopUsage report. Change the number of users dis‐
442 played. Default is 10.
443 Users=<OPT>
445 List of users jobs to include in report. Default is all.
446
449 FORMAT OPTIONS FOR CLUSTER REPORTS
450 AccountUtilizationByUser:
451 UserUtilizationByAccount:
452 Accounts, Cluster, Login, Proper, TresCount, Used
453 UserUtilizationByWckey:
455 WCKeyUtilizationByUser:
456 Cluster, Login, Proper, TresCount, Used, Wckey
457 Utilization:
459 Allocated, Cluster, Down, Idle, OverCommitted, PlannedDown, Re‐
460 ported, Reserved, TresCount, TresName
461 FORMAT OPTIONS FOR JOB REPORTS
464 SizesByAccount, SizesByAccountAndWckey:
465 Account, Cluster
466 SizesByWckey:
468 Wckey, Cluster
469 FORMAT OPTIONS FOR RESERVATION REPORTS
472 Utilization:
473 Allocated, Associations, Cluster, End, Flags, Idle, Name, Nodes,
474 ReservationId, Start, TotalTime, TresCount, TresName, TresTime
475 FORMAT OPTIONS FOR USER REPORTS
478 TopUsage:
479 Account, Cluster, Login, Proper, Used
480 All commands and options are case-insensitive.
483
486 Executing sreport sends a remote procedure call to slurmdbd. If enough
487 calls from sreport or other Slurm client commands that send remote pro‐
488 cedure calls to the slurmdbd daemon come in at once, it can result in a
489 degradation of performance of the slurmdbd daemon, possibly resulting
490 in a denial of service.
491 Do not run sreport or other Slurm client commands that send remote pro‐
493 cedure calls to slurmdbd from loops in shell scripts or other programs.
494 Ensure that programs limit calls to sreport to the minimum necessary
495 for the information you are trying to gather.
496
499 Some sreport options may be set via environment variables. These envi‐
500 ronment variables, along with their corresponding options, are listed
501 below. (Note: Command line options will always override these set‐
502 tings.)
503 SREPORT_CLUSTER Same as -M, --cluster
505 SREPORT_FEDERATION Same as --federation
507 SREPORT_LOCAL Same as --local
509 SREPORT_TRES Same as -t, --tres
511 SLURM_CONF The location of the Slurm configuration file.
513
516 Report number of jobs per account according to different job size bins:
517 $ sreport job sizesbyaccount
519 Report cluster utilization:
522 $ sreport cluster utilization
524 Report top usage:
527 $ sreport user top
529 Report number of jobs by user gore1 within the environ account:
532 $ sreport job sizesbyaccount All_Clusters users=gore1 account=environ PrintJobCount
534 Report cluster account utilization with the specified fields during the
537 specified 24 hour day of February 23, 2009, by user gore1:
538 $ sreport cluster AccountUtilizationByUser cluster=zeus user=gore1 start=2/23/09 end=2/24/09 format=Accounts,Cluster,TresCount,Login,Proper,Used
540 Report cluster account utilization by user in the LC account on cluster
543 zeus:
544 $ sreport cluster AccountUtilizationByUser cluster=zeus accounts=lc start=2/23/09 end=2/24/09
546 Report top usage in percent of the lc account during the specified
549 week:
550 $ sreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc
552
555 Copyright (C) 2009-2010 Lawrence Livermore National Security. Produced
556 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
557 Copyright (C) 2010-2021 SchedMD LLC.
558 This file is part of Slurm, a resource management program. For de‐
560 tails, see <https://slurm.schedmd.com/>.
561 Slurm is free software; you can redistribute it and/or modify it under
563 the terms of the GNU General Public License as published by the Free
564 Software Foundation; either version 2 of the License, or (at your op‐
565 tion) any later version.
566 Slurm is distributed in the hope that it will be useful, but WITHOUT
568 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
569 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
570 for more details.
571
574 sacct(1), slurmdbd(8)
575July 2021 Slurm Commands sreport(1)