1sreport(1) Slurm Commands sreport(1)
2
6 sreport - Generate reports from the slurm accounting data.
7
10 sreport [OPTIONS...] [COMMAND...]
11
14 sreport is used to generate reports of job usage and cluster utiliza‐
15 tion for Slurm jobs saved to the Slurm Database, slurmdbd.
16
19 -a, --all_clusters
20 Use all clusters instead of only the cluster from which the com‐
21 mand was executed.
22 -M, --cluster
24 The cluster(s) to generate reports for. Default is local clus‐
25 ter, unless the local cluster is currently part of a federation
26 and in that case generate a report for all clusters in the cur‐
27 rent federation. If the clusters included in a federation vary
28 through time, use this option to identify the clusters to be in‐
29 cluded in report. Implies --local.
30 --federation
32 Generate a report for the federation if a member of one.
33 -h, --help
35 Print a help message describing the usage of sreport.
36 --local
38 Generate a report for the local cluster, even if part of a fed‐
39 eration. Overrides --federation.
40 -n, --noheader
42 Don't display header when listing results.
43 -p, --parsable
45 Output will be '|' delimited with a '|' at the end.
46 -P, --parsable2
48 Output will be '|' delimited without a '|' at the end.
49 -Q, --quiet
51 Print no warning or informational messages, only error messages.
52 -t <time_format>
54 Specify the output time format. Time format options are case in‐
55 sensitive and may be abbreviated. The default format is Minutes.
56 Supported time format options are listed in the time command
57 section below.
58 -T, --tres=<tres_names>
60 Trackable resource (TRES) to report values for. By default CPU
61 resource use is reported (except for reservation reports. All
62 TRES types reserved by a reservation will be shown by default
63 unless overridden with this option). Multiple TRES names may be
64 separated using a comma separated list for all reports except
65 the job reports, which can only support a single TRES name, or
66 "ALL" for all TRES.
67 The "Reported" Billing TRES is calculated from the largest
69 Billing TRES of each node multiplied by the time frame. For ex‐
70 ample, if a node is part of multiple partitions and each has a
71 different TRESBillingWeights defined the Billing TRES for the
72 node will be the highest of the partitions. If TRESBilling‐
73 Weights is not defined on any partition for a node then the
74 Billing TRES will be equal to the number of CPUs on the node.
75 TRES node usage is no longer reported in percent format or in
77 Cluster Utilization. Please use TRES CPU instead. The main is‐
78 sue with using node is in most configurations multiple jobs are
79 able to run on the same node. This makes TRES node accounting
80 count the same node multiple times in the same period. In ex‐
81 clusive node configurations, CPU accounting returns the same us‐
82 age node accounting would.
83 -v, --verbose
85 Print detailed event logging.
86 -V , --version
88 Print version information and exit.
89
91 <keyword> may be omitted from the execute line and sreport will execute
92 in interactive mode. sreport will process commands as entered until ex‐
93 plicitly terminated.
94 exit Terminate the execution of sreport. Identical to the quit com‐
97 mand.
98 help Display a description of sreport options and commands.
100 nonparsable
102 Return output to normal after parsable or parsable2 has been
103 set.
104 parsable
106 Output will be | delimited with an ending '|'.
107 parsable2
109 Output will be | delimited without an ending '|'.
110 quiet Print no warning or informational messages, only fatal error
112 messages.
113 quit Terminate the execution of sreport. Identical to the exit com‐
115 mand.
116 time <time_format>
118 Specify the output time format. Time format options are case in‐
119 sensitive and may be abbreviated. The default format is Minutes.
120 Supported time format options include:
121 SecPer Seconds/Percentage of Total
123 MinPer Minutes/Percentage of Total
125 HourPer Hours/Percentage of Total
127 Seconds Seconds
129 Minutes Minutes
131 Hours Hours
133 Percent Percentage of Total
135 verbose
137 Enable detailed event logging.
138 version
140 Display the sreport version number.
141 !! Repeat the last command executed.
143
145 Valid report types are:
146 cluster <REPORT> <OPTIONS>
148 job <REPORT> <OPTIONS>
150 reservation <REPORT> <OPTIONS>
152 user <REPORT> <OPTIONS>
154 <REPORT> options for each type include:
156 cluster
158 AccountUtilizationByUser, UserUtilizationByAccount,
159 UserUtilizationByWckey, Utilization, WCKeyUtilization‐
160 ByUser
161 job SizesByAccount, SizesByAccountAndWcKey, SizesByWckey
163 reservation
165 Utilization
166 user TopUsage
168 NOTE: If OverSubscribe is configured to FORCE or YES in your slurm.conf
170 and the system is not configured to use preemption (PreemptMode=OFF)
171 accounting can easily grow to values greater than the actual utiliza‐
172 tion. It may be common on such systems to get error messages in the
173 slurmdbd log stating: "We have more allocated time than is possible."
174
177 cluster AccountUtilizationByUser
178 This report will display account utilization as it appears on
179 the hierarchical tree. Starting with the specified account or
180 the root account by default this report will list the underlying
181 usage with a sum on each level. Use the 'tree' option to span
182 the tree for better visibility.
183 NOTE: Idle reservation time will be split evenly among ac‐
185 counts/users given access to it. When a reservation is assigned
186 to whole accounts, the time will be counted in the association
187 for the accounts, not the user associations in the accounts. In
188 this case, the usage of a parent account can be larger than the
189 sum of its children.
190 cluster UserUtilizationByAccount
192 This report will display users by account in order of utiliza‐
193 tion without grouping multiple accounts by user into one, but
194 displaying them on separate lines.
195 cluster UserUtilizationByWCKey
197 This report will display users by wckey in order of utilization
198 without grouping multiple wckey by user into one, but displaying
199 them on separate lines.
200 cluster Utilization
202 This report will display total usage divided amongst Allocated,
203 Down, Planned Down, Idle, and Planned time for selected clus‐
204 ters.
205 Allocated
207 Time that nodes were in use with active jobs or an
209 active reservation. This does not include reser‐
210 vations created with the MAINT or IGNORE_JOBS
211 flags.
212 Down
214 Time that nodes were marked as Down or fully
216 Drained, or time that slurmctld was not responding
217 (assuming TrackSlurmctldDown is set in slur‐
218 mdbd.conf).
219 Idle
221 Time that nodes were not Allocated, Down, Planned‐
223 Down, or Planned.
224 OverCommitted
226 Time of eligible jobs waiting in the queue over
228 the Planned time. Unlike Planned, this has no
229 limit. It is typically useful to determine whether
230 your system is overloaded and by how much.
231 PlannedDown
233 Time that nodes were in use by a reservation cre‐
235 ated with the MAINT flag but not the IGNORE_JOBS
236 flag. Also, time that nodes were in the FUTURE
237 state or CLOUD nodes that were POWERED_DOWN.
238 Planned
240 Time that nodes were not Allocated, Down or
242 PlannedDown with eligible jobs in the queue that
243 were unable to start due to time or size con‐
244 straints. The maximum time is limited to the
245 amount of time possible on the system. If this
246 value is not of importance for you then the number
247 can be grouped with idle time.
248 TresCount
250 Configured number of TRES' on the reported nodes.
252 See also the TresName field.
253 Note: Reservations created with the IGNORE_JOBS flag are not
255 tracked in the Cluster Utilization report due to the fact that
256 allowing any current/active jobs to continue to run in the
257 reservation introduces the possibility for them to be accounted
258 for incorrectly. The jobs in these reservations will be tracked
259 as normal rather than being bundled in the reservation time, as
260 they are with reservations that do not have the IGNORE_JOBS
261 flag.
262 Note: The default view for the "Cluster Utilization" report in‐
264 cludes the following fields: Cluster, Allocated, Down, Planned‐
265 Down, Idle, Planned, Reported. You can include additional fields
266 like OverCommitted and TresCount fields with the Format option.
267 The TresName will also be included if using the -T, --tres
268 <tres_names> option.
269 cluster WCKeyUtilizationByUser
271 This report will display wckey utilization sorted by WCKey name
272 for each user on each cluster.
273 job SizesByAccount
275 This report will display the amount of time used for job ranges
276 specified by the 'grouping=' option. Only a single level in the
277 tree is displayed defaulting to the root dir. If you specify
278 other accounts with the 'account=' option sreport will use those
279 accounts as the root account and you will receive the aggregated
280 totals of each listed account plus their sub accounts.
281 job SizesByAccountAndWckey
283 This report is very similar to SizesByAccount with the differ‐
284 ence being each account is pair with wckeys so the identifier is
285 account:wckey instead of just account so there will most likely
286 be multiple accounts listed depending on the number of wckeys
287 used.
288 job SizesByWckey
290 This report will display the amount of time for each wckey for
291 job ranges specified by the 'grouping=' option.
292 reservation Utilization
294 This report will display total usage for reservations on the
295 systems. Note: Time requests on this report will not truncate
296 the time the reservation used, only the reservations that ran at
297 any time during the period requested.
298 user TopUsage
300 Displays the top users on a cluster, i.e. users with the highest
301 usage. By default users are sorted by CPUTime, but the -T,
302 --tres option will sort users by the first TRES specified.
303 Use the group option to group accounts together. The default is
305 to have a different line for each user account combination.
306 Each report type has various options...
308 OPTIONS FOR ALL REPORT TYPES
311 All_Clusters
314 Use all monitored clusters. Default is local cluster.
315 Clusters=<OPT>
317 List of clusters to include in report. Default is local
318 cluster.
319 End=<OPT>
321 Period ending for report. Default is 23:59:59 of previous
322 day. Valid time formats are...
323 HH:MM[:SS] [AM|PM]
325 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
326 MM/DD[/YY]-HH:MM[:SS]
327 YYYY-MM-DD[THH:MM[:SS]]
328 now[{+|-}count[seconds(default)|min‐
329 utes|hours|days|weeks]]
330 Format=<OPT>
332 Comma separated list of fields to display in report.
333 When using the format option for listing various fields
335 you can put a %NUMBER afterwards to specify how many
336 characters should be printed.
337 e.g. format=name%30 will print 30 characters of field
339 name right justified. A -30 will print 30 characters
340 left justified.
341 Start=<OPT>
343 Period start for report. Default is 00:00:00 of previous
344 day. Valid time formats are...
345 HH:MM[:SS] [AM|PM]
347 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
348 MM/DD[/YY]-HH:MM[:SS]
349 YYYY-MM-DD[THH:MM[:SS]]
350 now[{+|-}count[seconds(default)|min‐
351 utes|hours|days|weeks]]
352 OPTIONS SPECIFICALLY FOR CLUSTER REPORTS
354 Accounts=<OPT>
357 When used with the UserUtilizationByAccount, or AccountU‐
358 tilizationByUser, List of accounts to include in report.
359 Default is all.
360 Tree When used with the AccountUtilizationByUser report will
362 span the accounts as they are in the hierarchy.
363 Users=<OPT>
365 When used with any report other than Utilization, List of
366 users to include in report. Default is all.
367 Wckeys=<OPT>
369 When used with the UserUtilizationByWckey or WCKeyUti‐
370 lizationByUser, List of wckeys to include in report. De‐
371 fault is all.
372 OPTIONS SPECIFICALLY FOR JOB REPORTS
374 Accounts=<OPT>
377 List of accounts to use for the report. Default is all
378 which will show only one line corresponding to the totals
379 of all accounts in the hierarchy. This explanation does
380 not apply when ran with the FlatView or AcctAsParent op‐
381 tions.
382 AcctAsParent
384 When used with the SizesbyAccount(*) will take specified
385 accounts as parents and the next layer of accounts under
386 those specified will be displayed. Default is root if no
387 specific Accounts are requested. When FlatView is used,
388 this option is ignored.
389 FlatView
391 When used with the SizesbyAccount(*) will not group ac‐
392 counts in a hierarchical level, but print each account
393 where jobs ran on a separate line without any hierarchy.
394 GID=<OPT>
396 List of group ids to include in report. Default is all.
397 Grouping=<OPT>
399 Comma separated list of size groupings. (e.g.
400 50,100,150 would group job cpu count 1-49, 50-99,
401 100-149, > 150). grouping=individual will result in a
402 single column for each job size found.
403 Jobs=<OPT>
405 List of jobs/steps to include in report. Default is all.
406 Nodes=<OPT>
408 Only show jobs that ran on these nodes. Default is all.
409 Partitions=<OPT>
411 List of partitions jobs ran on to include in report. De‐
412 fault is all.
413 PrintJobCount
415 When used with the Sizes report will print number of jobs
416 ran instead of time used.
417 Users=<OPT>
419 List of users jobs to include in report. Default is all.
420 Wckeys=<OPT>
422 List of wckeys to use for the report. Default is all.
423 The SizesbyWckey report all users summed together. If
424 you want only certain users specify them with the Users=
425 option.
426 OPTIONS SPECIFICALLY FOR RESERVATION REPORTS
428 Names=<OPT>
431 List of reservations to use for the report. Default is
432 all.
433 Nodes=<OPT>
435 Only show reservations that used these nodes. Default is
436 all.
437 OPTIONS SPECIFICALLY FOR USER REPORTS
439 Accounts=<OPT>
442 List of accounts to use for the report. Default is all.
443 Group Group all accounts together for each user. Default is a
445 separate entry for each user and account reference.
446 TopCount=<OPT>
448 Used in the TopUsage report. Change the number of users
449 displayed. Default is 10.
450 Users=<OPT>
452 List of users jobs to include in report. Default is all.
453
455 FORMAT OPTIONS FOR CLUSTER REPORTS
456 AccountUtilizationByUser
458 Accounts, Cluster, Login, Proper, TresCount, Used
459 UserUtilizationByAccount
461 Accounts, Cluster, Login, Proper, TresCount, Used
462 UserUtilizationByWckey
464 Cluster, Login, Proper, TresCount, Used, Wckey
465 Utilization
467 Allocated, Cluster, Down, Idle, OverCommitted, Planned‐
468 Down, Reported, Planned, TresCount, TresName
469 WCKeyUtilizationByUser
471 Cluster, Login, Proper, TresCount, Used, Wckey
472 FORMAT OPTIONS FOR JOB REPORTS
474 SizesByAccount
477 Account, Cluster
478 SizesByAccountAndWckey
480 Account, Cluster
481 SizesByWckey
483 Wckey, Cluster
484 FORMAT OPTIONS FOR RESERVATION REPORTS
486 Utilization
489 Allocated, Associations, Cluster, End, Flags, Idle, Name,
490 Nodes, ReservationId, Start, TotalTime, TresCount, Tres‐
491 Name, TresTime
492 FORMAT OPTIONS FOR USER REPORTS
494 TopUsage
497 Account, Cluster, Login, Proper, Used
498 All commands and options are case-insensitive.
500
503 Executing sreport sends a remote procedure call to slurmdbd. If enough
504 calls from sreport or other Slurm client commands that send remote pro‐
505 cedure calls to the slurmdbd daemon come in at once, it can result in a
506 degradation of performance of the slurmdbd daemon, possibly resulting
507 in a denial of service.
508 Do not run sreport or other Slurm client commands that send remote pro‐
510 cedure calls to slurmdbd from loops in shell scripts or other programs.
511 Ensure that programs limit calls to sreport to the minimum necessary
512 for the information you are trying to gather.
513
516 Some sreport options may be set via environment variables. These envi‐
517 ronment variables, along with their corresponding options, are listed
518 below. (Note: Command line options will always override these set‐
519 tings.)
520 SREPORT_CLUSTER Same as -M, --cluster
522 SREPORT_FEDERATION Same as --federation
524 SREPORT_LOCAL Same as --local
526 SREPORT_TRES Same as -t, --tres
528 SLURM_CONF The location of the Slurm configuration file.
530
533 Report number of jobs per account according to different job size bins:
534 $ sreport job sizesbyaccount
536 Report cluster utilization:
539 $ sreport cluster utilization
541 Report top usage:
544 $ sreport user top
546 Report number of jobs by user gore1 within the environ account:
549 $ sreport job sizesbyaccount All_Clusters users=gore1 account=environ PrintJobCount
551 Report cluster account utilization with the specified fields during the
554 specified 24 hour day of February 23, 2009, by user gore1:
555 $ sreport cluster AccountUtilizationByUser cluster=zeus user=gore1 start=2/23/09 end=2/24/09 format=Accounts,Cluster,TresCount,Login,Proper,Used
557 Report cluster account utilization by user in the LC account on cluster
560 zeus:
561 $ sreport cluster AccountUtilizationByUser cluster=zeus accounts=lc start=2/23/09 end=2/24/09
563 Report top usage in percent of the lc account during the specified
566 week:
567 $ sreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc
569
572 Copyright (C) 2009-2010 Lawrence Livermore National Security. Produced
573 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
574 Copyright (C) 2010-2022 SchedMD LLC.
575 This file is part of Slurm, a resource management program. For de‐
577 tails, see <https://slurm.schedmd.com/>.
578 Slurm is free software; you can redistribute it and/or modify it under
580 the terms of the GNU General Public License as published by the Free
581 Software Foundation; either version 2 of the License, or (at your op‐
582 tion) any later version.
583 Slurm is distributed in the hope that it will be useful, but WITHOUT
585 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
586 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
587 for more details.
588
591 sacct(1), slurmdbd(8)
592October 2022 Slurm Commands sreport(1)