1sreport(1) Slurm Commands sreport(1)
2
6 sreport - Generate reports from the slurm accounting data.
7
10 sreport [OPTIONS...] [COMMAND...]
11
14 sreport is used to generate reports of job usage and cluster utiliza‐
15 tion for Slurm jobs saved to the Slurm Database, slurmdbd.
16
19 -a, --all_clusters
20 Use all clusters instead of only the cluster from which the com‐
21 mand was executed.
22 --federation
24 Generate a report for the federation if a member of one.
25 -h, --help
27 Print a help message describing the usage of sreport.
28 --local
30 Generate a report for the local cluster, even if part of a fed‐
31 eration. Overrides --federation
32 -M, --cluster
34 The cluster(s) to generate reports for. Default is local clus‐
35 ter, unless the local cluster is currently part of a federation
36 and in that case generate a report for all clusters in the cur‐
37 rent federation. If the clusters included in a federation vary
38 through time, use this option to identify the clusters to be
39 included in report. Implies --local.
40 -n, --noheader
42 Don't display header when listing results.
43 -p, --parsable
45 Output will be '|' delimited with a '|' at the end.
46 -P, --parsable2
48 Output will be '|' delimited without a '|' at the end.
49 -Q, --quiet
51 Print no warning or informational messages, only error messages.
52 -t <format>
54 Specify the output time format. Time format options are case
55 insensitive and may be abbreviated. The default format is Min‐
56 utes. Supported time format options are listed in the time com‐
57 mand section below.
58 -T, --tres <tres_names>
60 Trackable resource (TRES) to report values for. By default CPU
61 resource use is reported (except for reservation reports. All
62 TRES types reserved by a reservation will be shown by default
63 unless overridden with this option). Multiple TRES names may be
64 separated using a comma separated list for all reports except
65 the job reports, which can only support a single TRES name, or
66 "ALL" for all TRES.
67 The "Reported" Billing TRES is calculated from the largest
69 Billing TRES of each node multiplied by the time frame. For
70 example, if a node is part of multiple partitions and each has a
71 different TRESBillingWeights defined the Billing TRES for the
72 node will be the highest of the partitions. If TRESBilling‐
73 Weights is not defined on any partition for a node then the
74 Billing TRES will be equal to the number of CPUs on the node.
75 TRES node usage is no longer reported in percent format or in
77 Cluster Utilization. Please use TRES CPU instead. The main
78 issue with using node is in most configurations multiple jobs
79 are able to run on the same node. This makes TRES node account‐
80 ing count the same node multiple times in the same period. In
81 exclusive node configurations, CPU accounting returns the same
82 usage node accounting would.
83 -v, --verbose
86 Print detailed event logging.
87 -V , --version
89 Print version information and exit.
90
93 <keyword> may be omitted from the execute line and sreport will
94 execute in interactive mode. sreport will process commands as
95 entered until explicitly terminated.
96 exit Terminate the execution of sreport. Identical to the quit com‐
99 mand.
100 help Display a description of sreport options and commands.
103 nonparsable
106 Return output to normal after parsable or parsable2 has been
107 set.
108 parsable
111 Output will be | delimited with an ending '|'.
112 parsable2
115 Output will be | delimited without an ending '|'.
116 quiet Print no warning or informational messages, only fatal error
119 messages.
120 quit Terminate the execution of sreport. Identical to the exit com‐
123 mand.
124 time <time_format>
127 Specify the output time format. Time format options are case
128 insensitive and may be abbreviated. The default format is Min‐
129 utes. Supported time format options include:
130 SecPer Seconds/Percentage of Total
133 MinPer Minutes/Percentage of Total
135 HourPer Hours/Percentage of Total
137 Seconds Seconds
139 Minutes Minutes
141 Hours Hours
143 Percent Percentage of Total
145 verbose
148 Enable detailed event logging.
149 version
152 Display the sreport version number.
153 !! Repeat the last command executed.
156
159 Valid report types are:
160 cluster <REPORT> <OPTIONS>
161 job <REPORT> <OPTIONS>
163 reservation <REPORT> <OPTIONS>
165 user <REPORT> <OPTIONS>
167 <REPORT> options for each type include:
170 cluster - AccountUtilizationByUser, UserUtilizationByAccount,
172 UserUtilizationByWckey, Utilization, WCKeyUtilizationByUser
173 job - SizesByAccount, SizesByAccountAndWcKey, SizesByWckey
175 reservation - Utilization
177 user - TopUsage
179 NOTE: If OverSubscribe is configured to FORCE or YES in your slurm.conf
182 and the system is not configured to use preemption (PreemptMode=OFF)
183 accounting can easily grow to values greater than the actual utiliza‐
184 tion. It may be common on such systems to get error messages in the
185 slurmdbd log stating: "We have more allocated time than is possible."
186
189 cluster AccountUtilizationByUser
190 This report will display account utilization as it appears on
191 the hierarchical tree. Starting with the specified account or
192 the root account by default this report will list the underlying
193 usage with a sum on each level. Use the 'tree' option to span
194 the tree for better visibility.
195 NOTE: Idle reservation time will be split evenly among
197 accounts/users given access to it. When a reservation is
198 assigned to whole accounts, the time will be counted in the
199 association for the accounts, not the user associations in the
200 accounts. In this case, the usage of a parent account can be
201 larger than the sum of its children.
202 cluster UserUtilizationByAccount
204 This report will display users by account in order of utiliza‐
205 tion without grouping multiple accounts by user into one, but
206 displaying them on separate lines.
207 cluster UserUtilizationByWCKey
209 This report will display users by wckey in order of utilization
210 without grouping multiple wckey by user into one, but displaying
211 them on separate lines.
212 cluster Utilization
214 This report will display total usage divided amongst Allocated,
215 Down, Planned Down, Idle, and Reserved time for selected clus‐
216 ters.
217 Allocated Time that nodes were in use with active jobs or an
219 active reservation. This does not include reserva‐
220 tions created with the MAINT or IGNORE_JOBS flags.
221 Down Time that nodes were marked as Down or time that
223 slurmctld was not responding (assuming TrackSlurm‐
224 ctldDown is set in slurmdbd.conf).
225 Idle Time where nodes had no active jobs or reserva‐
227 tions.
228 Overcommited Time of eligible jobs waiting in the queue over the
230 Reserved time. Unlike Reserved, this has no limit.
231 It is typically useful to determine whether your
232 system is overloaded and by how much.
233 PlannedDown Time that nodes were in use by a reservation cre‐
235 ated with the MAINT flag but not the IGNORE_JOBS
236 flag.
237 Reserved Time that a node spent idle with eligible jobs in
239 the queue that were unable to start due to time or
240 size constraints. The maximum time is limited to
241 the amount of time possible on the system. If this
242 value is not of importance for you then the number
243 can be grouped with idle time.
244 TresCount Configured number of TRES' on the reported nodes.
246 See also the TresName field.
247 Note: Reservations created with the IGNORE_JOBS flag are not
249 tracked in the Cluster Utilization report due to the fact that
250 allowing any current/active jobs to continue to run in the
251 reservation introduces the possibility for them to be accounted
252 for incorrectly.
253 Note: The default view for the "Cluster Utilization" report
255 includes the following fields: Cluster, Allocated, Down,
256 PlannedDown, Idle, Reserved, Reported. You can include addi‐
257 tional fields like OverCommitted and TresCount fields with the
258 Format option. The TresName will also be included if using the
259 -T, --tres <tres_names> option.
260 cluster WCKeyUtilizationByUser
262 This report will display wckey utilization sorted by WCKey name
263 for each user on each cluster.
264 job SizesByAccount
267 This report will display the amount of time used for job ranges
268 specified by the 'grouping=' option. Only a single level in the
269 tree is displayed defaulting to the root dir. If you specify
270 other accounts with the 'account=' option sreport will use those
271 accounts as the root account and you will receive the aggregated
272 totals of each listed account plus their sub accounts.
273 job SizesByAccountAndWckey
275 This report is very similar to SizesByAccount with the differ‐
276 ence being each account is pair with wckeys so the identifier is
277 account:wckey instead of just account so there will most likely
278 be multiple accounts listed depending on the number of wckeys
279 used.
280 job SizesByWckey
282 This report will display the amount of time for each wckey for
283 job ranges specified by the 'grouping=' option.
284 reservation Utilization
287 This report will display total usage for reservations on the
288 systems. Note: Time requests on this report will not truncate
289 the time the reservation used, only the reservations that ran at
290 any time during the period requested.
291 user TopUsage
294 Displays the top users on a cluster. Use the group option to
295 group accounts together. The default is to have a different
296 line for each user account combination.
297 Each report type has various options...
300 OPTIONS FOR ALL REPORT TYPES
303 All_Clusters
306 Use all monitored clusters. Default is local cluster.
307 Clusters=<OPT>
309 List of clusters to include in report. Default is local clus‐
310 ter.
311 End=<OPT>
314 Period ending for report. Default is 23:59:59 of previous day.
315 Valid time formats are...
316 HH:MM[:SS] [AM|PM]
318 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
319 MM/DD[/YY]-HH:MM[:SS]
320 YYYY-MM-DD[THH:MM[:SS]]
321 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
322 Format=<OPT>
325 Comma separated list of fields to display in report.
326 When using the format option for listing various fields you can
328 put a %NUMBER afterwards to specify how many characters should
329 be printed.
330 e.g. format=name%30 will print 30 characters of field name right
332 justified. A -30 will print 30 characters left justified.
333 Start=<OPT>
336 Period start for report. Default is 00:00:00 of previous day.
337 Valid time formats are...
338 HH:MM[:SS] [AM|PM]
340 MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
341 MM/DD[/YY]-HH:MM[:SS]
342 YYYY-MM-DD[THH:MM[:SS]]
343 now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]
344 OPTIONS SPECIFICALLY FOR CLUSTER REPORTS
346 Accounts=<OPT>
349 When used with the UserUtilizationByAccount, or AccountUtiliza‐
350 tionByUser, List of accounts to include in report. Default is
351 all.
352 Tree When used with the AccountUtilizationByUser report will span the
354 accounts as they are in the hierarchy.
355 Users=<OPT>
357 When used with any report other than Utilization, List of users
358 to include in report. Default is all.
359 Wckeys=<OPT>
361 When used with the UserUtilizationByWckey or WCKeyUtilization‐
362 ByUser, List of wckeys to include in report. Default is all.
363 OPTIONS SPECIFICALLY FOR JOB REPORTS
366 Accounts=<OPT>
369 List of accounts to use for the report. Default is all which
370 will show only one line corresponding to the totals of all
371 accounts in the hierarchy. This explanation does not apply when
372 ran with the FlatView or AcctAsParent options.
373 AcctAsParent
375 When used with the SizesbyAccount(*) will take specified
376 accounts as parents and the next layer of accounts under those
377 specified will be displayed. Default is root if no specific
378 Accounts are requested. When FlatView is used, this option is
379 ignored.
380 FlatView
382 When used with the SizesbyAccount(*) will not group accounts in
383 a hierarchical level, but print each account where jobs ran on a
384 separate line without any hierarchy.
385 GID=<OPT>
387 List of group ids to include in report. Default is all.
388 Grouping=<OPT>
390 Comma separated list of size groupings. (e.g. 50,100,150 would
391 group job cpu count 1-49, 50-99, 100-149, > 150). group‐
392 ing=individual will result in a single column for each job size
393 found.
394 Jobs=<OPT>
396 List of jobs/steps to include in report. Default is all.
397 Nodes=<OPT>
399 Only show jobs that ran on these nodes. Default is all.
400 Partitions=<OPT>
402 List of partitions jobs ran on to include in report. Default is
403 all.
404 PrintJobCount
406 When used with the Sizes report will print number of jobs ran
407 instead of time used.
408 Users=<OPT>
410 List of users jobs to include in report. Default is all.
411 Wckeys=<OPT>
413 List of wckeys to use for the report. Default is all. The
414 SizesbyWckey report all users summed together. If you want only
415 certain users specify them with the Users= option.
416 OPTIONS SPECIFICALLY FOR RESERVATION REPORTS
419 Names=<OPT>
421 List of reservations to use for the report. Default is all.
422 Nodes=<OPT>
424 Only show reservations that used these nodes. Default is all.
425 OPTIONS SPECIFICALLY FOR USER REPORTS
429 Accounts=<OPT>
432 List of accounts to use for the report. Default is all.
433 Group Group all accounts together for each user. Default is a sepa‐
435 rate entry for each user and account reference.
436 TopCount=<OPT>
438 Used in the TopUsage report. Change the number of users dis‐
439 played. Default is 10.
440 Users=<OPT>
442 List of users jobs to include in report. Default is all.
443
446 FORMAT OPTIONS FOR CLUSTER REPORTS
447 AccountUtilizationByUser:
448 UserUtilizationByAccount:
449 Accounts, Cluster, Login, Proper, TresCount, Used
450 UserUtilizationByWckey:
452 WCKeyUtilizationByUser:
453 Cluster, Login, Proper, TresCount, Used, Wckey
454 Utilization:
456 Allocated, Cluster, Down, Idle, Overcommitted, PlannedDown,
457 Reported, Reserved, TresCount, TresName
458 FORMAT OPTIONS FOR JOB REPORTS
461 SizesByAccount, SizesByAccountAndWckey:
462 Account, Cluster
463 SizesByWckey:
465 Wckey, Cluster
466 FORMAT OPTIONS FOR RESERVATION REPORTS
469 Utilization:
470 Allocated, Associations, Cluster, End, Flags, Idle, Name, Nodes,
471 ReservationId, Start, TotalTime, TresCount, TresName, TresTime
472 FORMAT OPTIONS FOR USER REPORTS
475 TopUsage:
476 Account, Cluster, Login, Proper, Used
477 All commands and options are case-insensitive.
480
483 Executing sreport sends a remote procedure call to slurmdbd. If enough
484 calls from sreport or other Slurm client commands that send remote pro‐
485 cedure calls to the slurmdbd daemon come in at once, it can result in a
486 degradation of performance of the slurmdbd daemon, possibly resulting
487 in a denial of service.
488 Do not run sreport or other Slurm client commands that send remote pro‐
490 cedure calls to slurmdbd from loops in shell scripts or other programs.
491 Ensure that programs limit calls to sreport to the minimum necessary
492 for the information you are trying to gather.
493
496 Some sreport options may be set via environment variables. These envi‐
497 ronment variables, along with their corresponding options, are listed
498 below. (Note: commandline options will always override these settings)
499 SREPORT_CLUSTER Same as -M, --cluster
501 SREPORT_FEDERATION Same as --federation
503 SREPORT_LOCAL Same as --local
505 SREPORT_TRES Same as -t, --tres
507 SLURM_CONF The location of the Slurm configuration file.
509
512 sreport job sizesbyaccount
513 sreport cluster utilization
515 sreport user top
517 sreport job sizesbyaccount All_Clusters users=gore1 account=environ
519 PrintJobCount
520 Report number of jobs by user gore1 within the environ account
521 sreport cluster AccountUtilizationByUser cluster=zeus user=gore1
523 start=2/23/08 end=2/24/09 format=Accounts,Clus‐
524 ter,TresCount,Login,Proper,Used
525 Report cluster account utilization with the specified fields
526 during the specified 24 hour day of February 23, 2009, by user
527 gore1
528 sreport cluster AccountUtilizationByUser cluster=zeus accounts=lc
530 start=2/23/08 end=2/24/09
531 Report cluster account utilization by user in the LC account on
532 cluster zeus
533 sreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc
535 Report top usage in percent of the lc account during the speci‐
536 fied week
537
540 Copyright (C) 2009-2010 Lawrence Livermore National Security. Produced
541 at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
542 Copyright (C) 2010-2015 SchedMD LLC.
543 This file is part of Slurm, a resource management program. For
545 details, see <https://slurm.schedmd.com/>.
546 Slurm is free software; you can redistribute it and/or modify it under
548 the terms of the GNU General Public License as published by the Free
549 Software Foundation; either version 2 of the License, or (at your
550 option) any later version.
551 Slurm is distributed in the hope that it will be useful, but WITHOUT
553 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
554 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
555 for more details.
556
559 sacct(1), slurmdbd(8)
560November 2020 Slurm Commands sreport(1)