1sdiag(1) Slurm Commands sdiag(1)
2
6 sdiag - Scheduling diagnostic tool for Slurm
7
10 sdiag
11
14 sdiag shows information related to slurmctld execution about: threads,
15 agents, jobs, and scheduling algorithms. The goal is to obtain data
16 from slurmctld behaviour helping to adjust configuration parameters or
17 queues policies. The main reason behind is to know Slurm behaviour un‐
18 der systems with a high throughput.
19 It has two execution modes. The default mode --all shows several coun‐
21 ters and statistics explained later, and there is another execution op‐
22 tion --reset for resetting those values.
23 Values are reset at midnight UTC time by default.
25 The first block of information is related to global slurmctld execu‐
27 tion:
28 Server thread count
30 The number of current active slurmctld threads. A high number
31 would mean a high load processing events like job submissions,
32 jobs dispatching, jobs completing, etc. If this is often close
33 to MAX_SERVER_THREADS it could point to a potential bottleneck.
34 Agent queue size
37 Slurm design has scalability in mind and sending messages to
38 thousands of nodes is not a trivial task. The agent mechanism
39 helps to control communication between slurmctld and the slurmd
40 daemons for a best effort. This value denotes the count of en‐
41 queued outgoing RPC requests in an internal retry list.
42 Agent count
45 Number of agent threads. Each of these agent threads can create
46 in turn a group of up to 2 + AGENT_THREAD_COUNT active threads
47 at a time.
48 Agent thread count
51 Total count of active threads created by all the agent threads.
52 DBD Agent queue size
55 Slurm queues up the messages intended for the SlurmDBD and pro‐
56 cesses them in a separate thread. If the SlurmDBD, or database,
57 is down then this number will increase.
58 The max queue size is configured in the slurm.conf with MaxDB‐
60 DMsgs. If this number begins to grow more than half of the max
61 queue size, the slurmdbd and the database should be investigated
62 immediately.
63 Jobs submitted
66 Number of jobs submitted since last reset
67 Jobs started
70 Number of jobs started since last reset. This includes back‐
71 filled jobs.
72 Jobs completed
75 Number of jobs completed since last reset.
76 Jobs canceled
79 Number of jobs canceled since last reset.
80 Jobs failed
83 Number of jobs failed due to slurmd or other internal issues
84 since last reset.
85 Job states ts:
88 Lists the timestamp of when the following job state counts were
89 gathered.
90 Jobs pending:
93 Number of jobs pending at the given time of the time stamp
94 above.
95 Jobs running:
98 Number of jobs running at the given time of the time stamp
99 above.
100 Jobs running ts:
103 Time stamp of when the running job count was taken.
104 The next block of information is related to main scheduling algorithm
107 based on jobs priorities. A scheduling cycle implies to get the
108 job_write_lock lock, then trying to get resources for jobs pending,
109 starting from the most priority one and going in descendent order. Once
110 a job can not get the resources the loop keeps going but just for jobs
111 requesting other partitions. Jobs with dependencies or affected by ac‐
112 counts limits are not processed.
113 Last cycle
116 Time in microseconds for last scheduling cycle.
117 Max cycle
120 Maximum time in microseconds for any scheduling cycle since last
121 reset.
122 Total cycles
125 Total run time in microseconds for all scheduling cycles since
126 last reset. Scheduling is performed periodically and (depending
127 upon configuration) when a job is submitted or a job is com‐
128 pleted.
129 Mean cycle
132 Mean time in microseconds for all scheduling cycles since last
133 reset.
134 Mean depth cycle
137 Mean of cycle depth. Depth means number of jobs processed in a
138 scheduling cycle.
139 Cycles per minute
142 Counter of scheduling executions per minute.
143 Last queue length
146 Length of jobs pending queue.
147 The next block of information is related to backfilling scheduling al‐
150 gorithm. A backfilling scheduling cycle implies to get locks for jobs,
151 nodes and partitions objects then trying to get resources for jobs
152 pending. Jobs are processed based on priorities. If a job can not get
153 resources the algorithm calculates when it could get them obtaining a
154 future start time for the job. Then next job is processed and the al‐
155 gorithm tries to get resources for that job but avoiding to affect the
156 previous ones, and again it calculates the future start time if not
157 current resources available. The backfilling algorithm takes more time
158 for each new job to process since more priority jobs can not be af‐
159 fected. The algorithm itself takes measures for avoiding a long execu‐
160 tion cycle and for taking all the locks for too long.
161 Total backfilled jobs (since last slurm start)
164 Number of jobs started thanks to backfilling since last slurm
165 start.
166 Total backfilled jobs (since last stats cycle start)
169 Number of jobs started thanks to backfilling since last time
170 stats where reset. By default these values are reset at mid‐
171 night UTC time.
172 Total backfilled heterogeneous job components
175 Number of heterogeneous job components started thanks to back‐
176 filling since last Slurm start.
177 Total cycles
180 Number of backfill scheduling cycles since last reset
181 Last cycle when
184 Time when last backfill scheduling cycle happened in the format
185 "weekday Month MonthDay hour:minute.seconds year"
186 Last cycle
189 Time in microseconds of last backfill scheduling cycle. It
190 counts only execution time, removing sleep time inside a sched‐
191 uling cycle when it executes for an extended period time. Note
192 that locks are released during the sleep time so that other work
193 can proceed.
194 Max cycle
197 Time in microseconds of maximum backfill scheduling cycle execu‐
198 tion since last reset. It counts only execution time, removing
199 sleep time inside a scheduling cycle when it executes for an ex‐
200 tended period time. Note that locks are released during the
201 sleep time so that other work can proceed.
202 Mean cycle
205 Mean time in microseconds of backfilling scheduling cycles since
206 last reset.
207 Last depth cycle
210 Number of processed jobs during last backfilling scheduling cy‐
211 cle. It counts every job even if that job can not be started due
212 to dependencies or limits.
213 Last depth cycle (try sched)
216 Number of processed jobs during last backfilling scheduling cy‐
217 cle. It counts only jobs with a chance to start using available
218 resources. These jobs consume more scheduling time than jobs
219 which are found can not be started due to dependencies or lim‐
220 its.
221 Depth Mean
224 Mean count of jobs processed during all backfilling scheduling
225 cycles since last reset. Jobs which are found to be ineligible
226 to run when examined by the backfill scheduler are not counted
227 (e.g. jobs submitted to multiple partitions and already started,
228 jobs which have reached a QOS or account limit such as maximum
229 running jobs for an account, etc).
230 Depth Mean (try sched)
233 The subset of Depth Mean that the backfill scheduler attempted
234 to schedule.
235 Last queue length
238 Number of jobs pending to be processed by backfilling algorithm.
239 A job is counted once for each partition it is queued to use. A
240 pending job array will normally be counted as one job (tasks of
241 a job array which have already been started/requeued or individ‐
242 ually modified will already have individual job records and are
243 each counted as a separate job).
244 Queue length Mean
247 Mean count of jobs pending to be processed by backfilling algo‐
248 rithm. A job is counted once for each partition it requested.
249 A pending job array will normally be counted as one job (tasks
250 of a job array which have already been started/requeued or indi‐
251 vidually modified will already have individual job records and
252 are each counted as a separate job).
253 Last table size
256 Count of different time slots tested by the backfill scheduler
257 in its last iteration.
258 Mean table size
261 Mean count of different time slots tested by the backfill sched‐
262 uler. Larger counts increase the time required for the backfill
263 operation. The table size is influenced by many schuling param‐
264 eters, including: bf_min_age_reserve, bf_min_prio_reserve,
265 bf_resolution, and bf_window.
266 Latency for 1000 calls to gettimeofday()
269 Latency of 1000 calls to the gettimeofday() syscall in microsec‐
270 onds, as measured at controller startup.
271 The next blocks of information report the most frequently issued remote
274 procedure calls (RPCs), calls made for the Slurmctld daemon to perform
275 some action. The fourth block reports the RPCs issued by message type.
276 You will need to look up those RPC codes in the Slurm source code by
277 looking them up in the file src/common/slurm_protocol_defs.h. The re‐
278 port includes the number of times each RPC is invoked, the total time
279 consumed by all of those RPCs plus the average time consumed by each
280 RPC in microseconds. The fifth block reports the RPCs issued by user
281 ID, the total number of RPCs they have issued, the total time consumed
282 by all of those RPCs plus the average time consumed by each RPC in mi‐
283 croseconds. RPCs statistics are collected for the life of the slurm‐
284 ctld process unless explicitly --reset.
285 The sixth block of information, labeled Pending RPC Statistics, shows
288 information about pending outgoing RPCs on the slurmctld agent queue.
289 The first section of this block shows types of RPCs on the queue and
290 the count of each. The second section shows up to the first 25 individ‐
291 ual RPCs pending on the agent queue, including the type and the desti‐
292 nation host list. This information is cached and only refreshed on 30
293 second intervals.
294
297 -a, --all
298 Get and report information. This is the default mode of opera‐
299 tion.
300 -h, --help
303 Print description of options and exit.
304 -i, --sort-by-id
307 Sort Remote Procedure Call (RPC) data by message type ID and
308 user ID.
309 -M, --cluster=<string>
312 The cluster to issue commands to. Only one cluster name may be
313 specified. Note that the SlurmDBD must be up for this option to
314 work properly.
315 -r, --reset
318 Reset scheduler and RPC counters to 0. Only supported for Slurm
319 operators and administrators.
320 -t, --sort-by-time
323 Sort Remote Procedure Call (RPC) data by total run time.
324 -T, --sort-by-time2
327 Sort Remote Procedure Call (RPC) data by average run time.
328 --usage
331 Print list of options and exit.
332 -V, --version
335 Print current version number and exit.
336
339 Executing sdiag sends a remote procedure call to slurmctld. If enough
340 calls from sdiag or other Slurm client commands that send remote proce‐
341 dure calls to the slurmctld daemon come in at once, it can result in a
342 degradation of performance of the slurmctld daemon, possibly resulting
343 in a denial of service.
344 Do not run sdiag or other Slurm client commands that send remote proce‐
346 dure calls to slurmctld from loops in shell scripts or other programs.
347 Ensure that programs limit calls to sdiag to the minimum necessary for
348 the information you are trying to gather.
349
352 Some sdiag options may be set via environment variables. These environ‐
353 ment variables, along with their corresponding options, are listed be‐
354 low. (Note: commandline options will always override these settings)
355 SLURM_CLUSTERS Same as --cluster
357 SLURM_CONF The location of the Slurm configuration file.
360
363 Copyright (C) 2010-2011 Barcelona Supercomputing Center.
364 Copyright (C) 2010-2019 SchedMD LLC.
365 Slurm is free software; you can redistribute it and/or modify it under
367 the terms of the GNU General Public License as published by the Free
368 Software Foundation; either version 2 of the License, or (at your op‐
369 tion) any later version.
370 Slurm is distributed in the hope that it will be useful, but WITHOUT
372 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
373 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
374 for more details.
375
378 sinfo(1), squeue(1), scontrol(1), slurm.conf(5),
379April 2021 Slurm Commands sdiag(1)