1cpuunclaimed(8) System Manager's Manual cpuunclaimed(8)
2
6 cpuunclaimed - Sample CPU run queues and calculate unclaimed idle CPU.
7 Uses Linux eBPF/bcc.
8
10 cpuunclaimed [-T] [-j] [-J] [interval [count]]
11
13 This tool samples the length of the run queues and determine when there
14 are idle CPUs, yet queued threads waiting their turn. It reports the
15 amount of idle (yet unclaimed by waiting threads) CPU as a system-wide
16 percentage.
17 This situation can happen for a number of reasons:
19 - An application has been bound to some, but not all, CPUs, and
21 has runnable threads that cannot migrate to other CPUs due to
22 this configuration.
23 - CPU affinity: an optimization that leaves threads on CPUs where
25 the CPU caches are warm, even if this means short periods of
26 waiting while other CPUs are idle. The wait period is tunale
27 (see sysctl, kernel.sched*).
28 - Scheduler bugs.
30 An unclaimed idle of < 1% is likely to be CPU affinity, and not usually
32 a cause for concern. By leaving the CPU idle, overall throughput of the
33 system may be improved. This tool is best for identifying larger
34 issues, > 2%, due to the coarseness of its 99 Hertz samples.
35 This is an experimental tool that currently works by use of sampling to
37 keep overheads low. Tool assumptions:
38 - CPU samples consistently fire around the same offset. There will
40 sometimes be a lag as a sample is delayed by higher-priority
41 interrupts, but it is assumed the subsequent samples will catch
42 up to the expected offsets (as is seen in practice). You can use
43 -J to inspect sample offsets. Some systems can power down CPUs
44 when idle, and when they wake up again they may begin firing at
45 a skewed offset: this tool will detect the skew, print an error,
46 and exit.
47 - All CPUs are online (see ncpu).
49 If this identifies unclaimed CPU, you can double check it by dumping
51 raw samples (-j), as well as using other tracing tools to instrument
52 scheduler events (although this latter approach has much higher over‐
53 head).
54 Since this uses BPF, only the root user can use this tool.
56
58 CONFIG_BPF and bcc.
59
61 Sample and calculate unclaimed idle CPUs, output every 1 second
62 (default:
63 # cpuunclaimed
64 Print 5 second summaries, 10 times:
66 # cpuunclaimed 5 10
67 Print 1 second summaries with timestamps:
69 # cpuunclaimed -T 1
70 Raw dump of all samples (verbose), as comma-separated values:
72 # cpuunclaimed -j
73
75 %CPU CPU utilization as a system-wide percentage.
76 unclaimed idle
78 Percentage of CPU resources that were idle when work was queued
79 on other CPUs, as a system-wide percentage.
80 TIME Time (HH:MM:SS)
82 TIMESTAMP_ns
84 Timestamp, nanoseconds.
85 CPU# CPU ID.
87 OFFSET_ns_CPU#
89 Time offset that a sample fired within a sample group for this
90 CPU.
91
93 The overhead is expected to be low/negligible as this tool uses sam‐
94 pling at 99 Hertz (on all CPUs), which has a fixed and low cost, rather
95 than sampling every scheduler event as many other approaches use (which
96 can involve instrumenting millions of events per second). Sampled CPUs,
97 run queue lengths, and timestamps are written to ring buffers that are
98 periodically read by user space for reporting. Measure overhead in a
99 test environment.
100
102 This is from bcc.
103 https://github.com/iovisor/bcc
105 Also look in the bcc distribution for a companion _examples.txt file
107 containing example usage, output, and commentary for this tool.
108
110 Linux
111
113 Unstable - in development.
114
116 Brendan Gregg
117
119 runqlen(8)
120USER COMMANDS 2016-12-21 cpuunclaimed(8)