1offwaketime(8) System Manager's Manual offwaketime(8)
2
3
4
6 offwaketime - Summarize blocked time by off-CPU stack + waker stack.
7 Uses Linux eBPF/bcc.
8
10 offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K] [-d] [-f]
11 [--stack-storage-size STACK_STORAGE_SIZE] [-m MIN_BLOCK_TIME] [-M
12 MAX_BLOCK_TIME] [--state STATE] [duration]
13
15 This program shows kernel stack traces and task names that were blocked
16 and "off-CPU", along with the stack traces and task names for the
17 threads that woke them, and the total elapsed time from when they
18 blocked to when they were woken up. This combines the summaries from
19 both the offcputime and wakeuptime tools. The time measurement will be
20 very similar to off-CPU time, however, off-CPU time may include a lit‐
21 tle extra time spent waiting on a run queue to be scheduled. The com‐
22 bined stacks, task names, and total time is summarized in kernel con‐
23 text for efficiency, using an eBPF map.
24
25 The output summary will further help you identify reasons why threads
26 were blocking, and quantify the time from when they were blocked to wo‐
27 ken up. This spans all types of blocking activity: disk I/O, network
28 I/O, locks, page faults, swapping, sleeping, involuntary context
29 switches, etc.
30
31 This is complementary to CPU profiling (e.g., CPU flame graphs) which
32 shows the time spent on-CPU. This shows the time spent blocked off-CPU,
33 and the output, especially the -f format, can be used to generate an
34 "off-wake time flame graph".
35
36 See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
37
38
40 CONFIG_BPF and bcc.
41
43 -h Print usage message.
44
45 -f Print output in folded stack format.
46
47 -p PID Trace this process ID only (filtered in-kernel). Can be a comma
48 separated list of PIDS.
49
50 -t TID Trace this thread ID only (filtered in-kernel). Can be a comma
51 separated list of TIDS.
52
53 -u Only trace user threads (no kernel threads).
54
55 -k Only trace kernel threads (no user threads).
56
57 -U Show stacks from user space only (no kernel space stacks).
58
59 -K Show stacks from kernel space only (no user space stacks).
60
61 -d, --delimited
62 insert delimiter between kernel/user stacks
63
64 --stack-storage-size STACK_STORAGE_SIZE
65 Change the number of unique stack traces that can be stored and
66 displayed.
67
68 duration
69 Duration to trace, in seconds.
70
71 -m MIN_BLOCK_TIME
72 The amount of time in microseconds over which we store traces
73 (default 1)
74
75 -M MAX_BLOCK_TIME
76 The amount of time in microseconds under which we store traces
77 (default U64_MAX)
78
79 --state
80 Filter on this thread state bitmask (eg, 2 == TASK_UNINTERRUPT‐
81 IBLE). See include/linux/sched.h for states.
82
84 Trace all thread blocking events, and summarize (in-kernel) by user and
85 kernel off-CPU stack trace, waker stack traces, task names, and total
86 blocked time:
87 # offwaketime
88
89 Trace for 5 seconds only:
90 # offwaketime 5
91
92 Trace for 5 seconds, and emit output in folded stack format (suitable
93 for flame graphs), user-mode threads only:
94 # offwaketime -fu 5
95
96 Trace PID 185 only:
97 # offwaketime -p 185
98
100 This summarizes unique stack trace pairs in-kernel for efficiency, al‐
101 lowing it to trace a higher rate of events than methods that post-
102 process in user space. The stack trace and time data is only copied to
103 user space once, when the output is printed. While these techniques
104 greatly lower overhead, scheduler events are still a high frequency
105 event, as they can exceed 1 million events per second, and so caution
106 should still be used. Test before production use.
107
108 If the overhead is still a problem, take a look at the min block op‐
109 tion. If your aim is to chase down longer blocking events, then this
110 could be increased to filter shorter blocking events, further lowering
111 overhead.
112
114 This is from bcc.
115
116 https://github.com/iovisor/bcc
117
118 Also look in the bcc distribution for a companion _examples.txt file
119 containing example usage, output, and commentary for this tool.
120
122 Linux
123
125 Unstable - in development.
126
128 Brendan Gregg
129
131 offcputime(8), wakeuptime(8)
132
133
134
135USER COMMANDS 2016-01-30 offwaketime(8)