bcc-offwaketime(8)

1offwaketime(8)              System Manager's Manual             offwaketime(8)
2
3
4

NAME

6       offwaketime  -  Summarize  blocked time by off-CPU stack + waker stack.
7       Uses Linux eBPF/bcc.
8

SYNOPSIS

10       offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K]  [-f]  [--stack-
11       storage-size     STACK_STORAGE_SIZE]     [-m     MIN_BLOCK_TIME]    [-M
12       MAX_BLOCK_TIME] [duration]
13

DESCRIPTION

15       This program shows kernel stack traces and task names that were blocked
16       and  "off-CPU",  along  with  the  stack  traces and task names for the
17       threads that woke them, and the  total  elapsed  time  from  when  they
18       blocked  to  when they were woken up.  This combines the summaries from
19       both the offcputime and wakeuptime tools.  The time measurement will be
20       very  similar to off-CPU time, however, off-CPU time may include a lit‐
21       tle extra time spent waiting on a run queue to be scheduled.  The  com‐
22       bined  stacks,  task names, and total time is summarized in kernel con‐
23       text for efficiency, using an eBPF map.
24
25       The output summary will further help you identify reasons  why  threads
26       were  blocking,  and  quantify  the time from when they were blocked to
27       woken up.  This spans all types of blocking activity: disk I/O, network
28       I/O,  locks,  page  faults,  swapping,  sleeping,  involuntary  context
29       switches, etc.
30
31       This is complementary to CPU profiling (e.g., CPU flame  graphs)  which
32       shows the time spent on-CPU. This shows the time spent blocked off-CPU,
33       and the output, especially the -f format, can be used  to  generate  an
34       "off-wake time flame graph".
35
36       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
37
38

REQUIREMENTS

40       CONFIG_BPF and bcc.
41

OPTIONS

43       -h     Print usage message.
44
45       -f     Print output in folded stack format.
46
47       -p PID Trace this process ID only (filtered in-kernel).
48
49       -t TID Trace this thread ID only (filtered in-kernel).
50
51       -u     Only trace user threads (no kernel threads).
52
53       -k     Only trace kernel threads (no user threads).
54
55       -U     Show stacks from user space only (no kernel space stacks).
56
57       -K     Show stacks from kernel space only (no user space stacks).
58
59       --stack-storage-size STACK_STORAGE_SIZE
60              Change  the number of unique stack traces that can be stored and
61              displayed.
62
63       duration
64              Duration to trace, in seconds.
65
66       -m MIN_BLOCK_TIME
67              The amount of time in microseconds over which  we  store  traces
68              (default 1)
69
70       -M MAX_BLOCK_TIME
71              The  amount  of time in microseconds under which we store traces
72              (default U64_MAX)
73

EXAMPLES

75       Trace all thread blocking events, and summarize (in-kernel) by user and
76       kernel  off-CPU  stack trace, waker stack traces, task names, and total
77       blocked time:
78              # offwaketime
79
80       Trace for 5 seconds only:
81              # offwaketime 5
82
83       Trace for 5 seconds, and emit output in folded stack  format  (suitable
84       for flame graphs), user-mode threads only:
85              # offwaketime -fu 5
86
87       Trace PID 185 only:
88              # offwaketime -p 185
89

OVERHEAD

91       This  summarizes  unique  stack  trace  pairs in-kernel for efficiency,
92       allowing it to trace a higher rate of events than  methods  that  post-
93       process in user space.  The stack trace and time data is only copied to
94       user space once, when the output is  printed.  While  these  techniques
95       greatly  lower  overhead,  scheduler  events are still a high frequency
96       event, as they can exceed 1 million events per second, and  so  caution
97       should still be used. Test before production use.
98
99       If  the  overhead  is  still  a  problem,  take a look at the min block
100       option.  If your aim is to chase down longer blocking events, then this
101       could  be increased to filter shorter blocking events, further lowering
102       overhead.
103

SOURCE

105       This is from bcc.
106
107              https://github.com/iovisor/bcc
108
109       Also look in the bcc distribution for a  companion  _examples.txt  file
110       containing example usage, output, and commentary for this tool.
111

OS

113       Linux
114

STABILITY

116       Unstable - in development.
117

AUTHOR

119       Brendan Gregg
120