bcc-offwaketime(8)

1offwaketime(8)              System Manager's Manual             offwaketime(8)
2
3
4

NAME

6       offwaketime  -  Summarize  blocked time by off-CPU stack + waker stack.
7       Uses Linux eBPF/bcc.
8

SYNOPSIS

10       offwaketime [-h] [-p PID | -t TID | -u | -k] [-U | -K]  [-f]  [--stack-
11       storage-size     STACK_STORAGE_SIZE]     [-m     MIN_BLOCK_TIME]    [-M
12       MAX_BLOCK_TIME] [--state STATE] [duration]
13

DESCRIPTION

15       This program shows kernel stack traces and task names that were blocked
16       and  "off-CPU",  along  with  the  stack  traces and task names for the
17       threads that woke them, and the  total  elapsed  time  from  when  they
18       blocked  to  when they were woken up.  This combines the summaries from
19       both the offcputime and wakeuptime tools.  The time measurement will be
20       very  similar to off-CPU time, however, off-CPU time may include a lit‐
21       tle extra time spent waiting on a run queue to be scheduled.  The  com‐
22       bined  stacks,  task names, and total time is summarized in kernel con‐
23       text for efficiency, using an eBPF map.
24
25       The output summary will further help you identify reasons  why  threads
26       were  blocking,  and  quantify  the time from when they were blocked to
27       woken up.  This spans all types of blocking activity: disk I/O, network
28       I/O,  locks,  page  faults,  swapping,  sleeping,  involuntary  context
29       switches, etc.
30
31       This is complementary to CPU profiling (e.g., CPU flame  graphs)  which
32       shows the time spent on-CPU. This shows the time spent blocked off-CPU,
33       and the output, especially the -f format, can be used  to  generate  an
34       "off-wake time flame graph".
35
36       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
37
38

REQUIREMENTS

40       CONFIG_BPF and bcc.
41

OPTIONS

43       -h     Print usage message.
44
45       -f     Print output in folded stack format.
46
47       -p PID Trace  this process ID only (filtered in-kernel). Can be a comma
48              separated list of PIDS.
49
50       -t TID Trace this thread ID only (filtered in-kernel). Can be  a  comma
51              separated list of TIDS.
52
53       -u     Only trace user threads (no kernel threads).
54
55       -k     Only trace kernel threads (no user threads).
56
57       -U     Show stacks from user space only (no kernel space stacks).
58
59       -K     Show stacks from kernel space only (no user space stacks).
60
61       --stack-storage-size STACK_STORAGE_SIZE
62              Change  the number of unique stack traces that can be stored and
63              displayed.
64
65       duration
66              Duration to trace, in seconds.
67
68       -m MIN_BLOCK_TIME
69              The amount of time in microseconds over which  we  store  traces
70              (default 1)
71
72       -M MAX_BLOCK_TIME
73              The  amount  of time in microseconds under which we store traces
74              (default U64_MAX)
75
76       --state
77              Filter on this thread state bitmask (eg, 2 ==  TASK_UNINTERRUPT‐
78              IBLE).  See include/linux/sched.h for states.
79

EXAMPLES

81       Trace all thread blocking events, and summarize (in-kernel) by user and
82       kernel off-CPU stack trace, waker stack traces, task names,  and  total
83       blocked time:
84              # offwaketime
85
86       Trace for 5 seconds only:
87              # offwaketime 5
88
89       Trace  for  5 seconds, and emit output in folded stack format (suitable
90       for flame graphs), user-mode threads only:
91              # offwaketime -fu 5
92
93       Trace PID 185 only:
94              # offwaketime -p 185
95

OVERHEAD

97       This summarizes unique stack  trace  pairs  in-kernel  for  efficiency,
98       allowing  it  to  trace a higher rate of events than methods that post-
99       process in user space.  The stack trace and time data is only copied to
100       user  space  once,  when  the output is printed. While these techniques
101       greatly lower overhead, scheduler events are  still  a  high  frequency
102       event,  as  they can exceed 1 million events per second, and so caution
103       should still be used. Test before production use.
104
105       If the overhead is still a problem,  take  a  look  at  the  min  block
106       option.  If your aim is to chase down longer blocking events, then this
107       could be increased to filter shorter blocking events, further  lowering
108       overhead.
109

SOURCE

111       This is from bcc.
112
113              https://github.com/iovisor/bcc
114
115       Also  look  in  the bcc distribution for a companion _examples.txt file
116       containing example usage, output, and commentary for this tool.
117

OS

119       Linux
120

STABILITY

122       Unstable - in development.
123

AUTHOR

125       Brendan Gregg
126