bcc-offwaketime(8)

1offwaketime(8)              System Manager's Manual             offwaketime(8)
2
3
4

NAME

6       offwaketime  -  Summarize  blocked time by off-CPU stack + waker stack.
7       Uses Linux eBPF/bcc.
8

SYNOPSIS

10       offwaketime [-h] [-p PID | -t TID | -u  |  -k]  [-U  |  -K]  [-d]  [-f]
11       [--stack-storage-size   STACK_STORAGE_SIZE]   [-m  MIN_BLOCK_TIME]  [-M
12       MAX_BLOCK_TIME] [--state STATE] [duration]
13

DESCRIPTION

15       This program shows kernel stack traces and task names that were blocked
16       and  "off-CPU",  along  with  the  stack  traces and task names for the
17       threads that woke them, and the  total  elapsed  time  from  when  they
18       blocked  to  when they were woken up.  This combines the summaries from
19       both the offcputime and wakeuptime tools.  The time measurement will be
20       very  similar to off-CPU time, however, off-CPU time may include a lit‐
21       tle extra time spent waiting on a run queue to be scheduled.  The  com‐
22       bined  stacks,  task names, and total time is summarized in kernel con‐
23       text for efficiency, using an eBPF map.
24
25       The output summary will further help you identify reasons  why  threads
26       were blocking, and quantify the time from when they were blocked to wo‐
27       ken up.  This spans all types of blocking activity: disk  I/O,  network
28       I/O,  locks,  page  faults,  swapping,  sleeping,  involuntary  context
29       switches, etc.
30
31       This is complementary to CPU profiling (e.g., CPU flame  graphs)  which
32       shows the time spent on-CPU. This shows the time spent blocked off-CPU,
33       and the output, especially the -f format, can be used  to  generate  an
34       "off-wake time flame graph".
35
36       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
37
38

REQUIREMENTS

40       CONFIG_BPF and bcc.
41

OPTIONS

43       -h     Print usage message.
44
45       -f     Print output in folded stack format.
46
47       -p PID Trace  this process ID only (filtered in-kernel). Can be a comma
48              separated list of PIDS.
49
50       -t TID Trace this thread ID only (filtered in-kernel). Can be  a  comma
51              separated list of TIDS.
52
53       -u     Only trace user threads (no kernel threads).
54
55       -k     Only trace kernel threads (no user threads).
56
57       -U     Show stacks from user space only (no kernel space stacks).
58
59       -K     Show stacks from kernel space only (no user space stacks).
60
61       -d, --delimited
62              insert delimiter between kernel/user stacks
63
64       --stack-storage-size STACK_STORAGE_SIZE
65              Change  the number of unique stack traces that can be stored and
66              displayed.
67
68       duration
69              Duration to trace, in seconds.
70
71       -m MIN_BLOCK_TIME
72              The amount of time in microseconds over which  we  store  traces
73              (default 1)
74
75       -M MAX_BLOCK_TIME
76              The  amount  of time in microseconds under which we store traces
77              (default U64_MAX)
78
79       --state
80              Filter on this thread state bitmask (eg, 2 ==  TASK_UNINTERRUPT‐
81              IBLE).  See include/linux/sched.h for states.
82

EXAMPLES

84       Trace all thread blocking events, and summarize (in-kernel) by user and
85       kernel off-CPU stack trace, waker stack traces, task names,  and  total
86       blocked time:
87              # offwaketime
88
89       Trace for 5 seconds only:
90              # offwaketime 5
91
92       Trace  for  5 seconds, and emit output in folded stack format (suitable
93       for flame graphs), user-mode threads only:
94              # offwaketime -fu 5
95
96       Trace PID 185 only:
97              # offwaketime -p 185
98

OVERHEAD

100       This summarizes unique stack trace pairs in-kernel for efficiency,  al‐
101       lowing  it  to  trace  a  higher rate of events than methods that post-
102       process in user space.  The stack trace and time data is only copied to
103       user  space  once,  when  the output is printed. While these techniques
104       greatly lower overhead, scheduler events are  still  a  high  frequency
105       event,  as  they can exceed 1 million events per second, and so caution
106       should still be used. Test before production use.
107
108       If the overhead is still a problem, take a look at the  min  block  op‐
109       tion.   If  your aim is to chase down longer blocking events, then this
110       could be increased to filter shorter blocking events, further  lowering
111       overhead.
112

SOURCE

114       This is from bcc.
115
116              https://github.com/iovisor/bcc
117
118       Also  look  in  the bcc distribution for a companion _examples.txt file
119       containing example usage, output, and commentary for this tool.
120

OS

122       Linux
123

STABILITY

125       Unstable - in development.
126

AUTHOR

128       Brendan Gregg
129