bcc-offcputime(8)

1offcputime(8)               System Manager's Manual              offcputime(8)
2
3
4

NAME

6       offcputime  -  Summarize off-CPU time by kernel stack trace. Uses Linux
7       eBPF/bcc.
8

SYNOPSIS

10       offcputime [-h] [-p PID | -t TID |  -u  |  -k]  [-U  |  -K]  [-d]  [-f]
11       [--stack-storage-size   STACK_STORAGE_SIZE]   [-m  MIN_BLOCK_TIME]  [-M
12       MAX_BLOCK_TIME] [--state STATE] [duration]
13

DESCRIPTION

15       This program shows stack traces and task names that  were  blocked  and
16       "off-CPU", and the total duration they were not running: their "off-CPU
17       time".  It works by tracing when threads block and when they return  to
18       CPU,  measuring  both  the time they were off-CPU and the blocked stack
19       trace and the task name.  This data is summarized in the  kernel  using
20       an  eBPF map, and by summing the off-CPU time by unique stack trace and
21       task name.
22
23       The output summary will help you  identify  reasons  why  threads  were
24       blocking, and quantify the time they were off-CPU. This spans all types
25       of blocking activity: disk I/O, network I/O, locks, page faults, invol‐
26       untary context switches, etc.
27
28       This  is  complementary to CPU profiling (e.g., CPU flame graphs) which
29       shows the time spent on-CPU. This shows the time spent off-CPU, and the
30       output,  especially  the -f format, can be used to generate an "off-CPU
31       time flame graph".
32
33       See http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
34
35       This tool only works on Linux 4.6+. It uses the  new  `BPF_STACK_TRACE`
36       table  APIs  to generate the in-kernel stack traces.  For kernels older
37       than 4.6, see the version under tools/old.
38
39       Note: this tool only traces off-CPU times that began  and  ended  while
40       tracing.
41

REQUIREMENTS

43       CONFIG_BPF and bcc.
44

OPTIONS

46       -h     Print usage message.
47
48       -f     Print output in folded stack format.
49
50       -p PID Trace this process ID only (filtered in-kernel).
51
52       -t TID Trace this thread ID only (filtered in-kernel).
53
54       -u     Only trace user threads (no kernel threads).
55
56       -k     Only trace kernel threads (no user threads).
57
58       -U     Show stacks from user space only (no kernel space stacks).
59
60       -K     Show stacks from kernel space only (no user space stacks).
61
62       -d     Insert delimiter between kernel/user stacks.
63
64       -f     Output folded format.
65
66       --stack-storage-size STACK_STORAGE_SIZE
67              Change  the number of unique stack traces that can be stored and
68              displayed.
69
70       -m MIN_BLOCK_TIME
71              The minimum time in microseconds  over  which  we  store  traces
72              (default 1)
73
74       -M MAX_BLOCK_TIME
75              The  maximum  time  in  microseconds under which we store traces
76              (default U64_MAX)
77
78       --state
79              Filter on this thread state bitmask (eg, 2 ==  TASK_UNINTERRUPT‐
80              IBLE).  See include/linux/sched.h for states.
81
82       duration
83              Duration to trace, in seconds.
84

EXAMPLES

86       Trace  all  thread blocking events, and summarize (in-kernel) by kernel
87       stack trace and total off-CPU time:
88              # offcputime
89
90       Trace for 5 seconds only:
91              # offcputime 5
92
93       Trace for 5 seconds, and emit output in folded stack  format  (suitable
94       for flame graphs):
95              # offcputime -f 5
96
97       Trace PID 185 only:
98              # offcputime -p 185
99

OVERHEAD

101       This  summarizes unique stack traces in-kernel for efficiency, allowing
102       it to trace a higher rate of events than methods that  post-process  in
103       user  space. The stack trace and time data is only copied to user space
104       once, when the output is printed. While these techniques greatly  lower
105       overhead,  scheduler  events  are still a high frequency event, as they
106       can exceed 1 million events per second, and so caution should still  be
107       used. Test before production use.
108
109       If the overhead is still a problem, take a look at the MINBLOCK_US tun‐
110       able in the code. If your aim is to chase down longer blocking  events,
111       then this could be increased to filter shorter blocking events, further
112       lowering overhead.
113

SOURCE

115       This is from bcc.
116
117              https://github.com/iovisor/bcc
118
119       Also look in the bcc distribution for a  companion  _examples.txt  file
120       containing example usage, output, and commentary for this tool.
121

OS

123       Linux
124

STABILITY

126       Unstable - in development.
127

AUTHOR

129       Brendan Gregg
130