1BPFTRACE(8) System Manager's Manual BPFTRACE(8)
2
6 BPFtrace - the eBPF tracing language & frontend
7
9 bpftrace [OPTIONS] FILE
10 bpftrace [OPTIONS] -e ´program code´
11
13 BPFtrace is a high-level tracing language for Linux enhanced Berkeley
14 Packet Filter (eBPF) available in recent Linux kernels (4.x).
15 BPFtrace uses:
17 · LLVM as a backend to compile scripts to BPF-bytecode
19 · BCC for interacting with the Linux BPF system
21 As well as the existing Linux tracing capabilities:
25 ┌─────────┬─────────────┬──────────────┐
27 │ │ kernel │ userland │
28 ├─────────┼─────────────┼──────────────┤
29 │ static │ tracepoints │ USDT* probes │
30 ├─────────┼─────────────┼──────────────┤
31 │ dynamic │ kprobes │ uprobes │
32 └─────────┴─────────────┴──────────────┘
33 *USDT = user-level statically defined tracing
34 The BPFtrace language is inspired by awk and C, and predecessor tracers
36 such as DTrace and SystemTap.
37 See EXAMPLES and ONELINERS if you are impatient.
39 See PROBE TYPES and BUILTINS (variables/functions) for the bpftrace
40 language elements.
41
43 -l [searchterm]
44 List probes.
45 -e ´PROGRAM´
47 Execute PROGRAM.
48 -p PID Enable USDT probes on PID. Will terminate bpftrace on PID termi‐
50 nation. Note this is not a global PID filter on probes.
51 -c CMD Helper to run CMD. Equivalent to manually running CMD and then
53 giving passing the PID to -p. This is useful to ensure you've
54 traced at least the duration CMD's execution.
55 -v Verbose messages.
57 -d Debug info on dry run.
59 -dd Verbose debug info on dry run.
61
63 bpftrace -l ´*sleep*´
64 List probes containing "sleep".
65 bpftrace -e ´kprobe:do_nanosleep { printf("PID %d sleeping\n", pid); }´
67 Trace processes calling sleep.
68 bpftrace -c ´sleep 5´ -e ´kprobe:do_nanosleep { printf("PID %d sleep‐
70 ing\n", pid); }´
71 run "sleep 5" in a new process and then trace processes calling
72 sleep.
73 bpftrace -e ´tracepoint:raw_syscalls:sys_enter { @[comm]=count(); }´
75 Count syscalls by process name.
76
78 For brevity, just the the actual BPF code is shown below.
79 Usage: bpftrace -e ´bpf-code´
80 New processes with arguments:
82 tracepoint:syscalls:sys_enter_execve { join(args->argv); }
83 Files opened by process:
85 tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm,
86 str(args->filename)); }
87 Syscall count by program:
89 tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }
90 Syscall count by syscall:
92 tracepoint:syscalls:sys_enter_* { @[probe] = count(); }
93 Syscall count by process:
95 tracepoint:raw_syscalls:sys_enter { @[pid, comm] = count(); }
96 Read bytes by process:
98 tracepoint:syscalls:sys_exit_read /args->ret/ { @[comm] =
99 sum(args->ret); }
100 Read size distribution by process:
102 tracepoint:syscalls:sys_exit_read { @[comm] = hist(args->ret); }
103 Disk size by process:
105 tracepoint:block:block_rq_issue { printf("%d %s %d\n", pid,
106 comm, args->bytes); }
107 Pages paged in by process:
109 software:major-faults:1 { @[comm] = count(); }
110 Page faults by process:
112 software:faults:1 { @[comm] = count(); }
113 Profile user-level stacks at 99 Hertz, for PID 189:
115 profile:hz:99 /pid == 189/ { @[ustack] = count(); }
116
118 KPROBES
119 Attach a BPFtrace script to a kernel function, to be executed when that
120 function is called:
121 kprobe:vfs_read { ... }
123 UPROBES
125 Attach script to a userland function:
126 uprobe:/bin/bash:readline { ... }
128 TRACEPOINTS
130 Attach script to a statically defined tracepoint in the kernel:
131 tracepoint:sched:sched_switch { ... }
133 Tracepoints are guaranteed to be stable between kernel versions, unlike
135 kprobes.
136 SOFTWARE
138 Attach script to kernel software events, executing once every provided
139 count or use a default:
140 software:faults:100 software:faults:
142 HARDWARE
144 Attach script to hardware events (PMCs), executing once every provided
145 count or use a default:
146 hardware:cache-references:1000000 hardware:cache-references:
148 PROFILE
150 Run the script on all CPUs at specified time intervals:
151 profile:hz:99 { ... }
153 profile:s:1 { ... }
155 profile:ms:20 { ... }
157 profile:us:1500 { ... }
159 INTERVAL
161 Run the script once per interval, for printing interval output:
162 interval:s:1 { ... }
164 interval:ms:20 { ... }
166 MULTIPLE ATTACHMENT POINTS
168 A single probe can be attached to multiple events:
169 kprobe:vfs_read,kprobe:vfs_write { ... }
171 WILDCARDS
173 Some probe types allow wildcards to be used when attaching a probe:
174 kprobe:vfs_* { ... }
176 PREDICATES
178 Define conditions for which a probe should be executed:
179 kprobe:sys_open / uid == 0 / { ... }
181
183 The following variables and functions are available for use in bpftrace
184 scripts:
185 VARIABLES
187 pid Process ID (kernel tgid)
188 tid Thread ID (kernel pid)
190 cgroup Cgroup ID of the current process
192 uid User ID
194 gid Group ID
196 nsecs Nanosecond timestamp
198 cpu Processor ID
200 comm Process name
202 kstack Kernel stack trace
204 ustack User stack trace
206 arg0, arg1, ... etc.
208 Arguments to the function being traced
209 retval Return value from function being traced
211 func Name of the function currently being traced
213 probe Full name of the probe
215 curtask
217 Current task_struct as a u64.
218 rand Random number of type u32.
220 FUNCTIONS
222 hist(int n)
223 Produce a log2 histogram of values of n
224 lhist(int n, int min, int max, int step)
226 Produce a linear histogram of values of n
227 count()
229 Count the number of times this function is called
230 sum(int n)
232 Sum this value
233 min(int n)
235 Record the minimum value seen
236 max(int n)
238 Record the maximum value seen
239 avg(int n)
241 Average this value
242 stats(int n)
244 Return the count, average, and total for this value
245 delete(@x)
247 Delete the map element passed in as an argument
248 str(char *s)
250 Returns the string pointed to by s
251 printf(char *fmt, ...)
253 Print formatted to stdout
254 print(@x[, int top [, int div]])
256 Print a map, with optional top entry count and divisor
257 clear(@x)
259 Delete all key/values from a map
260 sym(void *p)
262 Resolve kernel address
263 usym(void *p)
265 Resolve user space address
266 kaddr(char *name)
268 Resolve kernel symbol name
269 uaddr(char *name)
271 Resolve user space symbol name
272 reg(char *name)
274 Returns the value stored in the named register
275 join(char *arr[])
277 Prints the string array
278 time(char *fmt)
280 Print the current time
281 system(char *fmt)
283 Execute shell command
284 exit() Quit bpftrace
286 kstack([StackMode mode, ][int level])
288 Kernel stack trace
289 ustack([StackMode mode, ][int level])
291 User stack trace
292
294 The official documentation can be found here:
295 https://github.com/iovisor/bpftrace/blob/master/docs
296
298 The first official talk by Alastair on bpftrace happened at the Tracing
299 Summit in Edinburgh, Oct 25th 2018.
300
302 Created by Alastair Robertson.
303 Manpage by Stephan Schuberth.
304
306 man -k bcc, after having installed the bpfcc-tools package under
307 Ubuntu.
308
310 Prior to contributing new tools, read the official checklist at:
311 https://github.com/iovisor/bpftrace/blob/master/CONTRIBUTING-TOOLS.md
312 October 2018 BPFTRACE(8)