1BPFTRACE(8) System Manager's Manual BPFTRACE(8)
2
6 bpftrace - the eBPF tracing language & frontend
7
9 bpftrace [OPTIONS] FILE
10 bpftrace [OPTIONS] -e ´program code´
11
13 bpftrace is a high-level tracing language for Linux enhanced Berkeley
14 Packet Filter (eBPF) available in recent Linux kernels (4.x).
15 bpftrace uses:
17 · LLVM as a backend to compile scripts to BPF-bytecode
19 · BCC for interacting with the Linux BPF system
21 As well as the existing Linux tracing capabilities:
25 ┌─────────┬─────────────┬──────────────┐
27 │ │ kernel │ userland │
28 ├─────────┼─────────────┼──────────────┤
29 │ static │ tracepoints │ USDT* probes │
30 ├─────────┼─────────────┼──────────────┤
31 │ dynamic │ kprobes │ uprobes │
32 └─────────┴─────────────┴──────────────┘
33 *USDT = user-level statically defined tracing
34 The bpftrace language is inspired by awk and C, and predecessor tracers
36 such as DTrace and SystemTap.
37 See EXAMPLES and ONELINERS if you are impatient.
39 See PROBE TYPES and BUILTINS (variables/functions) for the bpftrace
40 language elements.
41
43 -l [searchterm]
44 List probes.
45 -e ´PROGRAM´
47 Execute PROGRAM.
48 -p PID Enable USDT probes on PID. Will terminate bpftrace on PID termi‐
50 nation. Note this is not a global PID filter on probes.
51 -c CMD Helper to run CMD. Equivalent to manually running CMD and then
53 giving passing the PID to -p. This is useful to ensure you've
54 traced at least the duration CMD's execution.
55 --unsafe
57 Enable unsafe builtin functions. By default, bpftrace runs in
58 safe mode. Safe mode ensure programs cannot modify system state.
59 Unsafe builtin functions are marked as such in BUILTINS (func‐
60 tions).
61 -v Verbose messages.
63 -d Debug info on dry run.
65 -dd Verbose debug info on dry run.
67
69 bpftrace -l ´*sleep*´
70 List probes containing "sleep".
71 bpftrace -e ´kprobe:do_nanosleep { printf("PID %d sleeping\n", pid); }´
73 Trace processes calling sleep.
74 bpftrace -c ´sleep 5´ -e ´kprobe:do_nanosleep { printf("PID %d sleep‐
76 ing\n", pid); }´
77 run "sleep 5" in a new process and then trace processes calling
78 sleep.
79 bpftrace -e ´tracepoint:raw_syscalls:sys_enter { @[comm]=count(); }´
81 Count syscalls by process name.
82
84 For brevity, just the the actual BPF code is shown below.
85 Usage: bpftrace -e ´bpf-code´
86 New processes with arguments:
88 tracepoint:syscalls:sys_enter_execve { join(args->argv); }
89 Files opened by process:
91 tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm,
92 str(args->filename)); }
93 Syscall count by program:
95 tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }
96 Syscall count by syscall:
98 tracepoint:syscalls:sys_enter_* { @[probe] = count(); }
99 Syscall count by process:
101 tracepoint:raw_syscalls:sys_enter { @[pid, comm] = count(); }
102 Read bytes by process:
104 tracepoint:syscalls:sys_exit_read /args->ret/ { @[comm] =
105 sum(args->ret); }
106 Read size distribution by process:
108 tracepoint:syscalls:sys_exit_read { @[comm] = hist(args->ret); }
109 Disk size by process:
111 tracepoint:block:block_rq_issue { printf("%d %s %d\n", pid,
112 comm, args->bytes); }
113 Pages paged in by process:
115 software:major-faults:1 { @[comm] = count(); }
116 Page faults by process:
118 software:faults:1 { @[comm] = count(); }
119 Profile user-level stacks at 99 Hertz, for PID 189:
121 profile:hz:99 /pid == 189/ { @[ustack] = count(); }
122
124 KPROBES
125 Attach a bpftrace script to a kernel function, to be executed when that
126 function is called:
127 kprobe:vfs_read { ... }
129 UPROBES
131 Attach script to a userland function:
132 uprobe:/bin/bash:readline { ... }
134 TRACEPOINTS
136 Attach script to a statically defined tracepoint in the kernel:
137 tracepoint:sched:sched_switch { ... }
139 Tracepoints are guaranteed to be stable between kernel versions, unlike
141 kprobes.
142 SOFTWARE
144 Attach script to kernel software events, executing once every provided
145 count or use a default:
146 software:faults:100 software:faults:
148 HARDWARE
150 Attach script to hardware events (PMCs), executing once every provided
151 count or use a default:
152 hardware:cache-references:1000000 hardware:cache-references:
154 PROFILE
156 Run the script on all CPUs at specified time intervals:
157 profile:hz:99 { ... }
159 profile:s:1 { ... }
161 profile:ms:20 { ... }
163 profile:us:1500 { ... }
165 INTERVAL
167 Run the script once per interval, for printing interval output:
168 interval:s:1 { ... }
170 interval:ms:20 { ... }
172 MULTIPLE ATTACHMENT POINTS
174 A single probe can be attached to multiple events:
175 kprobe:vfs_read,kprobe:vfs_write { ... }
177 WILDCARDS
179 Some probe types allow wildcards to be used when attaching a probe:
180 kprobe:vfs_* { ... }
182 PREDICATES
184 Define conditions for which a probe should be executed:
185 kprobe:sys_open / uid == 0 / { ... }
187
189 The following variables and functions are available for use in bpftrace
190 scripts:
191 VARIABLES
193 pid Process ID (kernel tgid)
194 tid Thread ID (kernel pid)
196 cgroup Cgroup ID of the current process
198 uid User ID
200 gid Group ID
202 nsecs Nanosecond timestamp
204 cpu Processor ID
206 comm Process name
208 kstack Kernel stack trace
210 ustack User stack trace
212 arg0, arg1, ... etc.
214 Arguments to the function being traced
215 retval Return value from function being traced
217 func Name of the function currently being traced
219 probe Full name of the probe
221 curtask
223 Current task_struct as a u64.
224 rand Random number of type u32.
226 FUNCTIONS
228 hist(int n)
229 Produce a log2 histogram of values of n
230 lhist(int n, int min, int max, int step)
232 Produce a linear histogram of values of n
233 count()
235 Count the number of times this function is called
236 sum(int n)
238 Sum this value
239 min(int n)
241 Record the minimum value seen
242 max(int n)
244 Record the maximum value seen
245 avg(int n)
247 Average this value
248 stats(int n)
250 Return the count, average, and total for this value
251 delete(@x)
253 Delete the map element passed in as an argument
254 str(char *s)
256 Returns the string pointed to by s
257 printf(char *fmt, ...)
259 Print formatted to stdout
260 print(@x[, int top [, int div]])
262 Print a map, with optional top entry count and divisor
263 clear(@x)
265 Delete all key/values from a map
266 sym(void *p)
268 Resolve kernel address
269 usym(void *p)
271 Resolve user space address
272 kaddr(char *name)
274 Resolve kernel symbol name
275 uaddr(char *name)
277 Resolve user space symbol name
278 reg(char *name)
280 Returns the value stored in the named register
281 join(char *arr[])
283 Prints the string array
284 time(char *fmt)
286 Print the current time
287 cat(char *filename)
289 Print file content
290 ntop([int af, ]int|char[4|16] addr)
292 Convert IP address data to text
293 system(char *fmt) (unsafe)
295 Execute shell command
296 exit() Quit bpftrace
298 kstack([StackMode mode, ][int level])
300 Kernel stack trace
301 ustack([StackMode mode, ][int level])
303 User stack trace
304
306 The official documentation can be found here:
307 https://github.com/iovisor/bpftrace/blob/master/docs
308
310 The first official talk by Alastair on bpftrace happened at the Tracing
311 Summit in Edinburgh, Oct 25th 2018.
312
314 Created by Alastair Robertson.
315 Manpage by Stephan Schuberth.
316
318 man -k bcc, after having installed the bpfcc-tools package under
319 Ubuntu.
320
322 Prior to contributing new tools, read the official checklist at:
323 https://github.com/iovisor/bpftrace/blob/master/CONTRIBUTING-TOOLS.md
324 October 2018 BPFTRACE(8)