1BPFTRACE(8) System Manager's Manual BPFTRACE(8)
2
6 bpftrace - the eBPF tracing language & frontend
7
9 bpftrace [OPTIONS] FILE
10 bpftrace [OPTIONS] -e 'program code'
11
13 bpftrace is a high-level tracing language for Linux enhanced Berkeley
14 Packet Filter (eBPF) available in recent Linux kernels (4.x).
15 bpftrace uses:
17 · LLVM as a backend to compile scripts to BPF-bytecode
19 · BCC for interacting with the Linux BPF system
21 As well as the existing Linux tracing capabilities:
25 ┌─────────┬─────────────┬──────────────┐
27 │ │ kernel │ userland │
28 ├─────────┼─────────────┼──────────────┤
29 │ static │ tracepoints │ USDT* probes │
30 ├─────────┼─────────────┼──────────────┤
31 │ dynamic │ kprobes │ uprobes │
32 └─────────┴─────────────┴──────────────┘
33 *USDT = user-level statically defined tracing
34 The bpftrace language is inspired by awk and C, and predecessor tracers
36 such as DTrace and SystemTap.
37 See EXAMPLES and ONELINERS if you are impatient.
39 See PROBE TYPES and BUILTINS (variables/functions) for the bpftrace
40 language elements.
41
43 -l [searchterm]
44 List probes.
45 -e 'PROGRAM'
47 Execute PROGRAM.
48 -p PID Enable USDT probes on PID. Will terminate bpftrace on PID termi‐
50 nation. Note this is not a global PID filter on probes.
51 -c CMD Helper to run CMD. Equivalent to manually running CMD and then
53 giving passing the PID to -p. This is useful to ensure you've
54 traced at least the duration CMD's execution.
55 --unsafe
57 Enable unsafe builtin functions. By default, bpftrace runs in
58 safe mode. Safe mode ensure programs cannot modify system state.
59 Unsafe builtin functions are marked as such in BUILTINS (func‐
60 tions).
61 --btf Force BTF data processing if it's available. By default it's
63 enabled only if the user does not specify any types/includes.
64 -v Verbose messages.
66 -d Debug info on dry run.
68 -dd Verbose debug info on dry run.
70
72 bpftrace -l '*sleep*'
73 List probes containing "sleep".
74 bpftrace -e 'kprobe:do_nanosleep { printf("PID %d sleeping\n", pid); }'
76 Trace processes calling sleep.
77 bpftrace -c 'sleep 5' -e 'kprobe:do_nanosleep { printf("PID %d sleep‐
79 ing\n", pid); }'
80 run "sleep 5" in a new process and then trace processes calling
81 sleep.
82 bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm]=count(); }'
84 Count syscalls by process name.
85
87 For brevity, just the the actual BPF code is shown below.
88 Usage: bpftrace -e 'bpf-code'
89 New processes with arguments:
91 tracepoint:syscalls:sys_enter_execve { join(args->argv); }
92 Files opened by process:
94 tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm,
95 str(args->filename)); }
96 Syscall count by program:
98 tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }
99 Syscall count by syscall:
101 tracepoint:syscalls:sys_enter_* { @[probe] = count(); }
102 Syscall count by process:
104 tracepoint:raw_syscalls:sys_enter { @[pid, comm] = count(); }
105 Read bytes by process:
107 tracepoint:syscalls:sys_exit_read /args->ret/ { @[comm] =
108 sum(args->ret); }
109 Read size distribution by process:
111 tracepoint:syscalls:sys_exit_read { @[comm] = hist(args->ret); }
112 Disk size by process:
114 tracepoint:block:block_rq_issue { printf("%d %s %d\n", pid,
115 comm, args->bytes); }
116 Pages paged in by process:
118 software:major-faults:1 { @[comm] = count(); }
119 Page faults by process:
121 software:faults:1 { @[comm] = count(); }
122 Profile user-level stacks at 99 Hertz, for PID 189:
124 profile:hz:99 /pid == 189/ { @[ustack] = count(); }
125
127 KPROBES
128 Attach a bpftrace script to a kernel function, to be executed when that
129 function is called:
130 kprobe:vfs_read { ... }
132 UPROBES
134 Attach script to a userland function:
135 uprobe:/bin/bash:readline { ... }
137 TRACEPOINTS
139 Attach script to a statically defined tracepoint in the kernel:
140 tracepoint:sched:sched_switch { ... }
142 Tracepoints are guaranteed to be stable between kernel versions, unlike
144 kprobes.
145 SOFTWARE
147 Attach script to kernel software events, executing once every provided
148 count or use a default:
149 software:faults:100 software:faults:
151 HARDWARE
153 Attach script to hardware events (PMCs), executing once every provided
154 count or use a default:
155 hardware:cache-references:1000000 hardware:cache-references:
157 PROFILE
159 Run the script on all CPUs at specified time intervals:
160 profile:hz:99 { ... }
162 profile:s:1 { ... }
164 profile:ms:20 { ... }
166 profile:us:1500 { ... }
168 INTERVAL
170 Run the script once per interval, for printing interval output:
171 interval:s:1 { ... }
173 interval:ms:20 { ... }
175 MULTIPLE ATTACHMENT POINTS
177 A single probe can be attached to multiple events:
178 kprobe:vfs_read,kprobe:vfs_write { ... }
180 WILDCARDS
182 Some probe types allow wildcards to be used when attaching a probe:
183 kprobe:vfs_* { ... }
185 PREDICATES
187 Define conditions for which a probe should be executed:
188 kprobe:sys_open / uid == 0 / { ... }
190
192 The following variables and functions are available for use in bpftrace
193 scripts:
194 VARIABLES
196 pid Process ID (kernel tgid)
197 tid Thread ID (kernel pid)
199 cgroup Cgroup ID of the current process
201 uid User ID
203 gid Group ID
205 nsecs Nanosecond timestamp
207 cpu Processor ID
209 comm Process name
211 kstack Kernel stack trace
213 ustack User stack trace
215 arg0, arg1, ... etc.
217 Arguments to the function being traced
218 retval Return value from function being traced
220 func Name of the function currently being traced
222 probe Full name of the probe
224 curtask
226 Current task_struct as a u64.
227 rand Random number of type u32.
229 FUNCTIONS
231 hist(int n)
232 Produce a log2 histogram of values of n
233 lhist(int n, int min, int max, int step)
235 Produce a linear histogram of values of n
236 count()
238 Count the number of times this function is called
239 sum(int n)
241 Sum this value
242 min(int n)
244 Record the minimum value seen
245 max(int n)
247 Record the maximum value seen
248 avg(int n)
250 Average this value
251 stats(int n)
253 Return the count, average, and total for this value
254 delete(@x)
256 Delete the map element passed in as an argument
257 str(char *s)
259 Returns the string pointed to by s
260 printf(char *fmt, ...)
262 Print formatted to stdout
263 print(@x[, int top [, int div]])
265 Print a map, with optional top entry count and divisor
266 clear(@x)
268 Delete all key/values from a map
269 ksym(void *p)
271 Resolve kernel address
272 usym(void *p)
274 Resolve user space address
275 kaddr(char *name)
277 Resolve kernel symbol name
278 uaddr(char *name)
280 Resolve user space symbol name
281 reg(char *name)
283 Returns the value stored in the named register
284 join(char *arr[])
286 Prints the string array
287 time(char *fmt)
289 Print the current time
290 cat(char *filename)
292 Print file content
293 ntop([int af, ]int|char[4|16] addr)
295 Convert IP address data to text
296 system(char *fmt) (unsafe)
298 Execute shell command
299 exit() Quit bpftrace
301 kstack([StackMode mode, ][int level])
303 Kernel stack trace
304 ustack([StackMode mode, ][int level])
306 User stack trace
307
309 The official documentation can be found here:
310 https://github.com/iovisor/bpftrace/blob/master/docs
311
313 The first official talk by Alastair on bpftrace happened at the Tracing
314 Summit in Edinburgh, Oct 25th 2018.
315
317 Created by Alastair Robertson.
318 Manpage by Stephan Schuberth.
319
321 man -k bcc, after having installed the bpfcc-tools package under
322 Ubuntu.
323
325 Prior to contributing new tools, read the official checklist at:
326 https://github.com/iovisor/bpftrace/blob/master/CONTRIBUTING-TOOLS.md
327 October 2018 BPFTRACE(8)