1BPFTRACE(8) System Manager's Manual BPFTRACE(8)
2
6 bpftrace - the eBPF tracing language & frontend
7
9 bpftrace [OPTIONS] FILE
10 bpftrace [OPTIONS] -e 'program code'
11
13 bpftrace is a high-level tracing language for Linux enhanced Berkeley
14 Packet Filter (eBPF) available in recent Linux kernels (4.x).
15 bpftrace uses:
17 • LLVM as a backend to compile scripts to BPF-bytecode
19 • BCC for interacting with the Linux BPF system
21 As well as the existing Linux tracing capabilities:
25 ┌─────────┬─────────────┬──────────────┐
27 │ │ kernel │ userland │
28 ├─────────┼─────────────┼──────────────┤
29 │ static │ tracepoints │ USDT* probes │
30 ├─────────┼─────────────┼──────────────┤
31 │ dynamic │ kprobes │ uprobes │
32 └─────────┴─────────────┴──────────────┘
33 *USDT = user-level statically defined tracing
34 The bpftrace language is inspired by awk and C, and predecessor tracers
36 such as DTrace and SystemTap.
37 See EXAMPLES and ONELINERS if you are impatient.
39 See PROBE TYPES and BUILTINS (variables/functions) for the bpftrace
40 language elements.
41
43 -B MODE
44 The buffering mode to be used for output ('full', 'none').
45 -c CMD Helper to run CMD. Equivalent to manually running CMD and then
47 giving passing the PID to -p. This is useful to ensure you've
48 traced at least the duration CMD's execution.
49 -d Debug info dry run.
51 -dd Verbose Debug info dry run.
53 -e 'PROGRAM'
55 Execute PROGRAM.
56 -f FORMAT
58 The format to be used for output ('text'. 'json').
59 -h | --help
61 Show help message.
62 -I DIR Add the directory to the INCLUDE search path.
64 --include FILE
66 Add a #include file before preprocessing.
67 --info Print information about kernel BPF support.
69 -k Emit a warning when a bpf helper returns an error (except read
71 functions).
72 -kk Check all bpf helper functions.
74 -l [searchterm]
76 List probes.
77 -o FILE
79 Redirect bpftrace output to file.
80 -p PID Enable USDT probes on PID. Will terminate bpftrace on PID termi‐
82 nation. Note this is not a global PID filter on probes.
83 --unsafe
85 Enable unsafe builtin functions. By default, bpftrace runs in
86 safe mode. Safe mode ensure programs cannot modify system state.
87 Unsafe builtin functions are marked as such in BUILTINS (func‐
88 tions).
89 --usdt-file-activation
91 Activate usdt semaphores based on file path..
92 -v Verbose messages.
94 -V | --version
96 Print version of bpftrace.
97
99 bpftrace -l '*sleep*'
100 List probes containing "sleep".
101 bpftrace -e 'kprobe:do_nanosleep { printf("PID %d sleeping\n", pid); }'
103 Trace processes calling sleep.
104 bpftrace -c 'sleep 5' -e 'kprobe:do_nanosleep { printf("PID %d sleep‐
106 ing\n", pid); }'
107 run "sleep 5" in a new process and then trace processes calling
108 sleep.
109 bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm]=count(); }'
111 Count syscalls by process name.
112
114 For brevity, just the the actual BPF code is shown below.
115 Usage: bpftrace -e 'bpf-code'
116 New processes with arguments:
118 tracepoint:syscalls:sys_enter_execve { join(args->argv); }
119 Files opened by process:
121 tracepoint:syscalls:sys_enter_open { printf("%s %s\n", comm,
122 str(args->filename)); }
123 Syscall count by program:
125 tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }
126 Syscall count by syscall:
128 tracepoint:syscalls:sys_enter_* { @[probe] = count(); }
129 Syscall count by process:
131 tracepoint:raw_syscalls:sys_enter { @[pid, comm] = count(); }
132 Read bytes by process:
134 tracepoint:syscalls:sys_exit_read /args->ret/ { @[comm] =
135 sum(args->ret); }
136 Read size distribution by process:
138 tracepoint:syscalls:sys_exit_read { @[comm] = hist(args->ret); }
139 Disk size by process:
141 tracepoint:block:block_rq_issue { printf("%d %s %d\n", pid,
142 comm, args->bytes); }
143 Pages paged in by process:
145 software:major-faults:1 { @[comm] = count(); }
146 Page faults by process:
148 software:faults:1 { @[comm] = count(); }
149 Profile user-level stacks at 99 Hertz, for PID 189:
151 profile:hz:99 /pid == 189/ { @[ustack] = count(); }
152
154 KPROBES
155 Attach a bpftrace script to a kernel function, to be executed when that
156 function is called:
157 kprobe:vfs_read { ... }
159 UPROBES
161 Attach script to a userland function:
162 uprobe:/bin/bash:readline { ... }
164 TRACEPOINTS
166 Attach script to a statically defined tracepoint in the kernel:
167 tracepoint:sched:sched_switch { ... }
169 Tracepoints are guaranteed to be stable between kernel versions, unlike
171 kprobes.
172 SOFTWARE
174 Attach script to kernel software events, executing once every provided
175 count or use a default:
176 software:faults:100 software:faults:
178 HARDWARE
180 Attach script to hardware events (PMCs), executing once every provided
181 count or use a default:
182 hardware:cache-references:1000000 hardware:cache-references:
184 PROFILE
186 Run the script on all CPUs at specified time intervals:
187 profile:hz:99 { ... }
189 profile:s:1 { ... }
191 profile:ms:20 { ... }
193 profile:us:1500 { ... }
195 INTERVAL
197 Run the script once per interval, for printing interval output:
198 interval:s:1 { ... }
200 interval:ms:20 { ... }
202 MULTIPLE ATTACHMENT POINTS
204 A single probe can be attached to multiple events:
205 kprobe:vfs_read,kprobe:vfs_write { ... }
207 WILDCARDS
209 Some probe types allow wildcards to be used when attaching a probe:
210 kprobe:vfs_* { ... }
212 PREDICATES
214 Define conditions for which a probe should be executed:
215 kprobe:sys_open / uid == 0 / { ... }
217
219 The following variables and functions are available for use in bpftrace
220 scripts:
221 VARIABLES
223 pid Process ID (kernel tgid)
224 tid Thread ID (kernel pid)
226 cgroup Cgroup ID of the current process
228 uid User ID
230 gid Group ID
232 nsecs Nanosecond timestamp
234 elapsed
236 Nanoseconds since bpftrace initialization
237 cpu Processor ID
239 comm Process name
241 kstack Kernel stack trace
243 ustack User stack trace
245 arg0, arg1, ... etc.
247 Arguments to the function being traced
248 retval Return value from function being traced
250 func Name of the function currently being traced
252 probe Full name of the probe
254 curtask
256 Current task_struct as a u64.
257 rand Random number of type u32.
259 FUNCTIONS
261 hist(int n)
262 Produce a log2 histogram of values of n
263 lhist(int n, int min, int max, int step)
265 Produce a linear histogram of values of n
266 count()
268 Count the number of times this function is called
269 sum(int n)
271 Sum this value
272 min(int n)
274 Record the minimum value seen
275 max(int n)
277 Record the maximum value seen
278 avg(int n)
280 Average this value
281 stats(int n)
283 Return the count, average, and total for this value
284 delete(@x)
286 Delete the map element passed in as an argument
287 str(char *s)
289 Returns the string pointed to by s
290 printf(char *fmt, ...)
292 Print formatted to stdout
293 print(@x[, int top [, int div]])
295 Print a map, with optional top entry count and divisor
296 clear(@x)
298 Delete all key/values from a map
299 ksym(void *p)
301 Resolve kernel address
302 usym(void *p)
304 Resolve user space address
305 kaddr(char *name)
307 Resolve kernel symbol name
308 uaddr(char *name)
310 Resolve user space symbol name
311 reg(char *name)
313 Returns the value stored in the named register
314 join(char *arr[])
316 Prints the string array
317 time(char *fmt)
319 Print the current time
320 cat(char *filename)
322 Print file content
323 ntop([int af, ]int|char[4|16] addr)
325 Convert IP address data to text
326 system(char *fmt) (unsafe)
328 Execute shell command
329 exit() Quit bpftrace
331 kstack([StackMode mode, ][int level])
333 Kernel stack trace
334 ustack([StackMode mode, ][int level])
336 User stack trace
337
339 The official documentation can be found here:
340 https://github.com/iovisor/bpftrace/blob/master/docs
341
343 The first official talk by Alastair on bpftrace happened at the Tracing
344 Summit in Edinburgh, Oct 25th 2018.
345
347 Created by Alastair Robertson.
348 Manpage by Stephan Schuberth.
349
351 man -k bcc, after having installed the bpfcc-tools package under
352 Ubuntu.
353
355 Prior to contributing new tools, read the official checklist at:
356 https://github.com/iovisor/bpftrace/blob/master/CONTRIBUTING-TOOLS.md
357 October 2018 BPFTRACE(8)