1PERF-PROBE(1) perf Manual PERF-PROBE(1)
2
6 perf-probe - Define new dynamic tracepoints
7
9 perf probe [options] --add=PROBE [...]
10 or
11 perf probe [options] PROBE
12 or
13 perf probe [options] --del=[GROUP:]EVENT [...]
14 or
15 perf probe --list[=[GROUP:]EVENT]
16 or
17 perf probe [options] --line=LINE
18 or
19 perf probe [options] --vars=PROBEPOINT
20 or
21 perf probe [options] --funcs
22 or
23 perf probe [options] --definition=PROBE [...]
24
26 This command defines dynamic tracepoint events, by symbol and registers
27 without debuginfo, or by C expressions (C line numbers, C function
28 names, and C local variables) with debuginfo.
29
31 -k, --vmlinux=PATH
32 Specify vmlinux path which has debuginfo (Dwarf binary). Only when
33 using this with --definition, you can give an offline vmlinux file.
34 -m, --module=MODNAME|PATH
36 Specify module name in which perf-probe searches probe points or
37 lines. If a path of module file is passed, perf-probe treat it as
38 an offline module (this means you can add a probe on a module which
39 has not been loaded yet).
40 -s, --source=PATH
42 Specify path to kernel source.
43 -v, --verbose
45 Be more verbose (show parsed arguments, etc). Can not use with -q.
46 -q, --quiet
48 Do not show any warnings or messages. Can not use with -v.
49 -a, --add=
51 Define a probe event (see PROBE SYNTAX for detail).
52 -d, --del=
54 Delete probe events. This accepts glob wildcards(*, ?) and
55 character classes(e.g. [a-z], [!A-Z]).
56 -l, --list[=[GROUP:]EVENT]
58 List up current probe events. This can also accept filtering
59 patterns of event names. When this is used with --cache, perf shows
60 all cached probes instead of the live probes.
61 -L, --line=
63 Show source code lines which can be probed. This needs an argument
64 which specifies a range of the source code. (see LINE SYNTAX for
65 detail)
66 -V, --vars=
68 Show available local variables at given probe point. The argument
69 syntax is same as PROBE SYNTAX, but NO ARGs.
70 --externs
72 (Only for --vars) Show external defined variables in addition to
73 local variables.
74 --no-inlines
76 (Only for --add) Search only for non-inlined functions. The
77 functions which do not have instances are ignored.
78 -F, --funcs[=FILTER]
80 Show available functions in given module or kernel. With -x/--exec,
81 can also list functions in a user space executable / shared
82 library. This also can accept a FILTER rule argument.
83 -D, --definition=
85 Show trace-event definition converted from given probe-event
86 instead of write it into tracing/[k,u]probe_events.
87 --filter=FILTER
89 (Only for --vars and --funcs) Set filter. FILTER is a combination
90 of glob pattern, see FILTER PATTERN for detail. Default FILTER is
91 "!k???tab_* & !crc_*" for --vars, and "!_*" for --funcs. If several
92 filters are specified, only the last filter is used.
93 -f, --force
95 Forcibly add events with existing name.
96 -n, --dry-run
98 Dry run. With this option, --add and --del doesn’t execute actual
99 adding and removal operations.
100 --cache
102 (With --add) Cache the probes. Any events which successfully added
103 are also stored in the cache file. (With --list) Show cached
104 probes. (With --del) Remove cached probes.
105 --max-probes=NUM
107 Set the maximum number of probe points for an event. Default is
108 128.
109 --target-ns=PID: Obtain mount namespace information from the target
111 pid. This is used when creating a uprobe for a process that resides in
112 a different mount namespace from the perf(1) utility.
113 -x, --exec=PATH
115 Specify path to the executable or shared library file for user
116 space tracing. Can also be used with --funcs option.
117 --demangle
119 Demangle application symbols. --no-demangle is also available for
120 disabling demangling.
121 --demangle-kernel
123 Demangle kernel symbols. --no-demangle-kernel is also available for
124 disabling kernel demangling.
125 In absence of -m/-x options, perf probe checks if the first argument
127 after the options is an absolute path name. If its an absolute path,
128 perf probe uses it as a target module/target user space binary to
129 probe.
130
132 Probe points are defined by following syntax.
133 1) Define event based on function name
135 [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]
136 2) Define event based on source file with line number
138 [[GROUP:]EVENT=]SRC:ALN [ARG ...]
139 3) Define event based on source file with lazy pattern
141 [[GROUP:]EVENT=]SRC;PTN [ARG ...]
142 4) Pre-defined SDT events or cached event with name
144 %[sdt_PROVIDER:]SDTEVENT
145 or,
146 sdt_PROVIDER:SDTEVENT
147 EVENT specifies the name of new event, if omitted, it will be set the
149 name of the probed function, and for return probes, a "__return" suffix
150 is automatically added to the function name. You can also specify a
151 group name by GROUP, if omitted, set probe is used for kprobe and
152 probe_<bin> is used for uprobe. Note that using existing group name can
153 conflict with other events. Especially, using the group name reserved
154 for kernel modules can hide embedded events in the modules. FUNC
155 specifies a probed function name, and it may have one of the following
156 options; +OFFS is the offset from function entry address in bytes, :RLN
157 is the relative-line number from function entry line, and %return means
158 that it probes function return. And ;PTN means lazy matching pattern
159 (see LAZY MATCHING). Note that ;PTN must be the end of the probe point
160 definition. In addition, @SRC specifies a source file which has that
161 function. It is also possible to specify a probe point by the source
162 line number or lazy matching by using SRC:ALN or SRC;PTN syntax, where
163 SRC is the source file path, :ALN is the line number and ;PTN is the
164 lazy matching pattern. ARG specifies the arguments of this probe point,
165 (see PROBE ARGUMENT). SDTEVENT and PROVIDER is the pre-defined event
166 name which is defined by user SDT (Statically Defined Tracing) or the
167 pre-cached probes with event name. Note that before using the SDT
168 event, the target binary (on which SDT events are defined) must be
169 scanned by perf-buildid-cache(1) to make SDT events as cached events.
170 For details of the SDT, see below.
172 https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
173
175 In the probe syntax, =, @, +, : and ; are treated as a special
176 character. You can use a backslash (\) to escape the special
177 characters. This is useful if you need to probe on a specific versioned
178 symbols, like @GLIBC_... suffixes, or also you need to specify a source
179 file which includes the special characters. Note that usually single
180 backslash is consumed by shell, so you might need to pass double
181 backslash (\\) or wrapping with single quotes ('AAA\@BBB'). See
182 EXAMPLES how it is used.
183
185 Each probe argument follows below syntax.
186 [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@user]
188 NAME specifies the name of this argument (optional). You can use the
190 name of local variable, local data structure member (e.g. var→field,
191 var.field2), local array with fixed index (e.g. array[1], var→array[0],
192 var→pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax,
193 etc). Note that the name of this argument will be set as the last
194 member name if you specify a local data structure member (e.g. field2
195 for var→field1.field2.) $vars and $params special arguments are also
196 available for NAME, $vars is expanded to the local variables (including
197 function parameters) which can access at given probe point. $params is
198 expanded to only the function parameters. TYPE casts the type of this
199 argument (optional). If omitted, perf probe automatically set the type
200 based on debuginfo (*). Currently, basic types
201 (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers
202 (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are
203 supported. (see TYPES for detail) On x86 systems %REG is always the
204 short form of the register: for example %AX. %RAX or %EAX is not valid.
205 "@user" is a special attribute which means the LOCALVAR will be treated
206 as a user-space memory. This is only valid for kprobe event.
207
209 Basic types (u8/u16/u32/u64/s8/s16/s32/s64) and hexadecimal integers
210 (x8/x16/x32/x64) are integer types. Prefix s and u means those types
211 are signed and unsigned respectively, and x means that is shown in
212 hexadecimal format. Traced arguments are shown in decimal (sNN/uNN) or
213 hex (xNN). You can also use s or u to specify only signedness and leave
214 its size auto-detected by perf probe. Moreover, you can use x to
215 explicitly specify to be shown in hexadecimal (the size is also
216 auto-detected). String type is a special type, which fetches a
217 "null-terminated" string from kernel space. This means it will fail and
218 store NULL if the string container has been paged out. You can specify
219 string type only for the local variable or structure member which is an
220 array of or a pointer to char or unsigned char type. Bitfield is
221 another special type, which takes 3 parameters, bit-width, bit-offset,
222 and container-size (usually 32). The syntax is;
223 b<bit-width>@<bit-offset>/<container-size>
225
227 Line range is described by following syntax.
228 "FUNC[@SRC][:RLN[+NUM|-RLN2]]|SRC[:ALN[+NUM|-ALN2]]"
230 FUNC specifies the function name of showing lines. RLN is the start
232 line number from function entry line, and RLN2 is the end line number.
233 As same as probe syntax, SRC means the source file path, ALN is start
234 line number, and ALN2 is end line number in the file. It is also
235 possible to specify how many lines to show by using NUM. Moreover,
236 FUNC@SRC combination is good for searching a specific function when
237 several functions share same name. So, "source.c:100-120" shows lines
238 between 100th to l20th in source.c file. And "func:10+20" shows 20
239 lines from 10th line of func function.
240
242 The lazy line matching is similar to glob matching but ignoring spaces
243 in both of pattern and target. So this accepts wildcards(*, ?) and
244 character classes(e.g. [a-z], [!A-Z]).
245 e.g. a=* can matches a=b, a = b, a == b and so on.
247 This provides some sort of flexibility and robustness to probe point
249 definitions against minor code changes. For example, actual 10th line
250 of schedule() can be moved easily by modifying schedule(), but the same
251 line matching rq=cpu_rq* may still exist in the function.)
252
254 The filter pattern is a glob matching pattern(s) to filter variables.
255 In addition, you can use "!" for specifying filter-out rule. You also
256 can give several rules combined with "&" or "|", and fold those rules
257 as one rule by using "(" ")".
258 e.g. With --filter "foo* | bar*", perf probe -V shows variables which
260 start with "foo" or "bar". With --filter "!foo* & *bar", perf probe -V
261 shows variables which don’t start with "foo" and end with "bar", like
262 "fizzbar". But "foobar" is filtered out.
263
265 Display which lines in schedule() can be probed:
266 ./perf probe --line schedule
268 Add a probe on schedule() function 12th line with recording cpu local
270 variable:
271 ./perf probe schedule:12 cpu
273 or
274 ./perf probe --add='schedule:12 cpu'
275 Add one or more probes which has the name start with "schedule".
277 ./perf probe schedule*
279 or
280 ./perf probe --add='schedule*'
281 Add probes on lines in schedule() function which calls
283 update_rq_clock().
284 ./perf probe 'schedule;update_rq_clock*'
286 or
287 ./perf probe --add='schedule;update_rq_clock*'
288 Delete all probes on schedule().
290 ./perf probe --del='schedule*'
292 Add probes at zfree() function on /bin/zsh
294 ./perf probe -x /bin/zsh zfree or ./perf probe /bin/zsh zfree
296 Add probes at malloc() function on libc
298 ./perf probe -x /lib/libc.so.6 malloc or ./perf probe /lib/libc.so.6 malloc
300 Add a uprobe to a target process running in a different mount namespace
302 ./perf probe --target-ns <target pid> -x /lib64/libc.so.6 malloc
304 Add a USDT probe to a target process running in a different mount
306 namespace
307 ./perf probe --target-ns <target pid> -x /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/amd64/server/libjvm.so %sdt_hotspot:thread__sleep__end
309 Add a probe on specific versioned symbol by backslash escape
311 ./perf probe -x /lib64/libc-2.25.so 'malloc_get_state\@GLIBC_2.2.5'
313 Add a probe in a source file using special characters by backslash
315 escape
316 ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
318
320 Since perf probe depends on ftrace (tracefs) and kallsyms
321 (/proc/kallsyms), you have to care about the permission and some sysctl
322 knobs.
323 • Since tracefs and kallsyms requires root or privileged user to
325 access it, the following perf probe commands also require it;
326 --add, --del, --list (except for --cache option)
327 • The system admin can remount the tracefs with 755 (sudo mount -o
329 remount,mode=755 /sys/kernel/tracing/) to allow unprivileged user
330 to run the perf probe --list command.
331 • /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also
333 prevents perf probe to retrieve the important information from
334 kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users)
335 for the above commands. Since the user-space probe doesn’t need to
336 access kallsyms, this is only for probing the kernel function
337 (kprobes).
338 • Since the perf probe commands read the vmlinux (for kernel) and/or
340 the debuginfo file (including user-space application), you need to
341 ensure that you can read those files.
342
344 perf-trace(1), perf-record(1), perf-buildid-cache(1)
345perf 01/12/2023 PERF-PROBE(1)