1PERF-PROBE(1) perf Manual PERF-PROBE(1)
2
6 perf-probe - Define new dynamic tracepoints
7
9 perf probe [options] --add=PROBE [...]
10 or
11 perf probe [options] PROBE
12 or
13 perf probe [options] --del=[GROUP:]EVENT [...]
14 or
15 perf probe --list[=[GROUP:]EVENT]
16 or
17 perf probe [options] --line=LINE
18 or
19 perf probe [options] --vars=PROBEPOINT
20 or
21 perf probe [options] --funcs
22 or
23 perf probe [options] --definition=PROBE [...]
24
26 This command defines dynamic tracepoint events, by symbol and registers
27 without debuginfo, or by C expressions (C line numbers, C function
28 names, and C local variables) with debuginfo.
29
31 -k, --vmlinux=PATH
32 Specify vmlinux path which has debuginfo (Dwarf binary). Only when
33 using this with --definition, you can give an offline vmlinux file.
34 -m, --module=MODNAME|PATH
36 Specify module name in which perf-probe searches probe points or
37 lines. If a path of module file is passed, perf-probe treat it as
38 an offline module (this means you can add a probe on a module which
39 has not been loaded yet).
40 -s, --source=PATH
42 Specify path to kernel source.
43 -v, --verbose
45 Be more verbose (show parsed arguments, etc). Can not use with -q.
46 -q, --quiet
48 Be quiet (do not show any messages including errors). Can not use
49 with -v.
50 -a, --add=
52 Define a probe event (see PROBE SYNTAX for detail).
53 -d, --del=
55 Delete probe events. This accepts glob wildcards(*, ?) and
56 character classes(e.g. [a-z], [!A-Z]).
57 -l, --list[=[GROUP:]EVENT]
59 List up current probe events. This can also accept filtering
60 patterns of event names. When this is used with --cache, perf shows
61 all cached probes instead of the live probes.
62 -L, --line=
64 Show source code lines which can be probed. This needs an argument
65 which specifies a range of the source code. (see LINE SYNTAX for
66 detail)
67 -V, --vars=
69 Show available local variables at given probe point. The argument
70 syntax is same as PROBE SYNTAX, but NO ARGs.
71 --externs
73 (Only for --vars) Show external defined variables in addition to
74 local variables.
75 --no-inlines
77 (Only for --add) Search only for non-inlined functions. The
78 functions which do not have instances are ignored.
79 -F, --funcs[=FILTER]
81 Show available functions in given module or kernel. With -x/--exec,
82 can also list functions in a user space executable / shared
83 library. This also can accept a FILTER rule argument.
84 -D, --definition=
86 Show trace-event definition converted from given probe-event
87 instead of write it into tracing/[k,u]probe_events.
88 --filter=FILTER
90 (Only for --vars and --funcs) Set filter. FILTER is a combination
91 of glob pattern, see FILTER PATTERN for detail. Default FILTER is
92 "!k???tab_* & !crc_*" for --vars, and "!_*" for --funcs. If several
93 filters are specified, only the last filter is used.
94 -f, --force
96 Forcibly add events with existing name.
97 -n, --dry-run
99 Dry run. With this option, --add and --del doesn’t execute actual
100 adding and removal operations.
101 --cache
103 (With --add) Cache the probes. Any events which successfully added
104 are also stored in the cache file. (With --list) Show cached
105 probes. (With --del) Remove cached probes.
106 --max-probes=NUM
108 Set the maximum number of probe points for an event. Default is
109 128.
110 --target-ns=PID: Obtain mount namespace information from the target
112 pid. This is used when creating a uprobe for a process that resides in
113 a different mount namespace from the perf(1) utility.
114 -x, --exec=PATH
116 Specify path to the executable or shared library file for user
117 space tracing. Can also be used with --funcs option.
118 --demangle
120 Demangle application symbols. --no-demangle is also available for
121 disabling demangling.
122 --demangle-kernel
124 Demangle kernel symbols. --no-demangle-kernel is also available for
125 disabling kernel demangling.
126 In absence of -m/-x options, perf probe checks if the first argument
128 after the options is an absolute path name. If its an absolute path,
129 perf probe uses it as a target module/target user space binary to
130 probe.
131
133 Probe points are defined by following syntax.
134 1) Define event based on function name
136 [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]
137 2) Define event based on source file with line number
139 [[GROUP:]EVENT=]SRC:ALN [ARG ...]
140 3) Define event based on source file with lazy pattern
142 [[GROUP:]EVENT=]SRC;PTN [ARG ...]
143 4) Pre-defined SDT events or cached event with name
145 %[sdt_PROVIDER:]SDTEVENT
146 or,
147 sdt_PROVIDER:SDTEVENT
148 EVENT specifies the name of new event, if omitted, it will be set the
150 name of the probed function, and for return probes, a "__return" suffix
151 is automatically added to the function name. You can also specify a
152 group name by GROUP, if omitted, set probe is used for kprobe and
153 probe_<bin> is used for uprobe. Note that using existing group name can
154 conflict with other events. Especially, using the group name reserved
155 for kernel modules can hide embedded events in the modules. FUNC
156 specifies a probed function name, and it may have one of the following
157 options; +OFFS is the offset from function entry address in bytes, :RLN
158 is the relative-line number from function entry line, and %return means
159 that it probes function return. And ;PTN means lazy matching pattern
160 (see LAZY MATCHING). Note that ;PTN must be the end of the probe point
161 definition. In addition, @SRC specifies a source file which has that
162 function. It is also possible to specify a probe point by the source
163 line number or lazy matching by using SRC:ALN or SRC;PTN syntax, where
164 SRC is the source file path, :ALN is the line number and ;PTN is the
165 lazy matching pattern. ARG specifies the arguments of this probe point,
166 (see PROBE ARGUMENT). SDTEVENT and PROVIDER is the pre-defined event
167 name which is defined by user SDT (Statically Defined Tracing) or the
168 pre-cached probes with event name. Note that before using the SDT
169 event, the target binary (on which SDT events are defined) must be
170 scanned by perf-buildid-cache(1) to make SDT events as cached events.
171 For details of the SDT, see below.
173 https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
174
176 In the probe syntax, =, @, +, : and ; are treated as a special
177 character. You can use a backslash (\) to escape the special
178 characters. This is useful if you need to probe on a specific versioned
179 symbols, like @GLIBC_... suffixes, or also you need to specify a source
180 file which includes the special characters. Note that usually single
181 backslash is consumed by shell, so you might need to pass double
182 backslash (\\) or wrapping with single quotes ('AAA\@BBB'). See
183 EXAMPLES how it is used.
184
186 Each probe argument follows below syntax.
187 [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@user]
189 NAME specifies the name of this argument (optional). You can use the
191 name of local variable, local data structure member (e.g. var→field,
192 var.field2), local array with fixed index (e.g. array[1], var→array[0],
193 var→pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax,
194 etc). Note that the name of this argument will be set as the last
195 member name if you specify a local data structure member (e.g. field2
196 for var→field1.field2.) $vars and $params special arguments are also
197 available for NAME, $vars is expanded to the local variables (including
198 function parameters) which can access at given probe point. $params is
199 expanded to only the function parameters. TYPE casts the type of this
200 argument (optional). If omitted, perf probe automatically set the type
201 based on debuginfo (*). Currently, basic types
202 (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers
203 (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are
204 supported. (see TYPES for detail) On x86 systems %REG is always the
205 short form of the register: for example %AX. %RAX or %EAX is not valid.
206 "@user" is a special attribute which means the LOCALVAR will be treated
207 as a user-space memory. This is only valid for kprobe event.
208
210 Basic types (u8/u16/u32/u64/s8/s16/s32/s64) and hexadecimal integers
211 (x8/x16/x32/x64) are integer types. Prefix s and u means those types
212 are signed and unsigned respectively, and x means that is shown in
213 hexadecimal format. Traced arguments are shown in decimal (sNN/uNN) or
214 hex (xNN). You can also use s or u to specify only signedness and leave
215 its size auto-detected by perf probe. Moreover, you can use x to
216 explicitly specify to be shown in hexadecimal (the size is also
217 auto-detected). String type is a special type, which fetches a
218 "null-terminated" string from kernel space. This means it will fail and
219 store NULL if the string container has been paged out. You can specify
220 string type only for the local variable or structure member which is an
221 array of or a pointer to char or unsigned char type. Bitfield is
222 another special type, which takes 3 parameters, bit-width, bit-offset,
223 and container-size (usually 32). The syntax is;
224 b<bit-width>@<bit-offset>/<container-size>
226
228 Line range is described by following syntax.
229 "FUNC[@SRC][:RLN[+NUM|-RLN2]]|SRC[:ALN[+NUM|-ALN2]]"
231 FUNC specifies the function name of showing lines. RLN is the start
233 line number from function entry line, and RLN2 is the end line number.
234 As same as probe syntax, SRC means the source file path, ALN is start
235 line number, and ALN2 is end line number in the file. It is also
236 possible to specify how many lines to show by using NUM. Moreover,
237 FUNC@SRC combination is good for searching a specific function when
238 several functions share same name. So, "source.c:100-120" shows lines
239 between 100th to l20th in source.c file. And "func:10+20" shows 20
240 lines from 10th line of func function.
241
243 The lazy line matching is similar to glob matching but ignoring spaces
244 in both of pattern and target. So this accepts wildcards(*, ?) and
245 character classes(e.g. [a-z], [!A-Z]).
246 e.g. a=* can matches a=b, a = b, a == b and so on.
248 This provides some sort of flexibility and robustness to probe point
250 definitions against minor code changes. For example, actual 10th line
251 of schedule() can be moved easily by modifying schedule(), but the same
252 line matching rq=cpu_rq* may still exist in the function.)
253
255 The filter pattern is a glob matching pattern(s) to filter variables.
256 In addition, you can use "!" for specifying filter-out rule. You also
257 can give several rules combined with "&" or "|", and fold those rules
258 as one rule by using "(" ")".
259 e.g. With --filter "foo* | bar*", perf probe -V shows variables which
261 start with "foo" or "bar". With --filter "!foo* & *bar", perf probe -V
262 shows variables which don’t start with "foo" and end with "bar", like
263 "fizzbar". But "foobar" is filtered out.
264
266 Display which lines in schedule() can be probed:
267 ./perf probe --line schedule
269 Add a probe on schedule() function 12th line with recording cpu local
271 variable:
272 ./perf probe schedule:12 cpu
274 or
275 ./perf probe --add='schedule:12 cpu'
276 Add one or more probes which has the name start with "schedule".
278 ./perf probe schedule*
280 or
281 ./perf probe --add='schedule*'
282 Add probes on lines in schedule() function which calls
284 update_rq_clock().
285 ./perf probe 'schedule;update_rq_clock*'
287 or
288 ./perf probe --add='schedule;update_rq_clock*'
289 Delete all probes on schedule().
291 ./perf probe --del='schedule*'
293 Add probes at zfree() function on /bin/zsh
295 ./perf probe -x /bin/zsh zfree or ./perf probe /bin/zsh zfree
297 Add probes at malloc() function on libc
299 ./perf probe -x /lib/libc.so.6 malloc or ./perf probe /lib/libc.so.6 malloc
301 Add a uprobe to a target process running in a different mount namespace
303 ./perf probe --target-ns <target pid> -x /lib64/libc.so.6 malloc
305 Add a USDT probe to a target process running in a different mount
307 namespace
308 ./perf probe --target-ns <target pid> -x /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/amd64/server/libjvm.so %sdt_hotspot:thread__sleep__end
310 Add a probe on specific versioned symbol by backslash escape
312 ./perf probe -x /lib64/libc-2.25.so 'malloc_get_state\@GLIBC_2.2.5'
314 Add a probe in a source file using special characters by backslash
316 escape
317 ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
319
321 Since perf probe depends on ftrace (tracefs) and kallsyms
322 (/proc/kallsyms), you have to care about the permission and some sysctl
323 knobs.
324 • Since tracefs and kallsyms requires root or privileged user to
326 access it, the following perf probe commands also require it;
327 --add, --del, --list (except for --cache option)
328 • The system admin can remount the tracefs with 755 (sudo mount -o
330 remount,mode=755 /sys/kernel/tracing/) to allow unprivileged user
331 to run the perf probe --list command.
332 • /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also
334 prevents perf probe to retrieve the important information from
335 kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users)
336 for the above commands. Since the user-space probe doesn’t need to
337 access kallsyms, this is only for probing the kernel function
338 (kprobes).
339 • Since the perf probe commands read the vmlinux (for kernel) and/or
341 the debuginfo file (including user-space application), you need to
342 ensure that you can read those files.
343
345 perf-trace(1), perf-record(1), perf-buildid-cache(1)
346perf 06/14/2022 PERF-PROBE(1)