1PERF-PROBE(1)                     perf Manual                    PERF-PROBE(1)
2
3
4

NAME

6       perf-probe - Define new dynamic tracepoints
7

SYNOPSIS

9       perf probe [options] --add=PROBE [...]
10       or
11       perf probe [options] PROBE
12       or
13       perf probe [options] --del=[GROUP:]EVENT [...]
14       or
15       perf probe --list[=[GROUP:]EVENT]
16       or
17       perf probe [options] --line=LINE
18       or
19       perf probe [options] --vars=PROBEPOINT
20       or
21       perf probe [options] --funcs
22       or
23       perf probe [options] --definition=PROBE [...]
24

DESCRIPTION

26       This command defines dynamic tracepoint events, by symbol and registers
27       without debuginfo, or by C expressions (C line numbers, C function
28       names, and C local variables) with debuginfo.
29

OPTIONS

31       -k, --vmlinux=PATH
32           Specify vmlinux path which has debuginfo (Dwarf binary). Only when
33           using this with --definition, you can give an offline vmlinux file.
34
35       -m, --module=MODNAME|PATH
36           Specify module name in which perf-probe searches probe points or
37           lines. If a path of module file is passed, perf-probe treat it as
38           an offline module (this means you can add a probe on a module which
39           has not been loaded yet).
40
41       -s, --source=PATH
42           Specify path to kernel source.
43
44       -v, --verbose
45           Be more verbose (show parsed arguments, etc). Can not use with -q.
46
47       -q, --quiet
48           Do not show any warnings or messages. Can not use with -v.
49
50       -a, --add=
51           Define a probe event (see PROBE SYNTAX for detail).
52
53       -d, --del=
54           Delete probe events. This accepts glob wildcards(*, ?) and
55           character classes(e.g. [a-z], [!A-Z]).
56
57       -l, --list[=[GROUP:]EVENT]
58           List up current probe events. This can also accept filtering
59           patterns of event names. When this is used with --cache, perf shows
60           all cached probes instead of the live probes.
61
62       -L, --line=
63           Show source code lines which can be probed. This needs an argument
64           which specifies a range of the source code. (see LINE SYNTAX for
65           detail)
66
67       -V, --vars=
68           Show available local variables at given probe point. The argument
69           syntax is same as PROBE SYNTAX, but NO ARGs.
70
71       --externs
72           (Only for --vars) Show external defined variables in addition to
73           local variables.
74
75       --no-inlines
76           (Only for --add) Search only for non-inlined functions. The
77           functions which do not have instances are ignored.
78
79       -F, --funcs[=FILTER]
80           Show available functions in given module or kernel. With -x/--exec,
81           can also list functions in a user space executable / shared
82           library. This also can accept a FILTER rule argument.
83
84       -D, --definition=
85           Show trace-event definition converted from given probe-event
86           instead of write it into tracing/[k,u]probe_events.
87
88       --filter=FILTER
89           (Only for --vars and --funcs) Set filter. FILTER is a combination
90           of glob pattern, see FILTER PATTERN for detail. Default FILTER is
91           "!k???tab_* & !crc_*" for --vars, and "!_*" for --funcs. If several
92           filters are specified, only the last filter is used.
93
94       -f, --force
95           Forcibly add events with existing name.
96
97       -n, --dry-run
98           Dry run. With this option, --add and --del doesn’t execute actual
99           adding and removal operations.
100
101       --cache
102           (With --add) Cache the probes. Any events which successfully added
103           are also stored in the cache file. (With --list) Show cached
104           probes. (With --del) Remove cached probes.
105
106       --max-probes=NUM
107           Set the maximum number of probe points for an event. Default is
108           128.
109
110       --target-ns=PID: Obtain mount namespace information from the target
111       pid. This is used when creating a uprobe for a process that resides in
112       a different mount namespace from the perf(1) utility.
113
114       -x, --exec=PATH
115           Specify path to the executable or shared library file for user
116           space tracing. Can also be used with --funcs option.
117
118       --demangle
119           Demangle application symbols. --no-demangle is also available for
120           disabling demangling.
121
122       --demangle-kernel
123           Demangle kernel symbols. --no-demangle-kernel is also available for
124           disabling kernel demangling.
125
126       In absence of -m/-x options, perf probe checks if the first argument
127       after the options is an absolute path name. If its an absolute path,
128       perf probe uses it as a target module/target user space binary to
129       probe.
130

PROBE SYNTAX

132       Probe points are defined by following syntax.
133
134           1) Define event based on function name
135            [[GROUP:]EVENT=]FUNC[@SRC][:RLN|+OFFS|%return|;PTN] [ARG ...]
136
137           2) Define event based on source file with line number
138            [[GROUP:]EVENT=]SRC:ALN [ARG ...]
139
140           3) Define event based on source file with lazy pattern
141            [[GROUP:]EVENT=]SRC;PTN [ARG ...]
142
143           4) Pre-defined SDT events or cached event with name
144            %[sdt_PROVIDER:]SDTEVENT
145            or,
146            sdt_PROVIDER:SDTEVENT
147
148       EVENT specifies the name of new event, if omitted, it will be set the
149       name of the probed function, and for return probes, a "__return" suffix
150       is automatically added to the function name. You can also specify a
151       group name by GROUP, if omitted, set probe is used for kprobe and
152       probe_<bin> is used for uprobe. Note that using existing group name can
153       conflict with other events. Especially, using the group name reserved
154       for kernel modules can hide embedded events in the modules. FUNC
155       specifies a probed function name, and it may have one of the following
156       options; +OFFS is the offset from function entry address in bytes, :RLN
157       is the relative-line number from function entry line, and %return means
158       that it probes function return. And ;PTN means lazy matching pattern
159       (see LAZY MATCHING). Note that ;PTN must be the end of the probe point
160       definition. In addition, @SRC specifies a source file which has that
161       function. It is also possible to specify a probe point by the source
162       line number or lazy matching by using SRC:ALN or SRC;PTN syntax, where
163       SRC is the source file path, :ALN is the line number and ;PTN is the
164       lazy matching pattern. ARG specifies the arguments of this probe point,
165       (see PROBE ARGUMENT). SDTEVENT and PROVIDER is the pre-defined event
166       name which is defined by user SDT (Statically Defined Tracing) or the
167       pre-cached probes with event name. Note that before using the SDT
168       event, the target binary (on which SDT events are defined) must be
169       scanned by perf-buildid-cache(1) to make SDT events as cached events.
170
171       For details of the SDT, see below.
172       https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
173

ESCAPED CHARACTER

175       In the probe syntax, =, @, +, : and ; are treated as a special
176       character. You can use a backslash (\) to escape the special
177       characters. This is useful if you need to probe on a specific versioned
178       symbols, like @GLIBC_... suffixes, or also you need to specify a source
179       file which includes the special characters. Note that usually single
180       backslash is consumed by shell, so you might need to pass double
181       backslash (\\) or wrapping with single quotes ('AAA\@BBB'). See
182       EXAMPLES how it is used.
183

PROBE ARGUMENT

185       Each probe argument follows below syntax.
186
187           [NAME=]LOCALVAR|$retval|%REG|@SYMBOL[:TYPE][@user]
188
189       NAME specifies the name of this argument (optional). You can use the
190       name of local variable, local data structure member (e.g. var→field,
191       var.field2), local array with fixed index (e.g. array[1], var→array[0],
192       var→pointer[2]), or kprobe-tracer argument format (e.g. $retval, %ax,
193       etc). Note that the name of this argument will be set as the last
194       member name if you specify a local data structure member (e.g. field2
195       for var→field1.field2.) $vars and $params special arguments are also
196       available for NAME, $vars is expanded to the local variables (including
197       function parameters) which can access at given probe point. $params is
198       expanded to only the function parameters. TYPE casts the type of this
199       argument (optional). If omitted, perf probe automatically set the type
200       based on debuginfo (*). Currently, basic types
201       (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal integers
202       (x/x8/x16/x32/x64), signedness casting (u/s), "string" and bitfield are
203       supported. (see TYPES for detail) On x86 systems %REG is always the
204       short form of the register: for example %AX. %RAX or %EAX is not valid.
205       "@user" is a special attribute which means the LOCALVAR will be treated
206       as a user-space memory. This is only valid for kprobe event.
207

TYPES

209       Basic types (u8/u16/u32/u64/s8/s16/s32/s64) and hexadecimal integers
210       (x8/x16/x32/x64) are integer types. Prefix s and u means those types
211       are signed and unsigned respectively, and x means that is shown in
212       hexadecimal format. Traced arguments are shown in decimal (sNN/uNN) or
213       hex (xNN). You can also use s or u to specify only signedness and leave
214       its size auto-detected by perf probe. Moreover, you can use x to
215       explicitly specify to be shown in hexadecimal (the size is also
216       auto-detected). String type is a special type, which fetches a
217       "null-terminated" string from kernel space. This means it will fail and
218       store NULL if the string container has been paged out. You can specify
219       string type only for the local variable or structure member which is an
220       array of or a pointer to char or unsigned char type. Bitfield is
221       another special type, which takes 3 parameters, bit-width, bit-offset,
222       and container-size (usually 32). The syntax is;
223
224           b<bit-width>@<bit-offset>/<container-size>
225

LINE SYNTAX

227       Line range is described by following syntax.
228
229           "FUNC[@SRC][:RLN[+NUM|-RLN2]]|SRC[:ALN[+NUM|-ALN2]]"
230
231       FUNC specifies the function name of showing lines. RLN is the start
232       line number from function entry line, and RLN2 is the end line number.
233       As same as probe syntax, SRC means the source file path, ALN is start
234       line number, and ALN2 is end line number in the file. It is also
235       possible to specify how many lines to show by using NUM. Moreover,
236       FUNC@SRC combination is good for searching a specific function when
237       several functions share same name. So, "source.c:100-120" shows lines
238       between 100th to 120th in source.c file. And "func:10+20" shows 20
239       lines from 10th line of func function.
240

LAZY MATCHING

242       The lazy line matching is similar to glob matching but ignoring spaces
243       in both of pattern and target. So this accepts wildcards(*, ?) and
244       character classes(e.g. [a-z], [!A-Z]).
245
246       e.g. a=* can matches a=b, a = b, a == b and so on.
247
248       This provides some sort of flexibility and robustness to probe point
249       definitions against minor code changes. For example, actual 10th line
250       of schedule() can be moved easily by modifying schedule(), but the same
251       line matching rq=cpu_rq* may still exist in the function.)
252

FILTER PATTERN

254       The filter pattern is a glob matching pattern(s) to filter variables.
255       In addition, you can use "!" for specifying filter-out rule. You also
256       can give several rules combined with "&" or "|", and fold those rules
257       as one rule by using "(" ")".
258
259       e.g. With --filter "foo* | bar*", perf probe -V shows variables which
260       start with "foo" or "bar". With --filter "!foo* & *bar", perf probe -V
261       shows variables which don’t start with "foo" and end with "bar", like
262       "fizzbar". But "foobar" is filtered out.
263

EXAMPLES

265       Display which lines in schedule() can be probed:
266
267           ./perf probe --line schedule
268
269       Add a probe on schedule() function 12th line with recording cpu local
270       variable:
271
272           ./perf probe schedule:12 cpu
273           or
274           ./perf probe --add='schedule:12 cpu'
275
276       Add one or more probes which has the name start with "schedule".
277
278           ./perf probe schedule*
279           or
280           ./perf probe --add='schedule*'
281
282       Add probes on lines in schedule() function which calls
283       update_rq_clock().
284
285           ./perf probe 'schedule;update_rq_clock*'
286           or
287           ./perf probe --add='schedule;update_rq_clock*'
288
289       Delete all probes on schedule().
290
291           ./perf probe --del='schedule*'
292
293       Add probes at zfree() function on /bin/zsh
294
295           ./perf probe -x /bin/zsh zfree or ./perf probe /bin/zsh zfree
296
297       Add probes at malloc() function on libc
298
299           ./perf probe -x /lib/libc.so.6 malloc or ./perf probe /lib/libc.so.6 malloc
300
301       Add a uprobe to a target process running in a different mount namespace
302
303           ./perf probe --target-ns <target pid> -x /lib64/libc.so.6 malloc
304
305       Add a USDT probe to a target process running in a different mount
306       namespace
307
308           ./perf probe --target-ns <target pid> -x /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/amd64/server/libjvm.so %sdt_hotspot:thread__sleep__end
309
310       Add a probe on specific versioned symbol by backslash escape
311
312           ./perf probe -x /lib64/libc-2.25.so 'malloc_get_state\@GLIBC_2.2.5'
313
314       Add a probe in a source file using special characters by backslash
315       escape
316
317           ./perf probe -x /opt/test/a.out 'foo\+bar.c:4'
318

PERMISSIONS AND SYSCTL

320       Since perf probe depends on ftrace (tracefs) and kallsyms
321       (/proc/kallsyms), you have to care about the permission and some sysctl
322       knobs.
323
324       •   Since tracefs and kallsyms requires root or privileged user to
325           access it, the following perf probe commands also require it;
326           --add, --del, --list (except for --cache option)
327
328       •   The system admin can remount the tracefs with 755 (sudo mount -o
329           remount,mode=755 /sys/kernel/tracing/) to allow unprivileged user
330           to run the perf probe --list command.
331
332       •   /proc/sys/kernel/kptr_restrict = 2 (restrict all users) also
333           prevents perf probe to retrieve the important information from
334           kallsyms. You also need to set to 1 (restrict non CAP_SYSLOG users)
335           for the above commands. Since the user-space probe doesn’t need to
336           access kallsyms, this is only for probing the kernel function
337           (kprobes).
338
339       •   Since the perf probe commands read the vmlinux (for kernel) and/or
340           the debuginfo file (including user-space application), you need to
341           ensure that you can read those files.
342

SEE ALSO

344       perf-trace(1), perf-record(1), perf-buildid-cache(1)
345
346
347
348perf                              11/28/2023                     PERF-PROBE(1)
Impressum