1trace(8) System Manager's Manual trace(8)
2
6 trace - Trace a function and print its arguments or return value, op‐
7 tionally evaluating a filter. Uses Linux eBPF/bcc.
8
10 trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [--uid UID] [-v] [-Z
11 STRING_SIZE] [-S] [-M MAX_EVENTS] [-t]
12 [-u] [-T] [-C] [-c CGROUP_PATH] [-n NAME] [-f MSG_FILTER] [-B]
13 [-s SYM_FILE_LIST] [-K] [-U] [-a]
14 [-I header] [-A]
15 probe [probe ...]
16
19 trace probes functions you specify and displays trace messages if a
20 particular condition is met. You can control the message format to dis‐
21 play function arguments and return values.
22 Since this uses BPF, only the root user can use this tool.
24
26 CONFIG_BPF and bcc.
27
29 -h Print usage message.
30 -p PID Trace only functions in the process PID.
32 -L TID Trace only functions in the thread TID.
34 --uid UID
36 Trace only functions from user UID.
37 -v Display the generated BPF program, for debugging purposes.
39 -z STRING_SIZE
41 When collecting string arguments (of type char*), collect up to
42 STRING_SIZE characters. Longer strings will be truncated.
43 -s SYM_FILE_LIST
45 When collecting stack trace in build id format, use the comma
46 separated list for symbol resolution.
47 -S If set, trace messages from trace's own process. By default,
49 this is off to avoid tracing storms -- for example, if you trace
50 the write system call, and consider that trace is writing to the
51 standard output.
52 -M MAX_EVENTS
54 Print up to MAX_EVENTS trace messages and then exit.
55 -t Print times relative to the beginning of the trace (offsets), in
57 seconds.
58 -u Print UNIX timestamps instead of offsets from trace beginning,
60 requires -t.
61 -T Print the time column.
63 -C Print CPU id.
65 -c CGROUP_PATH
67 Trace only functions in processes under CGROUP_PATH hierarchy.
68 -n NAME
70 Only print process names containing this name.
71 -f MSG_FILTER
73 Only print message of event containing this string.
74 -B Treat argument of STRCMP helper as a binary value
76 -K Print the kernel stack for each event.
78 -U Print the user stack for each event.
80 -a Print virtual address in kernel and user stacks.
82 -I header
84 Additional header files to include in the BPF program. This is
85 needed if your filter or print expressions use types or data
86 structures that are not available in the standard headers. For
87 example: 'linux/mm.h'
88 -A Print aggregated amount of each trace. This should be used with
90 -M/--max-events together.
91 probe [probe ...]
93 One or more probes that attach to functions, filter conditions,
94 and print information. See PROBE SYNTAX below.
95
97 The general probe syntax is as follows:
98 [{p,r}]:[library]:function[+offset][(signature)] [(predicate)] ["format
100 string"[, arguments]]
101 {t:category:event,u:library:probe} [(predicate)] ["format string"[, ar‐
103 guments]]
104 {[{p,r}],t,u}
106 Probe type - "p" for function entry, "r" for function return,
107 "t" for kernel tracepoint, "u" for USDT probe. The default probe
108 type is "p".
109 [library]
111 Library containing the probe. Specify the full path to the .so
112 or executable file where the function to probe resides. Alterna‐
113 tively, you can specify just the lib name: for example, "c"
114 refers to libc. If no library name is specified, the kernel is
115 assumed. Also, you can specify an executable name (without a
116 full path) if it is in the PATH. For example, "bash".
117 category
119 The tracepoint category. For example, "sched" or "irq".
120 function
122 The function to probe. offset The offset after the address of
123 the function where the probe should injected. For example
124 "kfree_skb+56" in decimal or hexadecimal "kfree_skb+0x38" for‐
125 mat. Only works with kprobes and uprobes. Zero if omitted.
126 signature
128 The optional signature of the function to probe. This can make
129 it easier to access the function's arguments, instead of using
130 the "arg1", "arg2" etc. argument specifiers. For example,
131 "(struct timespec *ts)" in the signature position lets you use
132 "ts" in the filter or print expressions.
133 event The tracepoint event. For example, "block_rq_complete".
135 probe The USDT probe name. For example, "pthread_create".
137 [(predicate)]
139 The filter applied to the captured data. Only if the filter
140 evaluates as true, the trace message will be printed. The filter
141 can use any valid C expression that refers to the argument val‐
142 ues: arg1, arg2, etc., or to the return value retval in a return
143 probe. If necessary, use C cast operators to coerce the argu‐
144 ments to the desired type. For example, if arg1 is of type int,
145 use the expression ((int)arg1 < 0) to trace only invocations
146 where arg1 is negative. Note that only arg1-arg6 are supported,
147 and only if the function is using the standard x86_64 convention
148 where the first six arguments are in the RDI, RSI, RDX, RCX, R8,
149 R9 registers. If no predicate is specified, all function invoca‐
150 tions are traced.
151 The predicate expression may also use the STRCMP pseudo-function
153 to compare a predefined string to a string argument. For exam‐
154 ple: STRCMP("test", arg1). The order of arguments is important:
155 the first argument MUST be a quoted literal string, and the sec‐
156 ond argument can be a runtime string, most typically an argu‐
157 ment.
158 ["format string"[, arguments]]
160 A printf-style format string that will be used for the trace
161 message. You can use the following format specifiers: %s, %d,
162 %u, %lld, %llu, %hd, %hu, %c, %x, %llx -- with the same seman‐
163 tics as printf's. Make sure to pass the exact number of argu‐
164 ments as there are placeholders in the format string. The format
165 specifier replacements may be any C expressions, and may refer
166 to the same special keywords as in the predicate (arg1, arg2,
167 etc.).
168 In addition to the above format specifiers, you can also use %K
170 and %U when the expression is an address that potentially points
171 to executable code (i.e., a symbol). trace will resolve %K spec‐
172 ifiers to a kernel symbol, such as vfs__read, and will resolve
173 %U specifiers to a user-space symbol in that process, such as
174 sprintf.
175 In tracepoints, both the predicate and the arguments may refer
177 to the tracepoint format structure, which is stored in the spe‐
178 cial "args" variable. For example, the block:block_rq_complete
179 tracepoint can print or filter by args->nr_sector. To discover
180 the format of your tracepoint, use the tplist tool.
181 In USDT probes, the arg1, ..., argN variables refer to the
183 probe's arguments. To determine which arguments your probe has,
184 use the tplist tool.
185 The predicate expression and the format specifier replacements
187 for printing may also use the following special keywords: $pid,
188 $tgid to refer to the current process' pid and tgid; $uid, $gid
189 to refer to the current user's uid and gid; $cpu to refer to the
190 current processor number.
191
193 Trace all invocations of the open system call with the name of the file
194 (from userspace) being opened:
195 # trace '::do_sys_open "%s", arg2@user'
196 Trace all invocations of the read system call where the number of bytes
198 requested is greater than 20,000:
199 # trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'
200 Trace all malloc calls and print the size of the requested allocation:
202 # trace ':c:malloc "size = %d", arg1'
203 Trace returns from the readline function in bash and print the return
205 value as a string:
206 # trace 'r:bash:readline "%s", retval'
207 Trace the block:block_rq_complete tracepoint and print the number of
209 sectors completed:
210 # trace 't:block:block_rq_complete "%d sectors", args->nr_sec‐
211 tor'
212 Trace the pthread_create USDT probe from the pthread library and print
214 the address of the thread's start function:
215 # trace 'u:pthread:pthread_create "start addr = %llx", arg3'
216 Trace the nanosleep system call and print the sleep duration in
218 nanoseconds:
219 # trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns
220 , ts->tv_nsec'
221 Trace the inet_pton system call using build id mechanism and print the
223 stack
224 # trace -s /lib/x86_64-linux-gnu/libc.so.6,/bin/ping
225 'p:c:inet_pton' -U
226
228 This is from bcc.
229 https://github.com/iovisor/bcc
231 Also look in the bcc distribution for a companion _examples.txt file
233 containing example usage, output, and commentary for this tool.
234
236 Linux
237
239 Unstable - in development.
240
242 Sasha Goldshtein
243USER COMMANDS 2016-02-18 trace(8)