1trace(8) System Manager's Manual trace(8)
2
6 trace - Trace a function and print its arguments or return value,
7 optionally evaluating a filter. Uses Linux eBPF/bcc.
8
10 trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE]
11 [-S] [-s SYM_FILE_LIST]
12 [-M MAX_EVENTS] [-t] [-T] [-C] [-K] [-U] [-a] [-I header]
13 probe [probe ...]
14
16 trace probes functions you specify and displays trace messages if a
17 particular condition is met. You can control the message format to dis‐
18 play function arguments and return values.
19 Since this uses BPF, only the root user can use this tool.
21
23 CONFIG_BPF and bcc.
24
26 -h Print usage message.
27 -p PID Trace only functions in the process PID.
29 -L TID Trace only functions in the thread TID.
31 -v Display the generated BPF program, for debugging purposes.
33 -z STRING_SIZE
35 When collecting string arguments (of type char*), collect up to
36 STRING_SIZE characters. Longer strings will be truncated.
37 -s SYM_FILE_LIST
39 When collecting stack trace in build id format, use the coma
40 separated list for symbol resolution.
41 -S If set, trace messages from trace's own process. By default,
43 this is off to avoid tracing storms -- for example, if you trace
44 the write system call, and consider that trace is writing to the
45 standard output.
46 -M MAX_EVENTS
48 Print up to MAX_EVENTS trace messages and then exit.
49 -t Print times relative to the beginning of the trace (offsets), in
51 seconds.
52 -T Print the time column.
54 -C Print CPU id.
56 -B Treat argument of STRCMP helper as a binary value
58 -K Print the kernel stack for each event.
60 -U Print the user stack for each event. -a Print virtual address
62 in kernel and user stacks.
63 -I header
65 Additional header files to include in the BPF program. This is
66 needed if your filter or print expressions use types or data
67 structures that are not available in the standard headers. For
68 example: 'linux/mm.h'
69 probe [probe ...]
71 One or more probes that attach to functions, filter conditions,
72 and print information. See PROBE SYNTAX below.
73
75 The general probe syntax is as follows:
76 [{p,r}]:[library]:function[(signature)] [(predicate)] ["format
78 string"[, arguments]]
79 {t:category:event,u:library:probe} [(predicate)] ["format string"[,
81 arguments]]
82 {[{p,r}],t,u}
84 Probe type - "p" for function entry, "r" for function return,
85 "t" for kernel tracepoint, "u" for USDT probe. The default probe
86 type is "p".
87 [library]
89 Library containing the probe. Specify the full path to the .so
90 or executable file where the function to probe resides. Alterna‐
91 tively, you can specify just the lib name: for example, "c"
92 refers to libc. If no library name is specified, the kernel is
93 assumed. Also, you can specify an executable name (without a
94 full path) if it is in the PATH. For example, "bash".
95 category
97 The tracepoint category. For example, "sched" or "irq".
98 function
100 The function to probe.
101 signature
103 The optional signature of the function to probe. This can make
104 it easier to access the function's arguments, instead of using
105 the "arg1", "arg2" etc. argument specifiers. For example,
106 "(struct timespec *ts)" in the signature position lets you use
107 "ts" in the filter or print expressions.
108 event The tracepoint event. For example, "block_rq_complete".
110 probe The USDT probe name. For example, "pthread_create".
112 [(predicate)]
114 The filter applied to the captured data. Only if the filter
115 evaluates as true, the trace message will be printed. The filter
116 can use any valid C expression that refers to the argument val‐
117 ues: arg1, arg2, etc., or to the return value retval in a return
118 probe. If necessary, use C cast operators to coerce the argu‐
119 ments to the desired type. For example, if arg1 is of type int,
120 use the expression ((int)arg1 < 0) to trace only invocations
121 where arg1 is negative. Note that only arg1-arg6 are supported,
122 and only if the function is using the standard x86_64 convention
123 where the first six arguments are in the RDI, RSI, RDX, RCX, R8,
124 R9 registers. If no predicate is specified, all function invoca‐
125 tions are traced.
126 The predicate expression may also use the STRCMP pseudo-function
128 to compare a predefined string to a string argument. For exam‐
129 ple: STRCMP("test", arg1). The order of arguments is important:
130 the first argument MUST be a quoted literal string, and the sec‐
131 ond argument can be a runtime string, most typically an argu‐
132 ment.
133 ["format string"[, arguments]]
135 A printf-style format string that will be used for the trace
136 message. You can use the following format specifiers: %s, %d,
137 %u, %lld, %llu, %hd, %hu, %c, %x, %llx -- with the same seman‐
138 tics as printf's. Make sure to pass the exact number of argu‐
139 ments as there are placeholders in the format string. The format
140 specifier replacements may be any C expressions, and may refer
141 to the same special keywords as in the predicate (arg1, arg2,
142 etc.).
143 In addition to the above format specifiers, you can also use %K
145 and %U when the expression is an address that potentially points
146 to executable code (i.e., a symbol). trace will resolve %K spec‐
147 ifiers to a kernel symbol, such as vfs__read, and will resolve
148 %U specifiers to a user-space symbol in that process, such as
149 sprintf.
150 In tracepoints, both the predicate and the arguments may refer
152 to the tracepoint format structure, which is stored in the spe‐
153 cial "args" variable. For example, the block:block_rq_complete
154 tracepoint can print or filter by args->nr_sector. To discover
155 the format of your tracepoint, use the tplist tool.
156 In USDT probes, the arg1, ..., argN variables refer to the
158 probe's arguments. To determine which arguments your probe has,
159 use the tplist tool.
160 The predicate expression and the format specifier replacements
162 for printing may also use the following special keywords: $pid,
163 $tgid to refer to the current process' pid and tgid; $uid, $gid
164 to refer to the current user's uid and gid; $cpu to refer to the
165 current processor number.
166
168 Trace all invocations of the open system call with the name of the file
169 being opened:
170 # trace '::do_sys_open "%s", arg2'
171 Trace all invocations of the read system call where the number of bytes
173 requested is greater than 20,000:
174 # trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'
175 Trace all malloc calls and print the size of the requested allocation:
177 # trace ':c:malloc "size = %d", arg1'
178 Trace returns from the readline function in bash and print the return
180 value as a string:
181 # trace 'r:bash:readline "%s", retval'
182 Trace the block:block_rq_complete tracepoint and print the number of
184 sectors completed:
185 # trace 't:block:block_rq_complete "%d sectors", args->nr_sec‐
186 tor'
187 Trace the pthread_create USDT probe from the pthread library and print
189 the address of the thread's start function:
190 # trace 'u:pthread:pthread_create "start addr = %llx", arg3'
191 Trace the nanosleep system call and print the sleep duration in
193 nanoseconds:
194 # trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns
195 , ts->tv_nsec'
196 Trace the inet_pton system call using build id mechanism and print the
198 stack
199 # trace -s /lib/x86_64-linux-gnu/libc.so.6,/bin/ping
200 'p:c:inet_pton' -U
201
203 This is from bcc.
204 https://github.com/iovisor/bcc
206 Also look in the bcc distribution for a companion _examples.txt file
208 containing example usage, output, and commentary for this tool.
209
211 Linux
212
214 Unstable - in development.
215
217 Sasha Goldshtein
218USER COMMANDS 2016-02-18 trace(8)