1trace(8) System Manager's Manual trace(8)
2
6 trace - Trace a function and print its arguments or return value,
7 optionally evaluating a filter. Uses Linux eBPF/bcc.
8
10 trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE]
11 [-S]
12 [-M MAX_EVENTS] [-t] [-T] [-C] [-K] [-U] [-a] [-I header]
13 probe [probe ...]
14
16 trace probes functions you specify and displays trace messages if a
17 particular condition is met. You can control the message format to dis‐
18 play function arguments and return values.
19 Since this uses BPF, only the root user can use this tool.
21
23 CONFIG_BPF and bcc.
24
26 -h Print usage message.
27 -p PID Trace only functions in the process PID.
29 -L TID Trace only functions in the thread TID.
31 -v Display the generated BPF program, for debugging purposes.
33 -z STRING_SIZE
35 When collecting string arguments (of type char*), collect up to
36 STRING_SIZE characters. Longer strings will be truncated.
37 -S If set, trace messages from trace's own process. By default,
39 this is off to avoid tracing storms -- for example, if you trace
40 the write system call, and consider that trace is writing to the
41 standard output.
42 -M MAX_EVENTS
44 Print up to MAX_EVENTS trace messages and then exit.
45 -t Print times relative to the beginning of the trace (offsets), in
47 seconds.
48 -T Print the time column.
50 -C Print CPU id.
52 -B Treat argument of STRCMP helper as a binary value
54 -K Print the kernel stack for each event.
56 -U Print the user stack for each event. -a Print virtual address
58 in kernel and user stacks.
59 -I header
61 Additional header files to include in the BPF program. This is
62 needed if your filter or print expressions use types or data
63 structures that are not available in the standard headers. For
64 example: 'linux/mm.h'
65 probe [probe ...]
67 One or more probes that attach to functions, filter conditions,
68 and print information. See PROBE SYNTAX below.
69
71 The general probe syntax is as follows:
72 [{p,r}]:[library]:function[(signature)] [(predicate)] ["format
74 string"[, arguments]]
75 {t:category:event,u:library:probe} [(predicate)] ["format string"[,
77 arguments]]
78 {[{p,r}],t,u}
80 Probe type - "p" for function entry, "r" for function return,
81 "t" for kernel tracepoint, "u" for USDT probe. The default probe
82 type is "p".
83 [library]
85 Library containing the probe. Specify the full path to the .so
86 or executable file where the function to probe resides. Alterna‐
87 tively, you can specify just the lib name: for example, "c"
88 refers to libc. If no library name is specified, the kernel is
89 assumed. Also, you can specify an executable name (without a
90 full path) if it is in the PATH. For example, "bash".
91 category
93 The tracepoint category. For example, "sched" or "irq".
94 function
96 The function to probe.
97 signature
99 The optional signature of the function to probe. This can make
100 it easier to access the function's arguments, instead of using
101 the "arg1", "arg2" etc. argument specifiers. For example,
102 "(struct timespec *ts)" in the signature position lets you use
103 "ts" in the filter or print expressions.
104 event The tracepoint event. For example, "block_rq_complete".
106 probe The USDT probe name. For example, "pthread_create".
108 [(predicate)]
110 The filter applied to the captured data. Only if the filter
111 evaluates as true, the trace message will be printed. The filter
112 can use any valid C expression that refers to the argument val‐
113 ues: arg1, arg2, etc., or to the return value retval in a return
114 probe. If necessary, use C cast operators to coerce the argu‐
115 ments to the desired type. For example, if arg1 is of type int,
116 use the expression ((int)arg1 < 0) to trace only invocations
117 where arg1 is negative. Note that only arg1-arg6 are supported,
118 and only if the function is using the standard x86_64 convention
119 where the first six arguments are in the RDI, RSI, RDX, RCX, R8,
120 R9 registers. If no predicate is specified, all function invoca‐
121 tions are traced.
122 The predicate expression may also use the STRCMP pseudo-function
124 to compare a predefined string to a string argument. For exam‐
125 ple: STRCMP("test", arg1). The order of arguments is important:
126 the first argument MUST be a quoted literal string, and the sec‐
127 ond argument can be a runtime string, most typically an argu‐
128 ment.
129 ["format string"[, arguments]]
131 A printf-style format string that will be used for the trace
132 message. You can use the following format specifiers: %s, %d,
133 %u, %lld, %llu, %hd, %hu, %c, %x, %llx -- with the same seman‐
134 tics as printf's. Make sure to pass the exact number of argu‐
135 ments as there are placeholders in the format string. The format
136 specifier replacements may be any C expressions, and may refer
137 to the same special keywords as in the predicate (arg1, arg2,
138 etc.).
139 In addition to the above format specifiers, you can also use %K
141 and %U when the expression is an address that potentially points
142 to executable code (i.e., a symbol). trace will resolve %K spec‐
143 ifiers to a kernel symbol, such as vfs__read, and will resolve
144 %U specifiers to a user-space symbol in that process, such as
145 sprintf.
146 In tracepoints, both the predicate and the arguments may refer
148 to the tracepoint format structure, which is stored in the spe‐
149 cial "args" variable. For example, the block:block_rq_complete
150 tracepoint can print or filter by args->nr_sector. To discover
151 the format of your tracepoint, use the tplist tool.
152 In USDT probes, the arg1, ..., argN variables refer to the
154 probe's arguments. To determine which arguments your probe has,
155 use the tplist tool.
156 The predicate expression and the format specifier replacements
158 for printing may also use the following special keywords: $pid,
159 $tgid to refer to the current process' pid and tgid; $uid, $gid
160 to refer to the current user's uid and gid; $cpu to refer to the
161 current processor number.
162
164 Trace all invocations of the open system call with the name of the file
165 being opened:
166 # trace '::do_sys_open "%s", arg2'
167 Trace all invocations of the read system call where the number of bytes
169 requested is greater than 20,000:
170 # trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'
171 Trace all malloc calls and print the size of the requested allocation:
173 # trace ':c:malloc "size = %d", arg1'
174 Trace returns from the readline function in bash and print the return
176 value as a string:
177 # trace 'r:bash:readline "%s", retval'
178 Trace the block:block_rq_complete tracepoint and print the number of
180 sectors completed:
181 # trace 't:block:block_rq_complete "%d sectors", args->nr_sec‐
182 tor'
183 Trace the pthread_create USDT probe from the pthread library and print
185 the address of the thread's start function:
186 # trace 'u:pthread:pthread_create "start addr = %llx", arg3'
187 Trace the nanosleep system call and print the sleep duration in
189 nanoseconds:
190 # trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns
191 , ts->tv_nsec'
192
194 This is from bcc.
195 https://github.com/iovisor/bcc
197 Also look in the bcc distribution for a companion _examples.txt file
199 containing example usage, output, and commentary for this tool.
200
202 Linux
203
205 Unstable - in development.
206
208 Sasha Goldshtein
209USER COMMANDS 2016-02-18 trace(8)