1trace(8)                    System Manager's Manual                   trace(8)
2
3
4

NAME

6       trace  -  Trace  a  function  and  print its arguments or return value,
7       optionally evaluating a filter. Uses Linux eBPF/bcc.
8

SYNOPSIS

10       trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID]  [-v]  [-Z  STRING_SIZE]
11       [-S] [-s SYM_FILE_LIST]
12                [-M MAX_EVENTS] [-t] [-u] [-T] [-C] [-K] [-U] [-a] [-I header]
13                probe [probe ...]
14

DESCRIPTION

16       trace  probes  functions  you  specify and displays trace messages if a
17       particular condition is met. You can control the message format to dis‐
18       play function arguments and return values.
19
20       Since this uses BPF, only the root user can use this tool.
21

REQUIREMENTS

23       CONFIG_BPF and bcc.
24

OPTIONS

26       -h     Print usage message.
27
28       -p PID Trace only functions in the process PID.
29
30       -L TID Trace only functions in the thread TID.
31
32       -v     Display the generated BPF program, for debugging purposes.
33
34       -z STRING_SIZE
35              When  collecting string arguments (of type char*), collect up to
36              STRING_SIZE characters. Longer strings will be truncated.
37
38       -s SYM_FILE_LIST
39              When collecting stack trace in build id  format,  use  the  coma
40              separated list for symbol resolution.
41
42       -S     If  set,  trace  messages  from trace's own process. By default,
43              this is off to avoid tracing storms -- for example, if you trace
44              the write system call, and consider that trace is writing to the
45              standard output.
46
47       -M MAX_EVENTS
48              Print up to MAX_EVENTS trace messages and then exit.
49
50       -t     Print times relative to the beginning of the trace (offsets), in
51              seconds.
52
53       -u     Print  UNIX  timestamps instead of offsets from trace beginning,
54              requires -t.
55
56       -T     Print the time column.
57
58       -C     Print CPU id.
59
60       -c CGROUP_PATH
61              Trace only functions in processes under CGROUP_PATH hierarchy.
62
63       -n NAME
64              Only print process names containing this name.
65
66       -f MSG_FILTER
67              Only print message of event containing this string.
68
69       -B     Treat argument of STRCMP helper as a binary value
70
71       -K     Print the kernel stack for each event.
72
73       -U     Print the user stack for each event.  -a Print  virtual  address
74              in kernel and user stacks.
75
76       -I header
77              Additional  header  files to include in the BPF program. This is
78              needed if your filter or print expressions  use  types  or  data
79              structures  that  are not available in the standard headers. For
80              example: 'linux/mm.h'
81
82       probe [probe ...]
83              One or more probes that attach to functions, filter  conditions,
84              and print information. See PROBE SYNTAX below.
85

PROBE SYNTAX

87       The general probe syntax is as follows:
88
89       [{p,r}]:[library]:function[+offset][(signature)] [(predicate)] ["format
90       string"[, arguments]]
91
92       {t:category:event,u:library:probe}  [(predicate)]  ["format   string"[,
93       arguments]]
94
95       {[{p,r}],t,u}
96              Probe  type  -  "p" for function entry, "r" for function return,
97              "t" for kernel tracepoint, "u" for USDT probe. The default probe
98              type is "p".
99
100       [library]
101              Library  containing the probe.  Specify the full path to the .so
102              or executable file where the function to probe resides. Alterna‐
103              tively,  you  can  specify  just  the lib name: for example, "c"
104              refers to libc. If no library name is specified, the  kernel  is
105              assumed.  Also,  you  can  specify an executable name (without a
106              full path) if it is in the PATH.  For example, "bash".
107
108       category
109              The tracepoint category. For example, "sched" or "irq".
110
111       function
112              The function to probe.  offset The offset after the  address  of
113              the  function  where  the  probe  should  injected.  For example
114              "kfree_skb+56" in decimal or hexadecimal  "kfree_skb+0x38"  for‐
115              mat.  Only works with kprobes and uprobes. Zero if omitted.
116
117       signature
118              The  optional  signature of the function to probe. This can make
119              it easier to access the function's arguments, instead  of  using
120              the  "arg1",  "arg2"  etc.   argument  specifiers.  For example,
121              "(struct timespec *ts)" in the signature position lets  you  use
122              "ts" in the filter or print expressions.
123
124       event  The tracepoint event. For example, "block_rq_complete".
125
126       probe  The USDT probe name. For example, "pthread_create".
127
128       [(predicate)]
129              The  filter  applied  to  the  captured data. Only if the filter
130              evaluates as true, the trace message will be printed. The filter
131              can  use any valid C expression that refers to the argument val‐
132              ues: arg1, arg2, etc., or to the return value retval in a return
133              probe.  If  necessary,  use C cast operators to coerce the argu‐
134              ments to the desired type. For example, if arg1 is of type  int,
135              use  the  expression  ((int)arg1  < 0) to trace only invocations
136              where arg1 is negative.  Note that only arg1-arg6 are supported,
137              and only if the function is using the standard x86_64 convention
138              where the first six arguments are in the RDI, RSI, RDX, RCX, R8,
139              R9 registers. If no predicate is specified, all function invoca‐
140              tions are traced.
141
142              The predicate expression may also use the STRCMP pseudo-function
143              to  compare  a predefined string to a string argument. For exam‐
144              ple: STRCMP("test", arg1).  The order of arguments is important:
145              the first argument MUST be a quoted literal string, and the sec‐
146              ond argument can be a runtime string, most  typically  an  argu‐
147              ment.
148
149       ["format string"[, arguments]]
150              A  printf-style  format  string  that will be used for the trace
151              message. You can use the following format  specifiers:  %s,  %d,
152              %u,  %lld,  %llu, %hd, %hu, %c, %x, %llx -- with the same seman‐
153              tics as printf's. Make sure to pass the exact  number  of  argu‐
154              ments as there are placeholders in the format string. The format
155              specifier replacements may be any C expressions, and  may  refer
156              to  the  same  special keywords as in the predicate (arg1, arg2,
157              etc.).
158
159              In addition to the above format specifiers, you can also use  %K
160              and %U when the expression is an address that potentially points
161              to executable code (i.e., a symbol). trace will resolve %K spec‐
162              ifiers  to  a kernel symbol, such as vfs__read, and will resolve
163              %U specifiers to a user-space symbol in that  process,  such  as
164              sprintf.
165
166              In  tracepoints,  both the predicate and the arguments may refer
167              to the tracepoint format structure, which is stored in the  spe‐
168              cial  "args"  variable. For example, the block:block_rq_complete
169              tracepoint can print or filter by args->nr_sector.  To  discover
170              the format of your tracepoint, use the tplist tool.
171
172              In  USDT  probes,  the  arg1,  ...,  argN variables refer to the
173              probe's arguments.  To determine which arguments your probe has,
174              use the tplist tool.
175
176              The  predicate  expression and the format specifier replacements
177              for printing may also use the following special keywords:  $pid,
178              $tgid  to refer to the current process' pid and tgid; $uid, $gid
179              to refer to the current user's uid and gid; $cpu to refer to the
180              current processor number.
181

EXAMPLES

183       Trace all invocations of the open system call with the name of the file
184       being opened:
185              # trace '::do_sys_open "%s", arg2'
186
187       Trace all invocations of the read system call where the number of bytes
188       requested is greater than 20,000:
189              # trace '::sys_read (arg3 > 20000) "read %d bytes", arg3'
190
191       Trace all malloc calls and print the size of the requested allocation:
192              # trace ':c:malloc "size = %d", arg1'
193
194       Trace  returns  from the readline function in bash and print the return
195       value as a string:
196              # trace 'r:bash:readline "%s", retval'
197
198       Trace the block:block_rq_complete tracepoint and print  the  number  of
199       sectors completed:
200              #  trace  't:block:block_rq_complete "%d sectors", args->nr_sec‐
201              tor'
202
203       Trace the pthread_create USDT probe from the pthread library and  print
204       the address of the thread's start function:
205              # trace 'u:pthread:pthread_create "start addr = %llx", arg3'
206
207       Trace  the  nanosleep  system  call  and  print  the  sleep duration in
208       nanoseconds:
209              # trace 'p::SyS_nanosleep(struct timespec *ts) sleep for %lld ns
210              , ts->tv_nsec'
211
212       Trace  the inet_pton system call using build id mechanism and print the
213       stack
214              #     trace     -s     /lib/x86_64-linux-gnu/libc.so.6,/bin/ping
215              'p:c:inet_pton' -U
216

SOURCE

218       This is from bcc.
219
220              https://github.com/iovisor/bcc
221
222       Also  look  in  the bcc distribution for a companion _examples.txt file
223       containing example usage, output, and commentary for this tool.
224

OS

226       Linux
227

STABILITY

229       Unstable - in development.
230

AUTHOR

232       Sasha Goldshtein
233
234
235
236USER COMMANDS                     2016-02-18                          trace(8)
Impressum