1STRACE(1) General Commands Manual STRACE(1)
2
6 strace - trace system calls and signals
7
9 strace [-CdffhikqrtttTvVxxy] [-In] [-bexecve] [-eexpr]... [-acolumn]
10 [-ofile] [-sstrsize] [-Ppath]... -ppid... / [-D] [-Evar[=val]]...
11 [-uusername] command [args]
12 strace -c[df] [-In] [-bexecve] [-eexpr]... [-Ooverhead] [-Ssortby]
14 -ppid... / [-D] [-Evar[=val]]... [-uusername] command [args]
15
17 In the simplest case strace runs the specified command until it exits.
18 It intercepts and records the system calls which are called by a
19 process and the signals which are received by a process. The name of
20 each system call, its arguments and its return value are printed on
21 standard error or to the file specified with the -o option.
22 strace is a useful diagnostic, instructional, and debugging tool. Sys‐
24 tem administrators, diagnosticians and trouble-shooters will find it
25 invaluable for solving problems with programs for which the source is
26 not readily available since they do not need to be recompiled in order
27 to trace them. Students, hackers and the overly-curious will find that
28 a great deal can be learned about a system and its system calls by
29 tracing even ordinary programs. And programmers will find that since
30 system calls and signals are events that happen at the user/kernel
31 interface, a close examination of this boundary is very useful for bug
32 isolation, sanity checking and attempting to capture race conditions.
33 Each line in the trace contains the system call name, followed by its
35 arguments in parentheses and its return value. An example from strac‐
36 ing the command "cat /dev/null" is:
37 open("/dev/null", O_RDONLY) = 3
39 Errors (typically a return value of -1) have the errno symbol and error
41 string appended.
42 open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)
44 Signals are printed as signal symbol and decoded siginfo structure. An
46 excerpt from stracing and interrupting the command "sleep 666" is:
47 sigsuspend([] <unfinished ...>
49 --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=...} ---
50 +++ killed by SIGINT +++
51 If a system call is being executed and meanwhile another one is being
53 called from a different thread/process then strace will try to preserve
54 the order of those events and mark the ongoing call as being unfin‐
55 ished. When the call returns it will be marked as resumed.
56 [pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>
58 [pid 28779] clock_gettime(CLOCK_REALTIME, {1130322148, 939977000}) = 0
59 [pid 28772] <... select resumed> ) = 1 (in [3])
60 Interruption of a (restartable) system call by a signal delivery is
62 processed differently as kernel terminates the system call and also
63 arranges its immediate reexecution after the signal handler completes.
64 read(0, 0x7ffff72cf5cf, 1) = ? ERESTARTSYS (To be restarted)
66 --- SIGALRM ... ---
67 rt_sigreturn(0xe) = 0
68 read(0, "", 1) = 0
69 Arguments are printed in symbolic form with a passion. This example
71 shows the shell performing ">>xyzzy" output redirection:
72 open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3
74 Here the third argument of open is decoded by breaking down the flag
76 argument into its three bitwise-OR constituents and printing the mode
77 value in octal by tradition. Where traditional or native usage differs
78 from ANSI or POSIX, the latter forms are preferred. In some cases,
79 strace output has proven to be more readable than the source.
80 Structure pointers are dereferenced and the members are displayed as
82 appropriate. In all cases arguments are formatted in the most C-like
83 fashion possible. For example, the essence of the command "ls -l
84 /dev/null" is captured as:
85 lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
87 Notice how the 'struct stat' argument is dereferenced and how each mem‐
89 ber is displayed symbolically. In particular, observe how the st_mode
90 member is carefully decoded into a bitwise-OR of symbolic and numeric
91 values. Also notice in this example that the first argument to lstat
92 is an input to the system call and the second argument is an output.
93 Since output arguments are not modified if the system call fails, argu‐
94 ments may not always be dereferenced. For example, retrying the "ls
95 -l" example with a non-existent file produces the following line:
96 lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)
98 In this case the porch light is on but nobody is home.
100 Character pointers are dereferenced and printed as C strings. Non-
102 printing characters in strings are normally represented by ordinary C
103 escape codes. Only the first strsize (32 by default) bytes of strings
104 are printed; longer strings have an ellipsis appended following the
105 closing quote. Here is a line from "ls -l" where the getpwuid library
106 routine is reading the password file:
107 read(3, "root::0:0:System Administrator:/"..., 1024) = 422
109 While structures are annotated using curly braces, simple pointers and
111 arrays are printed using square brackets with commas separating ele‐
112 ments. Here is an example from the command "id" on a system with sup‐
113 plementary group ids:
114 getgroups(32, [100, 0]) = 2
116 On the other hand, bit-sets are also shown using square brackets but
118 set elements are separated only by a space. Here is the shell prepar‐
119 ing to execute an external command:
120 sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0
122 Here the second argument is a bit-set of two signals, SIGCHLD and SIGT‐
124 TOU. In some cases the bit-set is so full that printing out the unset
125 elements is more valuable. In that case, the bit-set is prefixed by a
126 tilde like this:
127 sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0
129 Here the second argument represents the full set of all signals.
131
133 -c Count time, calls, and errors for each system call
134 and report a summary on program exit. On Linux,
135 this attempts to show system time (CPU time spent
136 running in the kernel) independent of wall clock
137 time. If -c is used with -f or -F (below), only
138 aggregate totals for all traced processes are kept.
139 -C Like -c but also print regular output while pro‐
141 cesses are running.
142 -D Run tracer process as a detached grandchild, not as
144 parent of the tracee. This reduces the visible
145 effect of strace by keeping the tracee a direct
146 child of the calling process.
147 -d Show some debugging output of strace itself on the
149 standard error.
150 -f Trace child processes as they are created by cur‐
152 rently traced processes as a result of the fork(2),
153 vfork(2) and clone(2) system calls. Note that -p
154 PID -f will attach all threads of process PID if it
155 is multi-threaded, not only thread with thread_id =
156 PID.
157 -ff If the -o filename option is in effect, each pro‐
159 cesses trace is written to filename.pid where pid is
160 the numeric process id of each process. This is
161 incompatible with -c, since no per-process counts
162 are kept.
163 -F This option is now obsolete and it has the same
165 functionality as -f.
166 -h Print the help summary.
168 -i Print the instruction pointer at the time of the
170 system call.
171 -k Print the execution stack trace of the traced pro‐
173 cesses after each system call (experimental). This
174 option is available only if strace is built with
175 libunwind.
176 -q Suppress messages about attaching, detaching etc.
178 This happens automatically when output is redirected
179 to a file and the command is run directly instead of
180 attaching.
181 -qq If given twice, suppress messages about process exit
183 status.
184 -r Print a relative timestamp upon entry to each system
186 call. This records the time difference between the
187 beginning of successive system calls.
188 -t Prefix each line of the trace with the time of day.
190 -tt If given twice, the time printed will include the
192 microseconds.
193 -ttt If given thrice, the time printed will include the
195 microseconds and the leading portion will be printed
196 as the number of seconds since the epoch.
197 -T Show the time spent in system calls. This records
199 the time difference between the beginning and the
200 end of each system call.
201 -w Summarise the time difference between the beginning
203 and end of each system call. The default is to sum‐
204 marise the system time.
205 -v Print unabbreviated versions of environment, stat,
207 termios, etc. calls. These structures are very
208 common in calls and so the default behavior displays
209 a reasonable subset of structure members. Use this
210 option to get all of the gory details.
211 -V Print the version number of strace.
213 -x Print all non-ASCII strings in hexadecimal string
215 format.
216 -xx Print all strings in hexadecimal string format.
218 -y Print paths associated with file descriptor argu‐
220 ments.
221 -yy Print protocol specific information associated with
223 socket file descriptors.
224 -a column Align return values in a specific column (default
226 column 40).
227 -b syscall If specified syscall is reached, detach from traced
229 process. Currently, only execve syscall is sup‐
230 ported. This option is useful if you want to trace
231 multi-threaded process and therefore require -f, but
232 don't want to trace its (potentially very complex)
233 children.
234 -e expr A qualifying expression which modifies which events
236 to trace or how to trace them. The format of the
237 expression is:
238 [qualifier=][!]value1[,value2]...
240 where qualifier is one of trace, abbrev, verbose,
242 raw, signal, read, or write and value is a quali‐
243 fier-dependent symbol or number. The default quali‐
244 fier is trace. Using an exclamation mark negates
245 the set of values. For example, -e open means lit‐
246 erally -e trace=open which in turn means trace only
247 the open system call. By contrast, -e trace=!open
248 means to trace every system call except open. In
249 addition, the special values all and none have the
250 obvious meanings.
251 Note that some shells use the exclamation point for
253 history expansion even inside quoted arguments. If
254 so, you must escape the exclamation point with a
255 backslash.
256 -e trace=set
258 Trace only the specified set of system calls. The
259 -c option is useful for determining which system
260 calls might be useful to trace. For example,
261 trace=open,close,read,write means to only trace
262 those four system calls. Be careful when making
263 inferences about the user/kernel boundary if only a
264 subset of system calls are being monitored. The
265 default is trace=all.
266 -e trace=file
268 Trace all system calls which take a file name as an
269 argument. You can think of this as an abbreviation
270 for -e trace=open,stat,chmod,unlink,... which is
271 useful to seeing what files the process is referenc‐
272 ing. Furthermore, using the abbreviation will
273 ensure that you don't accidentally forget to include
274 a call like lstat in the list. Betchya woulda for‐
275 got that one.
276 -e trace=process
278 Trace all system calls which involve process manage‐
279 ment. This is useful for watching the fork, wait,
280 and exec steps of a process.
281 -e trace=network
283 Trace all the network related system calls.
284 -e trace=signal
286 Trace all signal related system calls.
287 -e trace=ipc
289 Trace all IPC related system calls.
290 -e trace=desc
292 Trace all file descriptor related system calls.
293 -e trace=memory
295 Trace all memory mapping related system calls.
296 -e abbrev=set
298 Abbreviate the output from printing each member of
299 large structures. The default is abbrev=all. The
300 -v option has the effect of abbrev=none.
301 -e verbose=set
303 Dereference structures for the specified set of sys‐
304 tem calls. The default is verbose=all.
305 -e raw=set Print raw, undecoded arguments for the specified set
307 of system calls. This option has the effect of
308 causing all arguments to be printed in hexadecimal.
309 This is mostly useful if you don't trust the decod‐
310 ing or you need to know the actual numeric value of
311 an argument.
312 -e signal=set
314 Trace only the specified subset of signals. The
315 default is signal=all. For example, signal =! SIGIO
316 (or signal=!io) causes SIGIO signals not to be
317 traced.
318 -e read=set Perform a full hexadecimal and ASCII dump of all the
320 data read from file descriptors listed in the speci‐
321 fied set. For example, to see all input activity on
322 file descriptors 3 and 5 use -e read=3,5. Note that
323 this is independent from the normal tracing of the
324 read(2) system call which is controlled by the
325 option -e trace=read.
326 -e write=set
328 Perform a full hexadecimal and ASCII dump of all the
329 data written to file descriptors listed in the spec‐
330 ified set. For example, to see all output activity
331 on file descriptors 3 and 5 use -e write=3,5. Note
332 that this is independent from the normal tracing of
333 the write(2) system call which is controlled by the
334 option -e trace=write.
335 -I interruptible
337 When strace can be interrupted by signals (such as
338 pressing ^C). 1: no signals are blocked; 2: fatal
339 signals are blocked while decoding syscall
340 (default); 3: fatal signals are always blocked
341 (default if '-o FILE PROG'); 4: fatal signals and
342 SIGTSTP (^Z) are always blocked (useful to make
343 strace -o FILE PROG not stop on ^Z).
344 -o filename Write the trace output to the file filename rather
346 than to stderr. Use filename.pid if -ff is used.
347 If the argument begins with '|' or with '!' then the
348 rest of the argument is treated as a command and all
349 output is piped to it. This is convenient for pip‐
350 ing the debugging output to a program without
351 affecting the redirections of executed programs.
352 -O overhead Set the overhead for tracing system calls to over‐
354 head microseconds. This is useful for overriding
355 the default heuristic for guessing how much time is
356 spent in mere measuring when timing system calls
357 using the -c option. The accuracy of the heuristic
358 can be gauged by timing a given program run without
359 tracing (using time(1)) and comparing the accumu‐
360 lated system call time to the total produced using
361 -c.
362 -p pid Attach to the process with the process ID pid and
364 begin tracing. The trace may be terminated at any
365 time by a keyboard interrupt signal (CTRL-C).
366 strace will respond by detaching itself from the
367 traced process(es) leaving it (them) to continue
368 running. Multiple -p options can be used to attach
369 to many processes in addition to command (which is
370 optional if at least one -p option is given). -p
371 "`pidof PROG`" syntax is supported.
372 -P path Trace only system calls accessing path. Multiple -P
374 options can be used to specify several paths.
375 -s strsize Specify the maximum string size to print (the
377 default is 32). Note that filenames are not consid‐
378 ered strings and are always printed in full.
379 -S sortby Sort the output of the histogram printed by the -c
381 option by the specified criterion. Legal values are
382 time, calls, name, and nothing (default is time).
383 -u username Run command with the user ID, group ID, and supple‐
385 mentary groups of username. This option is only
386 useful when running as root and enables the correct
387 execution of setuid and/or setgid binaries. Unless
388 this option is used setuid and setgid programs are
389 executed without effective privileges.
390 -E var=val Run command with var=val in its list of environment
392 variables.
393 -E var Remove var from the inherited list of environment
395 variables before passing it on to the command.
396
398 When command exits, strace exits with the same exit status. If
399 command is terminated by a signal, strace terminates itself with
400 the same signal, so that strace can be used as a wrapper process
401 transparent to the invoking parent process. Note that parent-
402 child relationship (signal stop notifications, getppid() value,
403 etc) between traced process and its parent are not preserved
404 unless -D is used.
405 When using -p without a command, the exit status of strace is
407 zero unless no processes has been attached or there was an unex‐
408 pected error in doing the tracing.
409
411 If strace is installed setuid to root then the invoking user
412 will be able to attach to and trace processes owned by any user.
413 In addition setuid and setgid programs will be executed and
414 traced with the correct effective privileges. Since only users
415 trusted with full root privileges should be allowed to do these
416 things, it only makes sense to install strace as setuid to root
417 when the users who can execute it are restricted to those users
418 who have this trust. For example, it makes sense to install a
419 special version of strace with mode 'rwsr-xr--', user root and
420 group trace, where members of the trace group are trusted users.
421 If you do use this feature, please remember to install a non-
422 setuid version of strace for ordinary lusers to use.
423
425 ltrace(1), time(1), ptrace(2), proc(5)
426
428 It is a pity that so much tracing clutter is produced by systems
429 employing shared libraries.
430 It is instructive to think about system call inputs and outputs
432 as data-flow across the user/kernel boundary. Because user-
433 space and kernel-space are separate and address-protected, it is
434 sometimes possible to make deductive inferences about process
435 behavior using inputs and outputs as propositions.
436 In some cases, a system call will differ from the documented
438 behavior or have a different name. For example, on System V-
439 derived systems the true time(2) system call does not take an
440 argument and the stat function is called xstat and takes an
441 extra leading argument. These discrepancies are normal but
442 idiosyncratic characteristics of the system call interface and
443 are accounted for by C library wrapper functions.
444 Some system calls have different names in different architec‐
446 tures and personalities. In these cases, system call filtering
447 and printing uses the names that match corresponding __NR_* ker‐
448 nel macros of the tracee's architecture and personality. There
449 are two exceptions from this general rule: arm_fadvise64_64(2)
450 ARM syscall and xtensa_fadvise64_64(2) Xtensa syscall are fil‐
451 tered and printed as fadvise64_64(2).
452 On some platforms a process that is attached to with the -p
454 option may observe a spurious EINTR return from the current sys‐
455 tem call that is not restartable. (Ideally, all system calls
456 should be restarted on strace attach, making the attach invisi‐
457 ble to the traced process, but a few system calls aren't.
458 Arguably, every instance of such behavior is a kernel bug.)
459 This may have an unpredictable effect on the process if the
460 process takes no action to restart the system call.
461
463 Programs that use the setuid bit do not have effective user ID
464 privileges while being traced.
465 A traced process runs slowly.
467 Traced processes which are descended from command may be left
469 running after an interrupt signal (CTRL-C).
470 The -i option is weakly supported.
472
474 The original strace was written by Paul Kranenburg for SunOS and
475 was inspired by its trace utility. The SunOS version of strace
476 was ported to Linux and enhanced by Branko Lankester, who also
477 wrote the Linux kernel support. Even though Paul released
478 strace 2.5 in 1992, Branko's work was based on Paul's strace 1.5
479 release from 1991. In 1993, Rick Sladkey merged strace 2.5 for
480 SunOS and the second release of strace for Linux, added many of
481 the features of truss(1) from SVR4, and produced an strace that
482 worked on both platforms. In 1994 Rick ported strace to SVR4
483 and Solaris and wrote the automatic configuration support. In
484 1995 he ported strace to Irix and tired of writing about himself
485 in the third person.
486
488 Problems with strace should be reported to the strace mailing
489 list at <strace-devel@lists.sourceforge.net>.
490 2010-03-30 STRACE(1)