1TRACE-CMD-RECORD(1) [FIXME: manual] TRACE-CMD-RECORD(1)
2
6 trace-cmd-record - record a trace from the Ftrace Linux internal tracer
7
9 trace-cmd record [OPTIONS] [command]
10
12 The trace-cmd(1) record command will set up the Ftrace Linux kernel
13 tracer to record the specified plugins or events that happen while the
14 command executes. If no command is given, then it will record until the
15 user hits Ctrl-C.
16 The record command of trace-cmd will set up the Ftrace tracer to start
18 tracing the various events or plugins that are given on the command
19 line. It will then create a number of tracing processes (one per CPU)
20 that will start recording from the kernel ring buffer straight into
21 temporary files. When the command is complete (or Ctrl-C is hit) all
22 the files will be combined into a trace.dat file that can later be read
23 (see trace-cmd-report(1)).
24
26 -p plugin
27 Specify a trace plugin. Plugins are special Ftrace tracers that
28 usually do more than just trace an event. Common plugins are
29 function, function_graph, preemptirqsoff, irqsoff, preemptoff, and
30 wakeup. A plugin must be supported by the running kernel. To see a
31 list of available plugins, see trace-cmd-list(1).
32 -e event
34 Specify an event to trace. Various static trace points have been
35 added to the Linux kernel. They are grouped by subsystem where you
36 can enable all events of a given subsystem or specify specific
37 events to be enabled. The event is of the format
38 "subsystem:event-name". You can also just specify the subsystem
39 without the :event-name or the event-name without the "subsystem:".
40 Using "-e sched_switch" will enable the "sched_switch" event where
41 as, "-e sched" will enable all events under the "sched" subsystem.
42 The ´event´ can also contain glob expressions. That is, "*stat*" will
44 select all events (or subsystems) that have the characters "stat" in their
45 names.
46 The keyword ´all´ can be used to enable all events.
48 -a
50 Every event that is being recorded has its output format file saved
51 in the output file to be able to display it later. But if other
52 events are enabled in the trace without trace-cmd’s knowledge, the
53 formats of those events will not be recorded and trace-cmd report
54 will not be able to display them. If this is the case, then specify
55 the -a option and the format for all events in the system will be
56 saved.
57 -T
59 Enable a stacktrace on each event. For example:
60 <idle>-0 [003] 58549.289091: sched_switch: kworker/0:1:0 [120] R ==> trace-cmd:2603 [120]
62 <idle>-0 [003] 58549.289092: kernel_stack: <stack trace>
63 => schedule (ffffffff814b260e)
64 => cpu_idle (ffffffff8100a38c)
65 => start_secondary (ffffffff814ab828)
66 --func-stack
68 Enable a stack trace on all functions. Note this is only applicable
69 for the "function" plugin tracer, and will only take effect if the
70 -l option is used and succeeds in limiting functions. If the
71 function tracer is not filtered, and the stack trace is enabled,
72 you can live lock the machine.
73 -f filter
75 Specify a filter for the previous event. This must come after a -e.
76 This will filter what events get recorded based on the content of
77 the event. Filtering is passed to the kernel directly so what
78 filtering is allowed may depend on what version of the kernel you
79 have. Basically, it will let you use C notation to check if an
80 event should be processed or not.
81 ==, >=, <=, >, <, &, |, && and ||
83 The above are usually safe to use to compare fields.
85 -v
87 This will cause all events specified after it on the command line
88 to not be traced. This is useful for selecting a subsystem to be
89 traced but to leave out various events. For Example: "-e sched -v
90 -e "*stat\*"" will enable all events in the sched subsystem except
91 those that have "stat" in their names.
92 Note: the *-v* option was taken from the way grep(1) inverts the following
94 matches.
95 -F
97 This will filter only the executable that is given on the command
98 line. If no command is given, then it will filter itself (pretty
99 pointless). Using -F will let you trace only events that are caused
100 by the given command.
101 -P pid
103 Similar to -F but lets you specify a process ID to trace.
104 -c
106 Used with either -F to trace the process´ children too.
107 -o output-file
109 By default, trace-cmd report will create a trace.dat file. You can
110 specify a different file to write to with the -o option.
111 -l function-name
113 This will limit the function and function_graph tracers to only
114 trace the given function name. More than one -l may be specified on
115 the command line to trace more than one function. The limited use
116 of glob expressions are also allowed. These are match* to only
117 filter functions that start with match. *match to only filter
118 functions that end with match. *match\* to only filter on
119 functions that contain match.
120 -g function-name
122 This option is for the function_graph plugin. It will graph the
123 given function. That is, it will only trace the function and all
124 functions that it calls. You can have more than one -g on the
125 command line.
126 -n function-name
128 This has the opposite effect of -l. The function given with the -n
129 option will not be traced. This takes precedence, that is, if you
130 include the same function for both -n and -l, it will not be
131 traced.
132 -d
134 Some tracer plugins enable the function tracer by default. Like the
135 latency tracers. This option prevents the function tracer from
136 being enabled at start up.
137 -O option
139 Ftrace has various options that can be enabled or disabled. This
140 allows you to set them. Appending the text no to an option disables
141 it. For example: "-O nograph-time" will disable the "graph-time"
142 Ftrace option.
143 -s interval
145 The processes that trace-cmd creates to record from the ring buffer
146 need to wake up to do the recording. Setting the interval to zero
147 will cause the processes to wakeup every time new data is written
148 into the buffer. But since Ftrace is recording kernel activity, the
149 act of this processes going back to sleep may cause new events into
150 the ring buffer which will wake the process back up. This will
151 needlessly add extra data into the ring buffer.
152 The ´interval´ metric is microseconds. The default is set to 1000 (1 ms).
154 This is the time each recording process will sleep before waking up to
155 record any new data that was written to the ring buffer.
156 -r priority
158 The priority to run the capture threads at. In a busy system the
159 trace capturing threads may be staved and events can be lost. This
160 increases the priority of those threads to the real time (FIFO)
161 priority. But use this option with care, it can also change the
162 behaviour of the system being traced.
163 -b size
165 This sets the ring buffer size to size kilobytes. Because the
166 Ftrace ring buffer is per CPU, this size is the size of each per
167 CPU ring buffer inside the kernel. Using "-b 10000" on a machine
168 with 4 CPUs will make Ftrace have a total buffer size of 40 Megs.
169 -k
171 By default, when trace-cmd is finished tracing, it will reset the
172 buffers and disable all the tracing that it enabled. This option
173 keeps trace-cmd from disabling the tracer and reseting the buffer.
174 This option is useful for debugging trace-cmd.
175 Note: usually trace-cmd will set the "tracing_on" file back to what it
177 was before it was called. This option will leave that file set to zero.
178 -i
180 By default, if an event is listed that trace-cmd does not find, it
181 will exit with an error. This option will just ignore events that
182 are listed on the command line but are not found on the system.
183 -N host:port
185 If another machine is running "trace-cmd listen", this option is
186 used to have the data sent to that machine with UDP packets.
187 Instead of writing to an output file, the data is sent off to a
188 remote box. This is ideal for embedded machines with little
189 storage, or having a single machine that will keep all the data in
190 a single repository.
191 Note: This option is not supported with latency tracer plugins:
193 wakeup, wakeup_rt, irqsoff, preemptoff and preemptirqsoff
194 -t
196 This option is used with -N, when there’s a need to send the live
197 data with TCP packets instead of UDP. Although TCP is not nearly as
198 fast as sending the UDP packets, but it may be needed if the
199 network is not that reliable, the amount of data is not that
200 intensive, and a guarantee is needed that all traced information is
201 transfered successfully.
202 --date
204 With the --date option, "trace-cmd" will write timestamps into the
205 trace buffer after it has finished recording. It will then map the
206 timestamp to gettimeofday which will allow wall time output from
207 the timestamps reading the created trace.dat file.
208
210 The basic way to trace all events:
211 # trace-cmd record -e all ls > /dev/null
213 # trace-cmd report
214 trace-cmd-13541 [003] 106260.693809: filemap_fault: address=0x128122 offset=0xce
215 trace-cmd-13543 [001] 106260.693809: kmalloc: call_site=81128dd4 ptr=0xffff88003dd83800 bytes_req=768 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_ZERO
216 ls-13545 [002] 106260.693809: kfree: call_site=810a7abb ptr=0x0
217 ls-13545 [002] 106260.693818: sys_exit_write: 0x1
218 To use the function tracer with sched switch tracing:
220 # trace-cmd record -p function -e sched_switch ls > /dev/null
222 # trace-cmd report
223 ls-13587 [002] 106467.860310: function: hrtick_start_fair <-- pick_next_task_fair
224 ls-13587 [002] 106467.860313: sched_switch: prev_comm=trace-cmd prev_pid=13587 prev_prio=120 prev_state=R ==> next_comm=trace-cmd next_pid=13583 next_prio=120
225 trace-cmd-13585 [001] 106467.860314: function: native_set_pte_at <-- __do_fault
226 trace-cmd-13586 [003] 106467.860314: function: up_read <-- do_page_fault
227 ls-13587 [002] 106467.860317: function: __phys_addr <-- schedule
228 trace-cmd-13585 [001] 106467.860318: function: _raw_spin_unlock <-- __do_fault
229 ls-13587 [002] 106467.860320: function: native_load_sp0 <-- __switch_to
230 trace-cmd-13586 [003] 106467.860322: function: down_read_trylock <-- do_page_fault
231 Here is a nice way to find what interrupts have the highest latency:
233 # trace-cmd record -p function_graph -e irq_handler_entry -l do_IRQ sleep 10
235 # trace-cmd report
236 <idle>-0 [000] 157412.933969: funcgraph_entry: | do_IRQ() {
237 <idle>-0 [000] 157412.933974: irq_handler_entry: irq=48 name=eth0
238 <idle>-0 [000] 157412.934004: funcgraph_exit: + 36.358 us | }
239 <idle>-0 [000] 157413.895004: funcgraph_entry: | do_IRQ() {
240 <idle>-0 [000] 157413.895011: irq_handler_entry: irq=48 name=eth0
241 <idle>-0 [000] 157413.895026: funcgraph_exit: + 24.014 us | }
242 <idle>-0 [000] 157415.891762: funcgraph_entry: | do_IRQ() {
243 <idle>-0 [000] 157415.891769: irq_handler_entry: irq=48 name=eth0
244 <idle>-0 [000] 157415.891784: funcgraph_exit: + 22.928 us | }
245 <idle>-0 [000] 157415.934869: funcgraph_entry: | do_IRQ() {
246 <idle>-0 [000] 157415.934874: irq_handler_entry: irq=48 name=eth0
247 <idle>-0 [000] 157415.934906: funcgraph_exit: + 37.512 us | }
248 <idle>-0 [000] 157417.888373: funcgraph_entry: | do_IRQ() {
249 <idle>-0 [000] 157417.888381: irq_handler_entry: irq=48 name=eth0
250 <idle>-0 [000] 157417.888398: funcgraph_exit: + 25.943 us | }
251
253 trace-cmd(1), trace-cmd-report(1), trace-cmd-start(1),
254 trace-cmd-stop(1), trace-cmd-extract(1), trace-cmd-reset(1),
255 trace-cmd-split(1), trace-cmd-list(1), trace-cmd-listen(1)
256
258 Written by Steven Rostedt, <rostedt@goodmis.org[1]>
259
261 git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
262
264 Copyright (C) 2010 Red Hat, Inc. Free use of this software is granted
265 under the terms of the GNU Public License (GPL).
266
268 1. rostedt@goodmis.org
269 mailto:rostedt@goodmis.org
270[FIXME: source] 03/23/2017 TRACE-CMD-RECORD(1)