1RTLA-TIMERLAT-HIST(1) RTLA-TIMERLAT-HIST(1)
2
6 rtla-timerlat-hist - Histograms of the operating system timer latency
7
9 rtla timerlat hist [OPTIONS] ...
10
12 The rtla timerlat tool is an interface for the timerlat tracer. The
13 timerlat tracer dispatches a kernel thread per-cpu. These threads set a
14 periodic timer to wake themselves up and go back to sleep. After the
15 wakeup, they collect and generate useful information for the debugging
16 of operating system timer latency.
17 The timerlat tracer outputs information in two ways. It periodically
19 prints the timer latency at the timer IRQ handler and the Thread han‐
20 dler. It also enable the trace of the most relevant information via os‐
21 noise: tracepoints.
22 The rtla timerlat hist displays a histogram of each tracer event occur‐
24 rence. This tool uses the periodic information, and the osnoise: trace‐
25 points are enabled when using the -T option.
26
28 -a, --auto us
29 Set the automatic trace mode. This mode sets some commonly used op‐
30 tions while debugging the system. It is equivalent to use -T us -s
31 us -t. By default, timerlat tracer uses FIFO:95 for timerlat
32 threads, thus equilavent to -P f:95.
33 -p, --period us
35 Set the timerlat tracer period in microseconds.
36 -i, --irq us
38 Stop trace if the IRQ latency is higher than the argument in us.
39 -T, --thread us
41 Stop trace if the Thread latency is higher than the argument in us.
42 -s, --stack us
44 Save the stack trace at the IRQ if a Thread latency is higher than
45 the argument in us.
46 --dma-latency us
48 Set the /dev/cpu_dma_latency to us, aiming to bound exit from
49 idle latencies. cyclictest sets this value to 0 by default, use
50 --dma-latency 0 to have similar results.
51 -b, --bucket-size N
53 Set the histogram bucket size (default 1).
54 -E, --entries N
56 Set the number of entries of the histogram (default 256).
57 --no-header
59 Do not print header.
60 --no-summary
62 Do not print summary.
63 --no-index
65 Do not print index.
66 --with-zeros
68 Print zero only entries.
69 -c, --cpus cpu-list
71 Set the osnoise tracer to run the sample threads in the cpu-list.
72 -d, --duration time[s|m|h|d]
74 Set the duration of the session.
75 -D, --debug
77 Print debug info.
78 -t, --trace[=file]
80 Save the stopped trace to [file|osnoise_trace.txt].
81 -e, --event sys:event
83 Enable an event in the trace (-t) session. The argument can be a
84 specific event, e.g., -e sched:sched_switch, or all events of a sys‐
85 tem group, e.g., -e sched. Multiple -e are allowed. It is only ac‐
86 tive when -t or -a are set.
87 --filter <filter>
89 Filter the previous -e sys:event event with <filter>. For further
90 information about event filtering see
91 https://www.kernel.org/doc/html/latest/trace/events.html#event-filtering.
92 --trigger <trigger>
94 Enable a trace event trigger to the previous -e sys:event. If
95 the hist: trigger is activated, the output histogram will be au‐
96 tomatically saved to a file named system_event_hist.txt. For
97 example, the command:
98 rtla <command> <mode> -t -e osnoise:irq_noise --trig‐
100 ger="hist:key=desc,duration/1000:sort=desc,dura‐
101 tion/1000:vals=hitcount"
102 Will automatically save the content of the histogram associated
104 to osnoise:irq_noise event in osnoise_irq_noise_hist.txt.
105 For further information about event trigger see
107 https://www.kernel.org/doc/html/latest/trace/events.html#event-triggers.
108 -P, --priority o:prio|r:prio|f:prio|d:runtime:period
110 Set scheduling parameters to the osnoise tracer threads, the format
111 to set the priority are:
112 • o:prio - use SCHED_OTHER with prio;
114 • r:prio - use SCHED_RR with prio;
116 • f:prio - use SCHED_FIFO with prio;
118 • d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with run‐
120 time and period in nanoseconds.
121 -h, --help
123 Print help menu.
124
126 In the example below, rtla timerlat hist is set to run for 10 minutes,
127 in the cpus 0-4, skipping zero only lines. Moreover, rtla timerlat hist
128 will change the priority of the timelat threads to run under
129 SCHED_DEADLINE priority, with a 10us runtime every 1ms period. The 1ms
130 period is also passed to the timerlat tracer:
131 [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms
133 # RTLA timerlat histogram
134 # Time unit is microseconds (us)
135 # Duration: 0 00:10:00
136 Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 IRQ-004 Thr-004
137 0 276489 0 206089 0 466018 0 481102 0 205546 0
138 1 318327 35487 388149 30024 94531 48382 83082 71078 388026 55730
139 2 3282 122584 4019 126527 28231 109012 23311 89309 4568 98739
140 3 940 11815 837 9863 6209 16227 6895 17196 910 9780
141 4 444 17287 424 11574 2097 38443 2169 36736 462 13476
142 5 206 43291 255 25581 1223 101908 1304 101137 236 28913
143 6 132 101501 96 64584 635 213774 757 215471 99 73453
144 7 74 169347 65 124758 350 57466 441 53639 69 148573
145 8 53 85183 31 156751 229 9052 306 9026 39 139907
146 9 22 10387 12 42762 161 2554 225 2689 19 26192
147 10 13 1898 8 5770 114 1247 128 1405 13 3772
148 11 9 560 9 924 71 686 76 765 8 713
149 12 4 256 2 360 50 411 64 474 3 278
150 13 2 167 2 172 43 256 53 350 4 180
151 14 1 88 1 116 15 198 42 223 0 115
152 15 2 63 3 94 11 139 20 150 0 58
153 16 2 37 0 56 5 78 10 102 0 39
154 17 0 18 0 28 4 57 8 80 0 15
155 18 0 8 0 17 2 50 6 56 0 12
156 19 0 9 0 5 0 19 0 48 0 18
157 20 0 4 0 8 0 11 2 27 0 4
158 21 0 2 0 3 1 9 1 18 0 6
159 22 0 1 0 3 1 7 0 3 0 5
160 23 0 2 0 4 0 2 0 7 0 2
161 24 0 2 0 2 1 3 0 3 0 5
162 25 0 0 0 1 0 1 0 1 0 3
163 26 0 1 0 0 0 2 0 2 0 0
164 27 0 0 0 3 0 1 0 0 0 1
165 28 0 0 0 3 0 0 0 1 0 0
166 29 0 0 0 2 0 2 0 1 0 3
167 30 0 1 0 0 0 0 0 0 0 0
168 31 0 1 0 0 0 0 0 2 0 2
169 32 0 0 0 1 0 2 0 0 0 0
170 33 0 0 0 2 0 0 0 0 0 1
171 34 0 0 0 0 0 0 0 0 0 2
172 35 0 1 0 1 0 0 0 0 0 1
173 36 0 1 0 0 0 1 0 1 0 0
174 37 0 0 0 1 0 0 0 0 0 0
175 40 0 0 0 0 0 1 0 1 0 0
176 41 0 0 0 0 0 0 0 0 0 1
177 42 0 0 0 0 0 0 0 0 0 1
178 44 0 0 0 0 0 1 0 0 0 0
179 46 0 0 0 0 0 0 0 1 0 0
180 47 0 0 0 0 0 0 0 0 0 1
181 50 0 0 0 0 0 0 0 0 0 1
182 54 0 0 0 1 0 0 0 0 0 0
183 58 0 0 0 1 0 0 0 0 0 0
184 over: 0 0 0 0 0 0 0 0 0 0
185 count: 600002 600002 600002 600002 600002 600002 600002 600002 600002 600002
186 min: 0 1 0 1 0 1 0 1 0 1
187 avg: 0 5 0 5 0 4 0 4 0 5
188 max: 16 36 15 58 24 44 21 46 13 50
189
191 rtla-timerlat(1), rtla-timerlat-top(1)
192 timerlat tracer documentation: <‐
194 https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html>
195
197 Written by Daniel Bristot de Oliveira <bristot@kernel.org>
198 RTLA-TIMERLAT-HIST(1)