1RTLA-TIMERLAT-HIST(1) RTLA-TIMERLAT-HIST(1)
2
6 rtla-timerlat-hist - Histograms of the operating system timer latency
7
9 rtla timerlat hist [OPTIONS] ...
10
12 The rtla timerlat tool is an interface for the timerlat tracer. The
13 timerlat tracer dispatches a kernel thread per-cpu. These threads set a
14 periodic timer to wake themselves up and go back to sleep. After the
15 wakeup, they collect and generate useful information for the debugging
16 of operating system timer latency.
17 The timerlat tracer outputs information in two ways. It periodically
19 prints the timer latency at the timer IRQ handler and the Thread han‐
20 dler. It also enable the trace of the most relevant information via os‐
21 noise: tracepoints.
22 The rtla timerlat hist displays a histogram of each tracer event occur‐
24 rence. This tool uses the periodic information, and the osnoise: trace‐
25 points are enabled when using the -T option.
26
28 -a, --auto us
29 Set the automatic trace mode. This mode sets some commonly used op‐
30 tions while debugging the system. It is equivalent to use -T us -s
31 us -t. By default, timerlat tracer uses FIFO:95 for timerlat
32 threads, thus equilavent to -P f:95.
33 -p, --period us
35 Set the timerlat tracer period in microseconds.
36 -i, --irq us
38 Stop trace if the IRQ latency is higher than the argument in us.
39 -T, --thread us
41 Stop trace if the Thread latency is higher than the argument in us.
42 -s, --stack us
44 Save the stack trace at the IRQ if a Thread latency is higher than
45 the argument in us.
46 --dma-latency us
48 Set the /dev/cpu_dma_latency to us, aiming to bound exit from
49 idle latencies. cyclictest sets this value to 0 by default, use
50 --dma-latency 0 to have similar results.
51 -u, --user-threads
53 Set timerlat to run without a workload, and then dispatches
54 user-space workloads to wait on the timerlat_fd. Once the workload
55 is awakes, it goes to sleep again adding so the measurement for the
56 kernel-to-user and user-to-kernel to the tracer output.
57 -b, --bucket-size N
59 Set the histogram bucket size (default 1).
60 -E, --entries N
62 Set the number of entries of the histogram (default 256).
63 --no-header
65 Do not print header.
66 --no-summary
68 Do not print summary.
69 --no-index
71 Do not print index.
72 --with-zeros
74 Print zero only entries.
75 -c, --cpus cpu-list
77 Set the osnoise tracer to run the sample threads in the cpu-list.
78 -H, --house-keeping cpu-list
80 Run rtla control threads only on the given cpu-list.
81 -d, --duration time[s|m|h|d]
83 Set the duration of the session.
84 -D, --debug
86 Print debug info.
87 -t, --trace[=file]
89 Save the stopped trace to [file|osnoise_trace.txt].
90 -e, --event sys:event
92 Enable an event in the trace (-t) session. The argument can be a
93 specific event, e.g., -e sched:sched_switch, or all events of a sys‐
94 tem group, e.g., -e sched. Multiple -e are allowed. It is only ac‐
95 tive when -t or -a are set.
96 --filter <filter>
98 Filter the previous -e sys:event event with <filter>. For further
99 information about event filtering see
100 https://www.kernel.org/doc/html/latest/trace/events.html#event-filtering.
101 --trigger <trigger>
103 Enable a trace event trigger to the previous -e sys:event. If
104 the hist: trigger is activated, the output histogram will be au‐
105 tomatically saved to a file named system_event_hist.txt. For
106 example, the command:
107 rtla <command> <mode> -t -e osnoise:irq_noise --trig‐
109 ger="hist:key=desc,duration/1000:sort=desc,dura‐
110 tion/1000:vals=hitcount"
111 Will automatically save the content of the histogram associated
113 to osnoise:irq_noise event in osnoise_irq_noise_hist.txt.
114 For further information about event trigger see
116 https://www.kernel.org/doc/html/latest/trace/events.html#event-triggers.
117 -P, --priority o:prio|r:prio|f:prio|d:runtime:period
119 Set scheduling parameters to the osnoise tracer threads, the format
120 to set the priority are:
121 • o:prio - use SCHED_OTHER with prio;
123 • r:prio - use SCHED_RR with prio;
125 • f:prio - use SCHED_FIFO with prio;
127 • d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with run‐
129 time and period in nanoseconds.
130 -C, --cgroup[=cgroup]
132 Set a cgroup to the tracer's threads. If the -C option is passed
133 without arguments, the tracer's thread will inherit rtla's cgroup.
134 Otherwise, the threads will be placed on the cgroup passed to the
135 option.
136 -h, --help
138 Print help menu.
139 --dump-tasks
141 prints the task running on all CPUs if stop conditions are met (de‐
142 pends on !--no-aa)
143 --no-aa
145 disable auto-analysis, reducing rtla timerlat cpu usage
146
148 In the example below, rtla timerlat hist is set to run for 10 minutes,
149 in the cpus 0-4, skipping zero only lines. Moreover, rtla timerlat hist
150 will change the priority of the timerlat threads to run under
151 SCHED_DEADLINE priority, with a 100us runtime every 1ms period. The 1ms
152 period is also passed to the timerlat tracer. Auto-analysis is disabled
153 to reduce overhead
154 [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1000 --no-aa
156 # RTLA timerlat histogram
157 # Time unit is microseconds (us)
158 # Duration: 0 00:10:00
159 Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 IRQ-004 Thr-004
160 0 276489 0 206089 0 466018 0 481102 0 205546 0
161 1 318327 35487 388149 30024 94531 48382 83082 71078 388026 55730
162 2 3282 122584 4019 126527 28231 109012 23311 89309 4568 98739
163 3 940 11815 837 9863 6209 16227 6895 17196 910 9780
164 4 444 17287 424 11574 2097 38443 2169 36736 462 13476
165 5 206 43291 255 25581 1223 101908 1304 101137 236 28913
166 6 132 101501 96 64584 635 213774 757 215471 99 73453
167 7 74 169347 65 124758 350 57466 441 53639 69 148573
168 8 53 85183 31 156751 229 9052 306 9026 39 139907
169 9 22 10387 12 42762 161 2554 225 2689 19 26192
170 10 13 1898 8 5770 114 1247 128 1405 13 3772
171 11 9 560 9 924 71 686 76 765 8 713
172 12 4 256 2 360 50 411 64 474 3 278
173 13 2 167 2 172 43 256 53 350 4 180
174 14 1 88 1 116 15 198 42 223 0 115
175 15 2 63 3 94 11 139 20 150 0 58
176 16 2 37 0 56 5 78 10 102 0 39
177 17 0 18 0 28 4 57 8 80 0 15
178 18 0 8 0 17 2 50 6 56 0 12
179 19 0 9 0 5 0 19 0 48 0 18
180 20 0 4 0 8 0 11 2 27 0 4
181 21 0 2 0 3 1 9 1 18 0 6
182 22 0 1 0 3 1 7 0 3 0 5
183 23 0 2 0 4 0 2 0 7 0 2
184 24 0 2 0 2 1 3 0 3 0 5
185 25 0 0 0 1 0 1 0 1 0 3
186 26 0 1 0 0 0 2 0 2 0 0
187 27 0 0 0 3 0 1 0 0 0 1
188 28 0 0 0 3 0 0 0 1 0 0
189 29 0 0 0 2 0 2 0 1 0 3
190 30 0 1 0 0 0 0 0 0 0 0
191 31 0 1 0 0 0 0 0 2 0 2
192 32 0 0 0 1 0 2 0 0 0 0
193 33 0 0 0 2 0 0 0 0 0 1
194 34 0 0 0 0 0 0 0 0 0 2
195 35 0 1 0 1 0 0 0 0 0 1
196 36 0 1 0 0 0 1 0 1 0 0
197 37 0 0 0 1 0 0 0 0 0 0
198 40 0 0 0 0 0 1 0 1 0 0
199 41 0 0 0 0 0 0 0 0 0 1
200 42 0 0 0 0 0 0 0 0 0 1
201 44 0 0 0 0 0 1 0 0 0 0
202 46 0 0 0 0 0 0 0 1 0 0
203 47 0 0 0 0 0 0 0 0 0 1
204 50 0 0 0 0 0 0 0 0 0 1
205 54 0 0 0 1 0 0 0 0 0 0
206 58 0 0 0 1 0 0 0 0 0 0
207 over: 0 0 0 0 0 0 0 0 0 0
208 count: 600002 600002 600002 600002 600002 600002 600002 600002 600002 600002
209 min: 0 1 0 1 0 1 0 1 0 1
210 avg: 0 5 0 5 0 4 0 4 0 5
211 max: 16 36 15 58 24 44 21 46 13 50
212
214 rtla-timerlat(1), rtla-timerlat-top(1)
215 timerlat tracer documentation: <‐
217 https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html>
218
220 Written by Daniel Bristot de Oliveira <bristot@kernel.org>
221 RTLA-TIMERLAT-HIST(1)