1trapstat(1M) System Administration Commands trapstat(1M)
2
6 trapstat - report trap statistics
7
9 /usr/sbin/trapstat [-t | -T | -e entry]
10 [-C processor_set_id | -c cpulist] [-P] [-a]
11 [-r rate] [ [interval [count]] | command | [args]]
12 /usr/sbin/trapstat -l
15
18 The trapstat utility gathers and displays run-time trap statistics on
19 UltraSPARC-based systems. The default output is a table of trap types
20 and CPU IDs, with each row of the table denoting a trap type and each
21 column of the table denoting a CPU. If standard output is a terminal,
22 the table contains as many columns of data as can fit within the termi‐
23 nal width; if standard output is not a terminal, the table contains at
24 most six columns of data. By default, data is gathered and and dis‐
25 played for all CPUs; if the data cannot fit in a single table, it is
26 printed across multiple tables. The set of CPUs for which data is gath‐
27 ered and displayed can be optionally specified with the -c or -C
28 option.
29 Unless the -r option or the -a option is specified, the value displayed
32 in each entry of the table corresponds to the number of traps per sec‐
33 ond. If the -r option is specified, the value corresponds to the number
34 of traps over the interval implied by the specified sampling rate; if
35 the -a option is specified, the value corresponds to the accumulated
36 number of traps since the invocation of trapstat.
37 By default, trapstat displays data once per second, and runs indefi‐
40 nitely; both of these behaviors can be optionally controlled with the
41 interval and count parameters, respectively. The interval is specified
42 in seconds; the count indicates the number of intervals to be executed
43 before exiting. Alternatively, command can be specified, in which case
44 trapstat executes the provided command and continues to run until the
45 command exits. A positive integer is assumed to be an interval; if the
46 desired command cannot be distinguished from an integer, the full path
47 of command must be specified.
48 UltraSPARC I (obsolete), II, and III handle translation lookaside buf‐
51 fer (TLB) misses by trapping to the operating system. TLB miss traps
52 can be a significant component of overall system performance for some
53 workloads; the -t option provides in-depth information on these traps.
54 When run with this option, trapstat displays both the rate of TLB miss
55 traps and the percentage of time spent processing those traps. Addi‐
56 tionally, TLB misses that hit in the translation storage buffer (TSB)
57 are differentiated from TLB misses that further miss in the TSB. (The
58 TSB is a software structure used as a translation entry cache to allow
59 the TLB to be quickly filled; it is discussed in detail in the Ultra‐
60 SPARC II User's Manual.) The TLB and TSB miss information is further
61 broken down into user- and kernel-mode misses.
62 Workloads with working sets that exceed the TLB reach may spend a sig‐
65 nificant amount of time missing in the TLB. To accommodate such work‐
66 loads, the operating system supports multiple page sizes: larger page
67 sizes increase the effective TLB reach and thereby reduce the number of
68 TLB misses. To provide insight into the relationship between page size
69 and TLB miss rate, trapstat optionally provides in-depth TLB miss
70 information broken down by page size using the -T option. The informa‐
71 tion provided by the -T option is a superset of that provided by the -t
72 option; only one of -t and -T can be specified.
73
75 The following options are supported:
76 -a Displays the number of traps as accumulating,
78 monotonically increasing values instead of per-
79 second or per-interval rates.
80 -c cpulist Enables trapstat only on the CPUs specified by
83 cpulist.
84 cpulist can be a single processor ID (for exam‐
86 ple, 4), a range of processor IDs (for example,
87 4-6), or a comma separated list of processor
88 IDs or processor ID ranges (for example, 4,5,6
89 or 4,6-8).
90 -C processor_set_id Enables trapstat only on the CPUs in the pro‐
93 cessor set specified by processor_set_id.
94 trapstat modifies its output to always reflect
96 the CPUs in the specified processor set. If a
97 CPU is added to the set, trapstat modifies its
98 output to include the added CPU; if a CPU is
99 removed from the set, trapstat modifies its
100 output to exclude the removed CPU. At most one
101 processor set can be specified.
102 -e entrylist Enables trapstat only for the trap table entry
105 or entries specified by entrylist. A trap table
106 entry can be specified by trap number or by
107 trap name (for example, the level-10 trap can
108 be specified as 74, 0x4A, 0x4a, or level-10).
109 entrylist can be a single trap table entry or a
111 comma separated list of trap table entries. If
112 the specified trap table entry is not valid,
113 trapstat prints a table of all valid trap table
114 entries and values. A list of valid trap table
115 entries is also found in The SPARC Architecture
116 Manual, Version 9 and the Sun Microelectronics
117 UltraSPARC II User's Manual. If the parsable
118 option (-P) is specified in addition to the -e
119 option, the format of the data is as follows:
120 Field Contents
125 1 Timestamp (nanoseconds since start)
126 2 CPU ID
127 3 Trap number (in hexadecimal)
128 4 Trap name
129 5 Trap rate per interval
130 Each field is separated with whitespace. If the
132 format is modified, it will be modified by
133 adding potentially new fields beginning with
134 field 6; exant fields will remain unchanged.
135 -l Lists trap table entries. By default, a table
138 is displayed containing all valid trap numbers,
139 their names and a brief description. The trap
140 name is used in both the default output and in
141 the entrylist parameter for the -e argument. If
142 the parsable option (-P) is specified in addi‐
143 tion to the -l option, the format of the data
144 is as follows:
145 Field Contents
150 1 Trap number in hexadecimal
151 2 Trap number in decimal
152 3 Trap name
153 Remaining Trap description
154 -P Generates parsable output. When run without
158 other data gathering modifying options (that
159 is, -e, -t or -T), trapstat's the parsable out‐
160 put has the following format:
161 Field Contents
166 1 Timestamp (nanoseconds since start)
167 2 CPU ID
168 3 Trap number (in hexadecimal)
169 4 Trap name
170 5 Trap rate per interval
171 Each field is separated with whitespace. If the
173 format is modified, it will be modified by
174 adding potentially new fields beginning with
175 field 6; extant fields will remain unchanged.
176 -r rate Explicitly sets the sampling rate to be rate
179 samples per second. If this option is speci‐
180 fied, trapstat's output changes from a traps-
181 per-second to traps-per-sampling-interval.
182 -t Enables TLB statistics.
185 A table is displayed with four principal col‐
187 umns of data: itlb-miss, itsb-miss, dtlb-miss,
188 and dtsb-miss. The columns contain both the
189 rate of the corresponding event and the per‐
190 centage of CPU time spent processing the event.
191 The percentage of CPU time is given only in
192 terms of a single CPU. The rows of the table
193 correspond to CPUs, with each CPU consuming two
194 rows: one row for user-mode events (denoted
195 with u) and one row for kernel-mode events
196 (denoted with k). For each row, the percentage
197 of CPU time is totalled and displayed in the
198 rightmost column. The CPUs are delineated with
199 a solid line. If the parsable option (-P) is
200 specified in addition to the -t option, the
201 format of the data is as follows:
202 Field Contents
207 1 Timestamp (nanoseconds since start)
208 2 CPU ID
209 3 Mode (k denotes kernel, u denotes user)
210 4 I-TLB misses
211 5 Percentage of time in I-TLB miss handler
212 6 I-TSB misses
213 7 Percentage of time in I-TSB miss handler
214 8 D-TLB misses
215 9 Percentage of time in D-TLB miss handler
216 10 D-TSB misses
217 11 Percentage of time in D-TSB miss handler
218 Each field is separated with whitespace. If the
220 format is modified, it will be modified by
221 adding potentially new fields beginning with
222 field 12; extant fields will remain unchanged.
223 -T Enables TLB statistics, with page size informa‐
226 tion. As with the -t option, a table is dis‐
227 played with four principal columns of data:
228 itlb-miss, itsb-miss, dtlb-miss, and dtsb-miss.
229 The columns contain both the absolute number of
230 the corresponding event, and the percentage of
231 CPU time spent processing the event. The per‐
232 centage of CPU time is given only in terms of a
233 single CPU. The rows of the table correspond to
234 CPUs, with each CPU consuming two sets of rows:
235 one set for user-level events (denoted with u)
236 and one set for kernel-level events (denoted
237 with k). Each set, in turn, contains as many
238 rows as there are page sizes supported (see
239 getpagesizes(3C)). For each row, the percentage
240 of CPU time is totalled and displayed in the
241 right-most column. The two sets are delineated
242 with a dashed line; CPUs are delineated with a
243 solid line. If the parsable option (-P) is
244 specified in addition to the -T option, the
245 format of the data is as follows:
246 Field Contents
251 1 Timestamp (nanoseconds since start)
252 2 CPU ID
253 3 Mode k denotes kernel, u denotes user)
254 4 Page size, in decimal
255 5 I-TLB misses
256 6 Percentage of time in I-TLB miss handler
257 7 I-TSB misses
258 8 Percentage of time in I-TSB miss handler
259 9 D-TLB misses
260 10 Percentage of time in D-TLB miss handler
261 11 D-TSB misses
262 12 Percentage of time in D-TSB miss handler
263 Each field is separated with whitespace. If the
265 format is modified, it will be modified by
266 adding potentially new fields beginning with
267 field 13; extant fields will remain unchanged.
268
271 Example 1 Using trapstat Without Options
272 When run without options, trapstat displays a table of trap types and
275 CPUs. At most six columns can fit in the default terminal width; if (as
276 in this example) there are more than six CPUs, multiple tables are dis‐
277 played:
278 example# trapstat
281 vct name | cpu0 cpu1 cpu4 cpu5 cpu8 cpu9
282 ------------------------+------------------------------------------------------
283 24 cleanwin | 6446 4837 6368 2153 2623 1321
284 41 level-1 | 100 0 0 0 1 0
285 44 level-4 | 0 1 1 1 0 0
286 45 level-5 | 0 0 0 0 0 0
287 47 level-7 | 0 0 0 0 9 0
288 49 level-9 | 100 100 100 100 100 100
289 4a level-10 | 100 0 0 0 0 0
290 4d level-13 | 6 10 7 16 13 11
291 4e level-14 | 100 0 0 0 1 0
292 60 int-vec | 2607 2740 2642 2922 2920 3033
293 64 itlb-miss | 3129 2475 3167 1037 1200 569
294 68 dtlb-miss | 121061 86162 109838 37386 45639 20269
295 6c dtlb-prot | 997 847 1061 379 406 184
296 84 spill-user-32 | 2809 2133 2739 200806 332776 454504
297 88 spill-user-64 | 45819 207856 93487 228529 68373 77590
298 8c spill-user-32-cln | 784 561 767 274 353 215
299 90 spill-user-64-cln | 9 37 17 39 12 13
300 98 spill-kern-64 | 62913 50145 63869 21916 28431 11738
301 a4 spill-asuser-32 | 1327 947 1288 460 572 335
302 a8 spill-asuser-64 | 26 48 18 54 10 14
303 ac spill-asuser-32-cln | 4580 3599 4555 1538 1978 857
304 b0 spill-asuser-64-cln | 26 0 0 2 0 0
305 c4 fill-user-32 | 2862 2161 2798 191746 318115 435850
306 c8 fill-user-64 | 45813 197781 89179 217668 63905 74281
307 cc fill-user-32-cln | 3802 2833 3733 10153 16419 19475
308 d0 fill-user-64-cln | 329 10105 4873 10603 4235 3649
309 d8 fill-kern-64 | 62519 49943 63611 21824 28328 11693
310 108 syscall-32 | 2285 1634 2278 737 957 383
311 126 self-xcall | 100 0 0 0 0 0
312 vct name | cpu12 cpu13 cpu14 cpu15
314 ------------------------+------------------------------------
315 24 cleanwin | 5435 4232 6302 6104
316 41 level-1 | 0 0 0 0
317 44 level-4 | 2 0 0 1
318 45 level-5 | 0 0 0 0
319 47 level-7 | 0 0 0 0
320 49 level-9 | 100 100 100 100
321 4a level-10 | 0 0 0 0
322 4d level-13 | 15 11 22 11
323 4e level-14 | 0 0 0 0
324 60 int-vec | 2813 2833 2738 2714
325 64 itlb-miss | 2636 1925 3133 3029
326 68 dtlb-miss | 90528 70639 107786 103425
327 6c dtlb-prot | 819 675 988 954
328 84 spill-user-32 | 175768 39933 2811 2742
329 88 spill-user-64 | 0 241348 96907 118298
330 8c spill-user-32-cln | 681 513 753 730
331 90 spill-user-64-cln | 0 42 16 20
332 98 spill-kern-64 | 52158 40914 62305 60141
333 a4 spill-asuser-32 | 1113 856 1251 1208
334 a8 spill-asuser-64 | 0 64 16 24
335 ac spill-asuser-32-cln | 3816 2942 4515 4381
336 b0 spill-asuser-64-cln | 0 0 0 0
337 c4 fill-user-32 | 170744 38444 2876 2784
338 c8 fill-user-64 | 0 230381 92941 111694
339 cc fill-user-32-cln | 8550 3790 3612 3553
340 d0 fill-user-64-cln | 0 10726 4495 5845
341 d8 fill-kern-64 | 51968 40760 62053 59922
342 108 syscall-32 | 1839 1495 2144 2083
343 126 self-xcall | 0 0 0 0
344 Example 2 Using trapset with CPU Filtering
348 The -c option can be used to limit the CPUs on which trapstat is
351 enabled. This example limits CPU 1 and CPUs 12 through 15.
352 example# trapstat -c 1,12-15
355 vct name | cpu1 cpu12 cpu13 cpu14 cpu15
358 ------------------------+---------------------------------------------
359 24 cleanwin | 6923 3072 2500 3518 2261
360 44 level-4 | 3 0 0 1 1
361 49 level-9 | 100 100 100 100 100
362 4d level-13 | 23 8 14 19 14
363 60 int-vec | 2559 2699 2752 2688 2792
364 64 itlb-miss | 3296 1548 1174 1698 1087
365 68 dtlb-miss | 114788 54313 43040 58336 38057
366 6c dtlb-prot | 1046 549 417 545 370
367 84 spill-user-32 | 66551 29480 301588 26522 213032
368 88 spill-user-64 | 0 318652 111239 299829 221716
369 8c spill-user-32-cln | 856 347 331 416 293
370 90 spill-user-64-cln | 0 55 21 59 39
371 98 spill-kern-64 | 66464 31803 24758 34004 22277
372 a4 spill-asuser-32 | 1423 569 560 698 483
373 a8 spill-asuser-64 | 0 74 32 98 46
374 ac spill-asuser-32-cln | 4875 2250 1728 2384 1584
375 b0 spill-asuser-64-cln | 0 2 0 1 0
376 c4 fill-user-32 | 64193 28418 287516 27055 202093
377 c8 fill-user-64 | 0 305016 106692 288542 210654
378 cc fill-user-32-cln | 6733 3520 15185 2396 12035
379 d0 fill-user-64-cln | 0 13226 3506 12933 11032
380 d8 fill-kern-64 | 66220 31680 24674 33892 22196
381 108 syscall-32 | 2446 967 817 1196 755
382 Example 3 Using trapstat with TLB Statistics
386 The -t option displays in-depth TLB statistics, including the amount of
389 time spent performing TLB miss processing. The following example shows
390 that the machine is spending 14.1 percent of its time just handling D-
391 TLB misses:
392 example# trapstat -t
395 cpu m| itlb-miss %tim itsb-miss %tim | dtlb-miss %tim dtsb-miss %tim |%tim
396 -----+-------------------------------+-------------------------------+----
397 0 u| 2571 0.3 0 0.0 | 10802 1.3 0 0.0 | 1.6
398 0 k| 0 0.0 0 0.0 | 106420 13.4 184 0.1 |13.6
399 -----+-------------------------------+-------------------------------+----
400 1 u| 3069 0.3 0 0.0 | 10983 1.2 100 0.0 | 1.6
401 1 k| 27 0.0 0 0.0 | 106974 12.6 19 0.0 |12.7
402 -----+-------------------------------+-------------------------------+----
403 2 u| 3033 0.3 0 0.0 | 11045 1.2 105 0.0 | 1.6
404 2 k| 43 0.0 0 0.0 | 107842 12.7 108 0.0 |12.8
405 -----+-------------------------------+-------------------------------+----
406 3 u| 2924 0.3 0 0.0 | 10380 1.2 121 0.0 | 1.6
407 3 k| 54 0.0 0 0.0 | 102682 12.2 16 0.0 |12.2
408 -----+-------------------------------+-------------------------------+----
409 4 u| 3064 0.3 0 0.0 | 10832 1.2 120 0.0 | 1.6
410 4 k| 31 0.0 0 0.0 | 107977 13.0 236 0.1 |13.1
411 =====+===============================+===============================+====
412 ttl | 14816 0.3 0 0.0 | 585937 14.1 1009 0.0 |14.5
413 Example 4 Using trapstat with TLB Statistics and Page Size Information
417 By specifying the -T option, trapstat shows TLB misses broken down by
420 page size. In this example, CPU 0 is spending 7.9 percent of its time
421 handling user-mode TLB misses on 8K pages, and another 2.3 percent of
422 its time handling user-mode TLB misses on 64K pages.
423 example# trapstat -T -c 0
426 cpu m size| itlb-miss %tim itsb-miss %tim | dtlb-miss %tim dtsb-miss %tim |%tim
427 ----------+-------------------------------+-------------------------------+----
428 0 u 8k| 1300 0.1 15 0.0 | 104897 7.9 90 0.0 | 8.0
429 0 u 64k| 0 0.0 0 0.0 | 29935 2.3 7 0.0 | 2.3
430 0 u 512k| 0 0.0 0 0.0 | 3569 0.2 2 0.0 | 0.2
431 0 u 4m| 0 0.0 0 0.0 | 233 0.0 2 0.0 | 0.0
432 - - - - - + - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - + - -
433 0 k 8k| 13 0.0 0 0.0 | 71733 6.5 110 0.0 | 6.5
434 0 k 64k| 0 0.0 0 0.0 | 0 0.0 0 0.0 | 0.0
435 0 k 512k| 0 0.0 0 0.0 | 0 0.0 206 0.1 | 0.1
436 0 k 4m| 0 0.0 0 0.0 | 0 0.0 0 0.0 | 0.0
437 ==========+===============================+===============================+====
438 ttl | 1313 0.1 15 0.0 | 210367 17.1 417 0.2 |17.5
439 Example 5 Using trapstat with Entry Filtering
443 By specifying the -e option, trapstat displays statistics for only spe‐
446 cific trap types. Using this option minimizes the probe effect when
447 seeking specific data. This example yields statistics for only the
448 dtlb-prot and syscall-32 traps on CPUs 12 through 15:
449 example# trapstat -e dtlb-prot,syscall-32 -c 12-15
452 vct name | cpu12 cpu13 cpu14 cpu15
453 ------------------------+------------------------------------
454 6c dtlb-prot | 817 754 1018 560
455 108 syscall-32 | 1426 1647 2186 1142
456 vct name | cpu12 cpu13 cpu14 cpu15
458 ------------------------+------------------------------------
459 6c dtlb-prot | 1085 996 800 707
460 108 syscall-32 | 2578 2167 1638 1452
461 Example 6 Using trapstat with a Higher Sampling Rate
465 The following example uses the -r option to specify a sampling rate of
468 1000 samples per second, and filter only for the level-10 trap. Addi‐
469 tionally, specifying the -P option yields parsable output.
470 Notice the timestamp difference between the level-10 events: 9,998,000
474 nanoseconds and 10,007,000 nanoseconds. These level-10 events corre‐
475 spond to the system clock, which by default ticks at 100 hertz (that
476 is, every 10,000,000 nanoseconds).
477 example# trapstat -e level-10 -P -r 1000
480 1070400 0 4a level-10 0
481 2048600 0 4a level-10 0
482 3030400 0 4a level-10 1
483 4035800 0 4a level-10 0
484 5027200 0 4a level-10 0
485 6027200 0 4a level-10 0
486 7027400 0 4a level-10 0
487 8028200 0 4a level-10 0
488 9026400 0 4a level-10 0
489 10029600 0 4a level-10 0
490 11028600 0 4a level-10 0
491 12024000 0 4a level-10 0
492 13028400 0 4a level-10 1
493 14031200 0 4a level-10 0
494 15027200 0 4a level-10 0
495 16027600 0 4a level-10 0
496 17025000 0 4a level-10 0
497 18026000 0 4a level-10 0
498 19027800 0 4a level-10 0
499 20025600 0 4a level-10 0
500 21025200 0 4a level-10 0
501 22025000 0 4a level-10 0
502 23035400 0 4a level-10 1
503 24027400 0 4a level-10 0
504 25026000 0 4a level-10 0
505 26027000 0 4a level-10 0
506
510 See attributes(5) for descriptions of the following attributes:
511 ┌───────────────────────────────────────────────────────────┐
516 │ ATTRIBUTE TYPE ATTRIBUTE VALUE │
517 │Availability SUNWcsu │
518 │Interface Stability │
519 │ Human Readable Output Unstable │
520 │ Parsable Output Evolving │
521 └───────────────────────────────────────────────────────────┘
522
524 lockstat(1M), pmap(1), psrset(1M), psrinfo(1M), pbind(1M), ppgsz(1),
525 getpagesizes(3C)
526 Sun Microelectronics UltraSPARC II User's Manual, January 1997,
529 STP1031,
530 The SPARC Architecture Manual, Version 9, 1994, Prentice-Hall.
533
535 When enabled, trapstat induces a varying probe effect, depending on the
536 type of information collected. While the precise probe effect depends
537 upon the specifics of the hardware, the following table can be used as
538 a rough guide:
539 Option Approximate probe effect
544 default 3-5% per trap
545 -e 3-5% per specified trap
546 -t, -T 40-45% per TLB miss trap hitting in the TSB, 25-30% per
547 TLB miss trap missing in the TSB
548 These probe effects are per trap not for the system as a whole. For
552 example, running trapstat with the default options on a system that
553 spends 7% of total time handling traps induces a performance degrada‐
554 tion of less than one half of one percent; running trapstat with the -t
555 or -T option on a system spending 5% of total time processing TLB
556 misses induce a performance degradation of no more than 2.5%.
557 When run with the -t or -T option, trapstat accounts for its probe
560 effect when calculating the %tim fields. This assures that the %tim
561 fields are a reasonably accurate indicator of the time a given workload
562 is spending handling TLB misses — regardless of the perturbing presence
563 of trapstat.
564 While the %tim fields include the explicit cost of executing the TLB
567 miss handler, they do not include the implicit costs of TLB miss traps
568 (for example, pipeline effects, cache pollution, etc). These implicit
569 costs become more significant as the trap rate grows; if high %tim val‐
570 ues are reported (greater than 50%), you can accurately infer that much
571 of the balance of time is being spent on the implicit costs of the TLB
572 miss traps.
573 Due to the potential system wide degradation induced, only the super-
576 user can run trapstat.
577 Due to the limitation of the underlying statistics gathering methodol‐
580 ogy, only one instance of trapstat can run at a time.
581SunOS 5.11 11 May 2004 trapstat(1M)