1PERF-INTEL-PT(1) perf Manual PERF-INTEL-PT(1)
2
3
4
6 perf-intel-pt - Support for Intel Processor Trace within perf tools
7
9 perf record -e intel_pt//
10
12 Intel Processor Trace (Intel PT) is an extension of Intel Architecture
13 that collects information about software execution such as control
14 flow, execution modes and timings and formats it into highly compressed
15 binary packets. Technical details are documented in the Intel 64 and
16 IA-32 Architectures Software Developer Manuals, Chapter 36 Intel
17 Processor Trace.
18
19 Intel PT is first supported in Intel Core M and 5th generation Intel
20 Core processors that are based on the Intel micro-architecture code
21 name Broadwell.
22
23 Trace data is collected by perf record and stored within the perf.data
24 file. See below for options to perf record.
25
26 Trace data must be decoded which involves walking the object code and
27 matching the trace data packets. For example a TNT packet only tells
28 whether a conditional branch was taken or not taken, so to make use of
29 that packet the decoder must know precisely which instruction was being
30 executed.
31
32 Decoding is done on-the-fly. The decoder outputs samples in the same
33 format as samples output by perf hardware events, for example as though
34 the "instructions" or "branches" events had been recorded. Presently 3
35 tools support this: perf script, perf report and perf inject. See below
36 for more information on using those tools.
37
38 The main distinguishing feature of Intel PT is that the decoder can
39 determine the exact flow of software execution. Intel PT can be used to
40 understand why and how did software get to a certain point, or behave a
41 certain way. The software does not have to be recompiled, so Intel PT
42 works with debug or release builds, however the executed images are
43 needed - which makes use in JIT-compiled environments, or with
44 self-modified code, a challenge. Also symbols need to be provided to
45 make sense of addresses.
46
47 A limitation of Intel PT is that it produces huge amounts of trace data
48 (hundreds of megabytes per second per core) which takes a long time to
49 decode, for example two or three orders of magnitude longer than it
50 took to collect. Another limitation is the performance impact of
51 tracing, something that will vary depending on the use-case and
52 architecture.
53
55 It is important to start small. That is because it is easy to capture
56 vastly more data than can possibly be processed.
57
58 The simplest thing to do with Intel PT is userspace profiling of small
59 programs. Data is captured with perf record e.g. to trace ls
60 userspace-only:
61
62 perf record -e intel_pt//u ls
63
64 And profiled with perf report e.g.
65
66 perf report
67
68 To also trace kernel space presents a problem, namely kernel
69 self-modifying code. A fairly good kernel image is available in
70 /proc/kcore but to get an accurate image a copy of /proc/kcore needs to
71 be made under the same conditions as the data capture. perf record can
72 make a copy of /proc/kcore if the option --kcore is used, but access to
73 /proc/kcore is restricted e.g.
74
75 sudo perf record -o pt_ls --kcore -e intel_pt// -- ls
76
77 which will create a directory named pt_ls and put the perf.data file
78 (named simply data) and copies of /proc/kcore, /proc/kallsyms and
79 /proc/modules into it. The other tools understand the directory format,
80 so to use perf report becomes:
81
82 sudo perf report -i pt_ls
83
84 Because samples are synthesized after-the-fact, the sampling period can
85 be selected for reporting. e.g. sample every microsecond
86
87 sudo perf report pt_ls --itrace=i1usge
88
89 See the sections below for more information about the --itrace option.
90
91 Beware the smaller the period, the more samples that are produced, and
92 the longer it takes to process them.
93
94 Also note that the coarseness of Intel PT timing information will start
95 to distort the statistical value of the sampling as the sampling period
96 becomes smaller.
97
98 To represent software control flow, "branches" samples are produced. By
99 default a branch sample is synthesized for every single branch. To get
100 an idea what data is available you can use the perf script tool with
101 all itrace sampling options, which will list all the samples.
102
103 perf record -e intel_pt//u ls
104 perf script --itrace=ibxwpe
105
106 An interesting field that is not printed by default is flags which can
107 be displayed as follows:
108
109 perf script --itrace=ibxwpe -F+flags
110
111 The flags are "bcrosyiABExghDt" which stand for branch, call, return,
112 conditional, system, asynchronous, interrupt, transaction abort, trace
113 begin, trace end, in transaction, VM-entry, VM-exit, interrupt
114 disabled, and interrupt disable toggle respectively.
115
116 perf script also supports higher level ways to dump instruction traces:
117
118 perf script --insn-trace --xed
119
120 Dump all instructions. This requires installing the xed tool (see XED
121 below) Dumping all instructions in a long trace can be fairly slow. It
122 is usually better to start with higher level decoding, like
123
124 perf script --call-trace
125
126 or
127
128 perf script --call-ret-trace
129
130 and then select a time range of interest. The time range can then be
131 examined in detail with
132
133 perf script --time starttime,stoptime --insn-trace --xed
134
135 While examining the trace it’s also useful to filter on specific CPUs
136 using the -C option
137
138 perf script --time starttime,stoptime --insn-trace --xed -C 1
139
140 Dump all instructions in time range on CPU 1.
141
142 Another interesting field that is not printed by default is ipc which
143 can be displayed as follows:
144
145 perf script --itrace=be -F+ipc
146
147 There are two ways that instructions-per-cycle (IPC) can be calculated
148 depending on the recording.
149
150 If the cyc config term (see config terms section below) was used, then
151 IPC is calculated using the cycle count from CYC packets, otherwise MTC
152 packets are used - refer to the mtc config term. When MTC is used,
153 however, the values are less accurate because the timing is less
154 accurate.
155
156 Because Intel PT does not update the cycle count on every branch or
157 instruction, the values will often be zero. When there are values, they
158 will be the number of instructions and number of cycles since the last
159 update, and thus represent the average IPC since the last IPC for that
160 event type. Note IPC for "branches" events is calculated separately
161 from IPC for "instructions" events.
162
163 Even with the cyc config term, it is possible to produce IPC
164 information for every change of timestamp, but at the expense of
165 accuracy. That is selected by specifying the itrace A option. Due to
166 the granularity of timestamps, the actual number of cycles increases
167 even though the cycles reported does not. The number of instructions is
168 known, but if IPC is reported, cycles can be too low and so IPC is too
169 high. Note that inaccuracy decreases as the period of sampling
170 increases i.e. if the number of cycles is too low by a small amount,
171 that becomes less significant if the number of cycles is large. It may
172 also be useful to use the A option in conjunction with
173 dlfilter-show-cycles.so to provide higher granularity cycle
174 information.
175
176 Also note that the IPC instruction count may or may not include the
177 current instruction. If the cycle count is associated with an
178 asynchronous branch (e.g. page fault or interrupt), then the
179 instruction count does not include the current instruction, otherwise
180 it does. That is consistent with whether or not that instruction has
181 retired when the cycle count is updated.
182
183 Another note, in the case of "branches" events, non-taken branches are
184 not presently sampled, so IPC values for them do not appear e.g. a CYC
185 packet with a TNT packet that starts with a non-taken branch. To see
186 every possible IPC value, "instructions" events can be used e.g.
187 --itrace=i0ns
188
189 While it is possible to create scripts to analyze the data, an
190 alternative approach is available to export the data to a sqlite or
191 postgresql database. Refer to script export-to-sqlite.py or
192 export-to-postgresql.py for more details, and to script
193 exported-sql-viewer.py for an example of using the database.
194
195 There is also script intel-pt-events.py which provides an example of
196 how to unpack the raw data for power events and PTWRITE. The script
197 also displays branches, and supports 2 additional modes selected by
198 option:
199
200 --insn-trace - instruction trace
201 --src-trace - source trace
202
203 As mentioned above, it is easy to capture too much data. One way to
204 limit the data captured is to use snapshot mode which is explained
205 further below. Refer to new snapshot option and Intel PT modes of
206 operation further below.
207
208 Another problem that will be experienced is decoder errors. They can be
209 caused by inability to access the executed image, self-modified or
210 JIT-ed code, or the inability to match side-band information (such as
211 context switches and mmaps) which results in the decoder not knowing
212 what code was executed.
213
214 There is also the problem of perf not being able to copy the data fast
215 enough, resulting in data lost because the buffer was full. See Buffer
216 handling below for more details.
217
219 new event
220 The Intel PT kernel driver creates a new PMU for Intel PT. PMU events
221 are selected by providing the PMU name followed by the "config"
222 separated by slashes. An enhancement has been made to allow default
223 "config" e.g. the option
224
225 -e intel_pt//
226
227 will use a default config value. Currently that is the same as
228
229 -e intel_pt/tsc,noretcomp=0/
230
231 which is the same as
232
233 -e intel_pt/tsc=1,noretcomp=0/
234
235 Note there are now new config terms - see section config terms further
236 below.
237
238 The config terms are listed in /sys/devices/intel_pt/format. They are
239 bit fields within the config member of the struct perf_event_attr which
240 is passed to the kernel by the perf_event_open system call. They
241 correspond to bit fields in the IA32_RTIT_CTL MSR. Here is a list of
242 them and their definitions:
243
244 $ grep -H . /sys/bus/event_source/devices/intel_pt/format/*
245 /sys/bus/event_source/devices/intel_pt/format/cyc:config:1
246 /sys/bus/event_source/devices/intel_pt/format/cyc_thresh:config:19-22
247 /sys/bus/event_source/devices/intel_pt/format/mtc:config:9
248 /sys/bus/event_source/devices/intel_pt/format/mtc_period:config:14-17
249 /sys/bus/event_source/devices/intel_pt/format/noretcomp:config:11
250 /sys/bus/event_source/devices/intel_pt/format/psb_period:config:24-27
251 /sys/bus/event_source/devices/intel_pt/format/tsc:config:10
252
253 Note that the default config must be overridden for each term i.e.
254
255 -e intel_pt/noretcomp=0/
256
257 is the same as:
258
259 -e intel_pt/tsc=1,noretcomp=0/
260
261 So, to disable TSC packets use:
262
263 -e intel_pt/tsc=0/
264
265 It is also possible to specify the config value explicitly:
266
267 -e intel_pt/config=0x400/
268
269 Note that, as with all events, the event is suffixed with event
270 modifiers:
271
272 u userspace
273 k kernel
274 h hypervisor
275 G guest
276 H host
277 p precise ip
278
279 h, G and H are for virtualization which are not used by Intel PT. p is
280 also not relevant to Intel PT. So only options u and k are meaningful
281 for Intel PT.
282
283 perf_event_attr is displayed if the -vv option is used e.g.
284
285 ------------------------------------------------------------
286 perf_event_attr:
287 type 6
288 size 112
289 config 0x400
290 { sample_period, sample_freq } 1
291 sample_type IP|TID|TIME|CPU|IDENTIFIER
292 read_format ID
293 disabled 1
294 inherit 1
295 exclude_kernel 1
296 exclude_hv 1
297 enable_on_exec 1
298 sample_id_all 1
299 ------------------------------------------------------------
300 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
301 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
302 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
303 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
304 ------------------------------------------------------------
305
306 config terms
307 The June 2015 version of Intel 64 and IA-32 Architectures Software
308 Developer Manuals, Chapter 36 Intel Processor Trace, defined new Intel
309 PT features. Some of the features are reflect in new config terms. All
310 the config terms are described below.
311
312 tsc Always supported. Produces TSC timestamp packets to provide timing
313 information. In some cases it is possible to decode without timing
314 information, for example a per-thread context that does not overlap
315 executable memory maps.
316
317 The default config selects tsc (i.e. tsc=1).
318
319 noretcomp Always supported. Disables "return compression" so a TIP
320 packet is produced when a function returns. Causes more packets to be
321 produced but might make decoding more reliable.
322
323 The default config does not select noretcomp (i.e. noretcomp=0).
324
325 psb_period Allows the frequency of PSB packets to be specified.
326
327 The PSB packet is a synchronization packet that provides a
328 starting point for decoding or recovery from errors.
329
330 Support for psb_period is indicated by:
331
332 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
333
334 which contains "1" if the feature is supported and "0"
335 otherwise.
336
337 Valid values are given by:
338
339 /sys/bus/event_source/devices/intel_pt/caps/psb_periods
340
341 which contains a hexadecimal value, the bits of which represent
342 valid values e.g. bit 2 set means value 2 is valid.
343
344 The psb_period value is converted to the approximate number of
345 trace bytes between PSB packets as:
346
347 2 ^ (value + 11)
348
349 e.g. value 3 means 16KiB bytes between PSBs
350
351 If an invalid value is entered, the error message
352 will give a list of valid values e.g.
353
354 $ perf record -e intel_pt/psb_period=15/u uname
355 Invalid psb_period for intel_pt. Valid values are: 0-5
356
357 If MTC packets are selected, the default config selects a value
358 of 3 (i.e. psb_period=3) or the nearest lower value that is
359 supported (0 is always supported). Otherwise the default is 0.
360
361 If decoding is expected to be reliable and the buffer is large
362 then a large PSB period can be used.
363
364 Because a TSC packet is produced with PSB, the PSB period can
365 also affect the granularity to timing information in the absence
366 of MTC or CYC.
367
368 mtc Produces MTC timing packets.
369
370 MTC packets provide finer grain timestamp information than TSC
371 packets. MTC packets record time using the hardware crystal
372 clock (CTC) which is related to TSC packets using a TMA packet.
373
374 Support for this feature is indicated by:
375
376 /sys/bus/event_source/devices/intel_pt/caps/mtc
377
378 which contains "1" if the feature is supported and
379 "0" otherwise.
380
381 The frequency of MTC packets can also be specified - see
382 mtc_period below.
383
384 mtc_period Specifies how frequently MTC packets are produced - see mtc
385 above for how to determine if MTC packets are supported.
386
387 Valid values are given by:
388
389 /sys/bus/event_source/devices/intel_pt/caps/mtc_periods
390
391 which contains a hexadecimal value, the bits of which represent
392 valid values e.g. bit 2 set means value 2 is valid.
393
394 The mtc_period value is converted to the MTC frequency as:
395
396 CTC-frequency / (2 ^ value)
397
398 e.g. value 3 means one eighth of CTC-frequency
399
400 Where CTC is the hardware crystal clock, the frequency of which
401 can be related to TSC via values provided in cpuid leaf 0x15.
402
403 If an invalid value is entered, the error message
404 will give a list of valid values e.g.
405
406 $ perf record -e intel_pt/mtc_period=15/u uname
407 Invalid mtc_period for intel_pt. Valid values are: 0,3,6,9
408
409 The default value is 3 or the nearest lower value
410 that is supported (0 is always supported).
411
412 cyc Produces CYC timing packets.
413
414 CYC packets provide even finer grain timestamp information than
415 MTC and TSC packets. A CYC packet contains the number of CPU
416 cycles since the last CYC packet. Unlike MTC and TSC packets,
417 CYC packets are only sent when another packet is also sent.
418
419 Support for this feature is indicated by:
420
421 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
422
423 which contains "1" if the feature is supported and
424 "0" otherwise.
425
426 The number of CYC packets produced can be reduced by specifying
427 a threshold - see cyc_thresh below.
428
429 cyc_thresh Specifies how frequently CYC packets are produced - see cyc
430 above for how to determine if CYC packets are supported.
431
432 Valid cyc_thresh values are given by:
433
434 /sys/bus/event_source/devices/intel_pt/caps/cycle_thresholds
435
436 which contains a hexadecimal value, the bits of which represent
437 valid values e.g. bit 2 set means value 2 is valid.
438
439 The cyc_thresh value represents the minimum number of CPU cycles
440 that must have passed before a CYC packet can be sent. The
441 number of CPU cycles is:
442
443 2 ^ (value - 1)
444
445 e.g. value 4 means 8 CPU cycles must pass before a CYC packet
446 can be sent. Note a CYC packet is still only sent when another
447 packet is sent, not at, e.g. every 8 CPU cycles.
448
449 If an invalid value is entered, the error message
450 will give a list of valid values e.g.
451
452 $ perf record -e intel_pt/cyc,cyc_thresh=15/u uname
453 Invalid cyc_thresh for intel_pt. Valid values are: 0-12
454
455 CYC packets are not requested by default.
456
457 pt Specifies pass-through which enables the branch config term.
458
459 The default config selects 'pt' if it is available, so a user will
460 never need to specify this term.
461
462 branch Enable branch tracing. Branch tracing is enabled by default so
463 to disable branch tracing use branch=0.
464
465 The default config selects 'branch' if it is available.
466
467 ptw Enable PTWRITE packets which are produced when a ptwrite
468 instruction is executed.
469
470 Support for this feature is indicated by:
471
472 /sys/bus/event_source/devices/intel_pt/caps/ptwrite
473
474 which contains "1" if the feature is supported and
475 "0" otherwise.
476
477 As an alternative, refer to "Emulated PTWRITE" further below.
478
479 fup_on_ptw Enable a FUP packet to follow the PTWRITE packet. The FUP
480 packet provides the address of the ptwrite instruction. In the absence
481 of fup_on_ptw, the decoder will use the address of the previous branch
482 if branch tracing is enabled, otherwise the address will be zero. Note
483 that fup_on_ptw will work even when branch tracing is disabled.
484
485 pwr_evt Enable power events. The power events provide information about
486 changes to the CPU C-state.
487
488 Support for this feature is indicated by:
489
490 /sys/bus/event_source/devices/intel_pt/caps/power_event_trace
491
492 which contains "1" if the feature is supported and
493 "0" otherwise.
494
495 event Enable Event Trace. The events provide information about
496 asynchronous events.
497
498 Support for this feature is indicated by:
499
500 /sys/bus/event_source/devices/intel_pt/caps/event_trace
501
502 which contains "1" if the feature is supported and
503 "0" otherwise.
504
505 notnt Disable TNT packets. Without TNT packets, it is not possible to
506 walk executable code to reconstruct control flow, however FUP, TIP,
507 TIP.PGE and TIP.PGD packets still indicate asynchronous control flow,
508 and (if return compression is disabled - see noretcomp) return
509 statements. The advantage of eliminating TNT packets is reducing the
510 size of the trace and corresponding tracing overhead.
511
512 Support for this feature is indicated by:
513
514 /sys/bus/event_source/devices/intel_pt/caps/tnt_disable
515
516 which contains "1" if the feature is supported and
517 "0" otherwise.
518
519 AUX area sampling option
520 To select Intel PT "sampling" the AUX area sampling option can be used:
521
522 --aux-sample
523
524 Optionally it can be followed by the sample size in bytes e.g.
525
526 --aux-sample=8192
527
528 In addition, the Intel PT event to sample must be defined e.g.
529
530 -e intel_pt//u
531
532 Samples on other events will be created containing Intel PT data e.g.
533 the following will create Intel PT samples on the branch-misses event,
534 note the events must be grouped using {}:
535
536 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}'
537
538 An alternative to --aux-sample is to add the config term
539 aux-sample-size to events. In this case, the grouping is implied e.g.
540
541 perf record -e intel_pt//u -e branch-misses/aux-sample-size=8192/u
542
543 is the same as:
544
545 perf record -e '{intel_pt//u,branch-misses/aux-sample-size=8192/u}'
546
547 but allows for also using an address filter e.g.:
548
549 perf record -e intel_pt//u --filter 'filter * @/bin/ls' -e branch-misses/aux-sample-size=8192/u -- ls
550
551 It is important to select a sample size that is big enough to contain
552 at least one PSB packet. If not a warning will be displayed:
553
554 Intel PT sample size (%zu) may be too small for PSB period (%zu)
555
556 The calculation used for that is: if sample_size ⟨ psb_period + 256
557 display the warning. When sampling is used, psb_period defaults to 0
558 (2KiB).
559
560 The default sample size is 4KiB.
561
562 The sample size is passed in aux_sample_size in struct perf_event_attr.
563 The sample size is limited by the maximum event size which is 64KiB. It
564 is difficult to know how big the event might be without the trace
565 sample attached, but the tool validates that the sample size is not
566 greater than 60KiB.
567
568 new snapshot option
569 The difference between full trace and snapshot from the kernel’s
570 perspective is that in full trace we don’t overwrite trace data that
571 the user hasn’t collected yet (and indicated that by advancing
572 aux_tail), whereas in snapshot mode we let the trace run and overwrite
573 older data in the buffer so that whenever something interesting
574 happens, we can stop it and grab a snapshot of what was going on around
575 that interesting moment.
576
577 To select snapshot mode a new option has been added:
578
579 -S
580
581 Optionally it can be followed by the snapshot size e.g.
582
583 -S0x100000
584
585 The default snapshot size is the auxtrace mmap size. If neither
586 auxtrace mmap size nor snapshot size is specified, then the default is
587 4MiB for privileged users (or if /proc/sys/kernel/perf_event_paranoid <
588 0), 128KiB for unprivileged users. If an unprivileged user does not
589 specify mmap pages, the mmap pages will be reduced as described in the
590 new auxtrace mmap size option section below.
591
592 The snapshot size is displayed if the option -vv is used e.g.
593
594 Intel PT snapshot size: %zu
595
596 new auxtrace mmap size option
597 Intel PT buffer size is specified by an addition to the -m option e.g.
598
599 -m,16
600
601 selects a buffer size of 16 pages i.e. 64KiB.
602
603 Note that the existing functionality of -m is unchanged. The auxtrace
604 mmap size is specified by the optional addition of a comma and the
605 value.
606
607 The default auxtrace mmap size for Intel PT is 4MiB/page_size for
608 privileged users (or if /proc/sys/kernel/perf_event_paranoid < 0),
609 128KiB for unprivileged users. If an unprivileged user does not specify
610 mmap pages, the mmap pages will be reduced from the default
611 512KiB/page_size to 256KiB/page_size, otherwise the user is likely to
612 get an error as they exceed their mlock limit (Max locked memory as
613 shown in /proc/self/limits). Note that perf does not count the first
614 512KiB (actually /proc/sys/kernel/perf_event_mlock_kb minus 1 page) per
615 cpu against the mlock limit so an unprivileged user is allowed 512KiB
616 per cpu plus their mlock limit (which defaults to 64KiB but is not
617 multiplied by the number of cpus).
618
619 In full-trace mode, powers of two are allowed for buffer size, with a
620 minimum size of 2 pages. In snapshot mode or sampling mode, it is the
621 same but the minimum size is 1 page.
622
623 The mmap size and auxtrace mmap size are displayed if the -vv option is
624 used e.g.
625
626 mmap length 528384
627 auxtrace mmap length 4198400
628
629 Intel PT modes of operation
630 Intel PT can be used in 3 modes: full-trace mode sample mode snapshot
631 mode
632
633 Full-trace mode traces continuously e.g.
634
635 perf record -e intel_pt//u uname
636
637 Sample mode attaches a Intel PT sample to other events e.g.
638
639 perf record --aux-sample -e intel_pt//u -e branch-misses:u
640
641 Snapshot mode captures the available data when a signal is sent or
642 "snapshot" control command is issued. e.g. using a signal
643
644 perf record -v -e intel_pt//u -S ./loopy 1000000000 &
645 [1] 11435
646 kill -USR2 11435
647 Recording AUX area tracing snapshot
648
649 Note that the signal sent is SIGUSR2. Note that "Recording AUX area
650 tracing snapshot" is displayed because the -v option is used.
651
652 The advantage of using "snapshot" control command is that the access is
653 controlled by access to a FIFO e.g.
654
655 $ mkfifo perf.control
656 $ mkfifo perf.ack
657 $ cat perf.ack &
658 [1] 15235
659 $ sudo ~/bin/perf record --control fifo:perf.control,perf.ack -S -e intel_pt//u -- sleep 60 &
660 [2] 15243
661 $ ps -e | grep perf
662 15244 pts/1 00:00:00 perf
663 $ kill -USR2 15244
664 bash: kill: (15244) - Operation not permitted
665 $ echo snapshot > perf.control
666 ack
667
668 The 3 Intel PT modes of operation cannot be used together.
669
670 Buffer handling
671 There may be buffer limitations (i.e. single ToPa entry) which means
672 that actual buffer sizes are limited to powers of 2 up to 4MiB
673 (MAX_ORDER). In order to provide other sizes, and in particular an
674 arbitrarily large size, multiple buffers are logically concatenated.
675 However an interrupt must be used to switch between buffers. That has
676 two potential problems: a) the interrupt may not be handled in time so
677 that the current buffer becomes full and some trace data is lost. b)
678 the interrupts may slow the system and affect the performance results.
679
680 If trace data is lost, the driver sets truncated in the PERF_RECORD_AUX
681 event which the tools report as an error.
682
683 In full-trace mode, the driver waits for data to be copied out before
684 allowing the (logical) buffer to wrap-around. If data is not copied out
685 quickly enough, again truncated is set in the PERF_RECORD_AUX event. If
686 the driver has to wait, the intel_pt event gets disabled. Because it is
687 difficult to know when that happens, perf tools always re-enable the
688 intel_pt event after copying out data.
689
690 Intel PT and build ids
691 By default "perf record" post-processes the event stream to find all
692 build ids for executables for all addresses sampled. Deliberately,
693 Intel PT is not decoded for that purpose (it would take too long).
694 Instead the build ids for all executables encountered (due to mmap,
695 comm or task events) are included in the perf.data file.
696
697 To see buildids included in the perf.data file use the command:
698
699 perf buildid-list
700
701 If the perf.data file contains Intel PT data, that is the same as:
702
703 perf buildid-list --with-hits
704
705 Snapshot mode and event disabling
706 In order to make a snapshot, the intel_pt event is disabled using an
707 IOCTL, namely PERF_EVENT_IOC_DISABLE. However doing that can also
708 disable the collection of side-band information. In order to prevent
709 that, a dummy software event has been introduced that permits tracking
710 events (like mmaps) to continue to be recorded while intel_pt is
711 disabled. That is important to ensure there is complete side-band
712 information to allow the decoding of subsequent snapshots.
713
714 A test has been created for that. To find the test:
715
716 perf test list
717 ...
718 23: Test using a dummy software event to keep tracking
719
720 To run the test:
721
722 perf test 23
723 23: Test using a dummy software event to keep tracking : Ok
724
725 perf record modes (nothing new here)
726 perf record essentially operates in one of three modes: per thread per
727 cpu workload only
728
729 "per thread" mode is selected by -t or by --per-thread (with -p or -u
730 or just a workload). "per cpu" is selected by -C or -a. "workload only"
731 mode is selected by not using the other options but providing a command
732 to run (i.e. the workload).
733
734 In per-thread mode an exact list of threads is traced. There is no
735 inheritance. Each thread has its own event buffer.
736
737 In per-cpu mode all processes (or processes from the selected cgroup
738 i.e. -G option, or processes selected with -p or -u) are traced. Each
739 cpu has its own buffer. Inheritance is allowed.
740
741 In workload-only mode, the workload is traced but with per-cpu buffers.
742 Inheritance is allowed. Note that you can now trace a workload in
743 per-thread mode by using the --per-thread option.
744
745 Privileged vs non-privileged users
746 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
747 users have memory limits imposed upon them. That affects what buffer
748 sizes they can have as outlined above.
749
750 The v4.2 kernel introduced support for a context switch metadata event,
751 PERF_RECORD_SWITCH, which allows unprivileged users to see when their
752 processes are scheduled out and in, just not by whom, which is left for
753 the PERF_RECORD_SWITCH_CPU_WIDE, that is only accessible in system wide
754 context, which in turn requires CAP_PERFMON or CAP_SYS_ADMIN.
755
756 Please see the 45ac1403f564 ("perf: Add PERF_RECORD_SWITCH to indicate
757 context switches") commit, that introduces these metadata events for
758 further info.
759
760 When working with kernels < v4.2, the following considerations must be
761 taken, as the sched:sched_switch tracepoints will be used to receive
762 such information:
763
764 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
765 users are not permitted to use tracepoints which means there is
766 insufficient side-band information to decode Intel PT in per-cpu mode,
767 and potentially workload-only mode too if the workload creates new
768 processes.
769
770 Note also, that to use tracepoints, read-access to debugfs is required.
771 So if debugfs is not mounted or the user does not have read-access, it
772 will again not be possible to decode Intel PT in per-cpu mode.
773
774 sched_switch tracepoint
775 The sched_switch tracepoint is used to provide side-band data for Intel
776 PT decoding in kernels where the PERF_RECORD_SWITCH metadata event
777 isn’t available.
778
779 The sched_switch events are automatically added. e.g. the second event
780 shown below:
781
782 $ perf record -vv -e intel_pt//u uname
783 ------------------------------------------------------------
784 perf_event_attr:
785 type 6
786 size 112
787 config 0x400
788 { sample_period, sample_freq } 1
789 sample_type IP|TID|TIME|CPU|IDENTIFIER
790 read_format ID
791 disabled 1
792 inherit 1
793 exclude_kernel 1
794 exclude_hv 1
795 enable_on_exec 1
796 sample_id_all 1
797 ------------------------------------------------------------
798 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
799 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
800 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
801 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
802 ------------------------------------------------------------
803 perf_event_attr:
804 type 2
805 size 112
806 config 0x108
807 { sample_period, sample_freq } 1
808 sample_type IP|TID|TIME|CPU|PERIOD|RAW|IDENTIFIER
809 read_format ID
810 inherit 1
811 sample_id_all 1
812 exclude_guest 1
813 ------------------------------------------------------------
814 sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8
815 sys_perf_event_open: pid -1 cpu 1 group_fd -1 flags 0x8
816 sys_perf_event_open: pid -1 cpu 2 group_fd -1 flags 0x8
817 sys_perf_event_open: pid -1 cpu 3 group_fd -1 flags 0x8
818 ------------------------------------------------------------
819 perf_event_attr:
820 type 1
821 size 112
822 config 0x9
823 { sample_period, sample_freq } 1
824 sample_type IP|TID|TIME|IDENTIFIER
825 read_format ID
826 disabled 1
827 inherit 1
828 exclude_kernel 1
829 exclude_hv 1
830 mmap 1
831 comm 1
832 enable_on_exec 1
833 task 1
834 sample_id_all 1
835 mmap2 1
836 comm_exec 1
837 ------------------------------------------------------------
838 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
839 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
840 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
841 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
842 mmap size 528384B
843 AUX area mmap length 4194304
844 perf event ring buffer mmapped per cpu
845 Synthesizing auxtrace information
846 Linux
847 [ perf record: Woken up 1 times to write data ]
848 [ perf record: Captured and wrote 0.042 MB perf.data ]
849
850 Note, the sched_switch event is only added if the user is permitted to
851 use it and only in per-cpu mode.
852
853 Note also, the sched_switch event is only added if TSC packets are
854 requested. That is because, in the absence of timing information, the
855 sched_switch events cannot be matched against the Intel PT trace.
856
858 By default, perf script will decode trace data found in the perf.data
859 file. This can be further controlled by new option --itrace.
860
861 New --itrace option
862 Having no option is the same as
863
864 --itrace
865
866 which, in turn, is the same as
867
868 --itrace=cepwx
869
870 The letters are:
871
872 i synthesize "instructions" events
873 b synthesize "branches" events
874 x synthesize "transactions" events
875 w synthesize "ptwrite" events
876 p synthesize "power" events (incl. PSB events)
877 c synthesize branches events (calls only)
878 r synthesize branches events (returns only)
879 o synthesize PEBS-via-PT events
880 I synthesize Event Trace events
881 e synthesize tracing error events
882 d create a debug log
883 g synthesize a call chain (use with i or x)
884 G synthesize a call chain on existing event records
885 l synthesize last branch entries (use with i or x)
886 L synthesize last branch entries on existing event records
887 s skip initial number of events
888 q quicker (less detailed) decoding
889 A approximate IPC
890 Z prefer to ignore timestamps (so-called "timeless" decoding)
891
892 "Instructions" events look like they were recorded by "perf record -e
893 instructions".
894
895 "Branches" events look like they were recorded by "perf record -e
896 branches". "c" and "r" can be combined to get calls and returns.
897
898 "Transactions" events correspond to the start or end of transactions.
899 The flags field can be used in perf script to determine whether the
900 event is a transaction start, commit or abort.
901
902 Note that "instructions", "branches" and "transactions" events depend
903 on code flow packets which can be disabled by using the config term
904 "branch=0". Refer to the config terms section above.
905
906 "ptwrite" events record the payload of the ptwrite instruction and
907 whether "fup_on_ptw" was used. "ptwrite" events depend on PTWRITE
908 packets which are recorded only if the "ptw" config term was used.
909 Refer to the config terms section above. perf script "synth" field
910 displays "ptwrite" information like this: "ip: 0 payload:
911 0x123456789abcdef0" where "ip" is 1 if "fup_on_ptw" was used.
912
913 "Power" events correspond to power event packets and CBR (core-to-bus
914 ratio) packets. While CBR packets are always recorded when tracing is
915 enabled, power event packets are recorded only if the "pwr_evt" config
916 term was used. Refer to the config terms section above. The power
917 events record information about C-state changes, whereas CBR is
918 indicative of CPU frequency. perf script "event,synth" fields display
919 information like this:
920
921 cbr: cbr: 22 freq: 2189 MHz (200%)
922 mwait: hints: 0x60 extensions: 0x1
923 pwre: hw: 0 cstate: 2 sub-cstate: 0
924 exstop: ip: 1
925 pwrx: deepest cstate: 2 last cstate: 2 wake reason: 0x4
926
927 Where:
928
929 "cbr" includes the frequency and the percentage of maximum non-turbo
930 "mwait" shows mwait hints and extensions
931 "pwre" shows C-state transitions (to a C-state deeper than C0) and
932 whether initiated by hardware
933 "exstop" indicates execution stopped and whether the IP was recorded
934 exactly,
935 "pwrx" indicates return to C0
936
937 For more details refer to the Intel 64 and IA-32 Architectures Software
938 Developer Manuals.
939
940 PSB events show when a PSB+ occurred and also the byte-offset in the
941 trace. Emitting a PSB+ can cause a CPU a slight delay. When doing
942 timing analysis of code with Intel PT, it is useful to know if a timing
943 bubble was caused by Intel PT or not.
944
945 Error events show where the decoder lost the trace. Error events are
946 quite important. Users must know if what they are seeing is a complete
947 picture or not. The "e" option may be followed by flags which affect
948 what errors will or will not be reported. Each flag must be preceded by
949 either + or -. The flags supported by Intel PT are:
950
951 -o Suppress overflow errors
952 -l Suppress trace data lost errors
953
954 For example, for errors but not overflow or data lost errors:
955
956 --itrace=e-o-l
957
958 The "d" option will cause the creation of a file "intel_pt.log"
959 containing all decoded packets and instructions. Note that this option
960 slows down the decoder and that the resulting file may be very large.
961 The "d" option may be followed by flags which affect what debug
962 messages will or will not be logged. Each flag must be preceded by
963 either + or -. The flags support by Intel PT are:
964
965 -a Suppress logging of perf events
966 +a Log all perf events
967 +e Output only on decoding errors (size configurable)
968 +o Output to stdout instead of "intel_pt.log"
969
970 By default, logged perf events are filtered by any specified time
971 ranges, but flag +a overrides that. The +e flag can be useful for
972 analyzing errors. By default, the log size in that case is 16384 bytes,
973 but can be altered by perf-config(1) e.g. perf config
974 itrace.debug-log-buffer-size=30000
975
976 In addition, the period of the "instructions" event can be specified.
977 e.g.
978
979 --itrace=i10us
980
981 sets the period to 10us i.e. one instruction sample is synthesized for
982 each 10 microseconds of trace. Alternatives to "us" are "ms"
983 (milliseconds), "ns" (nanoseconds), "t" (TSC ticks) or "i"
984 (instructions).
985
986 "ms", "us" and "ns" are converted to TSC ticks.
987
988 The timing information included with Intel PT does not give the time of
989 every instruction. Consequently, for the purpose of sampling, the
990 decoder estimates the time since the last timing packet based on 1 tick
991 per instruction. The time on the sample is not adjusted and reflects
992 the last known value of TSC.
993
994 For Intel PT, the default period is 100us.
995
996 Setting it to a zero period means "as often as possible".
997
998 In the case of Intel PT that is the same as a period of 1 and a unit of
999 instructions (i.e. --itrace=i1i).
1000
1001 Also the call chain size (default 16, max. 1024) for instructions or
1002 transactions events can be specified. e.g.
1003
1004 --itrace=ig32
1005 --itrace=xg32
1006
1007 Also the number of last branch entries (default 64, max. 1024) for
1008 instructions or transactions events can be specified. e.g.
1009
1010 --itrace=il10
1011 --itrace=xl10
1012
1013 Note that last branch entries are cleared for each sample, so there is
1014 no overlap from one sample to the next.
1015
1016 The G and L options are designed in particular for sample mode, and
1017 work much like g and l but add call chain and branch stack to the other
1018 selected events instead of synthesized events. For example, to record
1019 branch-misses events for ls and then add a call chain derived from the
1020 Intel PT trace:
1021
1022 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}' -- ls
1023 perf report --itrace=Ge
1024
1025 Although in fact G is a default for perf report, so that is the same as
1026 just:
1027
1028 perf report
1029
1030 One caveat with the G and L options is that they work poorly with
1031 "Large PEBS". Large PEBS means PEBS records will be accumulated by
1032 hardware and the written into the event buffer in one go. That reduces
1033 interrupts, but can give very late timestamps. Because the Intel PT
1034 trace is synchronized by timestamps, the PEBS events do not match the
1035 trace. Currently, Large PEBS is used only in certain circumstances: -
1036 hardware supports it - PEBS is used - event period is specified,
1037 instead of frequency - the sample type is limited to the following
1038 flags: PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_ADDR |
1039 PERF_SAMPLE_ID | PERF_SAMPLE_CPU | PERF_SAMPLE_STREAM_ID |
1040 PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_TRANSACTION
1041 | PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER
1042 | PERF_SAMPLE_PERIOD (and sometimes) | PERF_SAMPLE_TIME Because Intel
1043 PT sample mode uses a different sample type to the list above, Large
1044 PEBS is not used with Intel PT sample mode. To avoid Large PEBS in
1045 other cases, avoid specifying the event period i.e. avoid the perf
1046 record -c option, --count option, or period config term.
1047
1048 To disable trace decoding entirely, use the option --no-itrace.
1049
1050 It is also possible to skip events generated (instructions, branches,
1051 transactions) at the beginning. This is useful to ignore initialization
1052 code.
1053
1054 --itrace=i0nss1000000
1055
1056 skips the first million instructions.
1057
1058 The q option changes the way the trace is decoded. The decoding is much
1059 faster but much less detailed. Specifically, with the q option, the
1060 decoder does not decode TNT packets, and does not walk object code, but
1061 gets the ip from FUP and TIP packets. The q option can be used with the
1062 b and i options but the period is not used. The q option decodes more
1063 quickly, but is useful only if the control flow of interest is
1064 represented or indicated by FUP, TIP, TIP.PGE, or TIP.PGD packets
1065 (refer below). However the q option could be used to find time ranges
1066 that could then be decoded fully using the --time option.
1067
1068 What will not be decoded with the (single) q option:
1069
1070 • direct calls and jmps
1071
1072 • conditional branches
1073
1074 • non-branch instructions
1075
1076 What will be decoded with the (single) q option:
1077
1078 • asynchronous branches such as interrupts
1079
1080 • indirect branches
1081
1082 • function return target address if the noretcomp config term (refer
1083 config terms section) was used
1084
1085 • start of (control-flow) tracing
1086
1087 • end of (control-flow) tracing, if it is not out of context
1088
1089 • power events, ptwrite, transaction start and abort
1090
1091 • instruction pointer associated with PSB packets
1092
1093 Note the q option does not specify what events will be synthesized e.g.
1094 the p option must be used also to show power events.
1095
1096 Repeating the q option (double-q i.e. qq) results in even faster
1097 decoding and even less detail. The decoder decodes only extended PSB
1098 (PSB+) packets, getting the instruction pointer if there is a FUP
1099 packet within PSB+ (i.e. between PSB and PSBEND). Note PSB packets
1100 occur regularly in the trace based on the psb_period config term (refer
1101 config terms section). There will be a FUP packet if the PSB+ occurs
1102 while control flow is being traced.
1103
1104 What will not be decoded with the qq option:
1105
1106 • everything except instruction pointer associated with PSB packets
1107
1108 What will be decoded with the qq option:
1109
1110 • instruction pointer associated with PSB packets
1111
1112 The Z option is equivalent to having recorded a trace without TSC (i.e.
1113 config term tsc=0). It can be useful to avoid timestamp issues when
1114 decoding a trace of a virtual machine.
1115
1116 dlfilter-show-cycles.so
1117 Cycles can be displayed using dlfilter-show-cycles.so in which case the
1118 itrace A option can be useful to provide higher granularity cycle
1119 information:
1120
1121 perf script --itrace=A --call-trace --dlfilter dlfilter-show-cycles.so
1122
1123 To see a list of dlfilters:
1124
1125 perf script -v --list-dlfilters
1126
1127 See also perf-dlfilters(1)
1128
1129 dump option
1130 perf script has an option (-D) to "dump" the events i.e. display the
1131 binary data.
1132
1133 When -D is used, Intel PT packets are displayed. The packet decoder
1134 does not pay attention to PSB packets, but just decodes the bytes - so
1135 the packets seen by the actual decoder may not be identical in places
1136 where the data is corrupt. One example of that would be when the
1137 buffer-switching interrupt has been too slow, and the buffer has been
1138 filled completely. In that case, the last packet in the buffer might be
1139 truncated and immediately followed by a PSB as the trace continues in
1140 the next buffer.
1141
1142 To disable the display of Intel PT packets, combine the -D option with
1143 --no-itrace.
1144
1146 By default, perf report will decode trace data found in the perf.data
1147 file. This can be further controlled by new option --itrace exactly the
1148 same as perf script, with the exception that the default is
1149 --itrace=igxe.
1150
1152 perf inject also accepts the --itrace option in which case tracing data
1153 is removed and replaced with the synthesized events. e.g.
1154
1155 perf inject --itrace -i perf.data -o perf.data.new
1156
1157 Below is an example of using Intel PT with autofdo. It requires autofdo
1158 (https://github.com/google/autofdo) and gcc version 5. The bubble sort
1159 example is from the AutoFDO tutorial
1160 (https://gcc.gnu.org/wiki/AutoFDO/Tutorial) amended to take the number
1161 of elements as a parameter.
1162
1163 $ gcc-5 -O3 sort.c -o sort_optimized
1164 $ ./sort_optimized 30000
1165 Bubble sorting array of 30000 elements
1166 2254 ms
1167
1168 $ cat ~/.perfconfig
1169 [intel-pt]
1170 mispred-all = on
1171
1172 $ perf record -e intel_pt//u ./sort 3000
1173 Bubble sorting array of 3000 elements
1174 58 ms
1175 [ perf record: Woken up 2 times to write data ]
1176 [ perf record: Captured and wrote 3.939 MB perf.data ]
1177 $ perf inject -i perf.data -o inj --itrace=i100usle --strip
1178 $ ./create_gcov --binary=./sort --profile=inj --gcov=sort.gcov -gcov_version=1
1179 $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
1180 $ ./sort_autofdo 30000
1181 Bubble sorting array of 30000 elements
1182 2155 ms
1183
1184 Note there is currently no advantage to using Intel PT instead of LBR,
1185 but that may change in the future if greater use is made of the data.
1186
1188 Some hardware has the feature to redirect PEBS records to the Intel PT
1189 trace. Recording is selected by using the aux-output config term e.g.
1190
1191 perf record -c 10000 -e '{intel_pt/branch=0/,cycles/aux-output/ppp}' uname
1192
1193 Originally, software only supported redirecting at most one PEBS event
1194 because it was not able to differentiate one event from another. To
1195 overcome that, more recent kernels and perf tools add support for the
1196 PERF_RECORD_AUX_OUTPUT_HW_ID side-band event. To check for the presence
1197 of that event in a PEBS-via-PT trace:
1198
1199 perf script -D --no-itrace | grep PERF_RECORD_AUX_OUTPUT_HW_ID
1200
1201 To display PEBS events from the Intel PT trace, use the itrace o option
1202 e.g.
1203
1204 perf script --itrace=oe
1205
1207 For --xed the xed tool is needed. Here is how to install it:
1208
1209 $ git clone https://github.com/intelxed/mbuild.git mbuild
1210 $ git clone https://github.com/intelxed/xed
1211 $ cd xed
1212 $ ./mfile.py --share
1213 $ ./mfile.py examples
1214 $ sudo ./mfile.py --prefix=/usr/local install
1215 $ sudo ldconfig
1216 $ sudo cp obj/examples/xed /usr/local/bin
1217
1218 Basic xed testing:
1219
1220 $ xed | head -3
1221 ERROR: required argument(s) were missing
1222 Copyright (C) 2017, Intel Corporation. All rights reserved.
1223 XED version: [v10.0-328-g7d62c8c49b7b]
1224 $
1225
1227 Currently, kernel tracing is supported with either "timeless" decoding
1228 (i.e. no TSC timestamps) or VM Time Correlation. VM Time Correlation is
1229 an extra step using perf inject and requires unchanging VMX TSC Offset
1230 and no VMX TSC Scaling.
1231
1232 Other limitations and caveats
1233
1234 VMX controls may suppress packets needed for decoding resulting in decoding errors
1235 VMX controls may block the perf NMI to the host potentially resulting in lost trace data
1236 Guest kernel self-modifying code (e.g. jump labels or JIT-compiled eBPF) will result in decoding errors
1237 Guest thread information is unknown
1238 Guest VCPU is unknown but may be able to be inferred from the host thread
1239 Callchains are not supported
1240
1241 Example using "timeless" decoding
1242
1243 Start VM
1244
1245 $ sudo virsh start kubuntu20.04
1246 Domain kubuntu20.04 started
1247
1248 Mount the guest file system. Note sshfs needs -o direct_io to enable
1249 reading of proc files. root access is needed to read /proc/kcore.
1250
1251 $ mkdir vm0
1252 $ sshfs -o direct_io root@vm0:/ vm0
1253
1254 Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
1255
1256 $ perf buildid-cache -v --kcore vm0/proc/kcore
1257 kcore added to build-id cache directory /home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306
1258 $ KALLSYMS=/home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306/kallsyms
1259
1260 Find the VM process
1261
1262 $ ps -eLl | grep 'KVM\|PID'
1263 F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
1264 3 S 64055 1430 1 1440 1 80 0 - 1921718 - ? 00:02:47 CPU 0/KVM
1265 3 S 64055 1430 1 1441 1 80 0 - 1921718 - ? 00:02:41 CPU 1/KVM
1266 3 S 64055 1430 1 1442 1 80 0 - 1921718 - ? 00:02:38 CPU 2/KVM
1267 3 S 64055 1430 1 1443 2 80 0 - 1921718 - ? 00:03:18 CPU 3/KVM
1268
1269 Start an open-ended perf record, tracing the VM process, do something
1270 on the VM, and then ctrl-C to stop. TSC is not supported and tsc=0 must
1271 be specified. That means mtc is useless, so add mtc=0. However, IPC can
1272 still be determined, hence cyc=1 can be added. Only kernel decoding is
1273 supported, so k must be specified. Intel PT traces both the host and
1274 the guest so --guest and --host need to be specified. Without
1275 timestamps, --per-thread must be specified to distinguish threads.
1276
1277 $ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/tsc=0,mtc=0,cyc=1/k -p 1430 --per-thread
1278 ^C
1279 [ perf record: Woken up 1 times to write data ]
1280 [ perf record: Captured and wrote 5.829 MB ]
1281
1282 perf script can be used to provide an instruction trace
1283
1284 $ perf script --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
1285 CPU 0/KVM 1440 ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
1286 CPU 0/KVM 1440 ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
1287 CPU 0/KVM 1440 ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
1288 CPU 0/KVM 1440 ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
1289 CPU 0/KVM 1440 ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
1290 CPU 0/KVM 1440 ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
1291 CPU 0/KVM 1440 ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
1292 CPU 0/KVM 1440 ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
1293 CPU 0/KVM 1440 ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
1294 CPU 0/KVM 1440 ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
1295 CPU 0/KVM 1440 ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.11 (50/445)
1296 :1440 1440 ffffffffbb678b06 native_write_msr+0x6 ([guest.kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
1297 :1440 1440 ffffffffbb678b0b native_write_msr+0xb ([guest.kernel.kallsyms]) retq IPC: 0.04 (2/41)
1298 :1440 1440 ffffffffbb666646 lapic_next_deadline+0x26 ([guest.kernel.kallsyms]) data16 nop
1299 :1440 1440 ffffffffbb666648 lapic_next_deadline+0x28 ([guest.kernel.kallsyms]) xor %eax, %eax
1300 :1440 1440 ffffffffbb66664a lapic_next_deadline+0x2a ([guest.kernel.kallsyms]) popq %rbp
1301 :1440 1440 ffffffffbb66664b lapic_next_deadline+0x2b ([guest.kernel.kallsyms]) retq IPC: 0.16 (4/25)
1302 :1440 1440 ffffffffbb74607f clockevents_program_event+0x8f ([guest.kernel.kallsyms]) test %eax, %eax
1303 :1440 1440 ffffffffbb746081 clockevents_program_event+0x91 ([guest.kernel.kallsyms]) jz 0xffffffffbb74603c IPC: 0.06 (2/30)
1304 :1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx
1305 :1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12
1306
1307 Example using VM Time Correlation
1308
1309 Start VM
1310
1311 $ sudo virsh start kubuntu20.04
1312 Domain kubuntu20.04 started
1313
1314 Mount the guest file system. Note sshfs needs -o direct_io to enable
1315 reading of proc files. root access is needed to read /proc/kcore.
1316
1317 $ mkdir -p vm0
1318 $ sshfs -o direct_io root@vm0:/ vm0
1319
1320 Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
1321
1322 $ perf buildid-cache -v --kcore vm0/proc/kcore
1323 same kcore found in /home/user/.debug/[kernel.kcore]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777
1324 $ KALLSYMS=/home/user/.debug/\[kernel.kcore\]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777/kallsyms
1325
1326 Find the VM process
1327
1328 $ ps -eLl | grep 'KVM\|PID'
1329 F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
1330 3 S 64055 16998 1 17005 13 80 0 - 1818189 - ? 00:00:16 CPU 0/KVM
1331 3 S 64055 16998 1 17006 4 80 0 - 1818189 - ? 00:00:05 CPU 1/KVM
1332 3 S 64055 16998 1 17007 3 80 0 - 1818189 - ? 00:00:04 CPU 2/KVM
1333 3 S 64055 16998 1 17008 4 80 0 - 1818189 - ? 00:00:05 CPU 3/KVM
1334
1335 Start an open-ended perf record, tracing the VM process, do something
1336 on the VM, and then ctrl-C to stop. IPC can be determined, hence cyc=1
1337 can be added. Only kernel decoding is supported, so k must be
1338 specified. Intel PT traces both the host and the guest so --guest and
1339 --host need to be specified.
1340
1341 $ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/cyc=1/k -p 16998
1342 ^C[ perf record: Woken up 1 times to write data ]
1343 [ perf record: Captured and wrote 9.041 MB perf.data.kvm ]
1344
1345 Now perf inject can be used to determine the VMX TCS Offset. Note,
1346 Intel PT TSC packets are only 7-bytes, so the TSC Offset might differ
1347 from the actual value in the 8th byte. That will have no effect i.e.
1348 the resulting timestamps will be correct anyway.
1349
1350 $ perf inject -i perf.data.kvm --vm-time-correlation=dry-run
1351 ERROR: Unknown TSC Offset for VMCS 0x1bff6a
1352 VMCS: 0x1bff6a TSC Offset 0xffffe42722c64c41
1353 ERROR: Unknown TSC Offset for VMCS 0x1cbc08
1354 VMCS: 0x1cbc08 TSC Offset 0xffffe42722c64c41
1355 ERROR: Unknown TSC Offset for VMCS 0x1c3ce8
1356 VMCS: 0x1c3ce8 TSC Offset 0xffffe42722c64c41
1357 ERROR: Unknown TSC Offset for VMCS 0x1cbce9
1358 VMCS: 0x1cbce9 TSC Offset 0xffffe42722c64c41
1359
1360 Each virtual CPU has a different Virtual Machine Control Structure
1361 (VMCS) shown above with the calculated TSC Offset. For an unchanging
1362 TSC Offset they should all be the same for the same virtual machine.
1363
1364 Now that the TSC Offset is known, it can be provided to perf inject
1365
1366 $ perf inject -i perf.data.kvm --vm-time-correlation="dry-run 0xffffe42722c64c41"
1367
1368 Note the options for perf inject --vm-time-correlation are:
1369
1370 [ dry-run ] [ <TSC Offset> [ : <VMCS> [ , <VMCS> ]... ] ]...
1371
1372 So it is possible to specify different TSC Offsets for different VMCS.
1373 The option "dry-run" will cause the file to be processed but without
1374 updating it. Note it is also possible to get a intel_pt.log file by
1375 adding option --itrace=d
1376
1377 There were no errors so, do it for real
1378
1379 $ perf inject -i perf.data.kvm --vm-time-correlation=0xffffe42722c64c41 --force
1380
1381 perf script can be used to see if there are any decoder errors
1382
1383 $ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --itrace=e-o
1384
1385 There were none.
1386
1387 perf script can be used to provide an instruction trace showing
1388 timestamps
1389
1390 $ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
1391 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
1392 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
1393 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
1394 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
1395 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
1396 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
1397 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
1398 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
1399 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
1400 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
1401 CPU 1/KVM 17006 [001] 11500.262866075: ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.05 (40/769)
1402 :17006 17006 [001] 11500.262869216: ffffffff82200cb0 asm_sysvec_apic_timer_interrupt+0x0 ([guest.kernel.kallsyms]) clac
1403 :17006 17006 [001] 11500.262869216: ffffffff82200cb3 asm_sysvec_apic_timer_interrupt+0x3 ([guest.kernel.kallsyms]) pushq $0xffffffffffffffff
1404 :17006 17006 [001] 11500.262869216: ffffffff82200cb5 asm_sysvec_apic_timer_interrupt+0x5 ([guest.kernel.kallsyms]) callq 0xffffffff82201160
1405 :17006 17006 [001] 11500.262869216: ffffffff82201160 error_entry+0x0 ([guest.kernel.kallsyms]) cld
1406 :17006 17006 [001] 11500.262869216: ffffffff82201161 error_entry+0x1 ([guest.kernel.kallsyms]) pushq %rsi
1407 :17006 17006 [001] 11500.262869216: ffffffff82201162 error_entry+0x2 ([guest.kernel.kallsyms]) movq 0x8(%rsp), %rsi
1408 :17006 17006 [001] 11500.262869216: ffffffff82201167 error_entry+0x7 ([guest.kernel.kallsyms]) movq %rdi, 0x8(%rsp)
1409 :17006 17006 [001] 11500.262869216: ffffffff8220116c error_entry+0xc ([guest.kernel.kallsyms]) pushq %rdx
1410 :17006 17006 [001] 11500.262869216: ffffffff8220116d error_entry+0xd ([guest.kernel.kallsyms]) pushq %rcx
1411 :17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax
1412
1414 It is possible to use perf record to record sideband events within a
1415 virtual machine, so that an Intel PT trace on the host can be decoded.
1416 Sideband events from the guest perf.data file can be injected into the
1417 host perf.data file using perf inject.
1418
1419 Here is an example of the steps needed:
1420
1421 On the guest machine:
1422
1423 Check that no-kvmclock kernel command line option was used to boot:
1424
1425 Note, this is essential to enable time correlation between host and
1426 guest machines.
1427
1428 $ cat /proc/cmdline
1429 BOOT_IMAGE=/boot/vmlinuz-5.10.0-16-amd64 root=UUID=cb49c910-e573-47e0-bce7-79e293df8e1d ro no-kvmclock
1430
1431 There is no BPF support at present so, if possible, disable JIT
1432 compiling:
1433
1434 $ echo 0 | sudo tee /proc/sys/net/core/bpf_jit_enable
1435 0
1436
1437 Start perf record to collect sideband events:
1438
1439 $ sudo perf record -o guest-sideband-testing-guest-perf.data --sample-identifier --buildid-all --switch-events --kcore -a -e dummy
1440
1441 On the host machine:
1442
1443 Start perf record to collect Intel PT trace:
1444
1445 Note, the host trace will get very big, very fast, so the steps from
1446 starting to stopping the host trace really need to be done so that they
1447 happen in the shortest time possible.
1448
1449 $ sudo perf record -o guest-sideband-testing-host-perf.data -m,64M --kcore -a -e intel_pt/cyc/
1450
1451 On the guest machine:
1452
1453 Run a small test case, just uname in this example:
1454
1455 $ uname
1456 Linux
1457
1458 On the host machine:
1459
1460 Stop the Intel PT trace:
1461
1462 ^C
1463 [ perf record: Woken up 1 times to write data ]
1464 [ perf record: Captured and wrote 76.122 MB guest-sideband-testing-host-perf.data ]
1465
1466 On the guest machine:
1467
1468 Stop the Intel PT trace:
1469
1470 ^C
1471 [ perf record: Woken up 1 times to write data ]
1472 [ perf record: Captured and wrote 1.247 MB guest-sideband-testing-guest-perf.data ]
1473
1474 And then copy guest-sideband-testing-guest-perf.data to the host (not
1475 shown here).
1476
1477 On the host machine:
1478
1479 With the 2 perf.data recordings, and with their ownership changed to
1480 the user.
1481
1482 Identify the TSC Offset:
1483
1484 $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=dry-run
1485 VMCS: 0x103fc6 TSC Offset 0xfffffa6ae070cb20
1486 VMCS: 0x103ff2 TSC Offset 0xfffffa6ae070cb20
1487 VMCS: 0x10fdaa TSC Offset 0xfffffa6ae070cb20
1488 VMCS: 0x24d57c TSC Offset 0xfffffa6ae070cb20
1489
1490 Correct Intel PT TSC timestamps for the guest machine:
1491
1492 $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=0xfffffa6ae070cb20 --force
1493
1494 Identify the guest machine PID:
1495
1496 $ perf script -i guest-sideband-testing-host-perf.data --no-itrace --show-task-events | grep KVM
1497 CPU 0/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 0/KVM:13376/13381
1498 CPU 1/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 1/KVM:13376/13382
1499 CPU 2/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 2/KVM:13376/13383
1500 CPU 3/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 3/KVM:13376/13384
1501
1502 Note, the QEMU option -name debug-threads=on is needed so that thread
1503 names can be used to determine which thread is running which VCPU as
1504 above. libvirt seems to use this by default.
1505
1506 Create a guestmount, assuming the guest machine is vm_to_test:
1507
1508 $ mkdir -p ~/guestmount/13376
1509 $ sshfs -o direct_io vm_to_test:/ ~/guestmount/13376
1510
1511 Inject the guest perf.data file into the host perf.data file:
1512
1513 Note, due to the guestmount option, guest object files and debug files
1514 will be copied into the build ID cache from the guest machine, with the
1515 notable exception of VDSO. If needed, VDSO can be copied manually in a
1516 fashion similar to that used by the perf-archive script.
1517
1518 $ perf inject -i guest-sideband-testing-host-perf.data -o inj --guestmount ~/guestmount --guest-data=guest-sideband-testing-guest-perf.data,13376,0xfffffa6ae070cb20
1519
1520 Show an excerpt from the result. In this case the CPU and time range
1521 have been to chosen to show interaction between guest and host when
1522 uname is starting to run on the guest machine:
1523
1524 Notes:
1525
1526 • the CPU displayed, [002] in this case, is always the host CPU
1527
1528 • events happening in the virtual machine start with VM:13376
1529 VCPU:003, which shows the hypervisor PID 13376 and the VCPU number
1530
1531 • only calls and errors are displayed i.e. --itrace=ce
1532
1533 • branches entering and exiting the virtual machine are split, and
1534 show as 2 branches to/from "0 [unknown] ([unknown])"
1535
1536 $ perf script -i inj --itrace=ce -F+machine_pid,+vcpu,+addr,+pid,+tid,-period --ns --time 7919.408803365,7919.408804631 -C 2
1537 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms])
1538 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms])
1539 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms])
1540 CPU 3/KVM 13376/13384 [002] 7919.408803461: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown])
1541 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803461: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so)
1542 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803567: branches: 7f851c9b5a5a init_cacheinfo+0x3aa (/usr/lib/x86_64-linux-gnu/libc-2.31.so) => 0 [unknown] ([unknown])
1543 CPU 3/KVM 13376/13384 [002] 7919.408803567: branches: 0 [unknown] ([unknown]) => ffffffffc0f8ed80 vmx_vmexit+0x0 ([kernel.kallsyms])
1544 CPU 3/KVM 13376/13384 [002] 7919.408803596: branches: ffffffffc0f6619a vmx_vcpu_run+0x26a ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms])
1545 CPU 3/KVM 13376/13384 [002] 7919.408803801: branches: ffffffffc0f66445 vmx_vcpu_run+0x515 ([kernel.kallsyms]) => ffffffffb2290b30 native_write_msr+0x0 ([kernel.kallsyms])
1546 CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc0f661f8 vmx_vcpu_run+0x2c8 ([kernel.kallsyms]) => ffffffffc1092300 kvm_load_host_xsave_state+0x0 ([kernel.kallsyms])
1547 CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc1092327 kvm_load_host_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092220 kvm_load_host_xsave_state.part.0+0x0 ([kernel.kallsyms])
1548 CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662cf vmx_vcpu_run+0x39f ([kernel.kallsyms]) => ffffffffc0f63f90 vmx_recover_nmi_blocking+0x0 ([kernel.kallsyms])
1549 CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662e9 vmx_vcpu_run+0x3b9 ([kernel.kallsyms]) => ffffffffc0f619a0 __vmx_complete_interrupts+0x0 ([kernel.kallsyms])
1550 CPU 3/KVM 13376/13384 [002] 7919.408803872: branches: ffffffffc109cfb2 vcpu_enter_guest+0x752 ([kernel.kallsyms]) => ffffffffc0f5f570 vmx_handle_exit_irqoff+0x0 ([kernel.kallsyms])
1551 CPU 3/KVM 13376/13384 [002] 7919.408803881: branches: ffffffffc109d028 vcpu_enter_guest+0x7c8 ([kernel.kallsyms]) => ffffffffb234f900 __srcu_read_lock+0x0 ([kernel.kallsyms])
1552 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc109d06f vcpu_enter_guest+0x80f ([kernel.kallsyms]) => ffffffffc0f72e30 vmx_handle_exit+0x0 ([kernel.kallsyms])
1553 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72e3d vmx_handle_exit+0xd ([kernel.kallsyms]) => ffffffffc0f727c0 __vmx_handle_exit+0x0 ([kernel.kallsyms])
1554 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72b15 __vmx_handle_exit+0x355 ([kernel.kallsyms]) => ffffffffc0f60ae0 vmx_flush_pml_buffer+0x0 ([kernel.kallsyms])
1555 CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc0f72994 __vmx_handle_exit+0x1d4 ([kernel.kallsyms]) => ffffffffc10b7090 kvm_emulate_cpuid+0x0 ([kernel.kallsyms])
1556 CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc10b70f1 kvm_emulate_cpuid+0x61 ([kernel.kallsyms]) => ffffffffc10b6e10 kvm_cpuid+0x0 ([kernel.kallsyms])
1557 CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc10b7125 kvm_emulate_cpuid+0x95 ([kernel.kallsyms]) => ffffffffc1093110 kvm_skip_emulated_instruction+0x0 ([kernel.kallsyms])
1558 CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc109311f kvm_skip_emulated_instruction+0xf ([kernel.kallsyms]) => ffffffffc0f5e180 vmx_get_rflags+0x0 ([kernel.kallsyms])
1559 CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc109312a kvm_skip_emulated_instruction+0x1a ([kernel.kallsyms]) => ffffffffc0f5fd30 vmx_skip_emulated_instruction+0x0 ([kernel.kallsyms])
1560 CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc0f5fd79 vmx_skip_emulated_instruction+0x49 ([kernel.kallsyms]) => ffffffffc0f5fb50 skip_emulated_instruction+0x0 ([kernel.kallsyms])
1561 CPU 3/KVM 13376/13384 [002] 7919.408803956: branches: ffffffffc0f5fc68 skip_emulated_instruction+0x118 ([kernel.kallsyms]) => ffffffffc0f6a940 vmx_cache_reg+0x0 ([kernel.kallsyms])
1562 CPU 3/KVM 13376/13384 [002] 7919.408803964: branches: ffffffffc0f5fc11 skip_emulated_instruction+0xc1 ([kernel.kallsyms]) => ffffffffc0f5f9e0 vmx_set_interrupt_shadow+0x0 ([kernel.kallsyms])
1563 CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc109f8b1 vcpu_run+0x71 ([kernel.kallsyms]) => ffffffffc10ad2f0 kvm_cpu_has_pending_timer+0x0 ([kernel.kallsyms])
1564 CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc10ad2fb kvm_cpu_has_pending_timer+0xb ([kernel.kallsyms]) => ffffffffc10b0490 apic_has_pending_timer+0x0 ([kernel.kallsyms])
1565 CPU 3/KVM 13376/13384 [002] 7919.408803991: branches: ffffffffc109f899 vcpu_run+0x59 ([kernel.kallsyms]) => ffffffffc109c860 vcpu_enter_guest+0x0 ([kernel.kallsyms])
1566 CPU 3/KVM 13376/13384 [002] 7919.408803993: branches: ffffffffc109cd4c vcpu_enter_guest+0x4ec ([kernel.kallsyms]) => ffffffffc0f69140 vmx_prepare_switch_to_guest+0x0 ([kernel.kallsyms])
1567 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd7d vcpu_enter_guest+0x51d ([kernel.kallsyms]) => ffffffffb234f930 __srcu_read_unlock+0x0 ([kernel.kallsyms])
1568 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd9c vcpu_enter_guest+0x53c ([kernel.kallsyms]) => ffffffffc0f609b0 vmx_sync_pir_to_irr+0x0 ([kernel.kallsyms])
1569 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc0f60a6d vmx_sync_pir_to_irr+0xbd ([kernel.kallsyms]) => ffffffffc10adc20 kvm_lapic_find_highest_irr+0x0 ([kernel.kallsyms])
1570 CPU 3/KVM 13376/13384 [002] 7919.408804010: branches: ffffffffc0f60abd vmx_sync_pir_to_irr+0x10d ([kernel.kallsyms]) => ffffffffc0f60820 vmx_set_rvi+0x0 ([kernel.kallsyms])
1571 CPU 3/KVM 13376/13384 [002] 7919.408804019: branches: ffffffffc109ceca vcpu_enter_guest+0x66a ([kernel.kallsyms]) => ffffffffb2249840 fpregs_assert_state_consistent+0x0 ([kernel.kallsyms])
1572 CPU 3/KVM 13376/13384 [002] 7919.408804021: branches: ffffffffc109cf10 vcpu_enter_guest+0x6b0 ([kernel.kallsyms]) => ffffffffc0f65f30 vmx_vcpu_run+0x0 ([kernel.kallsyms])
1573 CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f6603b vmx_vcpu_run+0x10b ([kernel.kallsyms]) => ffffffffb229bed0 __get_current_cr3_fast+0x0 ([kernel.kallsyms])
1574 CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f66055 vmx_vcpu_run+0x125 ([kernel.kallsyms]) => ffffffffb2253050 cr4_read_shadow+0x0 ([kernel.kallsyms])
1575 CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc0f6608d vmx_vcpu_run+0x15d ([kernel.kallsyms]) => ffffffffc10921e0 kvm_load_guest_xsave_state+0x0 ([kernel.kallsyms])
1576 CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc1092207 kvm_load_guest_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092110 kvm_load_guest_xsave_state.part.0+0x0 ([kernel.kallsyms])
1577 CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffc0f660c6 vmx_vcpu_run+0x196 ([kernel.kallsyms]) => ffffffffb22061a0 perf_guest_get_msrs+0x0 ([kernel.kallsyms])
1578 CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffb22061a9 perf_guest_get_msrs+0x9 ([kernel.kallsyms]) => ffffffffb220cda0 intel_guest_get_msrs+0x0 ([kernel.kallsyms])
1579 CPU 3/KVM 13376/13384 [002] 7919.408804039: branches: ffffffffc0f66109 vmx_vcpu_run+0x1d9 ([kernel.kallsyms]) => ffffffffc0f652c0 clear_atomic_switch_msr+0x0 ([kernel.kallsyms])
1580 CPU 3/KVM 13376/13384 [002] 7919.408804040: branches: ffffffffc0f66119 vmx_vcpu_run+0x1e9 ([kernel.kallsyms]) => ffffffffc0f73f60 intel_pmu_lbr_is_enabled+0x0 ([kernel.kallsyms])
1581 CPU 3/KVM 13376/13384 [002] 7919.408804042: branches: ffffffffc0f73f81 intel_pmu_lbr_is_enabled+0x21 ([kernel.kallsyms]) => ffffffffc10b68e0 kvm_find_cpuid_entry+0x0 ([kernel.kallsyms])
1582 CPU 3/KVM 13376/13384 [002] 7919.408804045: branches: ffffffffc0f66454 vmx_vcpu_run+0x524 ([kernel.kallsyms]) => ffffffffc0f61ff0 vmx_update_hv_timer+0x0 ([kernel.kallsyms])
1583 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66142 vmx_vcpu_run+0x212 ([kernel.kallsyms]) => ffffffffc10af100 kvm_wait_lapic_expire+0x0 ([kernel.kallsyms])
1584 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66156 vmx_vcpu_run+0x226 ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms])
1585 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66161 vmx_vcpu_run+0x231 ([kernel.kallsyms]) => ffffffffc0f8eb20 vmx_vcpu_enter_exit+0x0 ([kernel.kallsyms])
1586 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f8eb44 vmx_vcpu_enter_exit+0x24 ([kernel.kallsyms]) => ffffffffb2353e10 rcu_note_context_switch+0x0 ([kernel.kallsyms])
1587 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffb2353e1c rcu_note_context_switch+0xc ([kernel.kallsyms]) => ffffffffb2353db0 rcu_qs+0x0 ([kernel.kallsyms])
1588 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms])
1589 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms])
1590 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms])
1591 CPU 3/KVM 13376/13384 [002] 7919.408804162: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown])
1592 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804162: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so)
1593 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804273: branches: 7f851cb7c0e4 _dl_init+0x74 (/usr/lib/x86_64-linux-gnu/ld-2.31.so) => 7f851cb7bf50 call_init.part.0+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)
1594 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: 55e0c00136f0 _start+0x0 (/usr/bin/uname) => ffffffff83200ac0 asm_exc_page_fault+0x0 ([kernel.kallsyms])
1595 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: ffffffff83200ac3 asm_exc_page_fault+0x3 ([kernel.kallsyms]) => ffffffff83201290 error_entry+0x0 ([kernel.kallsyms])
1596 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804534: branches: ffffffff832012fa error_entry+0x6a ([kernel.kallsyms]) => ffffffff830b59a0 sync_regs+0x0 ([kernel.kallsyms])
1597 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff83200ad9 asm_exc_page_fault+0x19 ([kernel.kallsyms]) => ffffffff830b8210 exc_page_fault+0x0 ([kernel.kallsyms])
1598 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b82a4 exc_page_fault+0x94 ([kernel.kallsyms]) => ffffffff830b80e0 __kvm_handle_async_pf+0x0 ([kernel.kallsyms])
1599 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b80ed __kvm_handle_async_pf+0xd ([kernel.kallsyms]) => ffffffff830b80c0 kvm_read_and_reset_apf_flags+0x0 ([kernel.kallsyms])
1600
1602 A common case for KVM test programs is that the test program acts as
1603 the hypervisor, creating, running and destroying the virtual machine,
1604 and providing the guest object code from its own object code. In this
1605 case, the VM is not running an OS, but only the functions loaded into
1606 it by the hypervisor test program, and conveniently, loaded at the same
1607 virtual addresses. To support that, option "--guest-code" has been
1608 added to perf script and perf kvm report.
1609
1610 Here is an example tracing a test program from the kernel’s KVM
1611 selftests:
1612
1613 # perf record --kcore -e intel_pt/cyc/ -- tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test
1614 [ perf record: Woken up 1 times to write data ]
1615 [ perf record: Captured and wrote 0.280 MB perf.data ]
1616 # perf script --guest-code --itrace=bep --ns -F-period,+addr,+flags
1617 [SNIP]
1618 tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux)
1619 tsc_msrs_test 18436 [007] 10897.962087733: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux)
1620 tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux)
1621 tsc_msrs_test 18436 [007] 10897.962087836: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown])
1622 [guest/18436] 18436 [007] 10897.962087836: branches: vmentry 0 [unknown] ([unknown]) => 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1623 [guest/18436] 18436 [007] 10897.962087836: branches: call 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1624 [guest/18436] 18436 [007] 10897.962088248: branches: vmexit 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown])
1625 tsc_msrs_test 18436 [007] 10897.962088248: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux)
1626 tsc_msrs_test 18436 [007] 10897.962088248: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux)
1627 tsc_msrs_test 18436 [007] 10897.962088256: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux)
1628 tsc_msrs_test 18436 [007] 10897.962088270: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux)
1629 [SNIP]
1630 tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux)
1631 tsc_msrs_test 18436 [007] 10897.962089321: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux)
1632 tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux)
1633 tsc_msrs_test 18436 [007] 10897.962089424: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown])
1634 [guest/18436] 18436 [007] 10897.962089424: branches: vmentry 0 [unknown] ([unknown]) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1635 [guest/18436] 18436 [007] 10897.962089701: branches: jmp 40dc1b ucall+0x7b (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc39 ucall+0x99 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1636 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1637 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1638 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc37 ucall+0x97 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc50 ucall+0xb0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1639 [guest/18436] 18436 [007] 10897.962089878: branches: vmexit 40dc55 ucall+0xb5 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown])
1640 tsc_msrs_test 18436 [007] 10897.962089878: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux)
1641 tsc_msrs_test 18436 [007] 10897.962089878: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux)
1642 tsc_msrs_test 18436 [007] 10897.962089887: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux)
1643 tsc_msrs_test 18436 [007] 10897.962089901: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux)
1644 [SNIP]
1645
1646 # perf kvm --guest-code --guest --host report -i perf.data --stdio | head -20
1647
1648 # To display the perf.data header info, please use --header/--header-only options.
1649 #
1650 #
1651 # Total Lost Samples: 0
1652 #
1653 # Samples: 12 of event 'instructions'
1654 # Event count (approx.): 2274583
1655 #
1656 # Children Self Command Shared Object Symbol
1657 # ........ ........ ............. .................... ...........................................
1658 #
1659 54.70% 0.00% tsc_msrs_test [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
1660 |
1661 ---entry_SYSCALL_64_after_hwframe
1662 do_syscall_64
1663 |
1664 |--29.44%--syscall_exit_to_user_mode
1665 | exit_to_user_mode_prepare
1666 | task_work_run
1667 | __fput
1668
1670 Event Trace records information about asynchronous events, for example
1671 interrupts, faults, VM exits and entries. The information is recorded
1672 in CFE and EVD packets, and also the Interrupt Flag is recorded on the
1673 MODE.Exec packet. The CFE packet contains a type field to identify one
1674 of the following:
1675
1676 1 INTR interrupt, fault, exception, NMI
1677 2 IRET interrupt return
1678 3 SMI system management interrupt
1679 4 RSM resume from system management mode
1680 5 SIPI startup interprocessor interrupt
1681 6 INIT INIT signal
1682 7 VMENTRY VM-Entry
1683 8 VMEXIT VM-Entry
1684 9 VMEXIT_INTR VM-Exit due to interrupt
1685 10 SHUTDOWN Shutdown
1686
1687 For more details, refer to the Intel 64 and IA-32 Architectures
1688 Software Developer Manuals (version 076 or later).
1689
1690 The capability to do Event Trace is indicated by the
1691 /sys/bus/event_source/devices/intel_pt/caps/event_trace file.
1692
1693 Event trace is selected for recording using the "event" config term.
1694 e.g.
1695
1696 perf record -e intel_pt/event/u uname
1697
1698 Event trace events are output using the --itrace I option. e.g.
1699
1700 perf script --itrace=Ie
1701
1702 perf script displays events containing CFE type, vector and event data,
1703 in the form:
1704
1705 evt: hw int (t) cfe: INTR IP: 1 vector: 3 PFA: 0x8877665544332211
1706
1707 The IP flag indicates if the event binds to an IP, which includes any
1708 case where flow control packet generation is enabled, as well as when
1709 CFE packet IP bit is set.
1710
1711 perf script displays events containing changes to the Interrupt Flag in
1712 the form:
1713
1714 iflag: t IFLAG: 1->0 via branch
1715
1716 where "via branch" indicates a branch (interrupt or return from
1717 interrupt) and "non branch" indicates an instruction such as CFI, STI
1718 or POPF).
1719
1720 In addition, the current state of the interrupt flag is indicated by
1721 the presence or absence of the "D" (interrupt disabled) perf script
1722 flag. If the interrupt flag is changed, then the "t" flag is also
1723 included i.e.
1724
1725 no flag, interrupts enabled IF=1
1726 t interrupts become disabled IF=1 -> IF=0
1727 D interrupts are disabled IF=0
1728 Dt interrupts become enabled IF=0 -> IF=1
1729
1730 The intel-pt-events.py script illustrates how to access Event Trace
1731 information using a Python script.
1732
1734 TNT packets are disabled using the "notnt" config term. e.g.
1735
1736 perf record -e intel_pt/notnt/u uname
1737
1738 In that case the --itrace q option is forced because walking executable
1739 code to reconstruct the control flow is not possible.
1740
1742 Later perf tools support a method to emulate the ptwrite instruction,
1743 which can be useful if hardware does not support the ptwrite
1744 instruction.
1745
1746 Instead of using the ptwrite instruction, a function is used which
1747 produces a trace that encodes the payload data into TNT packets. Here
1748 is an example of the function:
1749
1750 #include <stdint.h>
1751
1752 void perf_emulate_ptwrite(uint64_t x)
1753 __attribute__((externally_visible, noipa, no_instrument_function, naked));
1754
1755 #define PERF_EMULATE_PTWRITE_8_BITS \
1756 "1: shl %rax\n" \
1757 " jc 1f\n" \
1758 "1: shl %rax\n" \
1759 " jc 1f\n" \
1760 "1: shl %rax\n" \
1761 " jc 1f\n" \
1762 "1: shl %rax\n" \
1763 " jc 1f\n" \
1764 "1: shl %rax\n" \
1765 " jc 1f\n" \
1766 "1: shl %rax\n" \
1767 " jc 1f\n" \
1768 "1: shl %rax\n" \
1769 " jc 1f\n" \
1770 "1: shl %rax\n" \
1771 " jc 1f\n"
1772
1773 /* Undefined instruction */
1774 #define PERF_EMULATE_PTWRITE_UD2 ".byte 0x0f, 0x0b\n"
1775
1776 #define PERF_EMULATE_PTWRITE_MAGIC PERF_EMULATE_PTWRITE_UD2 ".ascii \"perf,ptwrite \"\n"
1777
1778 void perf_emulate_ptwrite(uint64_t x __attribute__ ((__unused__)))
1779 {
1780 /* Assumes SysV ABI : x passed in rdi */
1781 __asm__ volatile (
1782 "jmp 1f\n"
1783 PERF_EMULATE_PTWRITE_MAGIC
1784 "1: mov %rdi, %rax\n"
1785 PERF_EMULATE_PTWRITE_8_BITS
1786 PERF_EMULATE_PTWRITE_8_BITS
1787 PERF_EMULATE_PTWRITE_8_BITS
1788 PERF_EMULATE_PTWRITE_8_BITS
1789 PERF_EMULATE_PTWRITE_8_BITS
1790 PERF_EMULATE_PTWRITE_8_BITS
1791 PERF_EMULATE_PTWRITE_8_BITS
1792 PERF_EMULATE_PTWRITE_8_BITS
1793 "1: ret\n"
1794 );
1795 }
1796
1797 For example, a test program with the function above:
1798
1799 #include <stdio.h>
1800 #include <stdint.h>
1801 #include <stdlib.h>
1802
1803 #include "perf_emulate_ptwrite.h"
1804
1805 int main(int argc, char *argv[])
1806 {
1807 uint64_t x = 0;
1808
1809 if (argc > 1)
1810 x = strtoull(argv[1], NULL, 0);
1811 perf_emulate_ptwrite(x);
1812 return 0;
1813 }
1814
1815 Can be compiled and traced:
1816
1817 $ gcc -Wall -Wextra -O3 -g -o eg_ptw eg_ptw.c
1818 $ perf record -e intel_pt//u ./eg_ptw 0x1234567890abcdef
1819 [ perf record: Woken up 1 times to write data ]
1820 [ perf record: Captured and wrote 0.017 MB perf.data ]
1821 $ perf script --itrace=ew
1822 eg_ptw 19875 [007] 8061.235912: ptwrite: IP: 0 payload: 0x1234567890abcdef 55701249a196 perf_emulate_ptwrite+0x16 (/home/user/eg_ptw)
1823 $
1824
1826 Examples can be found on perf wiki page "Perf tools support for Intel®
1827 Processor Trace":
1828
1829 https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
1830
1832 perf-record(1), perf-script(1), perf-report(1), perf-inject(1)
1833
1834
1835
1836perf 01/12/2023 PERF-INTEL-PT(1)