1PERF-INTEL-PT(1) perf Manual PERF-INTEL-PT(1)
2
6 perf-intel-pt - Support for Intel Processor Trace within perf tools
7
9 perf record -e intel_pt//
10
12 Intel Processor Trace (Intel PT) is an extension of Intel Architecture
13 that collects information about software execution such as control
14 flow, execution modes and timings and formats it into highly compressed
15 binary packets. Technical details are documented in the Intel 64 and
16 IA-32 Architectures Software Developer Manuals, Chapter 36 Intel
17 Processor Trace.
18 Intel PT is first supported in Intel Core M and 5th generation Intel
20 Core processors that are based on the Intel micro-architecture code
21 name Broadwell.
22 Trace data is collected by perf record and stored within the perf.data
24 file. See below for options to perf record.
25 Trace data must be decoded which involves walking the object code and
27 matching the trace data packets. For example a TNT packet only tells
28 whether a conditional branch was taken or not taken, so to make use of
29 that packet the decoder must know precisely which instruction was being
30 executed.
31 Decoding is done on-the-fly. The decoder outputs samples in the same
33 format as samples output by perf hardware events, for example as though
34 the "instructions" or "branches" events had been recorded. Presently 3
35 tools support this: perf script, perf report and perf inject. See below
36 for more information on using those tools.
37 The main distinguishing feature of Intel PT is that the decoder can
39 determine the exact flow of software execution. Intel PT can be used to
40 understand why and how did software get to a certain point, or behave a
41 certain way. The software does not have to be recompiled, so Intel PT
42 works with debug or release builds, however the executed images are
43 needed - which makes use in JIT-compiled environments, or with
44 self-modified code, a challenge. Also symbols need to be provided to
45 make sense of addresses.
46 A limitation of Intel PT is that it produces huge amounts of trace data
48 (hundreds of megabytes per second per core) which takes a long time to
49 decode, for example two or three orders of magnitude longer than it
50 took to collect. Another limitation is the performance impact of
51 tracing, something that will vary depending on the use-case and
52 architecture.
53
55 It is important to start small. That is because it is easy to capture
56 vastly more data than can possibly be processed.
57 The simplest thing to do with Intel PT is userspace profiling of small
59 programs. Data is captured with perf record e.g. to trace ls
60 userspace-only:
61 perf record -e intel_pt//u ls
63 And profiled with perf report e.g.
65 perf report
67 To also trace kernel space presents a problem, namely kernel
69 self-modifying code. A fairly good kernel image is available in
70 /proc/kcore but to get an accurate image a copy of /proc/kcore needs to
71 be made under the same conditions as the data capture. perf record can
72 make a copy of /proc/kcore if the option --kcore is used, but access to
73 /proc/kcore is restricted e.g.
74 sudo perf record -o pt_ls --kcore -e intel_pt// -- ls
76 which will create a directory named pt_ls and put the perf.data file
78 (named simply data) and copies of /proc/kcore, /proc/kallsyms and
79 /proc/modules into it. The other tools understand the directory format,
80 so to use perf report becomes:
81 sudo perf report -i pt_ls
83 Because samples are synthesized after-the-fact, the sampling period can
85 be selected for reporting. e.g. sample every microsecond
86 sudo perf report pt_ls --itrace=i1usge
88 See the sections below for more information about the --itrace option.
90 Beware the smaller the period, the more samples that are produced, and
92 the longer it takes to process them.
93 Also note that the coarseness of Intel PT timing information will start
95 to distort the statistical value of the sampling as the sampling period
96 becomes smaller.
97 To represent software control flow, "branches" samples are produced. By
99 default a branch sample is synthesized for every single branch. To get
100 an idea what data is available you can use the perf script tool with
101 all itrace sampling options, which will list all the samples.
102 perf record -e intel_pt//u ls
104 perf script --itrace=ibxwpe
105 An interesting field that is not printed by default is flags which can
107 be displayed as follows:
108 perf script --itrace=ibxwpe -F+flags
110 The flags are "bcrosyiABExghDt" which stand for branch, call, return,
112 conditional, system, asynchronous, interrupt, transaction abort, trace
113 begin, trace end, in transaction, VM-entry, VM-exit, interrupt
114 disabled, and interrupt disable toggle respectively.
115 perf script also supports higher level ways to dump instruction traces:
117 perf script --insn-trace --xed
119 Dump all instructions. This requires installing the xed tool (see XED
121 below) Dumping all instructions in a long trace can be fairly slow. It
122 is usually better to start with higher level decoding, like
123 perf script --call-trace
125 or
127 perf script --call-ret-trace
129 and then select a time range of interest. The time range can then be
131 examined in detail with
132 perf script --time starttime,stoptime --insn-trace --xed
134 While examining the trace it’s also useful to filter on specific CPUs
136 using the -C option
137 perf script --time starttime,stoptime --insn-trace --xed -C 1
139 Dump all instructions in time range on CPU 1.
141 Another interesting field that is not printed by default is ipc which
143 can be displayed as follows:
144 perf script --itrace=be -F+ipc
146 There are two ways that instructions-per-cycle (IPC) can be calculated
148 depending on the recording.
149 If the cyc config term (see config terms section below) was used, then
151 IPC is calculated using the cycle count from CYC packets, otherwise MTC
152 packets are used - refer to the mtc config term. When MTC is used,
153 however, the values are less accurate because the timing is less
154 accurate.
155 Because Intel PT does not update the cycle count on every branch or
157 instruction, the values will often be zero. When there are values, they
158 will be the number of instructions and number of cycles since the last
159 update, and thus represent the average IPC since the last IPC for that
160 event type. Note IPC for "branches" events is calculated separately
161 from IPC for "instructions" events.
162 Even with the cyc config term, it is possible to produce IPC
164 information for every change of timestamp, but at the expense of
165 accuracy. That is selected by specifying the itrace A option. Due to
166 the granularity of timestamps, the actual number of cycles increases
167 even though the cycles reported does not. The number of instructions is
168 known, but if IPC is reported, cycles can be too low and so IPC is too
169 high. Note that inaccuracy decreases as the period of sampling
170 increases i.e. if the number of cycles is too low by a small amount,
171 that becomes less significant if the number of cycles is large. It may
172 also be useful to use the A option in conjunction with
173 dlfilter-show-cycles.so to provide higher granularity cycle
174 information.
175 Also note that the IPC instruction count may or may not include the
177 current instruction. If the cycle count is associated with an
178 asynchronous branch (e.g. page fault or interrupt), then the
179 instruction count does not include the current instruction, otherwise
180 it does. That is consistent with whether or not that instruction has
181 retired when the cycle count is updated.
182 Another note, in the case of "branches" events, non-taken branches are
184 not presently sampled, so IPC values for them do not appear e.g. a CYC
185 packet with a TNT packet that starts with a non-taken branch. To see
186 every possible IPC value, "instructions" events can be used e.g.
187 --itrace=i0ns
188 While it is possible to create scripts to analyze the data, an
190 alternative approach is available to export the data to a sqlite or
191 postgresql database. Refer to script export-to-sqlite.py or
192 export-to-postgresql.py for more details, and to script
193 exported-sql-viewer.py for an example of using the database.
194 There is also script intel-pt-events.py which provides an example of
196 how to unpack the raw data for power events and PTWRITE. The script
197 also displays branches, and supports 2 additional modes selected by
198 option:
199 --insn-trace - instruction trace
201 --src-trace - source trace
202 As mentioned above, it is easy to capture too much data. One way to
204 limit the data captured is to use snapshot mode which is explained
205 further below. Refer to new snapshot option and Intel PT modes of
206 operation further below.
207 Another problem that will be experienced is decoder errors. They can be
209 caused by inability to access the executed image, self-modified or
210 JIT-ed code, or the inability to match side-band information (such as
211 context switches and mmaps) which results in the decoder not knowing
212 what code was executed.
213 There is also the problem of perf not being able to copy the data fast
215 enough, resulting in data lost because the buffer was full. See Buffer
216 handling below for more details.
217
219 new event
220 The Intel PT kernel driver creates a new PMU for Intel PT. PMU events
221 are selected by providing the PMU name followed by the "config"
222 separated by slashes. An enhancement has been made to allow default
223 "config" e.g. the option
224 -e intel_pt//
226 will use a default config value. Currently that is the same as
228 -e intel_pt/tsc,noretcomp=0/
230 which is the same as
232 -e intel_pt/tsc=1,noretcomp=0/
234 Note there are now new config terms - see section config terms further
236 below.
237 The config terms are listed in /sys/devices/intel_pt/format. They are
239 bit fields within the config member of the struct perf_event_attr which
240 is passed to the kernel by the perf_event_open system call. They
241 correspond to bit fields in the IA32_RTIT_CTL MSR. Here is a list of
242 them and their definitions:
243 $ grep -H . /sys/bus/event_source/devices/intel_pt/format/*
245 /sys/bus/event_source/devices/intel_pt/format/cyc:config:1
246 /sys/bus/event_source/devices/intel_pt/format/cyc_thresh:config:19-22
247 /sys/bus/event_source/devices/intel_pt/format/mtc:config:9
248 /sys/bus/event_source/devices/intel_pt/format/mtc_period:config:14-17
249 /sys/bus/event_source/devices/intel_pt/format/noretcomp:config:11
250 /sys/bus/event_source/devices/intel_pt/format/psb_period:config:24-27
251 /sys/bus/event_source/devices/intel_pt/format/tsc:config:10
252 Note that the default config must be overridden for each term i.e.
254 -e intel_pt/noretcomp=0/
256 is the same as:
258 -e intel_pt/tsc=1,noretcomp=0/
260 So, to disable TSC packets use:
262 -e intel_pt/tsc=0/
264 It is also possible to specify the config value explicitly:
266 -e intel_pt/config=0x400/
268 Note that, as with all events, the event is suffixed with event
270 modifiers:
271 u userspace
273 k kernel
274 h hypervisor
275 G guest
276 H host
277 p precise ip
278 h, G and H are for virtualization which are not used by Intel PT. p is
280 also not relevant to Intel PT. So only options u and k are meaningful
281 for Intel PT.
282 perf_event_attr is displayed if the -vv option is used e.g.
284 ------------------------------------------------------------
286 perf_event_attr:
287 type 6
288 size 112
289 config 0x400
290 { sample_period, sample_freq } 1
291 sample_type IP|TID|TIME|CPU|IDENTIFIER
292 read_format ID
293 disabled 1
294 inherit 1
295 exclude_kernel 1
296 exclude_hv 1
297 enable_on_exec 1
298 sample_id_all 1
299 ------------------------------------------------------------
300 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
301 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
302 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
303 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
304 ------------------------------------------------------------
305 config terms
307 The June 2015 version of Intel 64 and IA-32 Architectures Software
308 Developer Manuals, Chapter 36 Intel Processor Trace, defined new Intel
309 PT features. Some of the features are reflect in new config terms. All
310 the config terms are described below.
311 tsc Always supported. Produces TSC timestamp packets to provide timing
313 information. In some cases it is possible to decode without timing
314 information, for example a per-thread context that does not overlap
315 executable memory maps.
316 The default config selects tsc (i.e. tsc=1).
318 noretcomp Always supported. Disables "return compression" so a TIP
320 packet is produced when a function returns. Causes more packets to be
321 produced but might make decoding more reliable.
322 The default config does not select noretcomp (i.e. noretcomp=0).
324 psb_period Allows the frequency of PSB packets to be specified.
326 The PSB packet is a synchronization packet that provides a
328 starting point for decoding or recovery from errors.
329 Support for psb_period is indicated by:
331 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
333 which contains "1" if the feature is supported and "0"
335 otherwise.
336 Valid values are given by:
338 /sys/bus/event_source/devices/intel_pt/caps/psb_periods
340 which contains a hexadecimal value, the bits of which represent
342 valid values e.g. bit 2 set means value 2 is valid.
343 The psb_period value is converted to the approximate number of
345 trace bytes between PSB packets as:
346 2 ^ (value + 11)
348 e.g. value 3 means 16KiB bytes between PSBs
350 If an invalid value is entered, the error message
352 will give a list of valid values e.g.
353 $ perf record -e intel_pt/psb_period=15/u uname
355 Invalid psb_period for intel_pt. Valid values are: 0-5
356 If MTC packets are selected, the default config selects a value
358 of 3 (i.e. psb_period=3) or the nearest lower value that is
359 supported (0 is always supported). Otherwise the default is 0.
360 If decoding is expected to be reliable and the buffer is large
362 then a large PSB period can be used.
363 Because a TSC packet is produced with PSB, the PSB period can
365 also affect the granularity to timing information in the absence
366 of MTC or CYC.
367 mtc Produces MTC timing packets.
369 MTC packets provide finer grain timestamp information than TSC
371 packets. MTC packets record time using the hardware crystal
372 clock (CTC) which is related to TSC packets using a TMA packet.
373 Support for this feature is indicated by:
375 /sys/bus/event_source/devices/intel_pt/caps/mtc
377 which contains "1" if the feature is supported and
379 "0" otherwise.
380 The frequency of MTC packets can also be specified - see
382 mtc_period below.
383 mtc_period Specifies how frequently MTC packets are produced - see mtc
385 above for how to determine if MTC packets are supported.
386 Valid values are given by:
388 /sys/bus/event_source/devices/intel_pt/caps/mtc_periods
390 which contains a hexadecimal value, the bits of which represent
392 valid values e.g. bit 2 set means value 2 is valid.
393 The mtc_period value is converted to the MTC frequency as:
395 CTC-frequency / (2 ^ value)
397 e.g. value 3 means one eighth of CTC-frequency
399 Where CTC is the hardware crystal clock, the frequency of which
401 can be related to TSC via values provided in cpuid leaf 0x15.
402 If an invalid value is entered, the error message
404 will give a list of valid values e.g.
405 $ perf record -e intel_pt/mtc_period=15/u uname
407 Invalid mtc_period for intel_pt. Valid values are: 0,3,6,9
408 The default value is 3 or the nearest lower value
410 that is supported (0 is always supported).
411 cyc Produces CYC timing packets.
413 CYC packets provide even finer grain timestamp information than
415 MTC and TSC packets. A CYC packet contains the number of CPU
416 cycles since the last CYC packet. Unlike MTC and TSC packets,
417 CYC packets are only sent when another packet is also sent.
418 Support for this feature is indicated by:
420 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
422 which contains "1" if the feature is supported and
424 "0" otherwise.
425 The number of CYC packets produced can be reduced by specifying
427 a threshold - see cyc_thresh below.
428 cyc_thresh Specifies how frequently CYC packets are produced - see cyc
430 above for how to determine if CYC packets are supported.
431 Valid cyc_thresh values are given by:
433 /sys/bus/event_source/devices/intel_pt/caps/cycle_thresholds
435 which contains a hexadecimal value, the bits of which represent
437 valid values e.g. bit 2 set means value 2 is valid.
438 The cyc_thresh value represents the minimum number of CPU cycles
440 that must have passed before a CYC packet can be sent. The
441 number of CPU cycles is:
442 2 ^ (value - 1)
444 e.g. value 4 means 8 CPU cycles must pass before a CYC packet
446 can be sent. Note a CYC packet is still only sent when another
447 packet is sent, not at, e.g. every 8 CPU cycles.
448 If an invalid value is entered, the error message
450 will give a list of valid values e.g.
451 $ perf record -e intel_pt/cyc,cyc_thresh=15/u uname
453 Invalid cyc_thresh for intel_pt. Valid values are: 0-12
454 CYC packets are not requested by default.
456 pt Specifies pass-through which enables the branch config term.
458 The default config selects 'pt' if it is available, so a user will
460 never need to specify this term.
461 branch Enable branch tracing. Branch tracing is enabled by default so
463 to disable branch tracing use branch=0.
464 The default config selects 'branch' if it is available.
466 ptw Enable PTWRITE packets which are produced when a ptwrite
468 instruction is executed.
469 Support for this feature is indicated by:
471 /sys/bus/event_source/devices/intel_pt/caps/ptwrite
473 which contains "1" if the feature is supported and
475 "0" otherwise.
476 As an alternative, refer to "Emulated PTWRITE" further below.
478 fup_on_ptw Enable a FUP packet to follow the PTWRITE packet. The FUP
480 packet provides the address of the ptwrite instruction. In the absence
481 of fup_on_ptw, the decoder will use the address of the previous branch
482 if branch tracing is enabled, otherwise the address will be zero. Note
483 that fup_on_ptw will work even when branch tracing is disabled.
484 pwr_evt Enable power events. The power events provide information about
486 changes to the CPU C-state.
487 Support for this feature is indicated by:
489 /sys/bus/event_source/devices/intel_pt/caps/power_event_trace
491 which contains "1" if the feature is supported and
493 "0" otherwise.
494 event Enable Event Trace. The events provide information about
496 asynchronous events.
497 Support for this feature is indicated by:
499 /sys/bus/event_source/devices/intel_pt/caps/event_trace
501 which contains "1" if the feature is supported and
503 "0" otherwise.
504 notnt Disable TNT packets. Without TNT packets, it is not possible to
506 walk executable code to reconstruct control flow, however FUP, TIP,
507 TIP.PGE and TIP.PGD packets still indicate asynchronous control flow,
508 and (if return compression is disabled - see noretcomp) return
509 statements. The advantage of eliminating TNT packets is reducing the
510 size of the trace and corresponding tracing overhead.
511 Support for this feature is indicated by:
513 /sys/bus/event_source/devices/intel_pt/caps/tnt_disable
515 which contains "1" if the feature is supported and
517 "0" otherwise.
518 AUX area sampling option
520 To select Intel PT "sampling" the AUX area sampling option can be used:
521 --aux-sample
523 Optionally it can be followed by the sample size in bytes e.g.
525 --aux-sample=8192
527 In addition, the Intel PT event to sample must be defined e.g.
529 -e intel_pt//u
531 Samples on other events will be created containing Intel PT data e.g.
533 the following will create Intel PT samples on the branch-misses event,
534 note the events must be grouped using {}:
535 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}'
537 An alternative to --aux-sample is to add the config term
539 aux-sample-size to events. In this case, the grouping is implied e.g.
540 perf record -e intel_pt//u -e branch-misses/aux-sample-size=8192/u
542 is the same as:
544 perf record -e '{intel_pt//u,branch-misses/aux-sample-size=8192/u}'
546 but allows for also using an address filter e.g.:
548 perf record -e intel_pt//u --filter 'filter * @/bin/ls' -e branch-misses/aux-sample-size=8192/u -- ls
550 It is important to select a sample size that is big enough to contain
552 at least one PSB packet. If not a warning will be displayed:
553 Intel PT sample size (%zu) may be too small for PSB period (%zu)
555 The calculation used for that is: if sample_size ⟨ psb_period + 256
557 display the warning. When sampling is used, psb_period defaults to 0
558 (2KiB).
559 The default sample size is 4KiB.
561 The sample size is passed in aux_sample_size in struct perf_event_attr.
563 The sample size is limited by the maximum event size which is 64KiB. It
564 is difficult to know how big the event might be without the trace
565 sample attached, but the tool validates that the sample size is not
566 greater than 60KiB.
567 new snapshot option
569 The difference between full trace and snapshot from the kernel’s
570 perspective is that in full trace we don’t overwrite trace data that
571 the user hasn’t collected yet (and indicated that by advancing
572 aux_tail), whereas in snapshot mode we let the trace run and overwrite
573 older data in the buffer so that whenever something interesting
574 happens, we can stop it and grab a snapshot of what was going on around
575 that interesting moment.
576 To select snapshot mode a new option has been added:
578 -S
580 Optionally it can be followed by the snapshot size e.g.
582 -S0x100000
584 The default snapshot size is the auxtrace mmap size. If neither
586 auxtrace mmap size nor snapshot size is specified, then the default is
587 4MiB for privileged users (or if /proc/sys/kernel/perf_event_paranoid <
588 0), 128KiB for unprivileged users. If an unprivileged user does not
589 specify mmap pages, the mmap pages will be reduced as described in the
590 new auxtrace mmap size option section below.
591 The snapshot size is displayed if the option -vv is used e.g.
593 Intel PT snapshot size: %zu
595 new auxtrace mmap size option
597 Intel PT buffer size is specified by an addition to the -m option e.g.
598 -m,16
600 selects a buffer size of 16 pages i.e. 64KiB.
602 Note that the existing functionality of -m is unchanged. The auxtrace
604 mmap size is specified by the optional addition of a comma and the
605 value.
606 The default auxtrace mmap size for Intel PT is 4MiB/page_size for
608 privileged users (or if /proc/sys/kernel/perf_event_paranoid < 0),
609 128KiB for unprivileged users. If an unprivileged user does not specify
610 mmap pages, the mmap pages will be reduced from the default
611 512KiB/page_size to 256KiB/page_size, otherwise the user is likely to
612 get an error as they exceed their mlock limit (Max locked memory as
613 shown in /proc/self/limits). Note that perf does not count the first
614 512KiB (actually /proc/sys/kernel/perf_event_mlock_kb minus 1 page) per
615 cpu against the mlock limit so an unprivileged user is allowed 512KiB
616 per cpu plus their mlock limit (which defaults to 64KiB but is not
617 multiplied by the number of cpus).
618 In full-trace mode, powers of two are allowed for buffer size, with a
620 minimum size of 2 pages. In snapshot mode or sampling mode, it is the
621 same but the minimum size is 1 page.
622 The mmap size and auxtrace mmap size are displayed if the -vv option is
624 used e.g.
625 mmap length 528384
627 auxtrace mmap length 4198400
628 Intel PT modes of operation
630 Intel PT can be used in 3 modes: full-trace mode sample mode snapshot
631 mode
632 Full-trace mode traces continuously e.g.
634 perf record -e intel_pt//u uname
636 Sample mode attaches a Intel PT sample to other events e.g.
638 perf record --aux-sample -e intel_pt//u -e branch-misses:u
640 Snapshot mode captures the available data when a signal is sent or
642 "snapshot" control command is issued. e.g. using a signal
643 perf record -v -e intel_pt//u -S ./loopy 1000000000 &
645 [1] 11435
646 kill -USR2 11435
647 Recording AUX area tracing snapshot
648 Note that the signal sent is SIGUSR2. Note that "Recording AUX area
650 tracing snapshot" is displayed because the -v option is used.
651 The advantage of using "snapshot" control command is that the access is
653 controlled by access to a FIFO e.g.
654 $ mkfifo perf.control
656 $ mkfifo perf.ack
657 $ cat perf.ack &
658 [1] 15235
659 $ sudo ~/bin/perf record --control fifo:perf.control,perf.ack -S -e intel_pt//u -- sleep 60 &
660 [2] 15243
661 $ ps -e | grep perf
662 15244 pts/1 00:00:00 perf
663 $ kill -USR2 15244
664 bash: kill: (15244) - Operation not permitted
665 $ echo snapshot > perf.control
666 ack
667 The 3 Intel PT modes of operation cannot be used together.
669 Buffer handling
671 There may be buffer limitations (i.e. single ToPa entry) which means
672 that actual buffer sizes are limited to powers of 2 up to 4MiB
673 (MAX_ORDER). In order to provide other sizes, and in particular an
674 arbitrarily large size, multiple buffers are logically concatenated.
675 However an interrupt must be used to switch between buffers. That has
676 two potential problems: a) the interrupt may not be handled in time so
677 that the current buffer becomes full and some trace data is lost. b)
678 the interrupts may slow the system and affect the performance results.
679 If trace data is lost, the driver sets truncated in the PERF_RECORD_AUX
681 event which the tools report as an error.
682 In full-trace mode, the driver waits for data to be copied out before
684 allowing the (logical) buffer to wrap-around. If data is not copied out
685 quickly enough, again truncated is set in the PERF_RECORD_AUX event. If
686 the driver has to wait, the intel_pt event gets disabled. Because it is
687 difficult to know when that happens, perf tools always re-enable the
688 intel_pt event after copying out data.
689 Intel PT and build ids
691 By default "perf record" post-processes the event stream to find all
692 build ids for executables for all addresses sampled. Deliberately,
693 Intel PT is not decoded for that purpose (it would take too long).
694 Instead the build ids for all executables encountered (due to mmap,
695 comm or task events) are included in the perf.data file.
696 To see buildids included in the perf.data file use the command:
698 perf buildid-list
700 If the perf.data file contains Intel PT data, that is the same as:
702 perf buildid-list --with-hits
704 Snapshot mode and event disabling
706 In order to make a snapshot, the intel_pt event is disabled using an
707 IOCTL, namely PERF_EVENT_IOC_DISABLE. However doing that can also
708 disable the collection of side-band information. In order to prevent
709 that, a dummy software event has been introduced that permits tracking
710 events (like mmaps) to continue to be recorded while intel_pt is
711 disabled. That is important to ensure there is complete side-band
712 information to allow the decoding of subsequent snapshots.
713 A test has been created for that. To find the test:
715 perf test list
717 ...
718 23: Test using a dummy software event to keep tracking
719 To run the test:
721 perf test 23
723 23: Test using a dummy software event to keep tracking : Ok
724 perf record modes (nothing new here)
726 perf record essentially operates in one of three modes: per thread per
727 cpu workload only
728 "per thread" mode is selected by -t or by --per-thread (with -p or -u
730 or just a workload). "per cpu" is selected by -C or -a. "workload only"
731 mode is selected by not using the other options but providing a command
732 to run (i.e. the workload).
733 In per-thread mode an exact list of threads is traced. There is no
735 inheritance. Each thread has its own event buffer.
736 In per-cpu mode all processes (or processes from the selected cgroup
738 i.e. -G option, or processes selected with -p or -u) are traced. Each
739 cpu has its own buffer. Inheritance is allowed.
740 In workload-only mode, the workload is traced but with per-cpu buffers.
742 Inheritance is allowed. Note that you can now trace a workload in
743 per-thread mode by using the --per-thread option.
744 Privileged vs non-privileged users
746 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
747 users have memory limits imposed upon them. That affects what buffer
748 sizes they can have as outlined above.
749 The v4.2 kernel introduced support for a context switch metadata event,
751 PERF_RECORD_SWITCH, which allows unprivileged users to see when their
752 processes are scheduled out and in, just not by whom, which is left for
753 the PERF_RECORD_SWITCH_CPU_WIDE, that is only accessible in system wide
754 context, which in turn requires CAP_PERFMON or CAP_SYS_ADMIN.
755 Please see the 45ac1403f564 ("perf: Add PERF_RECORD_SWITCH to indicate
757 context switches") commit, that introduces these metadata events for
758 further info.
759 When working with kernels < v4.2, the following considerations must be
761 taken, as the sched:sched_switch tracepoints will be used to receive
762 such information:
763 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
765 users are not permitted to use tracepoints which means there is
766 insufficient side-band information to decode Intel PT in per-cpu mode,
767 and potentially workload-only mode too if the workload creates new
768 processes.
769 Note also, that to use tracepoints, read-access to debugfs is required.
771 So if debugfs is not mounted or the user does not have read-access, it
772 will again not be possible to decode Intel PT in per-cpu mode.
773 sched_switch tracepoint
775 The sched_switch tracepoint is used to provide side-band data for Intel
776 PT decoding in kernels where the PERF_RECORD_SWITCH metadata event
777 isn’t available.
778 The sched_switch events are automatically added. e.g. the second event
780 shown below:
781 $ perf record -vv -e intel_pt//u uname
783 ------------------------------------------------------------
784 perf_event_attr:
785 type 6
786 size 112
787 config 0x400
788 { sample_period, sample_freq } 1
789 sample_type IP|TID|TIME|CPU|IDENTIFIER
790 read_format ID
791 disabled 1
792 inherit 1
793 exclude_kernel 1
794 exclude_hv 1
795 enable_on_exec 1
796 sample_id_all 1
797 ------------------------------------------------------------
798 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
799 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
800 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
801 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
802 ------------------------------------------------------------
803 perf_event_attr:
804 type 2
805 size 112
806 config 0x108
807 { sample_period, sample_freq } 1
808 sample_type IP|TID|TIME|CPU|PERIOD|RAW|IDENTIFIER
809 read_format ID
810 inherit 1
811 sample_id_all 1
812 exclude_guest 1
813 ------------------------------------------------------------
814 sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8
815 sys_perf_event_open: pid -1 cpu 1 group_fd -1 flags 0x8
816 sys_perf_event_open: pid -1 cpu 2 group_fd -1 flags 0x8
817 sys_perf_event_open: pid -1 cpu 3 group_fd -1 flags 0x8
818 ------------------------------------------------------------
819 perf_event_attr:
820 type 1
821 size 112
822 config 0x9
823 { sample_period, sample_freq } 1
824 sample_type IP|TID|TIME|IDENTIFIER
825 read_format ID
826 disabled 1
827 inherit 1
828 exclude_kernel 1
829 exclude_hv 1
830 mmap 1
831 comm 1
832 enable_on_exec 1
833 task 1
834 sample_id_all 1
835 mmap2 1
836 comm_exec 1
837 ------------------------------------------------------------
838 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
839 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
840 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
841 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
842 mmap size 528384B
843 AUX area mmap length 4194304
844 perf event ring buffer mmapped per cpu
845 Synthesizing auxtrace information
846 Linux
847 [ perf record: Woken up 1 times to write data ]
848 [ perf record: Captured and wrote 0.042 MB perf.data ]
849 Note, the sched_switch event is only added if the user is permitted to
851 use it and only in per-cpu mode.
852 Note also, the sched_switch event is only added if TSC packets are
854 requested. That is because, in the absence of timing information, the
855 sched_switch events cannot be matched against the Intel PT trace.
856
858 By default, perf script will decode trace data found in the perf.data
859 file. This can be further controlled by new option --itrace.
860 New --itrace option
862 Having no option is the same as
863 --itrace
865 which, in turn, is the same as
867 --itrace=cepwx
869 The letters are:
871 i synthesize "instructions" events
873 b synthesize "branches" events
874 x synthesize "transactions" events
875 w synthesize "ptwrite" events
876 p synthesize "power" events (incl. PSB events)
877 c synthesize branches events (calls only)
878 r synthesize branches events (returns only)
879 o synthesize PEBS-via-PT events
880 I synthesize Event Trace events
881 e synthesize tracing error events
882 d create a debug log
883 g synthesize a call chain (use with i or x)
884 G synthesize a call chain on existing event records
885 l synthesize last branch entries (use with i or x)
886 L synthesize last branch entries on existing event records
887 s skip initial number of events
888 q quicker (less detailed) decoding
889 A approximate IPC
890 Z prefer to ignore timestamps (so-called "timeless" decoding)
891 "Instructions" events look like they were recorded by "perf record -e
893 instructions".
894 "Branches" events look like they were recorded by "perf record -e
896 branches". "c" and "r" can be combined to get calls and returns.
897 "Transactions" events correspond to the start or end of transactions.
899 The flags field can be used in perf script to determine whether the
900 event is a transaction start, commit or abort.
901 Note that "instructions", "branches" and "transactions" events depend
903 on code flow packets which can be disabled by using the config term
904 "branch=0". Refer to the config terms section above.
905 "ptwrite" events record the payload of the ptwrite instruction and
907 whether "fup_on_ptw" was used. "ptwrite" events depend on PTWRITE
908 packets which are recorded only if the "ptw" config term was used.
909 Refer to the config terms section above. perf script "synth" field
910 displays "ptwrite" information like this: "ip: 0 payload:
911 0x123456789abcdef0" where "ip" is 1 if "fup_on_ptw" was used.
912 "Power" events correspond to power event packets and CBR (core-to-bus
914 ratio) packets. While CBR packets are always recorded when tracing is
915 enabled, power event packets are recorded only if the "pwr_evt" config
916 term was used. Refer to the config terms section above. The power
917 events record information about C-state changes, whereas CBR is
918 indicative of CPU frequency. perf script "event,synth" fields display
919 information like this:
920 cbr: cbr: 22 freq: 2189 MHz (200%)
922 mwait: hints: 0x60 extensions: 0x1
923 pwre: hw: 0 cstate: 2 sub-cstate: 0
924 exstop: ip: 1
925 pwrx: deepest cstate: 2 last cstate: 2 wake reason: 0x4
926 Where:
928 "cbr" includes the frequency and the percentage of maximum non-turbo
930 "mwait" shows mwait hints and extensions
931 "pwre" shows C-state transitions (to a C-state deeper than C0) and
932 whether initiated by hardware
933 "exstop" indicates execution stopped and whether the IP was recorded
934 exactly,
935 "pwrx" indicates return to C0
936 For more details refer to the Intel 64 and IA-32 Architectures Software
938 Developer Manuals.
939 PSB events show when a PSB+ occurred and also the byte-offset in the
941 trace. Emitting a PSB+ can cause a CPU a slight delay. When doing
942 timing analysis of code with Intel PT, it is useful to know if a timing
943 bubble was caused by Intel PT or not.
944 Error events show where the decoder lost the trace. Error events are
946 quite important. Users must know if what they are seeing is a complete
947 picture or not. The "e" option may be followed by flags which affect
948 what errors will or will not be reported. Each flag must be preceded by
949 either + or -. The flags supported by Intel PT are:
950 -o Suppress overflow errors
952 -l Suppress trace data lost errors
953 For example, for errors but not overflow or data lost errors:
955 --itrace=e-o-l
957 The "d" option will cause the creation of a file "intel_pt.log"
959 containing all decoded packets and instructions. Note that this option
960 slows down the decoder and that the resulting file may be very large.
961 The "d" option may be followed by flags which affect what debug
962 messages will or will not be logged. Each flag must be preceded by
963 either + or -. The flags support by Intel PT are:
964 -a Suppress logging of perf events
966 +a Log all perf events
967 +e Output only on decoding errors (size configurable)
968 +o Output to stdout instead of "intel_pt.log"
969 By default, logged perf events are filtered by any specified time
971 ranges, but flag +a overrides that. The +e flag can be useful for
972 analyzing errors. By default, the log size in that case is 16384 bytes,
973 but can be altered by perf-config(1) e.g. perf config
974 itrace.debug-log-buffer-size=30000
975 In addition, the period of the "instructions" event can be specified.
977 e.g.
978 --itrace=i10us
980 sets the period to 10us i.e. one instruction sample is synthesized for
982 each 10 microseconds of trace. Alternatives to "us" are "ms"
983 (milliseconds), "ns" (nanoseconds), "t" (TSC ticks) or "i"
984 (instructions).
985 "ms", "us" and "ns" are converted to TSC ticks.
987 The timing information included with Intel PT does not give the time of
989 every instruction. Consequently, for the purpose of sampling, the
990 decoder estimates the time since the last timing packet based on 1 tick
991 per instruction. The time on the sample is not adjusted and reflects
992 the last known value of TSC.
993 For Intel PT, the default period is 100us.
995 Setting it to a zero period means "as often as possible".
997 In the case of Intel PT that is the same as a period of 1 and a unit of
999 instructions (i.e. --itrace=i1i).
1000 Also the call chain size (default 16, max. 1024) for instructions or
1002 transactions events can be specified. e.g.
1003 --itrace=ig32
1005 --itrace=xg32
1006 Also the number of last branch entries (default 64, max. 1024) for
1008 instructions or transactions events can be specified. e.g.
1009 --itrace=il10
1011 --itrace=xl10
1012 Note that last branch entries are cleared for each sample, so there is
1014 no overlap from one sample to the next.
1015 The G and L options are designed in particular for sample mode, and
1017 work much like g and l but add call chain and branch stack to the other
1018 selected events instead of synthesized events. For example, to record
1019 branch-misses events for ls and then add a call chain derived from the
1020 Intel PT trace:
1021 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}' -- ls
1023 perf report --itrace=Ge
1024 Although in fact G is a default for perf report, so that is the same as
1026 just:
1027 perf report
1029 One caveat with the G and L options is that they work poorly with
1031 "Large PEBS". Large PEBS means PEBS records will be accumulated by
1032 hardware and the written into the event buffer in one go. That reduces
1033 interrupts, but can give very late timestamps. Because the Intel PT
1034 trace is synchronized by timestamps, the PEBS events do not match the
1035 trace. Currently, Large PEBS is used only in certain circumstances: -
1036 hardware supports it - PEBS is used - event period is specified,
1037 instead of frequency - the sample type is limited to the following
1038 flags: PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_ADDR |
1039 PERF_SAMPLE_ID | PERF_SAMPLE_CPU | PERF_SAMPLE_STREAM_ID |
1040 PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_TRANSACTION
1041 | PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER
1042 | PERF_SAMPLE_PERIOD (and sometimes) | PERF_SAMPLE_TIME Because Intel
1043 PT sample mode uses a different sample type to the list above, Large
1044 PEBS is not used with Intel PT sample mode. To avoid Large PEBS in
1045 other cases, avoid specifying the event period i.e. avoid the perf
1046 record -c option, --count option, or period config term.
1047 To disable trace decoding entirely, use the option --no-itrace.
1049 It is also possible to skip events generated (instructions, branches,
1051 transactions) at the beginning. This is useful to ignore initialization
1052 code.
1053 --itrace=i0nss1000000
1055 skips the first million instructions.
1057 The q option changes the way the trace is decoded. The decoding is much
1059 faster but much less detailed. Specifically, with the q option, the
1060 decoder does not decode TNT packets, and does not walk object code, but
1061 gets the ip from FUP and TIP packets. The q option can be used with the
1062 b and i options but the period is not used. The q option decodes more
1063 quickly, but is useful only if the control flow of interest is
1064 represented or indicated by FUP, TIP, TIP.PGE, or TIP.PGD packets
1065 (refer below). However the q option could be used to find time ranges
1066 that could then be decoded fully using the --time option.
1067 What will not be decoded with the (single) q option:
1069 • direct calls and jmps
1071 • conditional branches
1073 • non-branch instructions
1075 What will be decoded with the (single) q option:
1077 • asynchronous branches such as interrupts
1079 • indirect branches
1081 • function return target address if the noretcomp config term (refer
1083 config terms section) was used
1084 • start of (control-flow) tracing
1086 • end of (control-flow) tracing, if it is not out of context
1088 • power events, ptwrite, transaction start and abort
1090 • instruction pointer associated with PSB packets
1092 Note the q option does not specify what events will be synthesized e.g.
1094 the p option must be used also to show power events.
1095 Repeating the q option (double-q i.e. qq) results in even faster
1097 decoding and even less detail. The decoder decodes only extended PSB
1098 (PSB+) packets, getting the instruction pointer if there is a FUP
1099 packet within PSB+ (i.e. between PSB and PSBEND). Note PSB packets
1100 occur regularly in the trace based on the psb_period config term (refer
1101 config terms section). There will be a FUP packet if the PSB+ occurs
1102 while control flow is being traced.
1103 What will not be decoded with the qq option:
1105 • everything except instruction pointer associated with PSB packets
1107 What will be decoded with the qq option:
1109 • instruction pointer associated with PSB packets
1111 The Z option is equivalent to having recorded a trace without TSC (i.e.
1113 config term tsc=0). It can be useful to avoid timestamp issues when
1114 decoding a trace of a virtual machine.
1115 dlfilter-show-cycles.so
1117 Cycles can be displayed using dlfilter-show-cycles.so in which case the
1118 itrace A option can be useful to provide higher granularity cycle
1119 information:
1120 perf script --itrace=A --call-trace --dlfilter dlfilter-show-cycles.so
1122 To see a list of dlfilters:
1124 perf script -v --list-dlfilters
1126 See also perf-dlfilters(1)
1128 dump option
1130 perf script has an option (-D) to "dump" the events i.e. display the
1131 binary data.
1132 When -D is used, Intel PT packets are displayed. The packet decoder
1134 does not pay attention to PSB packets, but just decodes the bytes - so
1135 the packets seen by the actual decoder may not be identical in places
1136 where the data is corrupt. One example of that would be when the
1137 buffer-switching interrupt has been too slow, and the buffer has been
1138 filled completely. In that case, the last packet in the buffer might be
1139 truncated and immediately followed by a PSB as the trace continues in
1140 the next buffer.
1141 To disable the display of Intel PT packets, combine the -D option with
1143 --no-itrace.
1144
1146 By default, perf report will decode trace data found in the perf.data
1147 file. This can be further controlled by new option --itrace exactly the
1148 same as perf script, with the exception that the default is
1149 --itrace=igxe.
1150
1152 perf inject also accepts the --itrace option in which case tracing data
1153 is removed and replaced with the synthesized events. e.g.
1154 perf inject --itrace -i perf.data -o perf.data.new
1156 Below is an example of using Intel PT with autofdo. It requires autofdo
1158 (https://github.com/google/autofdo) and gcc version 5. The bubble sort
1159 example is from the AutoFDO tutorial
1160 (https://gcc.gnu.org/wiki/AutoFDO/Tutorial) amended to take the number
1161 of elements as a parameter.
1162 $ gcc-5 -O3 sort.c -o sort_optimized
1164 $ ./sort_optimized 30000
1165 Bubble sorting array of 30000 elements
1166 2254 ms
1167 $ cat ~/.perfconfig
1169 [intel-pt]
1170 mispred-all = on
1171 $ perf record -e intel_pt//u ./sort 3000
1173 Bubble sorting array of 3000 elements
1174 58 ms
1175 [ perf record: Woken up 2 times to write data ]
1176 [ perf record: Captured and wrote 3.939 MB perf.data ]
1177 $ perf inject -i perf.data -o inj --itrace=i100usle --strip
1178 $ ./create_gcov --binary=./sort --profile=inj --gcov=sort.gcov -gcov_version=1
1179 $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
1180 $ ./sort_autofdo 30000
1181 Bubble sorting array of 30000 elements
1182 2155 ms
1183 Note there is currently no advantage to using Intel PT instead of LBR,
1185 but that may change in the future if greater use is made of the data.
1186
1188 Some hardware has the feature to redirect PEBS records to the Intel PT
1189 trace. Recording is selected by using the aux-output config term e.g.
1190 perf record -c 10000 -e '{intel_pt/branch=0/,cycles/aux-output/ppp}' uname
1192 Originally, software only supported redirecting at most one PEBS event
1194 because it was not able to differentiate one event from another. To
1195 overcome that, more recent kernels and perf tools add support for the
1196 PERF_RECORD_AUX_OUTPUT_HW_ID side-band event. To check for the presence
1197 of that event in a PEBS-via-PT trace:
1198 perf script -D --no-itrace | grep PERF_RECORD_AUX_OUTPUT_HW_ID
1200 To display PEBS events from the Intel PT trace, use the itrace o option
1202 e.g.
1203 perf script --itrace=oe
1205
1207 For --xed the xed tool is needed. Here is how to install it:
1208 $ git clone https://github.com/intelxed/mbuild.git mbuild
1210 $ git clone https://github.com/intelxed/xed
1211 $ cd xed
1212 $ ./mfile.py --share
1213 $ ./mfile.py examples
1214 $ sudo ./mfile.py --prefix=/usr/local install
1215 $ sudo ldconfig
1216 $ sudo cp obj/examples/xed /usr/local/bin
1217 Basic xed testing:
1219 $ xed | head -3
1221 ERROR: required argument(s) were missing
1222 Copyright (C) 2017, Intel Corporation. All rights reserved.
1223 XED version: [v10.0-328-g7d62c8c49b7b]
1224 $
1225
1227 Currently, kernel tracing is supported with either "timeless" decoding
1228 (i.e. no TSC timestamps) or VM Time Correlation. VM Time Correlation is
1229 an extra step using perf inject and requires unchanging VMX TSC Offset
1230 and no VMX TSC Scaling.
1231 Other limitations and caveats
1233 VMX controls may suppress packets needed for decoding resulting in decoding errors
1235 VMX controls may block the perf NMI to the host potentially resulting in lost trace data
1236 Guest kernel self-modifying code (e.g. jump labels or JIT-compiled eBPF) will result in decoding errors
1237 Guest thread information is unknown
1238 Guest VCPU is unknown but may be able to be inferred from the host thread
1239 Callchains are not supported
1240 Example using "timeless" decoding
1242 Start VM
1244 $ sudo virsh start kubuntu20.04
1246 Domain kubuntu20.04 started
1247 Mount the guest file system. Note sshfs needs -o direct_io to enable
1249 reading of proc files. root access is needed to read /proc/kcore.
1250 $ mkdir vm0
1252 $ sshfs -o direct_io root@vm0:/ vm0
1253 Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
1255 $ perf buildid-cache -v --kcore vm0/proc/kcore
1257 kcore added to build-id cache directory /home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306
1258 $ KALLSYMS=/home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306/kallsyms
1259 Find the VM process
1261 $ ps -eLl | grep 'KVM\|PID'
1263 F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
1264 3 S 64055 1430 1 1440 1 80 0 - 1921718 - ? 00:02:47 CPU 0/KVM
1265 3 S 64055 1430 1 1441 1 80 0 - 1921718 - ? 00:02:41 CPU 1/KVM
1266 3 S 64055 1430 1 1442 1 80 0 - 1921718 - ? 00:02:38 CPU 2/KVM
1267 3 S 64055 1430 1 1443 2 80 0 - 1921718 - ? 00:03:18 CPU 3/KVM
1268 Start an open-ended perf record, tracing the VM process, do something
1270 on the VM, and then ctrl-C to stop. TSC is not supported and tsc=0 must
1271 be specified. That means mtc is useless, so add mtc=0. However, IPC can
1272 still be determined, hence cyc=1 can be added. Only kernel decoding is
1273 supported, so k must be specified. Intel PT traces both the host and
1274 the guest so --guest and --host need to be specified. Without
1275 timestamps, --per-thread must be specified to distinguish threads.
1276 $ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/tsc=0,mtc=0,cyc=1/k -p 1430 --per-thread
1278 ^C
1279 [ perf record: Woken up 1 times to write data ]
1280 [ perf record: Captured and wrote 5.829 MB ]
1281 perf script can be used to provide an instruction trace
1283 $ perf script --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
1285 CPU 0/KVM 1440 ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
1286 CPU 0/KVM 1440 ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
1287 CPU 0/KVM 1440 ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
1288 CPU 0/KVM 1440 ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
1289 CPU 0/KVM 1440 ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
1290 CPU 0/KVM 1440 ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
1291 CPU 0/KVM 1440 ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
1292 CPU 0/KVM 1440 ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
1293 CPU 0/KVM 1440 ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
1294 CPU 0/KVM 1440 ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
1295 CPU 0/KVM 1440 ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.11 (50/445)
1296 :1440 1440 ffffffffbb678b06 native_write_msr+0x6 ([guest.kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
1297 :1440 1440 ffffffffbb678b0b native_write_msr+0xb ([guest.kernel.kallsyms]) retq IPC: 0.04 (2/41)
1298 :1440 1440 ffffffffbb666646 lapic_next_deadline+0x26 ([guest.kernel.kallsyms]) data16 nop
1299 :1440 1440 ffffffffbb666648 lapic_next_deadline+0x28 ([guest.kernel.kallsyms]) xor %eax, %eax
1300 :1440 1440 ffffffffbb66664a lapic_next_deadline+0x2a ([guest.kernel.kallsyms]) popq %rbp
1301 :1440 1440 ffffffffbb66664b lapic_next_deadline+0x2b ([guest.kernel.kallsyms]) retq IPC: 0.16 (4/25)
1302 :1440 1440 ffffffffbb74607f clockevents_program_event+0x8f ([guest.kernel.kallsyms]) test %eax, %eax
1303 :1440 1440 ffffffffbb746081 clockevents_program_event+0x91 ([guest.kernel.kallsyms]) jz 0xffffffffbb74603c IPC: 0.06 (2/30)
1304 :1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx
1305 :1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12
1306 Example using VM Time Correlation
1308 Start VM
1310 $ sudo virsh start kubuntu20.04
1312 Domain kubuntu20.04 started
1313 Mount the guest file system. Note sshfs needs -o direct_io to enable
1315 reading of proc files. root access is needed to read /proc/kcore.
1316 $ mkdir -p vm0
1318 $ sshfs -o direct_io root@vm0:/ vm0
1319 Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
1321 $ perf buildid-cache -v --kcore vm0/proc/kcore
1323 same kcore found in /home/user/.debug/[kernel.kcore]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777
1324 $ KALLSYMS=/home/user/.debug/\[kernel.kcore\]/cc9c55a98c5e4ec0aeda69302554aabed5cd6491/2021021312450777/kallsyms
1325 Find the VM process
1327 $ ps -eLl | grep 'KVM\|PID'
1329 F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
1330 3 S 64055 16998 1 17005 13 80 0 - 1818189 - ? 00:00:16 CPU 0/KVM
1331 3 S 64055 16998 1 17006 4 80 0 - 1818189 - ? 00:00:05 CPU 1/KVM
1332 3 S 64055 16998 1 17007 3 80 0 - 1818189 - ? 00:00:04 CPU 2/KVM
1333 3 S 64055 16998 1 17008 4 80 0 - 1818189 - ? 00:00:05 CPU 3/KVM
1334 Start an open-ended perf record, tracing the VM process, do something
1336 on the VM, and then ctrl-C to stop. IPC can be determined, hence cyc=1
1337 can be added. Only kernel decoding is supported, so k must be
1338 specified. Intel PT traces both the host and the guest so --guest and
1339 --host need to be specified.
1340 $ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/cyc=1/k -p 16998
1342 ^C[ perf record: Woken up 1 times to write data ]
1343 [ perf record: Captured and wrote 9.041 MB perf.data.kvm ]
1344 Now perf inject can be used to determine the VMX TCS Offset. Note,
1346 Intel PT TSC packets are only 7-bytes, so the TSC Offset might differ
1347 from the actual value in the 8th byte. That will have no effect i.e.
1348 the resulting timestamps will be correct anyway.
1349 $ perf inject -i perf.data.kvm --vm-time-correlation=dry-run
1351 ERROR: Unknown TSC Offset for VMCS 0x1bff6a
1352 VMCS: 0x1bff6a TSC Offset 0xffffe42722c64c41
1353 ERROR: Unknown TSC Offset for VMCS 0x1cbc08
1354 VMCS: 0x1cbc08 TSC Offset 0xffffe42722c64c41
1355 ERROR: Unknown TSC Offset for VMCS 0x1c3ce8
1356 VMCS: 0x1c3ce8 TSC Offset 0xffffe42722c64c41
1357 ERROR: Unknown TSC Offset for VMCS 0x1cbce9
1358 VMCS: 0x1cbce9 TSC Offset 0xffffe42722c64c41
1359 Each virtual CPU has a different Virtual Machine Control Structure
1361 (VMCS) shown above with the calculated TSC Offset. For an unchanging
1362 TSC Offset they should all be the same for the same virtual machine.
1363 Now that the TSC Offset is known, it can be provided to perf inject
1365 $ perf inject -i perf.data.kvm --vm-time-correlation="dry-run 0xffffe42722c64c41"
1367 Note the options for perf inject --vm-time-correlation are:
1369 [ dry-run ] [ <TSC Offset> [ : <VMCS> [ , <VMCS> ]... ] ]...
1371 So it is possible to specify different TSC Offsets for different VMCS.
1373 The option "dry-run" will cause the file to be processed but without
1374 updating it. Note it is also possible to get a intel_pt.log file by
1375 adding option --itrace=d
1376 There were no errors so, do it for real
1378 $ perf inject -i perf.data.kvm --vm-time-correlation=0xffffe42722c64c41 --force
1380 perf script can be used to see if there are any decoder errors
1382 $ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --itrace=e-o
1384 There were none.
1386 perf script can be used to provide an instruction trace showing
1388 timestamps
1389 $ perf script -i perf.data.kvm --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
1391 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
1392 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
1393 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
1394 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
1395 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
1396 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
1397 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
1398 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
1399 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
1400 CPU 1/KVM 17006 [001] 11500.262865593: ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
1401 CPU 1/KVM 17006 [001] 11500.262866075: ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.05 (40/769)
1402 :17006 17006 [001] 11500.262869216: ffffffff82200cb0 asm_sysvec_apic_timer_interrupt+0x0 ([guest.kernel.kallsyms]) clac
1403 :17006 17006 [001] 11500.262869216: ffffffff82200cb3 asm_sysvec_apic_timer_interrupt+0x3 ([guest.kernel.kallsyms]) pushq $0xffffffffffffffff
1404 :17006 17006 [001] 11500.262869216: ffffffff82200cb5 asm_sysvec_apic_timer_interrupt+0x5 ([guest.kernel.kallsyms]) callq 0xffffffff82201160
1405 :17006 17006 [001] 11500.262869216: ffffffff82201160 error_entry+0x0 ([guest.kernel.kallsyms]) cld
1406 :17006 17006 [001] 11500.262869216: ffffffff82201161 error_entry+0x1 ([guest.kernel.kallsyms]) pushq %rsi
1407 :17006 17006 [001] 11500.262869216: ffffffff82201162 error_entry+0x2 ([guest.kernel.kallsyms]) movq 0x8(%rsp), %rsi
1408 :17006 17006 [001] 11500.262869216: ffffffff82201167 error_entry+0x7 ([guest.kernel.kallsyms]) movq %rdi, 0x8(%rsp)
1409 :17006 17006 [001] 11500.262869216: ffffffff8220116c error_entry+0xc ([guest.kernel.kallsyms]) pushq %rdx
1410 :17006 17006 [001] 11500.262869216: ffffffff8220116d error_entry+0xd ([guest.kernel.kallsyms]) pushq %rcx
1411 :17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax
1412
1414 It is possible to use perf record to record sideband events within a
1415 virtual machine, so that an Intel PT trace on the host can be decoded.
1416 Sideband events from the guest perf.data file can be injected into the
1417 host perf.data file using perf inject.
1418 Here is an example of the steps needed:
1420 On the guest machine:
1422 Check that no-kvmclock kernel command line option was used to boot:
1424 Note, this is essential to enable time correlation between host and
1426 guest machines.
1427 $ cat /proc/cmdline
1429 BOOT_IMAGE=/boot/vmlinuz-5.10.0-16-amd64 root=UUID=cb49c910-e573-47e0-bce7-79e293df8e1d ro no-kvmclock
1430 There is no BPF support at present so, if possible, disable JIT
1432 compiling:
1433 $ echo 0 | sudo tee /proc/sys/net/core/bpf_jit_enable
1435 0
1436 Start perf record to collect sideband events:
1438 $ sudo perf record -o guest-sideband-testing-guest-perf.data --sample-identifier --buildid-all --switch-events --kcore -a -e dummy
1440 On the host machine:
1442 Start perf record to collect Intel PT trace:
1444 Note, the host trace will get very big, very fast, so the steps from
1446 starting to stopping the host trace really need to be done so that they
1447 happen in the shortest time possible.
1448 $ sudo perf record -o guest-sideband-testing-host-perf.data -m,64M --kcore -a -e intel_pt/cyc/
1450 On the guest machine:
1452 Run a small test case, just uname in this example:
1454 $ uname
1456 Linux
1457 On the host machine:
1459 Stop the Intel PT trace:
1461 ^C
1463 [ perf record: Woken up 1 times to write data ]
1464 [ perf record: Captured and wrote 76.122 MB guest-sideband-testing-host-perf.data ]
1465 On the guest machine:
1467 Stop the Intel PT trace:
1469 ^C
1471 [ perf record: Woken up 1 times to write data ]
1472 [ perf record: Captured and wrote 1.247 MB guest-sideband-testing-guest-perf.data ]
1473 And then copy guest-sideband-testing-guest-perf.data to the host (not
1475 shown here).
1476 On the host machine:
1478 With the 2 perf.data recordings, and with their ownership changed to
1480 the user.
1481 Identify the TSC Offset:
1483 $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=dry-run
1485 VMCS: 0x103fc6 TSC Offset 0xfffffa6ae070cb20
1486 VMCS: 0x103ff2 TSC Offset 0xfffffa6ae070cb20
1487 VMCS: 0x10fdaa TSC Offset 0xfffffa6ae070cb20
1488 VMCS: 0x24d57c TSC Offset 0xfffffa6ae070cb20
1489 Correct Intel PT TSC timestamps for the guest machine:
1491 $ perf inject -i guest-sideband-testing-host-perf.data --vm-time-correlation=0xfffffa6ae070cb20 --force
1493 Identify the guest machine PID:
1495 $ perf script -i guest-sideband-testing-host-perf.data --no-itrace --show-task-events | grep KVM
1497 CPU 0/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 0/KVM:13376/13381
1498 CPU 1/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 1/KVM:13376/13382
1499 CPU 2/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 2/KVM:13376/13383
1500 CPU 3/KVM 0 [000] 0.000000: PERF_RECORD_COMM: CPU 3/KVM:13376/13384
1501 Note, the QEMU option -name debug-threads=on is needed so that thread
1503 names can be used to determine which thread is running which VCPU as
1504 above. libvirt seems to use this by default.
1505 Create a guestmount, assuming the guest machine is vm_to_test:
1507 $ mkdir -p ~/guestmount/13376
1509 $ sshfs -o direct_io vm_to_test:/ ~/guestmount/13376
1510 Inject the guest perf.data file into the host perf.data file:
1512 Note, due to the guestmount option, guest object files and debug files
1514 will be copied into the build ID cache from the guest machine, with the
1515 notable exception of VDSO. If needed, VDSO can be copied manually in a
1516 fashion similar to that used by the perf-archive script.
1517 $ perf inject -i guest-sideband-testing-host-perf.data -o inj --guestmount ~/guestmount --guest-data=guest-sideband-testing-guest-perf.data,13376,0xfffffa6ae070cb20
1519 Show an excerpt from the result. In this case the CPU and time range
1521 have been to chosen to show interaction between guest and host when
1522 uname is starting to run on the guest machine:
1523 Notes:
1525 • the CPU displayed, [002] in this case, is always the host CPU
1527 • events happening in the virtual machine start with VM:13376
1529 VCPU:003, which shows the hypervisor PID 13376 and the VCPU number
1530 • only calls and errors are displayed i.e. --itrace=ce
1532 • branches entering and exiting the virtual machine are split, and
1534 show as 2 branches to/from "0 [unknown] ([unknown])"
1535 $ perf script -i inj --itrace=ce -F+machine_pid,+vcpu,+addr,+pid,+tid,-period --ns --time 7919.408803365,7919.408804631 -C 2
1537 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms])
1538 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms])
1539 CPU 3/KVM 13376/13384 [002] 7919.408803365: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms])
1540 CPU 3/KVM 13376/13384 [002] 7919.408803461: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown])
1541 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803461: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so)
1542 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408803567: branches: 7f851c9b5a5a init_cacheinfo+0x3aa (/usr/lib/x86_64-linux-gnu/libc-2.31.so) => 0 [unknown] ([unknown])
1543 CPU 3/KVM 13376/13384 [002] 7919.408803567: branches: 0 [unknown] ([unknown]) => ffffffffc0f8ed80 vmx_vmexit+0x0 ([kernel.kallsyms])
1544 CPU 3/KVM 13376/13384 [002] 7919.408803596: branches: ffffffffc0f6619a vmx_vcpu_run+0x26a ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms])
1545 CPU 3/KVM 13376/13384 [002] 7919.408803801: branches: ffffffffc0f66445 vmx_vcpu_run+0x515 ([kernel.kallsyms]) => ffffffffb2290b30 native_write_msr+0x0 ([kernel.kallsyms])
1546 CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc0f661f8 vmx_vcpu_run+0x2c8 ([kernel.kallsyms]) => ffffffffc1092300 kvm_load_host_xsave_state+0x0 ([kernel.kallsyms])
1547 CPU 3/KVM 13376/13384 [002] 7919.408803850: branches: ffffffffc1092327 kvm_load_host_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092220 kvm_load_host_xsave_state.part.0+0x0 ([kernel.kallsyms])
1548 CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662cf vmx_vcpu_run+0x39f ([kernel.kallsyms]) => ffffffffc0f63f90 vmx_recover_nmi_blocking+0x0 ([kernel.kallsyms])
1549 CPU 3/KVM 13376/13384 [002] 7919.408803862: branches: ffffffffc0f662e9 vmx_vcpu_run+0x3b9 ([kernel.kallsyms]) => ffffffffc0f619a0 __vmx_complete_interrupts+0x0 ([kernel.kallsyms])
1550 CPU 3/KVM 13376/13384 [002] 7919.408803872: branches: ffffffffc109cfb2 vcpu_enter_guest+0x752 ([kernel.kallsyms]) => ffffffffc0f5f570 vmx_handle_exit_irqoff+0x0 ([kernel.kallsyms])
1551 CPU 3/KVM 13376/13384 [002] 7919.408803881: branches: ffffffffc109d028 vcpu_enter_guest+0x7c8 ([kernel.kallsyms]) => ffffffffb234f900 __srcu_read_lock+0x0 ([kernel.kallsyms])
1552 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc109d06f vcpu_enter_guest+0x80f ([kernel.kallsyms]) => ffffffffc0f72e30 vmx_handle_exit+0x0 ([kernel.kallsyms])
1553 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72e3d vmx_handle_exit+0xd ([kernel.kallsyms]) => ffffffffc0f727c0 __vmx_handle_exit+0x0 ([kernel.kallsyms])
1554 CPU 3/KVM 13376/13384 [002] 7919.408803897: branches: ffffffffc0f72b15 __vmx_handle_exit+0x355 ([kernel.kallsyms]) => ffffffffc0f60ae0 vmx_flush_pml_buffer+0x0 ([kernel.kallsyms])
1555 CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc0f72994 __vmx_handle_exit+0x1d4 ([kernel.kallsyms]) => ffffffffc10b7090 kvm_emulate_cpuid+0x0 ([kernel.kallsyms])
1556 CPU 3/KVM 13376/13384 [002] 7919.408803903: branches: ffffffffc10b70f1 kvm_emulate_cpuid+0x61 ([kernel.kallsyms]) => ffffffffc10b6e10 kvm_cpuid+0x0 ([kernel.kallsyms])
1557 CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc10b7125 kvm_emulate_cpuid+0x95 ([kernel.kallsyms]) => ffffffffc1093110 kvm_skip_emulated_instruction+0x0 ([kernel.kallsyms])
1558 CPU 3/KVM 13376/13384 [002] 7919.408803941: branches: ffffffffc109311f kvm_skip_emulated_instruction+0xf ([kernel.kallsyms]) => ffffffffc0f5e180 vmx_get_rflags+0x0 ([kernel.kallsyms])
1559 CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc109312a kvm_skip_emulated_instruction+0x1a ([kernel.kallsyms]) => ffffffffc0f5fd30 vmx_skip_emulated_instruction+0x0 ([kernel.kallsyms])
1560 CPU 3/KVM 13376/13384 [002] 7919.408803951: branches: ffffffffc0f5fd79 vmx_skip_emulated_instruction+0x49 ([kernel.kallsyms]) => ffffffffc0f5fb50 skip_emulated_instruction+0x0 ([kernel.kallsyms])
1561 CPU 3/KVM 13376/13384 [002] 7919.408803956: branches: ffffffffc0f5fc68 skip_emulated_instruction+0x118 ([kernel.kallsyms]) => ffffffffc0f6a940 vmx_cache_reg+0x0 ([kernel.kallsyms])
1562 CPU 3/KVM 13376/13384 [002] 7919.408803964: branches: ffffffffc0f5fc11 skip_emulated_instruction+0xc1 ([kernel.kallsyms]) => ffffffffc0f5f9e0 vmx_set_interrupt_shadow+0x0 ([kernel.kallsyms])
1563 CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc109f8b1 vcpu_run+0x71 ([kernel.kallsyms]) => ffffffffc10ad2f0 kvm_cpu_has_pending_timer+0x0 ([kernel.kallsyms])
1564 CPU 3/KVM 13376/13384 [002] 7919.408803980: branches: ffffffffc10ad2fb kvm_cpu_has_pending_timer+0xb ([kernel.kallsyms]) => ffffffffc10b0490 apic_has_pending_timer+0x0 ([kernel.kallsyms])
1565 CPU 3/KVM 13376/13384 [002] 7919.408803991: branches: ffffffffc109f899 vcpu_run+0x59 ([kernel.kallsyms]) => ffffffffc109c860 vcpu_enter_guest+0x0 ([kernel.kallsyms])
1566 CPU 3/KVM 13376/13384 [002] 7919.408803993: branches: ffffffffc109cd4c vcpu_enter_guest+0x4ec ([kernel.kallsyms]) => ffffffffc0f69140 vmx_prepare_switch_to_guest+0x0 ([kernel.kallsyms])
1567 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd7d vcpu_enter_guest+0x51d ([kernel.kallsyms]) => ffffffffb234f930 __srcu_read_unlock+0x0 ([kernel.kallsyms])
1568 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc109cd9c vcpu_enter_guest+0x53c ([kernel.kallsyms]) => ffffffffc0f609b0 vmx_sync_pir_to_irr+0x0 ([kernel.kallsyms])
1569 CPU 3/KVM 13376/13384 [002] 7919.408803996: branches: ffffffffc0f60a6d vmx_sync_pir_to_irr+0xbd ([kernel.kallsyms]) => ffffffffc10adc20 kvm_lapic_find_highest_irr+0x0 ([kernel.kallsyms])
1570 CPU 3/KVM 13376/13384 [002] 7919.408804010: branches: ffffffffc0f60abd vmx_sync_pir_to_irr+0x10d ([kernel.kallsyms]) => ffffffffc0f60820 vmx_set_rvi+0x0 ([kernel.kallsyms])
1571 CPU 3/KVM 13376/13384 [002] 7919.408804019: branches: ffffffffc109ceca vcpu_enter_guest+0x66a ([kernel.kallsyms]) => ffffffffb2249840 fpregs_assert_state_consistent+0x0 ([kernel.kallsyms])
1572 CPU 3/KVM 13376/13384 [002] 7919.408804021: branches: ffffffffc109cf10 vcpu_enter_guest+0x6b0 ([kernel.kallsyms]) => ffffffffc0f65f30 vmx_vcpu_run+0x0 ([kernel.kallsyms])
1573 CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f6603b vmx_vcpu_run+0x10b ([kernel.kallsyms]) => ffffffffb229bed0 __get_current_cr3_fast+0x0 ([kernel.kallsyms])
1574 CPU 3/KVM 13376/13384 [002] 7919.408804024: branches: ffffffffc0f66055 vmx_vcpu_run+0x125 ([kernel.kallsyms]) => ffffffffb2253050 cr4_read_shadow+0x0 ([kernel.kallsyms])
1575 CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc0f6608d vmx_vcpu_run+0x15d ([kernel.kallsyms]) => ffffffffc10921e0 kvm_load_guest_xsave_state+0x0 ([kernel.kallsyms])
1576 CPU 3/KVM 13376/13384 [002] 7919.408804030: branches: ffffffffc1092207 kvm_load_guest_xsave_state+0x27 ([kernel.kallsyms]) => ffffffffc1092110 kvm_load_guest_xsave_state.part.0+0x0 ([kernel.kallsyms])
1577 CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffc0f660c6 vmx_vcpu_run+0x196 ([kernel.kallsyms]) => ffffffffb22061a0 perf_guest_get_msrs+0x0 ([kernel.kallsyms])
1578 CPU 3/KVM 13376/13384 [002] 7919.408804032: branches: ffffffffb22061a9 perf_guest_get_msrs+0x9 ([kernel.kallsyms]) => ffffffffb220cda0 intel_guest_get_msrs+0x0 ([kernel.kallsyms])
1579 CPU 3/KVM 13376/13384 [002] 7919.408804039: branches: ffffffffc0f66109 vmx_vcpu_run+0x1d9 ([kernel.kallsyms]) => ffffffffc0f652c0 clear_atomic_switch_msr+0x0 ([kernel.kallsyms])
1580 CPU 3/KVM 13376/13384 [002] 7919.408804040: branches: ffffffffc0f66119 vmx_vcpu_run+0x1e9 ([kernel.kallsyms]) => ffffffffc0f73f60 intel_pmu_lbr_is_enabled+0x0 ([kernel.kallsyms])
1581 CPU 3/KVM 13376/13384 [002] 7919.408804042: branches: ffffffffc0f73f81 intel_pmu_lbr_is_enabled+0x21 ([kernel.kallsyms]) => ffffffffc10b68e0 kvm_find_cpuid_entry+0x0 ([kernel.kallsyms])
1582 CPU 3/KVM 13376/13384 [002] 7919.408804045: branches: ffffffffc0f66454 vmx_vcpu_run+0x524 ([kernel.kallsyms]) => ffffffffc0f61ff0 vmx_update_hv_timer+0x0 ([kernel.kallsyms])
1583 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66142 vmx_vcpu_run+0x212 ([kernel.kallsyms]) => ffffffffc10af100 kvm_wait_lapic_expire+0x0 ([kernel.kallsyms])
1584 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66156 vmx_vcpu_run+0x226 ([kernel.kallsyms]) => ffffffffb2255c60 x86_virt_spec_ctrl+0x0 ([kernel.kallsyms])
1585 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f66161 vmx_vcpu_run+0x231 ([kernel.kallsyms]) => ffffffffc0f8eb20 vmx_vcpu_enter_exit+0x0 ([kernel.kallsyms])
1586 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffc0f8eb44 vmx_vcpu_enter_exit+0x24 ([kernel.kallsyms]) => ffffffffb2353e10 rcu_note_context_switch+0x0 ([kernel.kallsyms])
1587 CPU 3/KVM 13376/13384 [002] 7919.408804057: branches: ffffffffb2353e1c rcu_note_context_switch+0xc ([kernel.kallsyms]) => ffffffffb2353db0 rcu_qs+0x0 ([kernel.kallsyms])
1588 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ebe0 vmx_vcpu_enter_exit+0xc0 ([kernel.kallsyms]) => ffffffffc0f8edc0 __vmx_vcpu_run+0x0 ([kernel.kallsyms])
1589 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8edd5 __vmx_vcpu_run+0x15 ([kernel.kallsyms]) => ffffffffc0f8eca0 vmx_update_host_rsp+0x0 ([kernel.kallsyms])
1590 CPU 3/KVM 13376/13384 [002] 7919.408804066: branches: ffffffffc0f8ee1b __vmx_vcpu_run+0x5b ([kernel.kallsyms]) => ffffffffc0f8ed60 vmx_vmenter+0x0 ([kernel.kallsyms])
1591 CPU 3/KVM 13376/13384 [002] 7919.408804162: branches: ffffffffc0f8ed62 vmx_vmenter+0x2 ([kernel.kallsyms]) => 0 [unknown] ([unknown])
1592 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804162: branches: 0 [unknown] ([unknown]) => 7f851c9b5a5c init_cacheinfo+0x3ac (/usr/lib/x86_64-linux-gnu/libc-2.31.so)
1593 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804273: branches: 7f851cb7c0e4 _dl_init+0x74 (/usr/lib/x86_64-linux-gnu/ld-2.31.so) => 7f851cb7bf50 call_init.part.0+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)
1594 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: 55e0c00136f0 _start+0x0 (/usr/bin/uname) => ffffffff83200ac0 asm_exc_page_fault+0x0 ([kernel.kallsyms])
1595 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804526: branches: ffffffff83200ac3 asm_exc_page_fault+0x3 ([kernel.kallsyms]) => ffffffff83201290 error_entry+0x0 ([kernel.kallsyms])
1596 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804534: branches: ffffffff832012fa error_entry+0x6a ([kernel.kallsyms]) => ffffffff830b59a0 sync_regs+0x0 ([kernel.kallsyms])
1597 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff83200ad9 asm_exc_page_fault+0x19 ([kernel.kallsyms]) => ffffffff830b8210 exc_page_fault+0x0 ([kernel.kallsyms])
1598 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b82a4 exc_page_fault+0x94 ([kernel.kallsyms]) => ffffffff830b80e0 __kvm_handle_async_pf+0x0 ([kernel.kallsyms])
1599 VM:13376 VCPU:003 uname 3404/3404 [002] 7919.408804631: branches: ffffffff830b80ed __kvm_handle_async_pf+0xd ([kernel.kallsyms]) => ffffffff830b80c0 kvm_read_and_reset_apf_flags+0x0 ([kernel.kallsyms])
1600
1602 A common case for KVM test programs is that the test program acts as
1603 the hypervisor, creating, running and destroying the virtual machine,
1604 and providing the guest object code from its own object code. In this
1605 case, the VM is not running an OS, but only the functions loaded into
1606 it by the hypervisor test program, and conveniently, loaded at the same
1607 virtual addresses. To support that, option "--guest-code" has been
1608 added to perf script and perf kvm report.
1609 Here is an example tracing a test program from the kernel’s KVM
1611 selftests:
1612 # perf record --kcore -e intel_pt/cyc/ -- tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test
1614 [ perf record: Woken up 1 times to write data ]
1615 [ perf record: Captured and wrote 0.280 MB perf.data ]
1616 # perf script --guest-code --itrace=bep --ns -F-period,+addr,+flags
1617 [SNIP]
1618 tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux)
1619 tsc_msrs_test 18436 [007] 10897.962087733: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux)
1620 tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux)
1621 tsc_msrs_test 18436 [007] 10897.962087836: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown])
1622 [guest/18436] 18436 [007] 10897.962087836: branches: vmentry 0 [unknown] ([unknown]) => 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1623 [guest/18436] 18436 [007] 10897.962087836: branches: call 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1624 [guest/18436] 18436 [007] 10897.962088248: branches: vmexit 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown])
1625 tsc_msrs_test 18436 [007] 10897.962088248: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux)
1626 tsc_msrs_test 18436 [007] 10897.962088248: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux)
1627 tsc_msrs_test 18436 [007] 10897.962088256: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux)
1628 tsc_msrs_test 18436 [007] 10897.962088270: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux)
1629 [SNIP]
1630 tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux)
1631 tsc_msrs_test 18436 [007] 10897.962089321: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux)
1632 tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux)
1633 tsc_msrs_test 18436 [007] 10897.962089424: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown])
1634 [guest/18436] 18436 [007] 10897.962089424: branches: vmentry 0 [unknown] ([unknown]) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1635 [guest/18436] 18436 [007] 10897.962089701: branches: jmp 40dc1b ucall+0x7b (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc39 ucall+0x99 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1636 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1637 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1638 [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc37 ucall+0x97 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc50 ucall+0xb0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test)
1639 [guest/18436] 18436 [007] 10897.962089878: branches: vmexit 40dc55 ucall+0xb5 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown])
1640 tsc_msrs_test 18436 [007] 10897.962089878: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux)
1641 tsc_msrs_test 18436 [007] 10897.962089878: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux)
1642 tsc_msrs_test 18436 [007] 10897.962089887: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux)
1643 tsc_msrs_test 18436 [007] 10897.962089901: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux)
1644 [SNIP]
1645 # perf kvm --guest-code --guest --host report -i perf.data --stdio | head -20
1647 # To display the perf.data header info, please use --header/--header-only options.
1649 #
1650 #
1651 # Total Lost Samples: 0
1652 #
1653 # Samples: 12 of event 'instructions'
1654 # Event count (approx.): 2274583
1655 #
1656 # Children Self Command Shared Object Symbol
1657 # ........ ........ ............. .................... ...........................................
1658 #
1659 54.70% 0.00% tsc_msrs_test [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
1660 |
1661 ---entry_SYSCALL_64_after_hwframe
1662 do_syscall_64
1663 |
1664 |--29.44%--syscall_exit_to_user_mode
1665 | exit_to_user_mode_prepare
1666 | task_work_run
1667 | __fput
1668
1670 Event Trace records information about asynchronous events, for example
1671 interrupts, faults, VM exits and entries. The information is recorded
1672 in CFE and EVD packets, and also the Interrupt Flag is recorded on the
1673 MODE.Exec packet. The CFE packet contains a type field to identify one
1674 of the following:
1675 1 INTR interrupt, fault, exception, NMI
1677 2 IRET interrupt return
1678 3 SMI system management interrupt
1679 4 RSM resume from system management mode
1680 5 SIPI startup interprocessor interrupt
1681 6 INIT INIT signal
1682 7 VMENTRY VM-Entry
1683 8 VMEXIT VM-Entry
1684 9 VMEXIT_INTR VM-Exit due to interrupt
1685 10 SHUTDOWN Shutdown
1686 For more details, refer to the Intel 64 and IA-32 Architectures
1688 Software Developer Manuals (version 076 or later).
1689 The capability to do Event Trace is indicated by the
1691 /sys/bus/event_source/devices/intel_pt/caps/event_trace file.
1692 Event trace is selected for recording using the "event" config term.
1694 e.g.
1695 perf record -e intel_pt/event/u uname
1697 Event trace events are output using the --itrace I option. e.g.
1699 perf script --itrace=Ie
1701 perf script displays events containing CFE type, vector and event data,
1703 in the form:
1704 evt: hw int (t) cfe: INTR IP: 1 vector: 3 PFA: 0x8877665544332211
1706 The IP flag indicates if the event binds to an IP, which includes any
1708 case where flow control packet generation is enabled, as well as when
1709 CFE packet IP bit is set.
1710 perf script displays events containing changes to the Interrupt Flag in
1712 the form:
1713 iflag: t IFLAG: 1->0 via branch
1715 where "via branch" indicates a branch (interrupt or return from
1717 interrupt) and "non branch" indicates an instruction such as CFI, STI
1718 or POPF).
1719 In addition, the current state of the interrupt flag is indicated by
1721 the presence or absence of the "D" (interrupt disabled) perf script
1722 flag. If the interrupt flag is changed, then the "t" flag is also
1723 included i.e.
1724 no flag, interrupts enabled IF=1
1726 t interrupts become disabled IF=1 -> IF=0
1727 D interrupts are disabled IF=0
1728 Dt interrupts become enabled IF=0 -> IF=1
1729 The intel-pt-events.py script illustrates how to access Event Trace
1731 information using a Python script.
1732
1734 TNT packets are disabled using the "notnt" config term. e.g.
1735 perf record -e intel_pt/notnt/u uname
1737 In that case the --itrace q option is forced because walking executable
1739 code to reconstruct the control flow is not possible.
1740
1742 Later perf tools support a method to emulate the ptwrite instruction,
1743 which can be useful if hardware does not support the ptwrite
1744 instruction.
1745 Instead of using the ptwrite instruction, a function is used which
1747 produces a trace that encodes the payload data into TNT packets. Here
1748 is an example of the function:
1749 #include <stdint.h>
1751 void perf_emulate_ptwrite(uint64_t x)
1753 __attribute__((externally_visible, noipa, no_instrument_function, naked));
1754 #define PERF_EMULATE_PTWRITE_8_BITS \
1756 "1: shl %rax\n" \
1757 " jc 1f\n" \
1758 "1: shl %rax\n" \
1759 " jc 1f\n" \
1760 "1: shl %rax\n" \
1761 " jc 1f\n" \
1762 "1: shl %rax\n" \
1763 " jc 1f\n" \
1764 "1: shl %rax\n" \
1765 " jc 1f\n" \
1766 "1: shl %rax\n" \
1767 " jc 1f\n" \
1768 "1: shl %rax\n" \
1769 " jc 1f\n" \
1770 "1: shl %rax\n" \
1771 " jc 1f\n"
1772 /* Undefined instruction */
1774 #define PERF_EMULATE_PTWRITE_UD2 ".byte 0x0f, 0x0b\n"
1775 #define PERF_EMULATE_PTWRITE_MAGIC PERF_EMULATE_PTWRITE_UD2 ".ascii \"perf,ptwrite \"\n"
1777 void perf_emulate_ptwrite(uint64_t x __attribute__ ((__unused__)))
1779 {
1780 /* Assumes SysV ABI : x passed in rdi */
1781 __asm__ volatile (
1782 "jmp 1f\n"
1783 PERF_EMULATE_PTWRITE_MAGIC
1784 "1: mov %rdi, %rax\n"
1785 PERF_EMULATE_PTWRITE_8_BITS
1786 PERF_EMULATE_PTWRITE_8_BITS
1787 PERF_EMULATE_PTWRITE_8_BITS
1788 PERF_EMULATE_PTWRITE_8_BITS
1789 PERF_EMULATE_PTWRITE_8_BITS
1790 PERF_EMULATE_PTWRITE_8_BITS
1791 PERF_EMULATE_PTWRITE_8_BITS
1792 PERF_EMULATE_PTWRITE_8_BITS
1793 "1: ret\n"
1794 );
1795 }
1796 For example, a test program with the function above:
1798 #include <stdio.h>
1800 #include <stdint.h>
1801 #include <stdlib.h>
1802 #include "perf_emulate_ptwrite.h"
1804 int main(int argc, char *argv[])
1806 {
1807 uint64_t x = 0;
1808 if (argc > 1)
1810 x = strtoull(argv[1], NULL, 0);
1811 perf_emulate_ptwrite(x);
1812 return 0;
1813 }
1814 Can be compiled and traced:
1816 $ gcc -Wall -Wextra -O3 -g -o eg_ptw eg_ptw.c
1818 $ perf record -e intel_pt//u ./eg_ptw 0x1234567890abcdef
1819 [ perf record: Woken up 1 times to write data ]
1820 [ perf record: Captured and wrote 0.017 MB perf.data ]
1821 $ perf script --itrace=ew
1822 eg_ptw 19875 [007] 8061.235912: ptwrite: IP: 0 payload: 0x1234567890abcdef 55701249a196 perf_emulate_ptwrite+0x16 (/home/user/eg_ptw)
1823 $
1824
1826 Examples can be found on perf wiki page "Perf tools support for Intel®
1827 Processor Trace":
1828 https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
1830
1832 perf-record(1), perf-script(1), perf-report(1), perf-inject(1)
1833perf 01/12/2023 PERF-INTEL-PT(1)