1PERF-INTEL-PT(1) perf Manual PERF-INTEL-PT(1)
2
6 perf-intel-pt - Support for Intel Processor Trace within perf tools
7
9 perf record -e intel_pt//
10
12 Intel Processor Trace (Intel PT) is an extension of Intel Architecture
13 that collects information about software execution such as control
14 flow, execution modes and timings and formats it into highly compressed
15 binary packets. Technical details are documented in the Intel 64 and
16 IA-32 Architectures Software Developer Manuals, Chapter 36 Intel
17 Processor Trace.
18 Intel PT is first supported in Intel Core M and 5th generation Intel
20 Core processors that are based on the Intel micro-architecture code
21 name Broadwell.
22 Trace data is collected by perf record and stored within the perf.data
24 file. See below for options to perf record.
25 Trace data must be decoded which involves walking the object code and
27 matching the trace data packets. For example a TNT packet only tells
28 whether a conditional branch was taken or not taken, so to make use of
29 that packet the decoder must know precisely which instruction was being
30 executed.
31 Decoding is done on-the-fly. The decoder outputs samples in the same
33 format as samples output by perf hardware events, for example as though
34 the "instructions" or "branches" events had been recorded. Presently 3
35 tools support this: perf script, perf report and perf inject. See below
36 for more information on using those tools.
37 The main distinguishing feature of Intel PT is that the decoder can
39 determine the exact flow of software execution. Intel PT can be used to
40 understand why and how did software get to a certain point, or behave a
41 certain way. The software does not have to be recompiled, so Intel PT
42 works with debug or release builds, however the executed images are
43 needed - which makes use in JIT-compiled environments, or with
44 self-modified code, a challenge. Also symbols need to be provided to
45 make sense of addresses.
46 A limitation of Intel PT is that it produces huge amounts of trace data
48 (hundreds of megabytes per second per core) which takes a long time to
49 decode, for example two or three orders of magnitude longer than it
50 took to collect. Another limitation is the performance impact of
51 tracing, something that will vary depending on the use-case and
52 architecture.
53
55 It is important to start small. That is because it is easy to capture
56 vastly more data than can possibly be processed.
57 The simplest thing to do with Intel PT is userspace profiling of small
59 programs. Data is captured with perf record e.g. to trace ls
60 userspace-only:
61 perf record -e intel_pt//u ls
63 And profiled with perf report e.g.
65 perf report
67 To also trace kernel space presents a problem, namely kernel
69 self-modifying code. A fairly good kernel image is available in
70 /proc/kcore but to get an accurate image a copy of /proc/kcore needs to
71 be made under the same conditions as the data capture. perf record can
72 make a copy of /proc/kcore if the option --kcore is used, but access to
73 /proc/kcore is restricted e.g.
74 sudo perf record -o pt_ls --kcore -e intel_pt// -- ls
76 which will create a directory named pt_ls and put the perf.data file
78 (named simply data) and copies of /proc/kcore, /proc/kallsyms and
79 /proc/modules into it. The other tools understand the directory format,
80 so to use perf report becomes:
81 sudo perf report -i pt_ls
83 Because samples are synthesized after-the-fact, the sampling period can
85 be selected for reporting. e.g. sample every microsecond
86 sudo perf report pt_ls --itrace=i1usge
88 See the sections below for more information about the --itrace option.
90 Beware the smaller the period, the more samples that are produced, and
92 the longer it takes to process them.
93 Also note that the coarseness of Intel PT timing information will start
95 to distort the statistical value of the sampling as the sampling period
96 becomes smaller.
97 To represent software control flow, "branches" samples are produced. By
99 default a branch sample is synthesized for every single branch. To get
100 an idea what data is available you can use the perf script tool with
101 all itrace sampling options, which will list all the samples.
102 perf record -e intel_pt//u ls
104 perf script --itrace=ibxwpe
105 An interesting field that is not printed by default is flags which can
107 be displayed as follows:
108 perf script --itrace=ibxwpe -F+flags
110 The flags are "bcrosyiABEx" which stand for branch, call, return,
112 conditional, system, asynchronous, interrupt, transaction abort, trace
113 begin, trace end, and in transaction, respectively.
114 perf script also supports higher level ways to dump instruction traces:
116 perf script --insn-trace --xed
118 Dump all instructions. This requires installing the xed tool (see XED
120 below) Dumping all instructions in a long trace can be fairly slow. It
121 is usually better to start with higher level decoding, like
122 perf script --call-trace
124 or
126 perf script --call-ret-trace
128 and then select a time range of interest. The time range can then be
130 examined in detail with
131 perf script --time starttime,stoptime --insn-trace --xed
133 While examining the trace it’s also useful to filter on specific CPUs
135 using the -C option
136 perf script --time starttime,stoptime --insn-trace --xed -C 1
138 Dump all instructions in time range on CPU 1.
140 Another interesting field that is not printed by default is ipc which
142 can be displayed as follows:
143 perf script --itrace=be -F+ipc
145 There are two ways that instructions-per-cycle (IPC) can be calculated
147 depending on the recording.
148 If the cyc config term (see config terms section below) was used, then
150 IPC is calculated using the cycle count from CYC packets, otherwise MTC
151 packets are used - refer to the mtc config term. When MTC is used,
152 however, the values are less accurate because the timing is less
153 accurate.
154 Because Intel PT does not update the cycle count on every branch or
156 instruction, the values will often be zero. When there are values, they
157 will be the number of instructions and number of cycles since the last
158 update, and thus represent the average IPC since the last IPC for that
159 event type. Note IPC for "branches" events is calculated separately
160 from IPC for "instructions" events.
161 Also note that the IPC instruction count may or may not include the
163 current instruction. If the cycle count is associated with an
164 asynchronous branch (e.g. page fault or interrupt), then the
165 instruction count does not include the current instruction, otherwise
166 it does. That is consistent with whether or not that instruction has
167 retired when the cycle count is updated.
168 Another note, in the case of "branches" events, non-taken branches are
170 not presently sampled, so IPC values for them do not appear e.g. a CYC
171 packet with a TNT packet that starts with a non-taken branch. To see
172 every possible IPC value, "instructions" events can be used e.g.
173 --itrace=i0ns
174 While it is possible to create scripts to analyze the data, an
176 alternative approach is available to export the data to a sqlite or
177 postgresql database. Refer to script export-to-sqlite.py or
178 export-to-postgresql.py for more details, and to script
179 exported-sql-viewer.py for an example of using the database.
180 There is also script intel-pt-events.py which provides an example of
182 how to unpack the raw data for power events and PTWRITE.
183 As mentioned above, it is easy to capture too much data. One way to
185 limit the data captured is to use snapshot mode which is explained
186 further below. Refer to new snapshot option and Intel PT modes of
187 operation further below.
188 Another problem that will be experienced is decoder errors. They can be
190 caused by inability to access the executed image, self-modified or
191 JIT-ed code, or the inability to match side-band information (such as
192 context switches and mmaps) which results in the decoder not knowing
193 what code was executed.
194 There is also the problem of perf not being able to copy the data fast
196 enough, resulting in data lost because the buffer was full. See Buffer
197 handling below for more details.
198
200 new event
201 The Intel PT kernel driver creates a new PMU for Intel PT. PMU events
202 are selected by providing the PMU name followed by the "config"
203 separated by slashes. An enhancement has been made to allow default
204 "config" e.g. the option
205 -e intel_pt//
207 will use a default config value. Currently that is the same as
209 -e intel_pt/tsc,noretcomp=0/
211 which is the same as
213 -e intel_pt/tsc=1,noretcomp=0/
215 Note there are now new config terms - see section config terms further
217 below.
218 The config terms are listed in /sys/devices/intel_pt/format. They are
220 bit fields within the config member of the struct perf_event_attr which
221 is passed to the kernel by the perf_event_open system call. They
222 correspond to bit fields in the IA32_RTIT_CTL MSR. Here is a list of
223 them and their definitions:
224 $ grep -H . /sys/bus/event_source/devices/intel_pt/format/*
226 /sys/bus/event_source/devices/intel_pt/format/cyc:config:1
227 /sys/bus/event_source/devices/intel_pt/format/cyc_thresh:config:19-22
228 /sys/bus/event_source/devices/intel_pt/format/mtc:config:9
229 /sys/bus/event_source/devices/intel_pt/format/mtc_period:config:14-17
230 /sys/bus/event_source/devices/intel_pt/format/noretcomp:config:11
231 /sys/bus/event_source/devices/intel_pt/format/psb_period:config:24-27
232 /sys/bus/event_source/devices/intel_pt/format/tsc:config:10
233 Note that the default config must be overridden for each term i.e.
235 -e intel_pt/noretcomp=0/
237 is the same as:
239 -e intel_pt/tsc=1,noretcomp=0/
241 So, to disable TSC packets use:
243 -e intel_pt/tsc=0/
245 It is also possible to specify the config value explicitly:
247 -e intel_pt/config=0x400/
249 Note that, as with all events, the event is suffixed with event
251 modifiers:
252 u userspace
254 k kernel
255 h hypervisor
256 G guest
257 H host
258 p precise ip
259 h, G and H are for virtualization which is not supported by Intel PT. p
261 is also not relevant to Intel PT. So only options u and k are
262 meaningful for Intel PT.
263 perf_event_attr is displayed if the -vv option is used e.g.
265 ------------------------------------------------------------
267 perf_event_attr:
268 type 6
269 size 112
270 config 0x400
271 { sample_period, sample_freq } 1
272 sample_type IP|TID|TIME|CPU|IDENTIFIER
273 read_format ID
274 disabled 1
275 inherit 1
276 exclude_kernel 1
277 exclude_hv 1
278 enable_on_exec 1
279 sample_id_all 1
280 ------------------------------------------------------------
281 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
282 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
283 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
284 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
285 ------------------------------------------------------------
286 config terms
288 The June 2015 version of Intel 64 and IA-32 Architectures Software
289 Developer Manuals, Chapter 36 Intel Processor Trace, defined new Intel
290 PT features. Some of the features are reflect in new config terms. All
291 the config terms are described below.
292 tsc Always supported. Produces TSC timestamp packets to provide timing
294 information. In some cases it is possible to decode without timing
295 information, for example a per-thread context that does not overlap
296 executable memory maps.
297 The default config selects tsc (i.e. tsc=1).
299 noretcomp Always supported. Disables "return compression" so a TIP
301 packet is produced when a function returns. Causes more packets to be
302 produced but might make decoding more reliable.
303 The default config does not select noretcomp (i.e. noretcomp=0).
305 psb_period Allows the frequency of PSB packets to be specified.
307 The PSB packet is a synchronization packet that provides a
309 starting point for decoding or recovery from errors.
310 Support for psb_period is indicated by:
312 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
314 which contains "1" if the feature is supported and "0"
316 otherwise.
317 Valid values are given by:
319 /sys/bus/event_source/devices/intel_pt/caps/psb_periods
321 which contains a hexadecimal value, the bits of which represent
323 valid values e.g. bit 2 set means value 2 is valid.
324 The psb_period value is converted to the approximate number of
326 trace bytes between PSB packets as:
327 2 ^ (value + 11)
329 e.g. value 3 means 16KiB bytes between PSBs
331 If an invalid value is entered, the error message
333 will give a list of valid values e.g.
334 $ perf record -e intel_pt/psb_period=15/u uname
336 Invalid psb_period for intel_pt. Valid values are: 0-5
337 If MTC packets are selected, the default config selects a value
339 of 3 (i.e. psb_period=3) or the nearest lower value that is
340 supported (0 is always supported). Otherwise the default is 0.
341 If decoding is expected to be reliable and the buffer is large
343 then a large PSB period can be used.
344 Because a TSC packet is produced with PSB, the PSB period can
346 also affect the granularity to timing information in the absence
347 of MTC or CYC.
348 mtc Produces MTC timing packets.
350 MTC packets provide finer grain timestamp information than TSC
352 packets. MTC packets record time using the hardware crystal
353 clock (CTC) which is related to TSC packets using a TMA packet.
354 Support for this feature is indicated by:
356 /sys/bus/event_source/devices/intel_pt/caps/mtc
358 which contains "1" if the feature is supported and
360 "0" otherwise.
361 The frequency of MTC packets can also be specified - see
363 mtc_period below.
364 mtc_period Specifies how frequently MTC packets are produced - see mtc
366 above for how to determine if MTC packets are supported.
367 Valid values are given by:
369 /sys/bus/event_source/devices/intel_pt/caps/mtc_periods
371 which contains a hexadecimal value, the bits of which represent
373 valid values e.g. bit 2 set means value 2 is valid.
374 The mtc_period value is converted to the MTC frequency as:
376 CTC-frequency / (2 ^ value)
378 e.g. value 3 means one eighth of CTC-frequency
380 Where CTC is the hardware crystal clock, the frequency of which
382 can be related to TSC via values provided in cpuid leaf 0x15.
383 If an invalid value is entered, the error message
385 will give a list of valid values e.g.
386 $ perf record -e intel_pt/mtc_period=15/u uname
388 Invalid mtc_period for intel_pt. Valid values are: 0,3,6,9
389 The default value is 3 or the nearest lower value
391 that is supported (0 is always supported).
392 cyc Produces CYC timing packets.
394 CYC packets provide even finer grain timestamp information than
396 MTC and TSC packets. A CYC packet contains the number of CPU
397 cycles since the last CYC packet. Unlike MTC and TSC packets,
398 CYC packets are only sent when another packet is also sent.
399 Support for this feature is indicated by:
401 /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
403 which contains "1" if the feature is supported and
405 "0" otherwise.
406 The number of CYC packets produced can be reduced by specifying
408 a threshold - see cyc_thresh below.
409 cyc_thresh Specifies how frequently CYC packets are produced - see cyc
411 above for how to determine if CYC packets are supported.
412 Valid cyc_thresh values are given by:
414 /sys/bus/event_source/devices/intel_pt/caps/cycle_thresholds
416 which contains a hexadecimal value, the bits of which represent
418 valid values e.g. bit 2 set means value 2 is valid.
419 The cyc_thresh value represents the minimum number of CPU cycles
421 that must have passed before a CYC packet can be sent. The
422 number of CPU cycles is:
423 2 ^ (value - 1)
425 e.g. value 4 means 8 CPU cycles must pass before a CYC packet
427 can be sent. Note a CYC packet is still only sent when another
428 packet is sent, not at, e.g. every 8 CPU cycles.
429 If an invalid value is entered, the error message
431 will give a list of valid values e.g.
432 $ perf record -e intel_pt/cyc,cyc_thresh=15/u uname
434 Invalid cyc_thresh for intel_pt. Valid values are: 0-12
435 CYC packets are not requested by default.
437 pt Specifies pass-through which enables the branch config term.
439 The default config selects 'pt' if it is available, so a user will
441 never need to specify this term.
442 branch Enable branch tracing. Branch tracing is enabled by default so
444 to disable branch tracing use branch=0.
445 The default config selects 'branch' if it is available.
447 ptw Enable PTWRITE packets which are produced when a ptwrite
449 instruction is executed.
450 Support for this feature is indicated by:
452 /sys/bus/event_source/devices/intel_pt/caps/ptwrite
454 which contains "1" if the feature is supported and
456 "0" otherwise.
457 fup_on_ptw Enable a FUP packet to follow the PTWRITE packet. The FUP
459 packet provides the address of the ptwrite instruction. In the absence
460 of fup_on_ptw, the decoder will use the address of the previous branch
461 if branch tracing is enabled, otherwise the address will be zero. Note
462 that fup_on_ptw will work even when branch tracing is disabled.
463 pwr_evt Enable power events. The power events provide information about
465 changes to the CPU C-state.
466 Support for this feature is indicated by:
468 /sys/bus/event_source/devices/intel_pt/caps/power_event_trace
470 which contains "1" if the feature is supported and
472 "0" otherwise.
473 AUX area sampling option
475 To select Intel PT "sampling" the AUX area sampling option can be used:
476 --aux-sample
478 Optionally it can be followed by the sample size in bytes e.g.
480 --aux-sample=8192
482 In addition, the Intel PT event to sample must be defined e.g.
484 -e intel_pt//u
486 Samples on other events will be created containing Intel PT data e.g.
488 the following will create Intel PT samples on the branch-misses event,
489 note the events must be grouped using {}:
490 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}'
492 An alternative to --aux-sample is to add the config term
494 aux-sample-size to events. In this case, the grouping is implied e.g.
495 perf record -e intel_pt//u -e branch-misses/aux-sample-size=8192/u
497 is the same as:
499 perf record -e '{intel_pt//u,branch-misses/aux-sample-size=8192/u}'
501 but allows for also using an address filter e.g.:
503 perf record -e intel_pt//u --filter 'filter * @/bin/ls' -e branch-misses/aux-sample-size=8192/u -- ls
505 It is important to select a sample size that is big enough to contain
507 at least one PSB packet. If not a warning will be displayed:
508 Intel PT sample size (%zu) may be too small for PSB period (%zu)
510 The calculation used for that is: if sample_size ⟨ psb_period + 256
512 display the warning. When sampling is used, psb_period defaults to 0
513 (2KiB).
514 The default sample size is 4KiB.
516 The sample size is passed in aux_sample_size in struct perf_event_attr.
518 The sample size is limited by the maximum event size which is 64KiB. It
519 is difficult to know how big the event might be without the trace
520 sample attached, but the tool validates that the sample size is not
521 greater than 60KiB.
522 new snapshot option
524 The difference between full trace and snapshot from the kernel’s
525 perspective is that in full trace we don’t overwrite trace data that
526 the user hasn’t collected yet (and indicated that by advancing
527 aux_tail), whereas in snapshot mode we let the trace run and overwrite
528 older data in the buffer so that whenever something interesting
529 happens, we can stop it and grab a snapshot of what was going on around
530 that interesting moment.
531 To select snapshot mode a new option has been added:
533 -S
535 Optionally it can be followed by the snapshot size e.g.
537 -S0x100000
539 The default snapshot size is the auxtrace mmap size. If neither
541 auxtrace mmap size nor snapshot size is specified, then the default is
542 4MiB for privileged users (or if /proc/sys/kernel/perf_event_paranoid <
543 0), 128KiB for unprivileged users. If an unprivileged user does not
544 specify mmap pages, the mmap pages will be reduced as described in the
545 new auxtrace mmap size option section below.
546 The snapshot size is displayed if the option -vv is used e.g.
548 Intel PT snapshot size: %zu
550 new auxtrace mmap size option
552 Intel PT buffer size is specified by an addition to the -m option e.g.
553 -m,16
555 selects a buffer size of 16 pages i.e. 64KiB.
557 Note that the existing functionality of -m is unchanged. The auxtrace
559 mmap size is specified by the optional addition of a comma and the
560 value.
561 The default auxtrace mmap size for Intel PT is 4MiB/page_size for
563 privileged users (or if /proc/sys/kernel/perf_event_paranoid < 0),
564 128KiB for unprivileged users. If an unprivileged user does not specify
565 mmap pages, the mmap pages will be reduced from the default
566 512KiB/page_size to 256KiB/page_size, otherwise the user is likely to
567 get an error as they exceed their mlock limit (Max locked memory as
568 shown in /proc/self/limits). Note that perf does not count the first
569 512KiB (actually /proc/sys/kernel/perf_event_mlock_kb minus 1 page) per
570 cpu against the mlock limit so an unprivileged user is allowed 512KiB
571 per cpu plus their mlock limit (which defaults to 64KiB but is not
572 multiplied by the number of cpus).
573 In full-trace mode, powers of two are allowed for buffer size, with a
575 minimum size of 2 pages. In snapshot mode or sampling mode, it is the
576 same but the minimum size is 1 page.
577 The mmap size and auxtrace mmap size are displayed if the -vv option is
579 used e.g.
580 mmap length 528384
582 auxtrace mmap length 4198400
583 Intel PT modes of operation
585 Intel PT can be used in 3 modes: full-trace mode sample mode snapshot
586 mode
587 Full-trace mode traces continuously e.g.
589 perf record -e intel_pt//u uname
591 Sample mode attaches a Intel PT sample to other events e.g.
593 perf record --aux-sample -e intel_pt//u -e branch-misses:u
595 Snapshot mode captures the available data when a signal is sent or
597 "snapshot" control command is issued. e.g. using a signal
598 perf record -v -e intel_pt//u -S ./loopy 1000000000 &
600 [1] 11435
601 kill -USR2 11435
602 Recording AUX area tracing snapshot
603 Note that the signal sent is SIGUSR2. Note that "Recording AUX area
605 tracing snapshot" is displayed because the -v option is used.
606 The advantage of using "snapshot" control command is that the access is
608 controlled by access to a FIFO e.g.
609 $ mkfifo perf.control
611 $ mkfifo perf.ack
612 $ cat perf.ack &
613 [1] 15235
614 $ sudo ~/bin/perf record --control fifo:perf.control,perf.ack -S -e intel_pt//u -- sleep 60 &
615 [2] 15243
616 $ ps -e | grep perf
617 15244 pts/1 00:00:00 perf
618 $ kill -USR2 15244
619 bash: kill: (15244) - Operation not permitted
620 $ echo snapshot > perf.control
621 ack
622 The 3 Intel PT modes of operation cannot be used together.
624 Buffer handling
626 There may be buffer limitations (i.e. single ToPa entry) which means
627 that actual buffer sizes are limited to powers of 2 up to 4MiB
628 (MAX_ORDER). In order to provide other sizes, and in particular an
629 arbitrarily large size, multiple buffers are logically concatenated.
630 However an interrupt must be used to switch between buffers. That has
631 two potential problems: a) the interrupt may not be handled in time so
632 that the current buffer becomes full and some trace data is lost. b)
633 the interrupts may slow the system and affect the performance results.
634 If trace data is lost, the driver sets truncated in the PERF_RECORD_AUX
636 event which the tools report as an error.
637 In full-trace mode, the driver waits for data to be copied out before
639 allowing the (logical) buffer to wrap-around. If data is not copied out
640 quickly enough, again truncated is set in the PERF_RECORD_AUX event. If
641 the driver has to wait, the intel_pt event gets disabled. Because it is
642 difficult to know when that happens, perf tools always re-enable the
643 intel_pt event after copying out data.
644 Intel PT and build ids
646 By default "perf record" post-processes the event stream to find all
647 build ids for executables for all addresses sampled. Deliberately,
648 Intel PT is not decoded for that purpose (it would take too long).
649 Instead the build ids for all executables encountered (due to mmap,
650 comm or task events) are included in the perf.data file.
651 To see buildids included in the perf.data file use the command:
653 perf buildid-list
655 If the perf.data file contains Intel PT data, that is the same as:
657 perf buildid-list --with-hits
659 Snapshot mode and event disabling
661 In order to make a snapshot, the intel_pt event is disabled using an
662 IOCTL, namely PERF_EVENT_IOC_DISABLE. However doing that can also
663 disable the collection of side-band information. In order to prevent
664 that, a dummy software event has been introduced that permits tracking
665 events (like mmaps) to continue to be recorded while intel_pt is
666 disabled. That is important to ensure there is complete side-band
667 information to allow the decoding of subsequent snapshots.
668 A test has been created for that. To find the test:
670 perf test list
672 ...
673 23: Test using a dummy software event to keep tracking
674 To run the test:
676 perf test 23
678 23: Test using a dummy software event to keep tracking : Ok
679 perf record modes (nothing new here)
681 perf record essentially operates in one of three modes: per thread per
682 cpu workload only
683 "per thread" mode is selected by -t or by --per-thread (with -p or -u
685 or just a workload). "per cpu" is selected by -C or -a. "workload only"
686 mode is selected by not using the other options but providing a command
687 to run (i.e. the workload).
688 In per-thread mode an exact list of threads is traced. There is no
690 inheritance. Each thread has its own event buffer.
691 In per-cpu mode all processes (or processes from the selected cgroup
693 i.e. -G option, or processes selected with -p or -u) are traced. Each
694 cpu has its own buffer. Inheritance is allowed.
695 In workload-only mode, the workload is traced but with per-cpu buffers.
697 Inheritance is allowed. Note that you can now trace a workload in
698 per-thread mode by using the --per-thread option.
699 Privileged vs non-privileged users
701 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
702 users have memory limits imposed upon them. That affects what buffer
703 sizes they can have as outlined above.
704 The v4.2 kernel introduced support for a context switch metadata event,
706 PERF_RECORD_SWITCH, which allows unprivileged users to see when their
707 processes are scheduled out and in, just not by whom, which is left for
708 the PERF_RECORD_SWITCH_CPU_WIDE, that is only accessible in system wide
709 context, which in turn requires CAP_PERFMON or CAP_SYS_ADMIN.
710 Please see the 45ac1403f564 ("perf: Add PERF_RECORD_SWITCH to indicate
712 context switches") commit, that introduces these metadata events for
713 further info.
714 When working with kernels < v4.2, the following considerations must be
716 taken, as the sched:sched_switch tracepoints will be used to receive
717 such information:
718 Unless /proc/sys/kernel/perf_event_paranoid is set to -1, unprivileged
720 users are not permitted to use tracepoints which means there is
721 insufficient side-band information to decode Intel PT in per-cpu mode,
722 and potentially workload-only mode too if the workload creates new
723 processes.
724 Note also, that to use tracepoints, read-access to debugfs is required.
726 So if debugfs is not mounted or the user does not have read-access, it
727 will again not be possible to decode Intel PT in per-cpu mode.
728 sched_switch tracepoint
730 The sched_switch tracepoint is used to provide side-band data for Intel
731 PT decoding in kernels where the PERF_RECORD_SWITCH metadata event
732 isn’t available.
733 The sched_switch events are automatically added. e.g. the second event
735 shown below:
736 $ perf record -vv -e intel_pt//u uname
738 ------------------------------------------------------------
739 perf_event_attr:
740 type 6
741 size 112
742 config 0x400
743 { sample_period, sample_freq } 1
744 sample_type IP|TID|TIME|CPU|IDENTIFIER
745 read_format ID
746 disabled 1
747 inherit 1
748 exclude_kernel 1
749 exclude_hv 1
750 enable_on_exec 1
751 sample_id_all 1
752 ------------------------------------------------------------
753 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
754 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
755 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
756 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
757 ------------------------------------------------------------
758 perf_event_attr:
759 type 2
760 size 112
761 config 0x108
762 { sample_period, sample_freq } 1
763 sample_type IP|TID|TIME|CPU|PERIOD|RAW|IDENTIFIER
764 read_format ID
765 inherit 1
766 sample_id_all 1
767 exclude_guest 1
768 ------------------------------------------------------------
769 sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8
770 sys_perf_event_open: pid -1 cpu 1 group_fd -1 flags 0x8
771 sys_perf_event_open: pid -1 cpu 2 group_fd -1 flags 0x8
772 sys_perf_event_open: pid -1 cpu 3 group_fd -1 flags 0x8
773 ------------------------------------------------------------
774 perf_event_attr:
775 type 1
776 size 112
777 config 0x9
778 { sample_period, sample_freq } 1
779 sample_type IP|TID|TIME|IDENTIFIER
780 read_format ID
781 disabled 1
782 inherit 1
783 exclude_kernel 1
784 exclude_hv 1
785 mmap 1
786 comm 1
787 enable_on_exec 1
788 task 1
789 sample_id_all 1
790 mmap2 1
791 comm_exec 1
792 ------------------------------------------------------------
793 sys_perf_event_open: pid 31104 cpu 0 group_fd -1 flags 0x8
794 sys_perf_event_open: pid 31104 cpu 1 group_fd -1 flags 0x8
795 sys_perf_event_open: pid 31104 cpu 2 group_fd -1 flags 0x8
796 sys_perf_event_open: pid 31104 cpu 3 group_fd -1 flags 0x8
797 mmap size 528384B
798 AUX area mmap length 4194304
799 perf event ring buffer mmapped per cpu
800 Synthesizing auxtrace information
801 Linux
802 [ perf record: Woken up 1 times to write data ]
803 [ perf record: Captured and wrote 0.042 MB perf.data ]
804 Note, the sched_switch event is only added if the user is permitted to
806 use it and only in per-cpu mode.
807 Note also, the sched_switch event is only added if TSC packets are
809 requested. That is because, in the absence of timing information, the
810 sched_switch events cannot be matched against the Intel PT trace.
811
813 By default, perf script will decode trace data found in the perf.data
814 file. This can be further controlled by new option --itrace.
815 New --itrace option
817 Having no option is the same as
818 --itrace
820 which, in turn, is the same as
822 --itrace=cepwx
824 The letters are:
826 i synthesize "instructions" events
828 b synthesize "branches" events
829 x synthesize "transactions" events
830 w synthesize "ptwrite" events
831 p synthesize "power" events (incl. PSB events)
832 c synthesize branches events (calls only)
833 r synthesize branches events (returns only)
834 e synthesize tracing error events
835 d create a debug log
836 g synthesize a call chain (use with i or x)
837 G synthesize a call chain on existing event records
838 l synthesize last branch entries (use with i or x)
839 L synthesize last branch entries on existing event records
840 s skip initial number of events
841 q quicker (less detailed) decoding
842 "Instructions" events look like they were recorded by "perf record -e
844 instructions".
845 "Branches" events look like they were recorded by "perf record -e
847 branches". "c" and "r" can be combined to get calls and returns.
848 "Transactions" events correspond to the start or end of transactions.
850 The flags field can be used in perf script to determine whether the
851 event is a tranasaction start, commit or abort.
852 Note that "instructions", "branches" and "transactions" events depend
854 on code flow packets which can be disabled by using the config term
855 "branch=0". Refer to the config terms section above.
856 "ptwrite" events record the payload of the ptwrite instruction and
858 whether "fup_on_ptw" was used. "ptwrite" events depend on PTWRITE
859 packets which are recorded only if the "ptw" config term was used.
860 Refer to the config terms section above. perf script "synth" field
861 displays "ptwrite" information like this: "ip: 0 payload:
862 0x123456789abcdef0" where "ip" is 1 if "fup_on_ptw" was used.
863 "Power" events correspond to power event packets and CBR (core-to-bus
865 ratio) packets. While CBR packets are always recorded when tracing is
866 enabled, power event packets are recorded only if the "pwr_evt" config
867 term was used. Refer to the config terms section above. The power
868 events record information about C-state changes, whereas CBR is
869 indicative of CPU frequency. perf script "event,synth" fields display
870 information like this: cbr: cbr: 22 freq: 2189 MHz (200%) mwait: hints:
871 0x60 extensions: 0x1 pwre: hw: 0 cstate: 2 sub-cstate: 0 exstop: ip: 1
872 pwrx: deepest cstate: 2 last cstate: 2 wake reason: 0x4 Where: "cbr"
873 includes the frequency and the percentage of maximum non-turbo "mwait"
874 shows mwait hints and extensions "pwre" shows C-state transitions (to a
875 C-state deeper than C0) and whether initiated by hardware "exstop"
876 indicates execution stopped and whether the IP was recorded exactly,
877 "pwrx" indicates return to C0 For more details refer to the Intel 64
878 and IA-32 Architectures Software Developer Manuals.
879 PSB events show when a PSB+ occurred and also the byte-offset in the
881 trace. Emitting a PSB+ can cause a CPU a slight delay. When doing
882 timing analysis of code with Intel PT, it is useful to know if a timing
883 bubble was caused by Intel PT or not.
884 Error events show where the decoder lost the trace. Error events are
886 quite important. Users must know if what they are seeing is a complete
887 picture or not. The "e" option may be followed by flags which affect
888 what errors will or will not be reported. Each flag must be preceded by
889 either + or -. The flags supported by Intel PT are: -o Suppress
890 overflow errors -l Suppress trace data lost errors For example, for
891 errors but not overflow or data lost errors:
892 --itrace=e-o-l
894 The "d" option will cause the creation of a file "intel_pt.log"
896 containing all decoded packets and instructions. Note that this option
897 slows down the decoder and that the resulting file may be very large.
898 The "d" option may be followed by flags which affect what debug
899 messages will or will not be logged. Each flag must be preceded by
900 either + or -. The flags support by Intel PT are: -a Suppress logging
901 of perf events +a Log all perf events By default, logged perf events
902 are filtered by any specified time ranges, but flag +a overrides that.
903 In addition, the period of the "instructions" event can be specified.
905 e.g.
906 --itrace=i10us
908 sets the period to 10us i.e. one instruction sample is synthesized for
910 each 10 microseconds of trace. Alternatives to "us" are "ms"
911 (milliseconds), "ns" (nanoseconds), "t" (TSC ticks) or "i"
912 (instructions).
913 "ms", "us" and "ns" are converted to TSC ticks.
915 The timing information included with Intel PT does not give the time of
917 every instruction. Consequently, for the purpose of sampling, the
918 decoder estimates the time since the last timing packet based on 1 tick
919 per instruction. The time on the sample is not adjusted and reflects
920 the last known value of TSC.
921 For Intel PT, the default period is 100us.
923 Setting it to a zero period means "as often as possible".
925 In the case of Intel PT that is the same as a period of 1 and a unit of
927 instructions (i.e. --itrace=i1i).
928 Also the call chain size (default 16, max. 1024) for instructions or
930 transactions events can be specified. e.g.
931 --itrace=ig32
933 --itrace=xg32
934 Also the number of last branch entries (default 64, max. 1024) for
936 instructions or transactions events can be specified. e.g.
937 --itrace=il10
939 --itrace=xl10
940 Note that last branch entries are cleared for each sample, so there is
942 no overlap from one sample to the next.
943 The G and L options are designed in particular for sample mode, and
945 work much like g and l but add call chain and branch stack to the other
946 selected events instead of synthesized events. For example, to record
947 branch-misses events for ls and then add a call chain derived from the
948 Intel PT trace:
949 perf record --aux-sample -e '{intel_pt//u,branch-misses:u}' -- ls
951 perf report --itrace=Ge
952 Although in fact G is a default for perf report, so that is the same as
954 just:
955 perf report
957 One caveat with the G and L options is that they work poorly with
959 "Large PEBS". Large PEBS means PEBS records will be accumulated by
960 hardware and the written into the event buffer in one go. That reduces
961 interrupts, but can give very late timestamps. Because the Intel PT
962 trace is synchronized by timestamps, the PEBS events do not match the
963 trace. Currently, Large PEBS is used only in certain circumstances: -
964 hardware supports it - PEBS is used - event period is specified,
965 instead of frequency - the sample type is limited to the following
966 flags: PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_ADDR |
967 PERF_SAMPLE_ID | PERF_SAMPLE_CPU | PERF_SAMPLE_STREAM_ID |
968 PERF_SAMPLE_DATA_SRC | PERF_SAMPLE_IDENTIFIER | PERF_SAMPLE_TRANSACTION
969 | PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_REGS_INTR | PERF_SAMPLE_REGS_USER
970 | PERF_SAMPLE_PERIOD (and sometimes) | PERF_SAMPLE_TIME Because Intel
971 PT sample mode uses a different sample type to the list above, Large
972 PEBS is not used with Intel PT sample mode. To avoid Large PEBS in
973 other cases, avoid specifying the event period i.e. avoid the perf
974 record -c option, --count option, or period config term.
975 To disable trace decoding entirely, use the option --no-itrace.
977 It is also possible to skip events generated (instructions, branches,
979 transactions) at the beginning. This is useful to ignore initialization
980 code.
981 --itrace=i0nss1000000
983 skips the first million instructions.
985 The q option changes the way the trace is decoded. The decoding is much
987 faster but much less detailed. Specifically, with the q option, the
988 decoder does not decode TNT packets, and does not walk object code, but
989 gets the ip from FUP and TIP packets. The q option can be used with the
990 b and i options but the period is not used. The q option decodes more
991 quickly, but is useful only if the control flow of interest is
992 represented or indicated by FUP, TIP, TIP.PGE, or TIP.PGD packets
993 (refer below). However the q option could be used to find time ranges
994 that could then be decoded fully using the --time option.
995 What will not be decoded with the (single) q option:
997 • direct calls and jmps
999 • conditional branches
1001 • non-branch instructions
1003 What will be decoded with the (single) q option:
1005 • asynchronous branches such as interrupts
1007 • indirect branches
1009 • function return target address if the noretcomp config term (refer
1011 config terms section) was used
1012 • start of (control-flow) tracing
1014 • end of (control-flow) tracing, if it is not out of context
1016 • power events, ptwrite, transaction start and abort
1018 • instruction pointer associated with PSB packets
1020 Note the q option does not specify what events will be synthesized e.g.
1022 the p option must be used also to show power events.
1023 Repeating the q option (double-q i.e. qq) results in even faster
1025 decoding and even less detail. The decoder decodes only extended PSB
1026 (PSB+) packets, getting the instruction pointer if there is a FUP
1027 packet within PSB+ (i.e. between PSB and PSBEND). Note PSB packets
1028 occur regularly in the trace based on the psb_period config term (refer
1029 config terms section). There will be a FUP packet if the PSB+ occurs
1030 while control flow is being traced.
1031 What will not be decoded with the qq option:
1033 • everything except instruction pointer associated with PSB packets
1035 What will be decoded with the qq option:
1037 • instruction pointer associated with PSB packets
1039 dump option
1041 perf script has an option (-D) to "dump" the events i.e. display the
1042 binary data.
1043 When -D is used, Intel PT packets are displayed. The packet decoder
1045 does not pay attention to PSB packets, but just decodes the bytes - so
1046 the packets seen by the actual decoder may not be identical in places
1047 where the data is corrupt. One example of that would be when the
1048 buffer-switching interrupt has been too slow, and the buffer has been
1049 filled completely. In that case, the last packet in the buffer might be
1050 truncated and immediately followed by a PSB as the trace continues in
1051 the next buffer.
1052 To disable the display of Intel PT packets, combine the -D option with
1054 --no-itrace.
1055
1057 By default, perf report will decode trace data found in the perf.data
1058 file. This can be further controlled by new option --itrace exactly the
1059 same as perf script, with the exception that the default is
1060 --itrace=igxe.
1061
1063 perf inject also accepts the --itrace option in which case tracing data
1064 is removed and replaced with the synthesized events. e.g.
1065 perf inject --itrace -i perf.data -o perf.data.new
1067 Below is an example of using Intel PT with autofdo. It requires autofdo
1069 (https://github.com/google/autofdo) and gcc version 5. The bubble sort
1070 example is from the AutoFDO tutorial
1071 (https://gcc.gnu.org/wiki/AutoFDO/Tutorial) amended to take the number
1072 of elements as a parameter.
1073 $ gcc-5 -O3 sort.c -o sort_optimized
1075 $ ./sort_optimized 30000
1076 Bubble sorting array of 30000 elements
1077 2254 ms
1078 $ cat ~/.perfconfig
1080 [intel-pt]
1081 mispred-all = on
1082 $ perf record -e intel_pt//u ./sort 3000
1084 Bubble sorting array of 3000 elements
1085 58 ms
1086 [ perf record: Woken up 2 times to write data ]
1087 [ perf record: Captured and wrote 3.939 MB perf.data ]
1088 $ perf inject -i perf.data -o inj --itrace=i100usle --strip
1089 $ ./create_gcov --binary=./sort --profile=inj --gcov=sort.gcov -gcov_version=1
1090 $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
1091 $ ./sort_autofdo 30000
1092 Bubble sorting array of 30000 elements
1093 2155 ms
1094 Note there is currently no advantage to using Intel PT instead of LBR,
1096 but that may change in the future if greater use is made of the data.
1097
1099 Some hardware has the feature to redirect PEBS records to the Intel PT
1100 trace. Recording is selected by using the aux-output config term e.g.
1101 perf record -c 10000 -e '{intel_pt/branch=0/,cycles/aux-output/ppp}' uname
1103 Note that currently, software only supports redirecting at most one
1105 PEBS event.
1106 To display PEBS events from the Intel PT trace, use the itrace o option
1108 e.g.
1109 perf script --itrace=oe
1111
1113 For --xed the xed tool is needed. Here is how to install it:
1114 $ git clone https://github.com/intelxed/mbuild.git mbuild
1116 $ git clone https://github.com/intelxed/xed
1117 $ cd xed
1118 $ ./mfile.py --share
1119 $ ./mfile.py examples
1120 $ sudo ./mfile.py --prefix=/usr/local install
1121 $ sudo ldconfig
1122 $ sudo cp obj/examples/xed /usr/local/bin
1123 Basic xed testing:
1125 $ xed | head -3
1127 ERROR: required argument(s) were missing
1128 Copyright (C) 2017, Intel Corporation. All rights reserved.
1129 XED version: [v10.0-328-g7d62c8c49b7b]
1130 $
1131
1133 Currently, only kernel tracing is supported and only with "timeless"
1134 decoding i.e. no TSC timestamps
1135 Other limitations and caveats
1137 VMX controls may suppress packets needed for decoding resulting in decoding errors
1139 VMX controls may block the perf NMI to the host potentially resulting in lost trace data
1140 Guest kernel self-modifying code (e.g. jump labels or JIT-compiled eBPF) will result in decoding errors
1141 Guest thread information is unknown
1142 Guest VCPU is unknown but may be able to be inferred from the host thread
1143 Callchains are not supported
1144 Example
1146 Start VM
1148 $ sudo virsh start kubuntu20.04
1150 Domain kubuntu20.04 started
1151 Mount the guest file system. Note sshfs needs -o direct_io to enable
1153 reading of proc files. root access is needed to read /proc/kcore.
1154 $ mkdir vm0
1156 $ sshfs -o direct_io root@vm0:/ vm0
1157 Copy the guest /proc/kallsyms, /proc/modules and /proc/kcore
1159 $ perf buildid-cache -v --kcore vm0/proc/kcore
1161 kcore added to build-id cache directory /home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306
1162 $ KALLSYMS=/home/user/.debug/[kernel.kcore]/9600f316a53a0f54278885e8d9710538ec5f6a08/2021021807494306/kallsyms
1163 Find the VM process
1165 $ ps -eLl | grep 'KVM\|PID'
1167 F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
1168 3 S 64055 1430 1 1440 1 80 0 - 1921718 - ? 00:02:47 CPU 0/KVM
1169 3 S 64055 1430 1 1441 1 80 0 - 1921718 - ? 00:02:41 CPU 1/KVM
1170 3 S 64055 1430 1 1442 1 80 0 - 1921718 - ? 00:02:38 CPU 2/KVM
1171 3 S 64055 1430 1 1443 2 80 0 - 1921718 - ? 00:03:18 CPU 3/KVM
1172 Start an open-ended perf record, tracing the VM process, do something
1174 on the VM, and then ctrl-C to stop. TSC is not supported and tsc=0 must
1175 be specified. That means mtc is useless, so add mtc=0. However, IPC can
1176 still be determined, hence cyc=1 can be added. Only kernel decoding is
1177 supported, so k must be specified. Intel PT traces both the host and
1178 the guest so --guest and --host need to be specified. Without
1179 timestamps, --per-thread must be specified to distinguish threads.
1180 $ sudo perf kvm --guest --host --guestkallsyms $KALLSYMS record --kcore -e intel_pt/tsc=0,mtc=0,cyc=1/k -p 1430 --per-thread
1182 ^C
1183 [ perf record: Woken up 1 times to write data ]
1184 [ perf record: Captured and wrote 5.829 MB ]
1185 perf script can be used to provide an instruction trace
1187 $ perf script --guestkallsyms $KALLSYMS --insn-trace --xed -F+ipc | grep -C10 vmresume | head -21
1189 CPU 0/KVM 1440 ffffffff82133cdd __vmx_vcpu_run+0x3d ([kernel.kallsyms]) movq 0x48(%rax), %r9
1190 CPU 0/KVM 1440 ffffffff82133ce1 __vmx_vcpu_run+0x41 ([kernel.kallsyms]) movq 0x50(%rax), %r10
1191 CPU 0/KVM 1440 ffffffff82133ce5 __vmx_vcpu_run+0x45 ([kernel.kallsyms]) movq 0x58(%rax), %r11
1192 CPU 0/KVM 1440 ffffffff82133ce9 __vmx_vcpu_run+0x49 ([kernel.kallsyms]) movq 0x60(%rax), %r12
1193 CPU 0/KVM 1440 ffffffff82133ced __vmx_vcpu_run+0x4d ([kernel.kallsyms]) movq 0x68(%rax), %r13
1194 CPU 0/KVM 1440 ffffffff82133cf1 __vmx_vcpu_run+0x51 ([kernel.kallsyms]) movq 0x70(%rax), %r14
1195 CPU 0/KVM 1440 ffffffff82133cf5 __vmx_vcpu_run+0x55 ([kernel.kallsyms]) movq 0x78(%rax), %r15
1196 CPU 0/KVM 1440 ffffffff82133cf9 __vmx_vcpu_run+0x59 ([kernel.kallsyms]) movq (%rax), %rax
1197 CPU 0/KVM 1440 ffffffff82133cfc __vmx_vcpu_run+0x5c ([kernel.kallsyms]) callq 0xffffffff82133c40
1198 CPU 0/KVM 1440 ffffffff82133c40 vmx_vmenter+0x0 ([kernel.kallsyms]) jz 0xffffffff82133c46
1199 CPU 0/KVM 1440 ffffffff82133c42 vmx_vmenter+0x2 ([kernel.kallsyms]) vmresume IPC: 0.11 (50/445)
1200 :1440 1440 ffffffffbb678b06 native_write_msr+0x6 ([guest.kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
1201 :1440 1440 ffffffffbb678b0b native_write_msr+0xb ([guest.kernel.kallsyms]) retq IPC: 0.04 (2/41)
1202 :1440 1440 ffffffffbb666646 lapic_next_deadline+0x26 ([guest.kernel.kallsyms]) data16 nop
1203 :1440 1440 ffffffffbb666648 lapic_next_deadline+0x28 ([guest.kernel.kallsyms]) xor %eax, %eax
1204 :1440 1440 ffffffffbb66664a lapic_next_deadline+0x2a ([guest.kernel.kallsyms]) popq %rbp
1205 :1440 1440 ffffffffbb66664b lapic_next_deadline+0x2b ([guest.kernel.kallsyms]) retq IPC: 0.16 (4/25)
1206 :1440 1440 ffffffffbb74607f clockevents_program_event+0x8f ([guest.kernel.kallsyms]) test %eax, %eax
1207 :1440 1440 ffffffffbb746081 clockevents_program_event+0x91 ([guest.kernel.kallsyms]) jz 0xffffffffbb74603c IPC: 0.06 (2/30)
1208 :1440 1440 ffffffffbb74603c clockevents_program_event+0x4c ([guest.kernel.kallsyms]) popq %rbx
1209 :1440 1440 ffffffffbb74603d clockevents_program_event+0x4d ([guest.kernel.kallsyms]) popq %r12
1210
1212 perf-record(1), perf-script(1), perf-report(1), perf-inject(1)
1213perf 06/03/2021 PERF-INTEL-PT(1)