1PQOS(8) System Manager's Manual PQOS(8)
2
6 pqos, pqos-msr, pqos-os - Intel(R) Resource Director Technology/AMD
7 PQoS monitoring and control tool
8
10 pqos [OPTIONS]...
11
13 Intel(R) Resource Director Technology/AMD PQoS is designed to monitor
14 and manage cpu resources and improve performance of applications and
15 virtual machines.
16 Intel(R) Resource Director Technology/AMD PQoS includes monitoring and
18 control technologies. Monitoring technologies include CMT (Cache Moni‐
19 toring Technology), which monitors occupancy of last level cache, and
20 MBM (Memory Bandwidth Monitoring). Control technologies include CAT
21 (Cache Allocation Technology), CDP (Code Data Prioritization) and MBA
22 (Memory Bandwidth Allocation).
23 pqos supports CMT and MBM on a per core or hardware thread basis. MBM
25 supports two types of events reporting local and remote memory band‐
26 width.
27 pqos-msr and pqos-os are simple pqos wrapper scripts that automatically
28 select the MSR or OS/Kernel library interface to program the technolo‐
29 gies.
30 Please see the -I option below for more information.
31 For hardware information please refer to the README located on:
33 https://github.com/intel/intel-cmt-cat/blob/master/README
34
36 pqos options are as follow:
37 -h, --help
39 show help
40 -v, --verbose
42 verbose mode
43 -V, --super-verbose
45 super-verbose mode
46 -l FILE, --log-file=FILE
48 log messages into selected log FILE
49 -s, --show
51 show the current allocation and monitoring configuration
52 -d, --display
54 display supported Intel(R) Resource Director Technology/AMD PQoS
55 capabilities
56 -D, --display-verbose
58 display supported Intel(R) Resource Director Technology/AMD PQoS
59 capabilities in verbose mode
60 -f FILE, --config-file=FILE
62 load commands from selected configuration FILE
63 -e CLASSDEF, --alloc-class=CLASSDEF
65 define the allocation classes on all CPU sockets. CLASSDEF for‐
66 mat is "TYPE:ID=DEFINITION;...".
67 define classes for selected CPU resources. CLASSDEF format is
68 "TYPE[@RESOURCE_ID]:ID=DEFINITION;...".
69 For CAT, TYPE is "llc" for the last level cache (aka l3) and
70 "l2" for level 2 cache, ID is a CLOS number and DEFINITION is a
71 bitmask.
72 For MBA, TYPE is "mba", ID is a CLOS number and DEFINITION is a
73 value between 1 and 100 representing the percentage of available
74 memory bandwidth.
75 For MBA CTRL, TYPE is "mba_max", ID is a CLOS number and DEFINI‐
76 TION is a value representing the requested memory bandwidth
77 specified in MBps.
78 RESOURCE_ID is a unique number that can represent a socket or
79 l2/l3 cache identifier. The RESOURCE_ID for each logical CPU can
80 be found using "pqos -s"
81 Note: When L2/L3 CDP is on, ID can be postfixed with 'D' for
82 data or 'C' for code.
83 Note: L2/L3 CDP is available on selected CPUs only.
84 Note: MBA CTRL is supported only by the OS interface and re‐
85 quires Linux and kernel version 4.18 or newer.
86 Some examples:
87 "-e llc:0=0xffff;llc:1=0x00ff"
88 "-e llc@0-1:2=0xff00;l2:2=0x3f;l2@2:1=0xf"
89 "-e llc:0d=0xfff;llc:0c=0xfff00"
90 "-e l2:0d=0xf;l2:0c=0xc"
91 "-e mba:1=30;mba@1:3=80"
92 "-e mba_max:1=6000;mba_max@1:3=10000"
93 Note:
94 "-e l2:2=0x3f" means that COS2 for all L2 cache clusters
95 is changed to 0x3f.
96 "-e l2@2:1=0xf" means that COS1 for L2 cache cluster 2 is
97 changed to 0xf.
98 "-e mba:1=30" means that COS1, on all sockets, can uti‐
99 lize up to 30% of available memory bandwidth.
100 "-e mba_max:1=6000" means that COS1, on all sockets, can
101 utilize up to 6000 MBps of memory bandwidth.
102 -a CLASS2ID, --alloc-assoc=CLASS2ID
104 associate allocation classes with cores or processes. CLASS2ID
105 format is "TYPE:ID=CORE_LIST;..." or "TYPE:ID=TASK_LIST;...".
106 For COS association required for CAT or MBA, TYPE is "cos",
107 "llc", "core" (for COS-core association) or "pid" (for COS-
108 process association) and ID is a class number. CORE_LIST is
109 comma or dash separated list of cores. TASK_LIST is comma or
110 dash separated list of process/task ID's.
111 For example:
112 "-a cos:0=0,2,4,6-10;cos:1=1;" associates cores 0, 2, 4,
113 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class 1.
114 "-a llc:0=0,2,4,6-10;llc:1=1;" associates cores 0, 2, 4,
115 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class 1.
116 "-a core:0=0,2,4,6-10;core:1=1;" associates cores 0, 2,
117 4, 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class
118 1.
119 "-I -a pid:0=3543,7643,4556;pid:1=7644;" associates
120 process ID 3543, 7643, 4556 with CAT class 0 and process
121 ID 7644 with class 1.
122 Notes:
123 "llc" TYPE label is considered deprecated, please use
124 "cos" or "core" instead.
125 The -I option must be used for PID association.
126 -R [CONFIG[,CONFIG]], --alloc-reset[=CONFIG[,CONFIG]]
128 reset allocation setting (L3 CAT, L2 CAT, MBA) and reconfigure
129 allocation. CONFIG is one of the following options:
130 l3cdp-on sets L3 CDP on
131 l3cdp-off sets L3 CDP off
132 l3cdp-any keeps current L3 CDP setting (default)
133 l3iordt-on sets L3 I/O RDT on
134 l3iordt-off sets L3 I/O RDT off
135 l3iordt-any keeps current L3 I/O RDT setting (de‐
136 fault)
137 l2cdp-on sets L2 CDP on
138 l2cdp-off sets L2 CDP off
139 l2cdp-any keeps current L2 CDP setting (default)
140 mbaCtrl-on sets MBA CTRL on
141 mbaCtrl-off sets MBA CTRL off
142 mbaCtrl-any keeps current MBA CTRL setting (default)
143 mba40-on enables MBA 4.0 extensions for all sockets
144 mba40-off disables MBA 4.0 extensions for all sockets
145 mba40-any keeps current MBA 4.0 setting (default)
146 -m EVTCORES, --mon-core=EVTCORES
148 select the cores and events for monitoring, EVT‐
149 CORES format is "EVENT:CORE_LIST". Valid EVENT
150 settings are:
151 - "llc" for CMT (LLC occupancy)
152 - "mbr" for MBR (remote memory bandwidth)
153 - "mbl" for MBL (local memory bandwidth)
154 - "mbt" for MBT (total memory bandwidth)
155 - "all" or "" for all detected event types (ex‐
156 cept MBT)
157 CORE_LIST is comma or dash separated list of
158 cores.
159 Example "-m all:0,2,4-10;llc:1,3;mbr:11-12".
160 Core statistics can be grouped by enclosing the
161 core list in square brackets.
162 Example "-m llc:[0-3];all:[4,5,6];mbr:[0-3],7,8".
163 -p [EVTPIDS], --mon-pid[=EVTPIDS]
165 select top 10 most active (CPU utilizing) process
166 ids to monitor or select the process ids and
167 events to monitor, EVTPIDS format is
168 "EVENT:PID_LIST".
169 See -m option for valid EVENT settings. PID_LIST
170 is comma separated list of process ids.
171 Examples:
172 "-p llc:22,25673"
173 "-p all:892,4588-4592"
174 Process's IDs can be grouped by enclosing them in
176 square brackets.
177 Examples:
178 "-p llc:[22,25673]"
179 "-p all:892,[4588-4592]"
180 Note:
182 Requires Linux and kernel versions 4.10 and
183 newer.
184 The -I option must be used for PID monitor‐
185 ing.
186 It is not possible to track both processes
187 and cores at the same time.
188 --mon-uncore[=EVTUNCORE]
191 select sockets and uncore events for monitoring,
192 EVTUNCORE format is 'EVENT:SOCKET_LIST. Socket's
193 IDs can be grouped by enclosing them in square
194 brackets.
195 Examples:
196 "--mon-uncore"
197 "--mon-uncore=all:0"
198 Note: It is not possible to track both sockets and
200 cores at the same time.
201 -T, --mon-top
203 enable top like monitoring output sorted by high‐
204 est LLC occupancy
205 --mon-dev=EVTDEVICES"
207 select I/O RDT devices and events to monitor, EVT‐
208 DEVICES format is
209 See -m option for valid EVENT settings. DE‐
210 VICE_LIST is comma separated list of I/O RDT de‐
211 vices.
212 Examples:
213 "--mon-dev all:0000:0010:04.0@1"
214 "--mon-dev llc:0000:0010:05.0"
215 --mon-channel=EVTCHANNELS
217 select I/O RDT channels and events to monitor,
218 EVTCHANNELS format is
219 See -m option for valid EVENT settings. CHAN‐
220 NEL_LIST is comma separated list of I/O RDT chan‐
221 nels.
222 Channels can be grouped by enclosing them in
223 square brackets.
224 -o FILE, --mon-file FILE
226 select output FILE to store monitored data in, the
227 default is 'stdout'
228 -u TYPE, --mon-file-type=TYPE
230 select the output format TYPE for monitored data.
231 Supported TYPE settings are: "text" (default),
232 "xml" and "csv".
233 -i INTERVAL, --mon-interval=INTERVAL
235 define monitoring sampling INTERVAL in 100ms
236 units, 1=100ms, default 10=10x100ms=1s
237 -t SECONDS, --mon-time=SECONDS
239 define monitoring time in seconds, use 'inf' or
240 'infinite' for infinite monitoring. Use CTRL+C to
241 stop monitoring at any time.
242 -r, --mon-reset[=CONFIG[,CONFIG]]
244 reset monitoring and use all RMID's in the system
245 and reconfigure allocation. CONFIG is one of the
246 following options:
247 l3iordt-on sets L3 I/O RDT on
248 l3iordt-off sets L3 I/O RDT off
249 l3iordt-any keeps current L3 I/O RDT
250 setting (default)
251 --disable-mon-ipc
253 Disable IPC monitoring
254 --disable-mon-llc_miss
256 Disable LLC misses monitoring
257 -H, --profile-list
259 list supported allocation profiles
260 -c PROFILE, --profile-set=PROFILE
262 select a PROFILE from predefined allocation
263 classes, use -H to list available profiles
264 -I, --iface-os
266 set the library interface to use the kernel imple‐
267 mentation. If not set the default implementation
268 is to program the MSR's directly.
269 --iface=INTERFACE
271 set the library interface to automatically de‐
272 tected one ('auto'), MSR ('msr') or kernel inter‐
273 face ('os'). INTERFACE can be set to either
274 'auto' (default), 'msr' or 'os'. If automatic de‐
275 tection is selected ('auto'), it:
276 1) Takes RDT_IFACE environment variable into ac‐
277 count if this variable is set
278 2) Selects OS interface if the kernel interface is
279 supported
280 3) Selects MSR interface otherwise
281
283 CMT, MBM and CAT are configured using Model Specific Registers (MSRs).
284 The pqos software executes in user space, and access to the MSRs is ob‐
285 tained through a standard Linux* interface. The msr file interface is
286 protected and requires root privileges. The msr driver might not be
287 auto-loaded and on some modular kernels the driver may need to be
288 loaded manually:
289 For Linux:
291 sudo modprobe msr
292 For FreeBSD:
294 sudo kldload cpuctl
295 Interface enforcement:
297 If you require system wide interface enforcement you can do so by set‐
298 ting the "RDT_IFACE" environment variable.
299
301 msr(4)
302
304 pqos was written by Tomasz Kantecki <tomasz.kantecki@intel.com>, Marcel
305 Cornu <marcel.d.cornu@intel.com>, Aaron Hetherington <aaron.hethering‐
306 ton@intel.com>, Michal Aleksinski <michalx.aleksinski@intel.com>, Woj‐
307 ciech Andralojc <wojciechx.andralojc@intel.com>, Adrian Boczkowski
308 <adrianx.boczkowski@intel.com>
309 This is free software; see the source for copying conditions. There is
311 NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
312 PURPOSE.
313 Apr 19, 2022 PQOS(8)