1HWLOC(7) hwloc HWLOC(7)
2
6 hwloc - General information about hwloc ("hardware locality").
7
9 hwloc provides command line tools and a C API to obtain the hierarchi‐
10 cal map of key computing elements, such as: NUMA memory nodes, shared
11 caches, processor packages, processor cores, and processor "threads".
12 hwloc also gathers various attributes such as cache and memory informa‐
13 tion, and is portable across a variety of different operating systems
14 and platforms.
15 Definitions
17 hwloc has some specific definitions for terms that are used in this man
18 page and other hwloc documentation.
19 hwloc CPU set:
21 A set of processors included in an hwloc object, expressed as a
22 bitmask indexed by the physical numbers of the CPUs (as announced
23 by the OS). The hwloc definition of "CPU set" does not carry any
24 of the same connotations as Linux's "CPU set" (e.g., process
25 affinity, cgroup, etc.).
26 hwloc node set:
28 A set of NUMA memory nodes near an hwloc object, expressed as a
29 bitmask indexed by the physical numbers of the NUMA nodes (as an‐
30 nounced by the OS).
31 Linux CPU set:
33 See http://www.mjmwired.net/kernel/Documentation/cpusets.txt for a
34 discussion of Linux CPU sets. A super-short-ignoring-many-details
35 description (taken from that page) is:
36 "Cpusets provide a mechanism for assigning a set of CPUs and Mem‐
38 ory Nodes to a set of tasks."
39 Linux Cgroup:
41 See http://www.mjmwired.net/kernel/Documentation/cgroups.txt for a
42 discussion of Linux control groups. A super-short-ignoring-many-
43 details description (taken from that page) is:
44 "Control Groups provide a mechanism for aggregating/partitioning
46 sets of tasks, and all their future children, into hierarchical
47 groups with specialized behaviour."
48 To be clear, hwloc supports all of the above concepts. It is simply
50 worth noting that they are different things.
51 Location Specification
53 Locations refer to specific regions within a topology. Before reading
54 the rest of this man page, it may be useful to read lstopo(1) and/or
55 run lstopo on your machine to see the reported topology tree. Seeing
56 and understanding a topology tree will definitely help in understanding
57 the concepts that are discussed below.
58 Locations can be specified in multiple ways:
60 Tuples: Tuples of hwloc "objects" and associated indexes can be spec‐
62 ified in the form object:index. hwloc objects represent
63 types of mapped items (e.g., packages, cores, etc.) in a
64 topology tree; indexes are non-negative integers that specify
65 a unique physical object in a topology tree. Both concepts
66 are described in detail, below.
67 Indexes may also be specified as ranges. x-y enumerates from
69 index x to y. x:y enumerates y objects starting from index x
70 (wrapping around the end of the index range if needed). x-
71 enumerates all objects starting from index x. all, odd, and
72 even are also supported for listing all objects, or only
73 those with odd or even indexes.
74 Chaining multiple tuples together in the more general form
76 object1:index[.object2:index2[...]] is permissable. While
77 the first tuple's object may appear anywhere in the topology,
78 the Nth tuple's object must have a shallower topology depth
79 than the (N+1)th tuple's object. Put simply: as you move
80 right in a tuple chain, objects must go deeper in the topol‐
81 ogy tree. When using logical indexes (which is the default),
82 indexes specified in chained tuples are relative to the scope
83 of the parent object. For example, "package:0.core:1" refers
84 to the second core in the first package.
85 When using OS/physical indexes, the first object matching the
87 given index is used.
88 PCI and OS devices may also be designed using their identi‐
90 fier. For example, "pci=02:03.1" is the PCI device with bus
91 ID "02:03.1". "os=eth0" is the network interface whose soft‐
92 ware name is "eth0". PCI devices may also be filtered based
93 on their vendor and/or device IDs, for instance
94 "pci[15b3:]:2" for the third Mellanox PCI device (vendor ID
95 0x15b3). OS devices may also be filtered based on their sub‐
96 type, for instance "os[gpu]:all" for all GPU OS devices.
97 Hex: For tools that manipulate object as sets (e.g. hwloc-calc and
99 hwloc-bind), locations can also be specified as hexidecimal
100 bitmasks prefixed with "0x". Commas must be used to separate
101 the hex digits into blocks of 8, such as
102 "0xffc0140,0x00020110". Leading zeros in each block do not
103 need to be specified. For example, "0xffc0140,0x20110" is
104 equivalent to the prior example, and "0x0000000f" is exactly
105 equivalent to "0xf". Intermediate blocks of 8 digits that
106 are all zeoro can be left empty; "0xff0,,0x13" is equivalent
107 to "0xff0,0x00000000,0x13". If the location is prefixed with
108 the special string "0xf...f", then all unspecified bits are
109 set (as if the set were infinite). For example, "0xf...f,0x1"
110 sets both the first bit and all bits starting with the 33rd.
111 The string "0xf...f" -- with no other specified values --
112 sets all bits.
113 "all" and "root" are special locations consisting in the root object in
115 tree. It contains the entire current topology.
116 Some tools directly operate on these objects (e.g. hwloc-info and
118 hwloc-annotate). They do not support hexadecimal locations because
119 each location may correspond to multiple objects. For instance, there
120 can be exactly one L3 cache per package and NUMA node, which means it's
121 the same location. If multiple locations are given on the command-
122 line, these tools will operation on each location individually and con‐
123 secutively.
124 Some other tools internally manipulate objects as sets (e.g. hwloc-calc
126 and hwloc-bind). They translate each input location into a hexidecimal
127 location. When I/O or Misc objects are used, they are translated into
128 the set of processors (or NUMA nodes) that are close to the given ob‐
129 ject (because I/O or Misc objects do not contain processors or NUMA
130 nodes).
131 If multiple locations are specified on the command-line (delimited by
133 whitespace), they are combined (the overall location is wider). If
134 prefixed with "~", the given location will be cleared instead of added
135 to the current list of locations. If prefixed with "x", the given lo‐
136 cation will be and'ed instead of added to the current list. If pre‐
137 fixed with "^", the given location will be xor'ed.
138 More complex operations may be performed by using hwloc-calc to compute
140 intermediate values.
141 hwloc Objects
143 Objects in tuples can be any of the following strings (listed from
144 "biggest" to "smallest"):
145 machine A set of processors and memory.
147 numanode A NUMA node; a set of processors around memory which the pro‐
149 cessors can directly access. If hbm is used instead of nu‐
150 manode in locations, command-line tools only consider high-
151 bandwidth memory nodes such as Intel Xeon Phi MCDRAM.
152 package Typically a physical package or chip, that goes into a pack‐
154 age, it is a grouping of one or more processors.
155 l1cache ... l5cache
157 A data (or unified) cache.
158 l1icache ... l3icache
160 An instruction cache.
161 core A single, physical processing unit which may still contain
163 multiple logical processors, such as hardware threads.
164 pu Short for processor unit (not process!). The smallest physi‐
166 cal execution unit that hwloc recognizes. For example, there
167 may be multiple PUs on a core (e.g., hardware threads).
168 osdev, pcidev, bridge, and misc may also be used to specify special de‐
170 vices although some of them have dedicated identification ways as ex‐
171 plained in Location Specification.
172 Finally, note that an object can be denoted by its numeric "depth" in
174 the topology graph.
175 hwloc Indexes
177 Indexes are integer values that uniquely specify a given object of a
178 specific type. Indexes can be expressed either as logical values or
179 physical values. Most hwloc utilities accept logical indexes by de‐
180 fault. Passing --physical switches to physical/OS indexes. Both logi‐
181 cal and physical indexes are described on this man page.
182 Logical indexes are relative to the object order in the output from the
184 lstopo command. They always start with 0 and increment by 1 for each
185 successive object.
186 Physical indexes are how the operating system refers to objects. Note
188 that while physical indexes are non-negative integer values, the hard‐
189 ware and/or operating system may choose arbitrary values -- they may
190 not start with 0, and successive objects may not have consecutive val‐
191 ues.
192 For example, if the first few lines of lstopo -p output are the follow‐
194 ing:
195 Machine (47GB)
197 NUMANode P#0 (24GB) + Package P#0 + L3 (12MB)
198 L2 (256KB) + L1 (32KB) + Core P#0 + PU P#0
199 L2 (256KB) + L1 (32KB) + Core P#1 + PU P#0
200 L2 (256KB) + L1 (32KB) + Core P#2 + PU P#0
201 L2 (256KB) + L1 (32KB) + Core P#8 + PU P#0
202 L2 (256KB) + L1 (32KB) + Core P#9 + PU P#0
203 L2 (256KB) + L1 (32KB) + Core P#10 + PU P#0
204 NUMANode P#1 (24GB) + Package P#1 + L3 (12MB)
205 L2 (256KB) + L1 (32KB) + Core P#0 + PU P#0
206 L2 (256KB) + L1 (32KB) + Core P#1 + PU P#0
207 L2 (256KB) + L1 (32KB) + Core P#2 + PU P#0
208 L2 (256KB) + L1 (32KB) + Core P#8 + PU P#0
209 L2 (256KB) + L1 (32KB) + Core P#9 + PU P#0
210 L2 (256KB) + L1 (32KB) + Core P#10 + PU P#0
211 In this example, the first core on the second package is logically num‐
213 ber 6 (i.e., logically the 7th core, starting from 0). Its physical
214 index is 0, but note that another core also has a physical index of 0.
215 Hence, physical indexes may only be relevant within the scope of their
216 parent (or set of ancestors). In this example, to uniquely identify
217 logical core 6 with physical indexes, you must specify (at a minimum)
218 both a package and a core: package 1, core 0.
219 Index values, regardless of whether they are logical or physical, can
221 be expressed in several different forms (where X, Y, and N are positive
222 integers):
223 X The object with index value X.
225 X-Y All the objects with index values >= X and <= Y.
227 X- All the objects with index values >= X.
229 X:N N objects starting with index X, possibly wrapping around the
231 end of the level.
232 all A special index value indicating all valid index values.
234 odd A special index value indicating all valid odd index values.
236 even A special index value indicating all valid even index values.
238 REMEMBER: hwloc's command line tools accept logical indexes for loca‐
240 tion values by default. Use --physical and --logical to switch from
241 one mode to another.
242
244 hwloc's command line tool documentation: lstopo(1), hwloc-bind(1),
245 hwloc-calc(1), hwloc-distrib(1), hwloc-ps(1).
246 hwloc has many C API functions, each of which have their own man page.
248 Some top-level man pages are also provided, grouping similar functions
249 together. A few good places to start might include: hwlocality_ob‐
250 jects(3), hwlocality_types(3), hwlocality_creation(3), hwlocal‐
251 ity_cpuset(3), hwlocality_information(3), and hwlocality_binding(3).
252 For a listing of all available hwloc man pages, look at all "hwloc*"
254 files in the man1 and man3 directories.
2552.5.0 Jun 14, 2021 HWLOC(7)