1HWLOC(7) hwloc HWLOC(7)
2
6 hwloc - General information about hwloc ("hardware locality").
7
9 hwloc provides command line tools and a C API to obtain the hierarchi‐
10 cal map of key computing elements, such as: NUMA memory nodes, shared
11 caches, processor packages, processor cores, and processor "threads".
12 hwloc also gathers various attributes such as cache and memory informa‐
13 tion, and is portable across a variety of different operating systems
14 and platforms.
15 Definitions
17 hwloc has some specific definitions for terms that are used in this man
18 page and other hwloc documentation.
19 hwloc CPU set:
21 A set of processors included in an hwloc object, expressed as a
22 bitmask indexed by the physical numbers of the CPUs (as announced
23 by the OS). The hwloc definition of "CPU set" does not carry any
24 of the same connotations as Linux's "CPU set" (e.g., process
25 affinity, cgroup, etc.).
26 hwloc node set:
28 A set of NUMA memory nodes near an hwloc object, expressed as a
29 bitmask indexed by the physical numbers of the NUMA nodes (as an‐
30 nounced by the OS).
31 Linux CPU set:
33 See http://www.mjmwired.net/kernel/Documentation/cpusets.txt for a
34 discussion of Linux CPU sets. A super-short-ignoring-many-details
35 description (taken from that page) is:
36 "Cpusets provide a mechanism for assigning a set of CPUs and Mem‐
38 ory Nodes to a set of tasks."
39 Linux Cgroup:
41 See http://www.mjmwired.net/kernel/Documentation/cgroups.txt for a
42 discussion of Linux control groups. A super-short-ignoring-many-
43 details description (taken from that page) is:
44 "Control Groups provide a mechanism for aggregating/partitioning
46 sets of tasks, and all their future children, into hierarchical
47 groups with specialized behaviour."
48 To be clear, hwloc supports all of the above concepts. It is simply
50 worth noting that they are different things.
51 Location Specification
53 Locations refer to specific regions within a topology. Before reading
54 the rest of this man page, it may be useful to read lstopo(1) and/or
55 run lstopo on your machine to see the reported topology tree. Seeing
56 and understanding a topology tree will definitely help in understanding
57 the concepts that are discussed below.
58 Locations can be specified in multiple ways:
60 Tuples: Tuples of hwloc "objects" and associated indexes can be spec‐
62 ified in the form object:index. hwloc objects represent
63 types of mapped items (e.g., packages, cores, etc.) in a
64 topology tree; indexes are non-negative integers that specify
65 a unique physical object in a topology tree. Both concepts
66 are described in detail, below.
67 Some filters may be added after the type to further specify
69 which objects are wanted. <type>[subtype=<subtype>] selects
70 objects matching the given type and also its subtype string
71 attribute. For instance NUMA[HBM] selects NUMA nodes of sub‐
72 type "HBM". The prefix subtype= may be avoided if there is
73 no ambiguity. NUMA[tier=X] selects NUMA nodes of tier <X>
74 ("MemoryTier" info attribute).
75 Indexes may also be specified as ranges. x-y enumerates from
77 index x to y. x:y enumerates y objects starting from index x
78 (wrapping around the end of the index range if needed). x-
79 enumerates all objects starting from index x. all, odd, and
80 even are also supported for listing all objects, or only
81 those with odd or even indexes.
82 Chaining multiple tuples together in the more general form
84 object1:index[.object2:index2[...]] is permissable. While
85 the first tuple's object may appear anywhere in the topology,
86 the Nth tuple's object must have a shallower topology depth
87 than the (N+1)th tuple's object. Put simply: as you move
88 right in a tuple chain, objects must go deeper in the topol‐
89 ogy tree. When using logical indexes (which is the default),
90 indexes specified in chained tuples are relative to the scope
91 of the parent object. For example, "package:0.core:1" refers
92 to the second core in the first package.
93 When using OS/physical indexes, the first object matching the
95 given index is used.
96 PCI and OS devices may also be designed using their identi‐
98 fier. For example, "pci=02:03.1" is the PCI device with bus
99 ID "02:03.1". "os=eth0" is the network interface whose soft‐
100 ware name is "eth0". PCI devices may also be filtered based
101 on their vendor and/or device IDs, for instance
102 "pci[15b3:]:2" for the third Mellanox PCI device (vendor ID
103 0x15b3). OS devices may also be filtered based on their sub‐
104 type, for instance "os[gpu]:all" for all GPU OS devices.
105 Hex: For tools that manipulate object as sets (e.g. hwloc-calc and
107 hwloc-bind), locations can also be specified as hexidecimal
108 bitmasks prefixed with "0x". Commas must be used to separate
109 the hex digits into blocks of 8, such as
110 "0xffc0140,0x00020110". Leading zeros in each block do not
111 need to be specified. For example, "0xffc0140,0x20110" is
112 equivalent to the prior example, and "0x0000000f" is exactly
113 equivalent to "0xf". Intermediate blocks of 8 digits that
114 are all zeoro can be left empty; "0xff0,,0x13" is equivalent
115 to "0xff0,0x00000000,0x13". If the location is prefixed with
116 the special string "0xf...f", then all unspecified bits are
117 set (as if the set were infinite). For example, "0xf...f,0x1"
118 sets both the first bit and all bits starting with the 33rd.
119 The string "0xf...f" -- with no other specified values --
120 sets all bits.
121 "all" and "root" are special locations consisting in the root object in
123 tree. It contains the entire current topology.
124 Some tools directly operate on these objects (e.g. hwloc-info and
126 hwloc-annotate). They do not support hexadecimal locations because
127 each location may correspond to multiple objects. For instance, there
128 can be exactly one L3 cache per package and NUMA node, which means it's
129 the same location. If multiple locations are given on the command-
130 line, these tools will operation on each location individually and con‐
131 secutively.
132 Some other tools internally manipulate objects as sets (e.g. hwloc-calc
134 and hwloc-bind). They translate each input location into a hexidecimal
135 location. When I/O or Misc objects are used, they are translated into
136 the set of processors (or NUMA nodes) that are close to the given ob‐
137 ject (because I/O or Misc objects do not contain processors or NUMA
138 nodes).
139 If multiple locations are specified on the command-line (delimited by
141 whitespace), they are combined (the overall location is wider). If
142 prefixed with "~", the given location will be cleared instead of added
143 to the current list of locations. If prefixed with "x", the given lo‐
144 cation will be and'ed instead of added to the current list. If pre‐
145 fixed with "^", the given location will be xor'ed.
146 More complex operations may be performed by using hwloc-calc to compute
148 intermediate values.
149 hwloc Objects
151 Objects in tuples can be any of the following strings (listed from
152 "biggest" to "smallest"):
153 machine A set of processors and memory.
155 numanode A NUMA node; a set of processors around memory which the pro‐
157 cessors can directly access. If numa[hbm] is used instead of
158 numanode in locations, command-line tools only consider NUMA
159 nodes marked as high-bandwidth memory (subtype "HBM").
160 package Typically a physical package or chip, that goes into a pack‐
162 age, it is a grouping of one or more processors.
163 l1cache ... l5cache
165 A data (or unified) cache.
166 l1icache ... l3icache
168 An instruction cache.
169 core A single, physical processing unit which may still contain
171 multiple logical processors, such as hardware threads.
172 pu Short for processor unit (not process!). The smallest physi‐
174 cal execution unit that hwloc recognizes. For example, there
175 may be multiple PUs on a core (e.g., hardware threads).
176 osdev, pcidev, bridge, and misc may also be used to specify special de‐
178 vices although some of them have dedicated identification ways as ex‐
179 plained in Location Specification.
180 Finally, note that an object can be denoted by its numeric "depth" in
182 the topology graph.
183 hwloc Indexes
185 Indexes are integer values that uniquely specify a given object of a
186 specific type. Indexes can be expressed either as logical values or
187 physical values. Most hwloc utilities accept logical indexes by de‐
188 fault. Passing --physical switches to physical/OS indexes. Both logi‐
189 cal and physical indexes are described on this man page.
190 Logical indexes are relative to the object order in the output from the
192 lstopo command. They always start with 0 and increment by 1 for each
193 successive object.
194 Physical indexes are how the operating system refers to objects. Note
196 that while physical indexes are non-negative integer values, the hard‐
197 ware and/or operating system may choose arbitrary values -- they may
198 not start with 0, and successive objects may not have consecutive val‐
199 ues.
200 For example, if the first few lines of lstopo -p output are the follow‐
202 ing:
203 Machine (47GB)
205 NUMANode P#0 (24GB) + Package P#0 + L3 (12MB)
206 L2 (256KB) + L1 (32KB) + Core P#0 + PU P#0
207 L2 (256KB) + L1 (32KB) + Core P#1 + PU P#0
208 L2 (256KB) + L1 (32KB) + Core P#2 + PU P#0
209 L2 (256KB) + L1 (32KB) + Core P#8 + PU P#0
210 L2 (256KB) + L1 (32KB) + Core P#9 + PU P#0
211 L2 (256KB) + L1 (32KB) + Core P#10 + PU P#0
212 NUMANode P#1 (24GB) + Package P#1 + L3 (12MB)
213 L2 (256KB) + L1 (32KB) + Core P#0 + PU P#0
214 L2 (256KB) + L1 (32KB) + Core P#1 + PU P#0
215 L2 (256KB) + L1 (32KB) + Core P#2 + PU P#0
216 L2 (256KB) + L1 (32KB) + Core P#8 + PU P#0
217 L2 (256KB) + L1 (32KB) + Core P#9 + PU P#0
218 L2 (256KB) + L1 (32KB) + Core P#10 + PU P#0
219 In this example, the first core on the second package is logically num‐
221 ber 6 (i.e., logically the 7th core, starting from 0). Its physical
222 index is 0, but note that another core also has a physical index of 0.
223 Hence, physical indexes may only be relevant within the scope of their
224 parent (or set of ancestors). In this example, to uniquely identify
225 logical core 6 with physical indexes, you must specify (at a minimum)
226 both a package and a core: package 1, core 0.
227 Index values, regardless of whether they are logical or physical, can
229 be expressed in several different forms (where X, Y, and N are positive
230 integers):
231 X The object with index value X.
233 X-Y All the objects with index values >= X and <= Y.
235 X- All the objects with index values >= X.
237 X:N N objects starting with index X, possibly wrapping around the
239 end of the level.
240 all A special index value indicating all valid index values.
242 odd A special index value indicating all valid odd index values.
244 even A special index value indicating all valid even index values.
246 REMEMBER: hwloc's command line tools accept logical indexes for loca‐
248 tion values by default. Use --physical and --logical to switch from
249 one mode to another.
250
252 hwloc's command line tool documentation: lstopo(1), hwloc-bind(1),
253 hwloc-calc(1), hwloc-distrib(1), hwloc-ps(1).
254 hwloc has many C API functions, each of which have their own man page.
256 Some top-level man pages are also provided, grouping similar functions
257 together. A few good places to start might include: hwlocality_ob‐
258 jects(3), hwlocality_types(3), hwlocality_creation(3), hwlocal‐
259 ity_cpuset(3), hwlocality_information(3), and hwlocality_binding(3).
260 For a listing of all available hwloc man pages, look at all "hwloc*"
262 files in the man1 and man3 directories.
2632.10.0 Dec 04, 2023 HWLOC(7)