1IBNETDISCOVER(8) Open IB Diagnostics IBNETDISCOVER(8)
2
6 IBNETDISCOVER - discover InfiniBand topology
7
9 ibnetdiscover [options] [<topology-file>]
10
12 ibnetdiscover performs IB subnet discovery and outputs a human readable
13 topology file. GUIDs, node types, and port numbers are displayed as
14 well as port LIDs and NodeDescriptions. All nodes (and links) are dis‐
15 played (full topology). Optionally, this utility can be used to list
16 the current connected nodes by nodetype. The output is printed to
17 standard output unless a topology file is specified.
18
20 -l, --list List of connected nodes
21 -g, --grouping Show grouping. Grouping correlates IB nodes by differ‐
23 ent vendor specific schemes. It may also show the switch external
24 ports correspondence.
25 -H, --Hca_list List of connected CAs
27 -S, --Switch_list List of connected switches
29 -R, --Router_list List of connected routers
31 -s, --show Show progress information during discovery.
33 -f, --full Show full information (ports' speed and width, vlcap)
35 -p, --ports Obtain a ports report which is a list of connected ports
37 with relevant information (like LID, portnum, GUID, width, speed, and
38 NodeDescription).
39 -m, --max_hops Report max hops discovered.
41 --outstanding_smps, -o <val>
43 Specify the number of outstanding SMP's which should be issued
44 during the scan
45 Default: 2
47 Cache File flags
49 --cache <filename> Cache the ibnetdiscover network data in the speci‐
50 fied filename. This cache may be used by other tools for later analy‐
51 sis.
52 --load-cache <filename> Load and use the cached ibnetdiscover data
54 stored in the specified filename. May be useful for outputting and
55 learning about other fabrics or a previous state of a fabric.
56 --diff <filename> Load cached ibnetdiscover data and do a diff compari‐
58 son to the current network or another cache. A special diff output for
59 ibnetdiscover output will be displayed showing differences between the
60 old and current fabric. By default, the following are compared for
61 differences: switches, channel adapters, routers, and port connections.
62 --diffcheck <key(s)> Specify what diff checks should be done in the
64 --diff option above. Comma separate multiple diff check key(s). The
65 available diff checks are: sw = switches, ca = channel adapters, router
66 = routers, port = port connections, lid = lids, nodedesc = node
67 descriptions. Note that port, lid, and nodedesc are checked only for
68 the node types that are specified (e.g. sw, ca, router). If port is
69 specified alongside lid or nodedesc, remote port lids and node descrip‐
70 tions will also be compared.
71 Port Selection flags
73 -C, --Ca <ca_name> use the specified ca_name.
74 -P, --Port <ca_port> use the specified ca_port.
76 Local port Selection
78 Multiple port/Multiple CA support: when no IB device or port is speci‐
79 fied (see the "local umad parameters" below), the libibumad library
80 selects the port to use by the following criteria:
81 1. the first port that is ACTIVE.
83 2. if not found, the first port that is UP (physical link up).
85 If a port and/or CA name is specified, the libibumad library
87 attempts to fulfill the user request, and will fail if it is not
88 possible.
89 For example:
91 ibaddr # use the first port (criteria #1 above)
93 ibaddr -C mthca1 # pick the best port from "mthca1" only.
94 ibaddr -P 2 # use the second (active/up) port from the first available IB device.
95 ibaddr -C mthca0 -P 2 # use the specified port only.
96 Configuration flags
98 --config, -z <config_file> Specify alternate config file.
99 Default: /etc/infiniband-diags/ibdiag.conf
100 --outstanding_smps, -o <val>
102 Specify the number of outstanding SMP's which should be issued
103 during the scan
104 Default: 2
106 --node-name-map <node-name-map> Specify a node name map.
108 This file maps GUIDs to more user friendly names. See FILES sec‐
109 tion.
110 -t, --timeout <timeout_ms> override the default timeout for the
112 solicited mads.
113 -y, --m_key <key>
115 use the specified M_key for requests. If non-numeric value (like
116 'x') is specified then a value will be prompted for.
117 Debugging flags
119 -d raise the IB debugging level. May be used several times (-ddd
120 or -d -d -d).
121 -e show send and receive errors (timeouts and others)
123 -h, --help show the usage message
125 -v, --verbose
127 increase the application verbosity level. May be used several
128 times (-vv or -v -v -v)
129 -V, --version show the version info.
131
133 CONFIG FILE
134 /etc/infiniband-diags/ibdiag.conf
135 A global config file is provided to set some of the common options for
137 all tools. See supplied config file for details.
138 NODE NAME MAP FILE FORMAT
140 The node name map is used to specify user friendly names for nodes in
141 the output. GUIDs are used to perform the lookup.
142 This functionality is provided by the opensm-libs package. See
144 opensm(8) for the file location for your installation.
145 Generically:
147 # comment
149 <guid> "<name>"
150 Example:
152 # IB1
154 # Line cards
155 0x0008f104003f125c "IB1 (Rack 11 slot 1 ) ISR9288/ISR9096 Voltaire sLB-24D"
156 0x0008f104003f125d "IB1 (Rack 11 slot 1 ) ISR9288/ISR9096 Voltaire sLB-24D"
157 0x0008f104003f10d2 "IB1 (Rack 11 slot 2 ) ISR9288/ISR9096 Voltaire sLB-24D"
158 0x0008f104003f10d3 "IB1 (Rack 11 slot 2 ) ISR9288/ISR9096 Voltaire sLB-24D"
159 0x0008f104003f10bf "IB1 (Rack 11 slot 12 ) ISR9288/ISR9096 Voltaire sLB-24D"
160 # Spines
162 0x0008f10400400e2d "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
163 0x0008f10400400e2e "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
164 0x0008f10400400e2f "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
165 0x0008f10400400e31 "IB1 (Rack 11 spine 2 ) ISR9288 Voltaire sFB-12D"
166 0x0008f10400400e32 "IB1 (Rack 11 spine 2 ) ISR9288 Voltaire sFB-12D"
167 # GUID Node Name
169 0x0008f10400411a08 "SW1 (Rack 3) ISR9024 Voltaire 9024D"
170 0x0008f10400411a28 "SW2 (Rack 3) ISR9024 Voltaire 9024D"
171 0x0008f10400411a34 "SW3 (Rack 3) ISR9024 Voltaire 9024D"
172 0x0008f104004119d0 "SW4 (Rack 3) ISR9024 Voltaire 9024D"
173 TOPOLOGY FILE FORMAT
175 The topology file format is human readable and largely intuitive. Most
176 identifiers are given textual names like vendor ID (vendid), device ID
177 (device ID), GUIDs of various types (sysimgguid, caguid, switchguid,
178 etc.). PortGUIDs are shown in parentheses (). For switches, this is
179 shown on the switchguid line. For CA and router ports, it is shown on
180 the connectivity lines. The IB node is identified followed by the num‐
181 ber of ports and a quoted the node GUID. On the right of this line is
182 a comment (#) followed by the NodeDescription in quotes. If the node
183 is a switch, this line also contains whether switch port 0 is base or
184 enhanced, and the LID and LMC of port 0. Subsequent lines pertaining
185 to this node show the connectivity. On the left is the port number of
186 the current node. On the right is the peer node (node at other end of
187 link). It is identified in quotes with nodetype followed by - followed
188 by NodeGUID with the port number in square brackets. Further on the
189 right is a comment (#). What follows the comment is dependent on the
190 node type. If it it a switch node, it is followed by the NodeDescrip‐
191 tion in quotes and the LID of the peer node. If it is a CA or router
192 node, it is followed by the local LID and LMC and then followed by the
193 NodeDescription in quotes and the LID of the peer node. The active
194 link width and speed are then appended to the end of this output line.
195 An example of this is:
197 #
199 # Topology file: generated on Tue Jun 5 14:15:10 2007
200 #
201 # Max of 3 hops discovered
202 # Initiated from node 0008f10403960558 port 0008f10403960559
203 Non-Chassis Nodes
205 vendid=0x8f1
207 devid=0x5a06
208 sysimgguid=0x5442ba00003000
209 switchguid=0x5442ba00003080(5442ba00003080)
210 Switch 24 "S-005442ba00003080" # "ISR9024 Voltaire" base port 0 lid 6 lmc 0
211 [22] "H-0008f10403961354"[1](8f10403961355) # "MT23108 InfiniHost Mellanox Technologies" lid 4 4xSDR
212 [10] "S-0008f10400410015"[1] # "SW-6IB4 Voltaire" lid 3 4xSDR
213 [8] "H-0008f10403960558"[2](8f1040396055a) # "MT23108 InfiniHost Mellanox Technologies" lid 14 4xSDR
214 [6] "S-0008f10400410015"[3] # "SW-6IB4 Voltaire" lid 3 4xSDR
215 [12] "H-0008f10403960558"[1](8f10403960559) # "MT23108 InfiniHost Mellanox Technologies" lid 10 4xSDR
216 vendid=0x8f1
218 devid=0x5a05
219 switchguid=0x8f10400410015(8f10400410015)
220 Switch 8 "S-0008f10400410015" # "SW-6IB4 Voltaire" base port 0 lid 3 lmc 0
221 [6] "H-0008f10403960984"[1](8f10403960985) # "MT23108 InfiniHost Mellanox Technologies" lid 16 4xSDR
222 [4] "H-005442b100004900"[1](5442b100004901) # "MT23108 InfiniHost Mellanox Technologies" lid 12 4xSDR
223 [1] "S-005442ba00003080"[10] # "ISR9024 Voltaire" lid 6 1xSDR
224 [3] "S-005442ba00003080"[6] # "ISR9024 Voltaire" lid 6 4xSDR
225 vendid=0x2c9
227 devid=0x5a44
228 caguid=0x8f10403960984
229 Ca 2 "H-0008f10403960984" # "MT23108 InfiniHost Mellanox Technologies"
230 [1](8f10403960985) "S-0008f10400410015"[6] # lid 16 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR
231 vendid=0x2c9
233 devid=0x5a44
234 caguid=0x5442b100004900
235 Ca 2 "H-005442b100004900" # "MT23108 InfiniHost Mellanox Technologies"
236 [1](5442b100004901) "S-0008f10400410015"[4] # lid 12 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR
237 vendid=0x2c9
239 devid=0x5a44
240 caguid=0x8f10403961354
241 Ca 2 "H-0008f10403961354" # "MT23108 InfiniHost Mellanox Technologies"
242 [1](8f10403961355) "S-005442ba00003080"[22] # lid 4 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR
243 vendid=0x2c9
245 devid=0x5a44
246 caguid=0x8f10403960558
247 Ca 2 "H-0008f10403960558" # "MT23108 InfiniHost Mellanox Technologies"
248 [2](8f1040396055a) "S-005442ba00003080"[8] # lid 14 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR
249 [1](8f10403960559) "S-005442ba00003080"[12] # lid 10 lmc 1 "ISR9024 Voltaire" lid 6 1xSDR
250 When grouping is used, IB nodes are organized into chassis which are
252 numbered. Nodes which cannot be determined to be in a chassis are dis‐
253 played as "Non-Chassis Nodes". External ports are also shown on the
254 connectivity lines.
255
257 Hal Rosenstock
258 < halr@voltaire.com >
259 Ira Weiny
261 < ira.weiny@intel.com >
262 IBNETDISCOVER(8)