1INFINIBAND-DIAGS(8) Open IB Diagnostics INFINIBAND-DIAGS(8)
2
6 INFINIBAND-DIAGS -
7
9 DESCRIPTION
10 infiniband-diags is a set of utilities designed to help configure,
11 debug, and maintain infiniband fabrics. Many tools and utilities are
12 provided. Some with similar functionality.
13 The base utilities use directed route MAD's to perform their opera‐
15 tions. They may therefore work even in unconfigured subnets. Other,
16 higher level utilities, require LID routed MAD's and to some extent
17 SA/SM access.
18 THE USE OF SMPs (QP0)
20 Many of the tools in this package rely on the use of SMPs via QP0 to
21 acquire data directly from the SMA. While this mode of operation is
22 not technically in compliance with the InfiniBand specification, prac‐
23 tical experience has found that this level of diagnostics is valuable
24 when working with a fabric which is broken or only partially config‐
25 ured. For this reason many of these tools may require the use of an
26 MKey or operation from Virtual Machines may be restricted for security
27 reasons.
28 COMMON OPTIONS
30 Most OpenIB diagnostics take some of the following common flags. The
31 exact list of supported flags per utility can be found in the documen‐
32 tation for those commands.
33 Addressing Flags
35 The -D and -G option have two forms:
36 -D, --Direct The address specified is a directed route
38 Examples:
40 [options] -D [options] "0" # self port
41 [options] -D [options] "0,1,2,1,4" # out via port 1, then 2, ...
42 (Note the second number in the path specified must match the port being
44 used. This can be specified using the port selection flag '-P' or the
45 port found through the automatic selection process.)
46 -D, --Direct <dr_path> The address specified is a directed route
48 Examples:
50 -D "0" # self port
51 -D "0,1,2,1,4" # out via port 1, then 2, ...
52 (Note the second number in the path specified must match the port being
54 used. This can be specified using the port selection flag '-P' or the
55 port found through the automatic selection process.)
56 -G, --Guid The address specified is a Port GUID
58 --port-guid, -G <port_guid> Specify a port_guid
60 -L, --Lid The address specified is a LID
62 -s, --sm_port <smlid> use 'smlid' as the target lid for SA queries.
64 Port Selection flags
66 -C, --Ca <ca_name> use the specified ca_name.
67 -P, --Port <ca_port> use the specified ca_port.
69 Local port Selection
71 Multiple port/Multiple CA support: when no IB device or port is speci‐
72 fied (see the "local umad parameters" below), the libibumad library
73 selects the port to use by the following criteria:
74 1. the first port that is ACTIVE.
76 2. if not found, the first port that is UP (physical link up).
78 If a port and/or CA name is specified, the libibumad library
80 attempts to fulfill the user request, and will fail if it is not
81 possible.
82 For example:
84 ibaddr # use the first port (criteria #1 above)
86 ibaddr -C mthca1 # pick the best port from "mthca1" only.
87 ibaddr -P 2 # use the second (active/up) port from the first available IB device.
88 ibaddr -C mthca0 -P 2 # use the specified port only.
89 Debugging flags
91 -d raise the IB debugging level. May be used several times (-ddd
92 or -d -d -d).
93 -e show send and receive errors (timeouts and others)
95 -h, --help show the usage message
97 -v, --verbose
99 increase the application verbosity level. May be used several
100 times (-vv or -v -v -v)
101 -V, --version show the version info.
103 Configuration flags
105 -t, --timeout <timeout_ms> override the default timeout for the
106 solicited mads.
107 --outstanding_smps, -o <val>
109 Specify the number of outstanding SMP's which should be issued
110 during the scan
111 Default: 2
113 --node-name-map <node-name-map> Specify a node name map.
115 This file maps GUIDs to more user friendly names. See FILES sec‐
116 tion.
117 --config, -z <config_file> Specify alternate config file.
119 Default: /etc/infiniband-diags/ibdiag.conf
120 COMMON FILES
122 The following config files are common amongst many of the utilities.
123 CONFIG FILE
125 /etc/infiniband-diags/ibdiag.conf
126 A global config file is provided to set some of the common options for
128 all tools. See supplied config file for details.
129 NODE NAME MAP FILE FORMAT
131 The node name map is used to specify user friendly names for nodes in
132 the output. GUIDs are used to perform the lookup.
133 This functionality is provided by the opensm-libs package. See
135 opensm(8) for the file location for your installation.
136 Generically:
138 # comment
140 <guid> "<name>"
141 Example:
143 # IB1
145 # Line cards
146 0x0008f104003f125c "IB1 (Rack 11 slot 1 ) ISR9288/ISR9096 Voltaire sLB-24D"
147 0x0008f104003f125d "IB1 (Rack 11 slot 1 ) ISR9288/ISR9096 Voltaire sLB-24D"
148 0x0008f104003f10d2 "IB1 (Rack 11 slot 2 ) ISR9288/ISR9096 Voltaire sLB-24D"
149 0x0008f104003f10d3 "IB1 (Rack 11 slot 2 ) ISR9288/ISR9096 Voltaire sLB-24D"
150 0x0008f104003f10bf "IB1 (Rack 11 slot 12 ) ISR9288/ISR9096 Voltaire sLB-24D"
151 # Spines
153 0x0008f10400400e2d "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
154 0x0008f10400400e2e "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
155 0x0008f10400400e2f "IB1 (Rack 11 spine 1 ) ISR9288 Voltaire sFB-12D"
156 0x0008f10400400e31 "IB1 (Rack 11 spine 2 ) ISR9288 Voltaire sFB-12D"
157 0x0008f10400400e32 "IB1 (Rack 11 spine 2 ) ISR9288 Voltaire sFB-12D"
158 # GUID Node Name
160 0x0008f10400411a08 "SW1 (Rack 3) ISR9024 Voltaire 9024D"
161 0x0008f10400411a28 "SW2 (Rack 3) ISR9024 Voltaire 9024D"
162 0x0008f10400411a34 "SW3 (Rack 3) ISR9024 Voltaire 9024D"
163 0x0008f104004119d0 "SW4 (Rack 3) ISR9024 Voltaire 9024D"
164 TOPOLOGY FILE FORMAT
166 The topology file format is human readable and largely intuitive. Most
167 identifiers are given textual names like vendor ID (vendid), device ID
168 (device ID), GUIDs of various types (sysimgguid, caguid, switchguid,
169 etc.). PortGUIDs are shown in parentheses (). For switches, this is
170 shown on the switchguid line. For CA and router ports, it is shown on
171 the connectivity lines. The IB node is identified followed by the num‐
172 ber of ports and a quoted the node GUID. On the right of this line is
173 a comment (#) followed by the NodeDescription in quotes. If the node
174 is a switch, this line also contains whether switch port 0 is base or
175 enhanced, and the LID and LMC of port 0. Subsequent lines pertaining
176 to this node show the connectivity. On the left is the port number of
177 the current node. On the right is the peer node (node at other end of
178 link). It is identified in quotes with nodetype followed by - followed
179 by NodeGUID with the port number in square brackets. Further on the
180 right is a comment (#). What follows the comment is dependent on the
181 node type. If it it a switch node, it is followed by the NodeDescrip‐
182 tion in quotes and the LID of the peer node. If it is a CA or router
183 node, it is followed by the local LID and LMC and then followed by the
184 NodeDescription in quotes and the LID of the peer node. The active
185 link width and speed are then appended to the end of this output line.
186 An example of this is:
188 #
190 # Topology file: generated on Tue Jun 5 14:15:10 2007
191 #
192 # Max of 3 hops discovered
193 # Initiated from node 0008f10403960558 port 0008f10403960559
194 Non-Chassis Nodes
196 vendid=0x8f1
198 devid=0x5a06
199 sysimgguid=0x5442ba00003000
200 switchguid=0x5442ba00003080(5442ba00003080)
201 Switch 24 "S-005442ba00003080" # "ISR9024 Voltaire" base port 0 lid 6 lmc 0
202 [22] "H-0008f10403961354"[1](8f10403961355) # "MT23108 InfiniHost Mellanox Technologies" lid 4 4xSDR
203 [10] "S-0008f10400410015"[1] # "SW-6IB4 Voltaire" lid 3 4xSDR
204 [8] "H-0008f10403960558"[2](8f1040396055a) # "MT23108 InfiniHost Mellanox Technologies" lid 14 4xSDR
205 [6] "S-0008f10400410015"[3] # "SW-6IB4 Voltaire" lid 3 4xSDR
206 [12] "H-0008f10403960558"[1](8f10403960559) # "MT23108 InfiniHost Mellanox Technologies" lid 10 4xSDR
207 vendid=0x8f1
209 devid=0x5a05
210 switchguid=0x8f10400410015(8f10400410015)
211 Switch 8 "S-0008f10400410015" # "SW-6IB4 Voltaire" base port 0 lid 3 lmc 0
212 [6] "H-0008f10403960984"[1](8f10403960985) # "MT23108 InfiniHost Mellanox Technologies" lid 16 4xSDR
213 [4] "H-005442b100004900"[1](5442b100004901) # "MT23108 InfiniHost Mellanox Technologies" lid 12 4xSDR
214 [1] "S-005442ba00003080"[10] # "ISR9024 Voltaire" lid 6 1xSDR
215 [3] "S-005442ba00003080"[6] # "ISR9024 Voltaire" lid 6 4xSDR
216 vendid=0x2c9
218 devid=0x5a44
219 caguid=0x8f10403960984
220 Ca 2 "H-0008f10403960984" # "MT23108 InfiniHost Mellanox Technologies"
221 [1](8f10403960985) "S-0008f10400410015"[6] # lid 16 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR
222 vendid=0x2c9
224 devid=0x5a44
225 caguid=0x5442b100004900
226 Ca 2 "H-005442b100004900" # "MT23108 InfiniHost Mellanox Technologies"
227 [1](5442b100004901) "S-0008f10400410015"[4] # lid 12 lmc 1 "SW-6IB4 Voltaire" lid 3 4xSDR
228 vendid=0x2c9
230 devid=0x5a44
231 caguid=0x8f10403961354
232 Ca 2 "H-0008f10403961354" # "MT23108 InfiniHost Mellanox Technologies"
233 [1](8f10403961355) "S-005442ba00003080"[22] # lid 4 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR
234 vendid=0x2c9
236 devid=0x5a44
237 caguid=0x8f10403960558
238 Ca 2 "H-0008f10403960558" # "MT23108 InfiniHost Mellanox Technologies"
239 [2](8f1040396055a) "S-005442ba00003080"[8] # lid 14 lmc 1 "ISR9024 Voltaire" lid 6 4xSDR
240 [1](8f10403960559) "S-005442ba00003080"[12] # lid 10 lmc 1 "ISR9024 Voltaire" lid 6 1xSDR
241 When grouping is used, IB nodes are organized into chassis which are
243 numbered. Nodes which cannot be determined to be in a chassis are dis‐
244 played as "Non-Chassis Nodes". External ports are also shown on the
245 connectivity lines.
246 Utilities list
248 Basic fabric conectivity
249 See: ibnetdiscover, iblinkinfo
250 Node information
252 See: ibnodes, ibswitches, ibhosts, ibrouters
253 Port information
255 See: ibportstate, ibaddr
256 Switch Forwarding Table info
258 See: ibtracert, ibroute, dump_lfts, dump_mfts, check_lft_balance,
259 ibfindnodesusing
260 Peformance counters
262 See: ibqueryerrors, perfquery
263 Local HCA info
265 See: ibstat, ibstatus
266 Connectivity check
268 See: ibping, ibsysstat
269 Low level query tools
271 See: smpquery, smpdump, saquery, sminfo
272 Fabric verification tools
274 See: ibidsverify
275 Backwards compatibility scripts
277 The following scripts have been identified as redundant and/or lower
278 performing as compared to the above scripts. They are provided as
279 legacy scripts when --enable-compat-utils is specified at build time.
280 ibcheckerrors, ibclearcounters, ibclearerrors, ibdatacounters ibcheck‐
282 net, ibchecknode, ibcheckport, ibcheckportstate, ibcheckportwidth,
283 ibcheckstate, ibcheckwidth, ibswportwatch, ibprintca, ibprintrt,
284 ibprintswitch, set_nodedesc.sh
285 AUTHORS
287 Ira Weiny
288 < ira.weiny@intel.com >
289 INFINIBAND-DIAGS(8)