1TORUS-2QOS.CONF(5) OpenIB Management TORUS-2QOS.CONF(5)
2
6 torus-2QoS.conf - Torus-2QoS configuration for OpenSM subnet manager
7
9 The file torus-2QoS.conf contains configuration information that is
10 specific to the OpenSM routing engine torus-2QoS. Blank lines and
11 lines where the first non-whitespace character is "#" are ignored. A
12 token is any contiguous group of non-whitespace characters. Any tokens
13 on a line following the recognized configuration tokens described below
14 are ignored.
15 [torus|mesh] x_radix[m|M|t|T] y_radix[m|M|t|T] z_radix[m|M|t|T]
17 Either torus or mesh must be the first keyword in the configura‐
18 tion, and sets the topology that torus-2QoS will try to con‐
19 struct. A 2D topology can be configured by specifying one of
20 x_radix, y_radix, or z_radix as 1. An individual dimension can
21 be configured as mesh (open) or torus (looped) by suffixing its
22 radix specification with one of m, M, t, or T. Thus, "mesh 3T 4
23 5" and "torus 3 4M 5M" both specify the same topology.
24 Note that although torus-2QoS can route mesh fabrics, its abil‐
26 ity to route around failed components is severely compromised on
27 such fabrics. A failed fabric component is very likely to cause
28 a disjoint ring; see UNICAST ROUTING in torus-2QoS(8).
29 xp_link sw0_GUID sw1_GUID
31 yp_link sw0_GUID sw1_GUID
32 zp_link sw0_GUID sw1_GUID
33 xm_link sw0_GUID sw1_GUID
34 ym_link sw0_GUID sw1_GUID
35 zm_link sw0_GUID sw1_GUID
36 These keywords are used to seed the torus/mesh topology. For
37 example, "xp_link 0x2000 0x2001" specifies that a link from the
38 switch with node GUID 0x2000 to the switch with node GUID 0x2001
39 would point in the positive x direction, while "xm_link 0x2000
40 0x2001" specifies that a link from the switch with node GUID
41 0x2000 to the switch with node GUID 0x2001 would point in the
42 negative x direction. All the link keywords for a given seed
43 must specify the same "from" switch.
44 In general, it is not necessary to configure both the positive
46 and negative directions for a given coordinate; either is suffi‐
47 cient. However, the algorithm used for topology discovery needs
48 extra information for torus dimensions of radix four (see TOPOL‐
49 OGY DISCOVERY in torus-2QoS(8)). For such cases both the posi‐
50 tive and negative coordinate directions must be specified.
51 Based on the topology specified via the torus/mesh keyword,
53 torus-2QoS will detect and log when it has insufficient seed
54 configuration.
55 x_dateline position
57 y_dateline position
58 z_dateline position
59 In order for torus-2QoS to provide the guarantee that path SL
60 values do not change under any conditions for which it can still
61 route the fabric, its idea of dateline position must not change
62 relative to physical switch locations. The dateline keywords
63 provide the means to configure such behavior.
64 The dateline for a torus dimension is always between the switch
66 with coordinate 0 and the switch with coordinate radix-1 for
67 that dimension. By default, the common switch in a torus seed
68 is taken as the origin of the coordinate system used to describe
69 switch location. The position parameter for a dateline keyword
70 moves the origin (and hence the dateline) the specified amount
71 relative to the common switch in a torus seed.
72 next_seed
74 If any of the switches used to specify a seed were to fail
75 torus-2QoS would be unable to complete topology discovery suc‐
76 cessfully. The next_seed keyword specifies that the following
77 link and dateline keywords apply to a new seed specification.
78 For maximum resiliency, no seed specification should share a
80 switch with any other seed specification. Multiple seed speci‐
81 fications should use dateline configuration to ensure that
82 torus-2QoS can grant path SL values that are constant, regard‐
83 less of which seed was used to initiate topology discovery.
84 portgroup_max_ports max_ports
86 This keyword specifies the maximum number of parallel inter-
87 switch links, and also the maximum number of host ports per
88 switch, that torus-2QoS can accommodate. The default value is
89 16. Torus-2QoS will log an error message during topology dis‐
90 covery if this parameter needs to be increased. If this keyword
91 appears multiple times, the last instance prevails.
92 Note that the switch management port (switch port 0) gets put
94 into the same port group with the host ports, so if you have 16
95 host ports per switch, portgroup_max_ports would need to be at
96 least 17.
97 port_order p1 p2 p3 ...
99 This keyword specifies the order in which CA ports on a destina‐
100 tion switch are visited when computing routes. When the fabric
101 contains switches connected with multiple parallel links, routes
102 are distributed in a round-robin fashion across such links, and
103 so changing the order that CA ports are visited changes the dis‐
104 tribution of routes across such links. This may be advantageous
105 for some specific traffic patterns.
106 The default is to visit CA ports in increasing port order on
108 destination switches.
109 Duplicate values in the list will be ignored.
111 max_changes max
113 This keyword specifies the maximum number of torus changes
114 reported. The default value is 32.
115
117 # Look for a 2D (since x radix is one) 4x5 torus.
119 torus 1 4 5
120 # y is radix-4 torus dimension, need both
122 # ym_link and yp_link configuration.
123 yp_link 0x200000 0x200005 # sw @ y=0,z=0 -> sw @ y=1,z=0
124 ym_link 0x200000 0x20000f # sw @ y=0,z=0 -> sw @ y=3,z=0
125 # z is not radix-4 torus dimension, only need one of
127 # zm_link or zp_link configuration.
128 zp_link 0x200000 0x200001 # sw @ y=0,z=0 -> sw @ y=0,z=1
129 next_seed
131 yp_link 0x20000b 0x200010 # sw @ y=2,z=1 -> sw @ y=3,z=1
133 ym_link 0x20000b 0x200006 # sw @ y=2,z=1 -> sw @ y=1,z=1
134 zp_link 0x20000b 0x20000c # sw @ y=2,z=1 -> sw @ y=2,z=2
135 y_dateline -2 # Move the dateline for this seed
137 z_dateline -1 # back to its original position.
138 # If OpenSM failover is configured, for maximum resiliency
140 # one instance should run on a host attached to a switch
141 # from the first seed, and another instance should run
142 # on a host attached to a switch from the second seed.
143 # Both instances should use this torus-2QoS.conf to ensure
144 # path SL values do not change in the event of SM failover.
145 # port_order defines the order on which the ports would be
147 # chosen for routing.
148 port_order 7 10 8 11 9 12 25 28 26 29 27 30
149
152 /etc/rdma/torus-2QoS.conf
153 Default torus-2QoS config file.
154
156 opensm(8), torus-2QoS(8).
157OpenIB January 4, 2013 TORUS-2QOS.CONF(5)