1STACD.CONF(5) STACD.CONF(5)
2
6 stacd.conf - stacd(8) configuration file
7
9 /etc/stas/stacd.conf
10
12 When stacd(8) starts up, it reads its configuration from stacd.conf.
13
15 stacd.conf is a plain text file divided into sections, with
16 configuration entries in the style key=value. A space immediately
17 before or after the = is ignored. Empty lines and lines starting with #
18 are ignored, which may be used for commenting.
19
21 [Global] section
22 The following options are available in the [Global]section:
23 tron=
25 Trace ON. Takes a boolean argument. If true, enables full code
26 tracing. The trace will be displayed in the system log such as
27 systemd's journal. Defaults to false.
28 hdr-digest=
30 Enable Protocol Data Unit (PDU) Header Digest. Takes a boolean
31 argument. NVMe/TCP facilitates an optional PDU Header digest.
32 Digests are calculated using the CRC32C algorithm. If true, Header
33 Digests are inserted in PDUs and checked for errors. Defaults to
34 false.
35 data-digest=
37 Enable Protocol Data Unit (PDU) Data Digest. Takes a boolean
38 argument. NVMe/TCP facilitates an optional PDU Data digest. Digests
39 are calculated using the CRC32C algorithm. If true, Data Digests
40 are inserted in PDUs and checked for errors. Defaults to false.
41 kato=
43 Keep Alive Timeout (KATO) in seconds. Takes an unsigned integer.
44 This field specifies the timeout value for the Keep Alive feature
45 in seconds. Defaults to 30 seconds for Discovery Controller
46 connections and 120 seconds for I/O Controller connections.
47 ip-family=
49 Takes a string argument. With this you can specify whether IPv4,
50 IPv6, or both are supported when connecting to a Controller.
51 Connections will not be attempted to IP addresses (whether
52 discovered or manually configured with the 'controller') if those
53 IP addresses are disabled by this option. If an invalid value is
54 entered, then "ipv4+ipv6" will be used by default.
55 Choices are ipv4, ipv6, or ipv4+ipv6.
57 Defaults to ipv4+ipv6.
59 ignore-iface=
61 Takes a boolean argument. This option controls how connections with
62 I/O Controllers (IOC) are made.
63 There is no guarantee that there will be a route to reach that IOC.
65 However, we can use the socket option SO_BINDTODEVICE to force the
66 connection to be made on a specific interface instead of letting
67 the routing tables decide where to make the connection.
68 This option determines whether stacd will use SO_BINDTODEVICE to
70 force connections on an interface or just rely on the routing
71 tables. The default is to use SO_BINDTODEVICE, in other words,
72 stacd does not ignore the interface.
73 BACKGROUND: By default, stacd will connect to IOCs on the same
75 interface that was used to retrieve the discovery log pages. If
76 stafd discovers a DC on an interface using mDNS, and stafd connects
77 to that DC and retrieves the log pages, it is expected that the
78 storage subsystems listed in the log pages are reachable on the
79 same interface where the DC was discovered.
80 For example, let's say a DC is discovered on interface ens102. Then
82 all the subsystems listed in the log pages retrieved from that DC
83 must be reachable on interface ens102. If this doesn't work, for
84 example you cannot "ping -I ens102 [storage-ip]", then the most
85 likely explanation is that arp proxy is not enabled on the switch
86 that the host is connected to on interface ens102. Whatever you do,
87 resist the temptation to manually set up the routing tables or to
88 add alternate routes going over a different interface than the one
89 where the DC is located. That simply won't work. Make sure arp
90 proxy is enabled on the switch first.
91 Setting routes won't work because, by default, stacd uses the
93 SO_BINDTODEVICE socket option when it connects to IOCs. This option
94 is used to force a socket connection to be made on a specific
95 interface instead of letting the routing tables decide where to
96 connect the socket. Even if you were to manually configure an
97 alternate route on a different interface, the connections (i.e.
98 host to IOC) will still be made on the interface where the DC was
99 discovered by stafd.
100 Defaults to false.
102 udev-rule=
104 Takes a string argument enabled or disabled. This option determines
105 whether nvme-cli's udev rule will be executed or ignored.
106 A udev rule gets installed with nvme-cli that tells the udev daemon
108 (udevd) to look for Asynchronous Event Notifications (AEN)
109 indicating a change of Discovery Log Page Entries (DPLE). The udev
110 rule is installed as:
111 /usr/lib/udev/rules.d/70-nvmf-autoconnect.rules
112 When an AEN is detected, udevd simply instructs systemd to start a
114 one-shot service that will retrieve the changed DPLEs and connect
115 to all the I/O Controllers (IOC) listed in the DPLEs. This is
116 basically the same as performing nvme-cli's "connect-all" command.
117 Unfortunately, stafd and stacd also perform the same operations
119 when an AEN is received. This results in a race condition between
120 udevd and stafd/stacd.
121 This is not really a problem. stafd and stacd are designed to
123 handle this type of race condition and will conclude, eventually,
124 that the connections succeeded. The only downside is that there may
125 be error messages printed to the syslog when the race condition
126 happens. These messages are printed by the kernel because two
127 processes are trying to connect to the same IOC at the same time.
128 One of them will be rejected by the kernel, but the other will
129 succeed.
130 The udev-rule option allows a user to disable nvme-cli's udev rule
132 so that udevd will not act on received AENs. Instead, only
133 stafd/stacd will be allowed to react to AENs and set up IOC
134 connections.
135 Defaults to enabled, which means that udevd and stafd/stacd will
137 react to AENs. It also means that the race condition will happen by
138 default and error messages will be printed to the syslog.
139 sticky-connections=
141 Keep existing connections to I/O controllers (IOC). Takes a string
142 argument enabled or disabled.
143 The parameter sticky-connections determines how stacd reacts to the
145 removal of an IOC Discovery Page Entry (DLPE) or the removal of a
146 controller= entry in /etc/stas/stacd.conf. In other words, whether
147 it should immediately disconnect from IOC when the DPLE/controller=
148 is removed, or whether it should maintain the connection.
149 Table 1. List of terms used in the following text:
151 ┌─────────────────┬───────────────────────────┐
152 │Term │ Description │
153 ├─────────────────┼───────────────────────────┤
154 │Manual Config │ Refers to manually adding │
155 │ │ entries to stacd.conf │
156 ├─────────────────┼───────────────────────────┤
157 │Automatic Config │ Refers to receiving │
158 │ │ configuration from a │
159 │ │ Discovery Controller (DC) │
160 │ │ as DLPEs │
161 ├─────────────────┼───────────────────────────┤
162 │External Config │ Refers to configuration │
163 │ │ done outside of the │
164 │ │ nvme-stas framework, for │
165 │ │ example using nvme-cli │
166 │ │ commands │
167 └─────────────────┴───────────────────────────┘
168 IOC connection creation. There are 3 ways to configure IOC
170 connections on a host:
171 1. Manual Config by adding controller= entries to the
173 [Controllers] section (see below).
174 2. Automatic Config received in the form of DLPEs from a remote
176 DC.
177 3. External Config using nvme-cli (e.g. "nvme connect")
179 Zoning and DLPEs. Zoning configuration is performed at Discovery
181 Controllers (DC). A zone is used to specify the list of IOC that a
182 host is allowed to access. The zone contains a list of hosts and
183 the IOC that these hosts can access. Users can add or remove IOC
184 and/or hosts from zones.
185 DCs notify hosts of zoning configuration changes by sending
187 Asynchronous Event Notifications (AEN) indicating a "Change of
188 Discovery Log Page (DLP)". The host uses these AENs as a trigger to
189 retrieve the new list of DLPEs by issuing a Get DLP command. This
190 happens in real time, which means that a host that was previously
191 connected to an IOC may suddenly be told that it is no longer
192 allowed to connect to that IOC and should disconnect from it.
193 IOC connection removal. There are 3 ways to remove controller
195 connections to an IOC:
196 1. Manual Config.
198 1. by adding blacklist= entries to the [Controllers] section
200 (see below).
201 2. by removing controller= entries from the [Controllers]
203 section.
204 2. Automatic Config. As explained above, changing zoning at a DC
207 will result in the host getting a new list of DLPEs. On DLPE
208 removal, the host should remove the connection to the IOC
209 matching that DLPE.
210 3. External Config using nvme-cli (e.g. "nvme disconnect" or "nvme
212 disconnect-all")
213 Some users may prefer for the IOC to be "sticky" and only be
214 removed manually (nvme-cli or blacklist=) or removed by a system
215 reboot. They don't want for IOC connections to be removed
216 unexpectedly on DLPE removal. This is where sticky-connections=
217 comes into play.
218 sticky-connections= tells stacd whether to keep connections to IOC
220 even if their DPLEs have been removed or the controller= entries in
221 stacd.conf have been removed.
222 With sticky-connections=disabled (default). stacd immediately
224 disconnects from a previously connected IOC if the response to a
225 Get DLP command no longer contains a DLPE matching that IOC or a
226 controller= entry in stacd.conf is removed.
227 Ongoing I/O transactions will be terminated immediately as well.
229 There is no way to tell what happens to the data being exchanged
230 when such an abrupt termination happens. If a host was in the
231 middle of writing to a storage subsystem, there is a good chance
232 that incomplete and potentially corrupt data will be left on the
233 remote storage.
234 NOTE. This mode implies that nvme-stas will only allow Manually
236 Configured or Automatically Configured IOC connections to exist.
237 Externally Configured connections using nvme-cli that do not match
238 any Manual Config (stacd.conf) or Automatic Config (DLPEs) will get
239 deleted immediately by stacd.
240 With sticky-connections=enabled. stacd does not disconnect from
242 IOCs when a DPLE is removed or a controller= entry is removed from
243 stacd.conf.
244 Instead, users can issue the nvme-cli command "nvme disconnect",
246 add a blacklist= entry to stacd.conf, or wait until the next system
247 reboot at which time all connections will be removed.
248 [Controllers] section
250 The following options are available in the [Controllers] section:
251 controller=
253 Controllers are specified with the controller option. This option
254 may be specified more than once to specify more than one
255 controller. The format is one line per Controller composed of a
256 series of fields separated by semi-colons as follows:
257 controller=transport=[trtype];traddr=[traddr];trsvcid=[trsvcid];host-traddr=[traddr],host-iface=[iface];nqn=[nqn]
259 Fields
262 transport=
263 This is a mandatory field that specifies the network fabric
264 being used for a NVMe-over-Fabrics network. Current trtype
265 values understood are:
266 Table 2. Transport type
268 ┌───────┬────────────────────────────┐
269 │trtype │ Definition │
270 ├───────┼────────────────────────────┤
271 │rdma │ The network fabric is an │
272 │ │ rdma network (RoCE, iWARP, │
273 │ │ Infiniband, basic rdma, │
274 │ │ etc) │
275 ├───────┼────────────────────────────┤
276 │fc │ The network fabric is a │
277 │ │ Fibre Channel network. │
278 ├───────┼────────────────────────────┤
279 │tcp │ The network fabric is a │
280 │ │ TCP/IP network. │
281 ├───────┼────────────────────────────┤
282 │loop │ Connect to a NVMe over │
283 │ │ Fabrics target on the │
284 │ │ local host │
285 └───────┴────────────────────────────┘
286 traddr=
288 This is a mandatory field that specifies the network
289 address of the Controller. For transports using IP
290 addressing (e.g. rdma) this should be an IP-based address
291 (ex. IPv4, IPv6). It could also be a resolvable host name
292 (e.g. localhost).
293 trsvcid=
295 This is an optional field that specifies the transport
296 service id. For transports using IP addressing (e.g. rdma,
297 tcp) this field is the port number.
298 Depending on the transport type, this field will default to
300 either 8009 or 4420 as follows.
301 UDP port 4420 and TCP port 4420 have been assigned by IANA
303 for use by NVMe over Fabrics. NVMe/RoCEv2 controllers use
304 UDP port 4420 by default. NVMe/iWARP controllers use TCP
305 port 4420 by default.
306 TCP port 4420 has been assigned for use by NVMe over
308 Fabrics and TCP port 8009 has been assigned by IANA for use
309 by NVMe over Fabrics discovery. TCP port 8009 is the
310 default TCP port for NVMe/TCP discovery controllers. There
311 is no default TCP port for NVMe/TCP I/O controllers, the
312 Transport Service Identifier (TRSVCID) field in the
313 Discovery Log Entry indicates the TCP port to use.
314 The TCP ports that may be used for NVMe/TCP I/O controllers
316 include TCP port 4420, and the Dynamic and/or Private TCP
317 ports (i.e., ports in the TCP port number range from 49152
318 to 65535). NVMe/TCP I/O controllers should not use TCP port
319 8009. TCP port 4420 shall not be used for both NVMe/iWARP
320 and NVMe/TCP at the same IP address on the same network.
321 Ref: IANA Service names port numbers[1]
323 nqn=
325 This is an optional field that specifies the Discovery
326 Controller's NVMe Qualified Name. If not specified, this
327 will default to the well-known DC NQN:
328 nqn.2014-08.org.nvmexpress.discovery.
329 host-traddr=
331 This is an optional field that specifies the network
332 address used on the host to connect to the Controller. For
333 TCP, this sets the source address on the socket.
334 host-iface=
336 This is an optional field that specifies the network
337 interface used on the host to connect to the Controller
338 (e.g. IP eth1, enp2s0, enx78e7d1ea46da). This forces the
339 connection to be made on a specific interface instead of
340 letting the system decide.
341 Examples:
342 controller = transport=tcp;traddr=localhost;trsvcid=8009
344 controller = transport=tcp;traddr=[2001:db8:::370:7334];host-iface=enp0s8
345 controller = transport=fc;traddr=nn-0x204600a098cbcac6:pn-0x204700a098cbcac6
346 blacklist=
350 Blacklisted controllers can be specified with the blacklist option.
351 Using mDNS to automatically discover and connect to controllers,
352 can result in unintentional connections being made. This keyword
353 allows configuring the controllers that should not be connected to
354 (whatever the reason may be).
355 The syntax is the same as for "controller", except that the key
357 host-traddr does not apply. Multiple blacklist keywords may appear
358 in the config file to specify more than 1 blacklisted controller.
359 Note 1: A minimal match approach is used to eliminate unwanted
361 controllers. That is, you do not need to specify all the parameters
362 to identify a controller. Just specifying the host-iface, for
363 example, can be used to blacklist all controllers on an interface.
364 Note 2: blacklist takes precedence over controller. A controller
366 specified by the controller keyword, can be eliminated by the
367 blacklist keyword.
368 Examples:
370 blacklist = transport=tcp;traddr=fe80::2c6e:dee7:857:26bb # Eliminate a specific address
372 blacklist = host-iface=enp0s8 # Eliminate everything on this interface
373
377 stacd(8)
378
380 1. IANA Service names port numbers
381 https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=nvme
382nvme-stas 1.1.5 STACD.CONF(5)