1fi_usnic(7) Libfabric v1.17.0 fi_usnic(7)
2
6 fi_usnic - The usNIC Fabric Provider
7
9 The usnic provider is designed to run over the Cisco VIC (virtualized
10 NIC) hardware on Cisco UCS servers. It utilizes the Cisco usNIC
11 (userspace NIC) capabilities of the VIC to enable ultra low latency and
12 other offload capabilities on Ethernet networks.
13
15 • The usnic libfabric provider requires the use of the “libnl” library.
16 • There are two versions of libnl generally available: v1 and v3; the
18 usnic provider can use either version.
19 • If you are building libfabric/the usnic provider from source, you
21 will need to have the libnl header files available (e.g., if you
22 are installing libnl from RPM or other packaging system, install
23 the “-devel” versions of the package).
24 • If you have libnl (either v1 or v3) installed in a non-standard lo‐
26 cation (e.g., not in /usr/lib or /usr/lib64), you may need to tell
27 libfabric’s configure where to find libnl via the --with-libnl=DIR
28 command line option (where DIR is the installation prefix of the
29 libnl package).
30 • The most common way to use the libfabric usnic provider is via an MPI
32 implementation that uses libfabric (and the usnic provider) as a low‐
33 er layer transport. MPI applications do not need to know anything
34 about libfabric or usnic in this use case – the MPI implementation
35 hides all these details from the application.
36 • If you are writing applications directly to the libfabric API:
38 • FI_EP_DGRAM endpoints are the best supported method of utilizing
40 the usNIC interface. Specifically, the FI_EP_DGRAM endpoint type
41 has been extensively tested as the underlying layer for Open MPI’s
42 usnic BTL.
43 • FI_EP_MSG and FI_EP_RDM endpoints are implemented, but are only
45 lightly tested. It is likely that there are still some bugs in
46 these endpoint types. In particular, there are known bugs in RDM
47 support in the presence of congestion or packet loss (issue 1621).
48 RMA is not yet supported.
49 • fi_provider(7) lists requirements for all providers. The following
51 limitations exist in the usnic provider:
52 • multicast operations are not supported on FI_EP_DGRAM and
54 FI_EP_RDM endpoints.
55 • FI_EP_MSG endpoints only support connect, accept, and getname CM
57 operations.
58 • Passive endpoints only support listen, setname, and getname CM
60 operations.
61 • FI_EP_DGRAM endpoints support fi_sendmsg() and fi_recvmsg(), but
63 some flags are ignored. fi_sendmsg() supports FI_INJECT and
64 FI_COMPLETION. fi_recvmsg() supports FI_MORE.
65 • Address vectors only support FI_AV_MAP.
67 • No counters are supported.
69 • The tag matching interface is not supported.
71 • FI_MSG_PREFIX is only supported on FI_EP_DGRAM and usage is lim‐
73 ited to releases 1.1 and beyond.
74 • fi_control with FI_GETWAIT may only be used on CQs that have been
76 bound to an endpoint. If fi_control is used on an unbound CQ, it
77 will return -FI_EOPBADSTATE.
78 • There is limited support for data returned as part of an erro‐
80 neous asynchronous operation. EQs will return error data for CM
81 operations, CQs do not support returning error data.
82 • As of 1.5, usNIC supports fi_mr_regv, and fi_mr_regattr. Support
84 is limited to a single iov.
85 • Atomic operations are not supported.
87 • Resource management is not supported. The application is responsi‐
89 ble for resource protection.
90 • The usnic libfabric provider supports extensions that provide in‐
92 formation and functionality beyond the standard libfabric inter‐
93 face. See the “USNIC EXTENSIONS” section, below.
94
96 The usnic libfabric provider exports extensions for additional VIC, us‐
97 NIC, and Ethernet capabilities not provided by the standard libfabric
98 interface.
99 These extensions are available via the “fi_ext_usnic.h” header file.
101 Fabric Extension: getinfo
103 Version 2 of the “fabric getinfo” extension was introduced in Libfabric
104 release v1.3.0 and can be used to retrieve IP and SR-IOV information
105 about a usNIC device obtained from the fi_getinfo(3) function.
106 The “fabric getinfo” extension is obtained by calling fi_open_ops and
108 requesting FI_USNIC_FABRIC_OPS_1 to get the usNIC fabric extension op‐
109 erations. The getinfo function accepts a version parameter that can be
110 used to select different versions of the extension. The information
111 returned by the “fabric getinfo” extension is accessible through a
112 fi_usnic_info struct that uses a version tagged union. The accessed
113 union member must correspond with the requested version. It is recom‐
114 mended that applications explicitly request a version rather than using
115 the header provided FI_EXT_USNIC_INFO_VERSION. Although there is a
116 version 1 of the extension, its use is discouraged, and it may not be
117 available in future releases.
118 Compatibility issues
120 The addition of version 2 of the extension caused an alignment issue
121 that could lead to invalid data in the v1 portion of the structure.
122 This means that the alignment difference manifests when an application
123 using v1 of the extension is compiled with Libfabric v1.1.x or v1.2.x,
124 but then runs with Libfabric.so that is v1.3.x or higher (and vice ver‐
125 sa).
126 The v1.4.0 release of Libfabric introduced a padding field to explicit‐
128 ly maintain compatibility with the v1.3.0 release. If the issue is en‐
129 countered, then it is recommended that you upgrade to a release con‐
130 taining version 2 of the extension, or recompile with a patched version
131 of an older release.
132 #include <rdma/fi_ext_usnic.h>
134 struct fi_usnic_info {
136 uint32_t ui_version;
137 uint8_t ui_pad0[4];
138 union {
139 struct fi_usnic_info_v1 v1;
140 struct fi_usnic_info_v2 v2;
141 } ui;
142 } __attribute__((packed));
143 int getinfo(uint32_t version, struct fid_fabric *fabric,
145 struct fi_usnic_info *info);
146 version
148 Version of getinfo to be used
149 fabric Fabric descriptor
151 info Upon successful return, this parameter will contain information
153 about the fabric.
154 • Version 2
156 struct fi_usnic_cap {
158 const char *uc_capability;
159 int uc_present;
160 } __attribute__((packed));
161 struct fi_usnic_info_v2 {
163 uint32_t ui_link_speed;
164 uint32_t ui_netmask_be;
165 char ui_ifname[IFNAMSIZ];
166 unsigned ui_num_vf;
167 unsigned ui_qp_per_vf;
168 unsigned ui_cq_per_vf;
169 char ui_devname[FI_EXT_USNIC_MAX_DEVNAME];
171 uint8_t ui_mac_addr[6];
172 uint8_t ui_pad0[2];
174 uint32_t ui_ipaddr_be;
176 uint32_t ui_prefixlen;
177 uint32_t ui_mtu;
178 uint8_t ui_link_up;
179 uint8_t ui_pad1[3];
181 uint32_t ui_vendor_id;
183 uint32_t ui_vendor_part_id;
184 uint32_t ui_device_id;
185 char ui_firmware[64];
186 unsigned ui_intr_per_vf;
188 unsigned ui_max_cq;
189 unsigned ui_max_qp;
190 unsigned ui_max_cqe;
192 unsigned ui_max_send_credits;
193 unsigned ui_max_recv_credits;
194 const char *ui_nicname;
196 const char *ui_pid;
197 struct fi_usnic_cap **ui_caps;
199 } __attribute__((packed));
200 • Version 1
202 struct fi_usnic_info_v1 {
204 uint32_t ui_link_speed;
205 uint32_t ui_netmask_be;
206 char ui_ifname[IFNAMSIZ];
207 uint32_t ui_num_vf;
209 uint32_t ui_qp_per_vf;
210 uint32_t ui_cq_per_vf;
211 } __attribute__((packed));
212 Version 1 of the “fabric getinfo” extension can be used by explicitly
214 requesting it in the call to getinfo and accessing the v1 portion of
215 the fi_usnic_info.ui union. Use of version 1 is not recommended and it
216 may be removed from future releases.
217 The following is an example of how to utilize version 2 of the usnic
219 “fabric getinfo” extension.
220 #include <stdio.h>
222 #include <rdma/fabric.h>
223 /* The usNIC extensions are all in the
225 rdma/fi_ext_usnic.h header */
226 #include <rdma/fi_ext_usnic.h>
227 int main(int argc, char *argv[]) {
229 struct fi_info *info;
230 struct fi_info *info_list;
231 struct fi_info hints = {0};
232 struct fi_ep_attr ep_attr = {0};
233 struct fi_fabric_attr fabric_attr = {0};
234 fabric_attr.prov_name = "usnic";
236 ep_attr.type = FI_EP_DGRAM;
237 hints.caps = FI_MSG;
239 hints.mode = FI_LOCAL_MR | FI_MSG_PREFIX;
240 hints.addr_format = FI_SOCKADDR;
241 hints.ep_attr = &ep_attr;
242 hints.fabric_attr = &fabric_attr;
243 /* Find all usnic providers */
245 fi_getinfo(FI_VERSION(1, 0), NULL, 0, 0, &hints, &info_list);
246 for (info = info_list; NULL != info; info = info->next) {
248 /* Open the fabric on the interface */
249 struct fid_fabric *fabric;
250 fi_fabric(info->fabric_attr, &fabric, NULL);
251 /* Pass FI_USNIC_FABRIC_OPS_1 to get usnic ops
253 on the fabric */
254 struct fi_usnic_ops_fabric *usnic_fabric_ops;
255 fi_open_ops(&fabric->fid, FI_USNIC_FABRIC_OPS_1, 0,
256 (void **) &usnic_fabric_ops, NULL);
257 /* Now use the returned usnic ops structure to call
259 usnic extensions. The following extension queries
260 some IP and SR-IOV characteristics about the
261 usNIC device. */
262 struct fi_usnic_info usnic_info;
263 /* Explicitly request version 2. */
265 usnic_fabric_ops->getinfo(2, fabric, &usnic_info);
266 printf("Fabric interface %s is %s:\n"
268 "\tNetmask: 0x%08x\n\tLink speed: %d\n"
269 "\tSR-IOV VFs: %d\n\tQPs per SR-IOV VF: %d\n"
270 "\tCQs per SR-IOV VF: %d\n",
271 info->fabric_attr->name,
272 usnic_info.ui.v2.ui_ifname,
273 usnic_info.ui.v2.ui_netmask_be,
274 usnic_info.ui.v2.ui_link_speed,
275 usnic_info.ui.v2.ui_num_vf,
276 usnic_info.ui.v2.ui_qp_per_vf,
277 usnic_info.ui.v2.ui_cq_per_vf);
278 fi_close(&fabric->fid);
280 }
281 fi_freeinfo(info_list);
283 return 0;
284 }
285 Adress Vector Extension: get_distance
287 The “address vector get_distance” extension was introduced in Libfabric
288 release v1.0.0 and can be used to retrieve the network distance of an
289 address.
290 The “get_distance” extension is obtained by calling fi_open_ops and re‐
292 questing FI_USNIC_AV_OPS_1 to get the usNIC address vector extension
293 operations.
294 int get_distance(struct fid_av *av, void *addr, int *metric);
296 av Address vector
298 addr Destination address
300 metric On output this will contain -1 if the destination host is un‐
302 reachable, 0 is the destination host is locally connected, and 1
303 otherwise.
304 See fi_ext_usnic.h for more details.
306
308 New naming convention for fabric/domain starting with libfabric v1.4
309 The release of libfabric v1.4 introduced a new naming convention for
310 fabric and domain. However the usNIC provider remains backward compat‐
311 ible with applications supporting the old scheme and decides which one
312 to use based on the version passed to fi_getinfo:
313 • When FI_VERSION(1,4) or higher is used:
315 • fabric name is the network address with the CIDR notation (i.e.,
317 a.b.c.d/e)
318 • domain name is the usNIC Linux interface name (i.e., usnic_X)
320 • When a lower version number is used, like FI_VERSION(1, 3), it fol‐
322 lows the same behavior the usNIC provider exhibited in libfabric <=
323 v1.3:
324 • fabric name is the usNIC Linux interface name (i.e., usnic_X)
326 • domain name is NULL
328
330 fabric(7), fi_open_ops(3), fi_provider(7),
331
333 OpenFabrics.
334Libfabric Programmer’s Manual 2022-12-11 fi_usnic(7)