1fi_usnic(7) Libfabric v1.6.1 fi_usnic(7)
2
6 fi_usnic - The usNIC Fabric Provider
7
9 The usnic provider is designed to run over the Cisco VIC (virtualized
10 NIC) hardware on Cisco UCS servers. It utilizes the Cisco usNIC
11 (userspace NIC) capabilities of the VIC to enable ultra low latency and
12 other offload capabilities on Ethernet networks.
13
15 · The usnic libfabric provider requires the use of the "libnl" library.
16 · There are two versions of libnl generally available: v1 and v3; the
18 usnic provider can use either version.
19 · If you are building libfabric/the usnic provider from source, you
21 will need to have the libnl header files available (e.g., if you
22 are installing libnl from RPM or other packaging system, install
23 the "-devel" versions of the package).
24 · If you have libnl (either v1 or v3) installed in a non-standard
26 location (e.g., not in /usr/lib or /usr/lib64), you may need to
27 tell libfabric's configure where to find libnl via the
28 --with-libnl=DIR command line option (where DIR is the installation
29 prefix of the libnl package).
30 · The most common way to use the libfabric usnic provider is via an MPI
32 implementation that uses libfabric (and the usnic provider) as a
33 lower layer transport. MPI applications do not need to know anything
34 about libfabric or usnic in this use case -- the MPI implementation
35 hides all these details from the application.
36 · If you are writing applications directly to the libfabric API:
38 · FI_EP_DGRAM endpoints are the best supported method of utilizing
40 the usNIC interface. Specifically, the FI_EP_DGRAM endpoint type
41 has been extensively tested as the underlying layer for Open MPI's
42 usnic BTL.
43 · FI_EP_MSG and FI_EP_RDM endpoints are implemented, but are only
45 lightly tested. It is likely that there are still some bugs in
46 these endpoint types. In particular, there are known bugs in RDM
47 support in the presence of congestion or packet loss (issue 1621).
48 RMA is not yet supported.
49 · fi_provider(7) lists requirements for all providers. The following
51 limitations exist in the usnic provider:
52 · multicast operations are not supported on FI_EP_DGRAM and
54 FI_EP_RDM endpoints.
55 · FI_EP_MSG endpoints only support connect, accept, and getname CM
57 operations.
58 · Passive endpoints only support listen, setname, and getname CM
60 operations.
61 · FI_EP_DGRAM endpoints support fi_sendmsg() and fi_recvmsg(), but
63 some flags are ignored. fi_sendmsg() supports FI_INJECT and
64 FI_COMPLETION. fi_recvmsg() supports FI_MORE.
65 · Address vectors only support FI_AV_MAP.
67 · No counters are supported.
69 · The tag matching interface is not supported.
71 · FI_MSG_PREFIX is only supported on FI_EP_DGRAM and usage is lim‐
73 ited to releases 1.1 and beyond.
74 · fi_control with FI_GETWAIT may only be used on CQs that have been
76 bound to an endpoint. If fi_control is used on an unbound CQ, it
77 will return -FI_EOPBADSTATE.
78 · There is limited support for data returned as part of an erro‐
80 neous asynchronous operation. EQs will return error data for CM
81 operations, CQs do not support returning error data.
82 · As of 1.5, usNIC supports fi_mr_regv, and fi_mr_regattr. Support
84 is limited to a single iov.
85 · Atomic operations are not supported.
87 · Resource management is not supported. The application is responsi‐
89 ble for resource protection.
90 · The usnic libfabric provider supports extensions that provide
92 information and functionality beyond the standard libfabric inter‐
93 face. See the "USNIC EXTENSIONS" section, below.
94
96 The usnic libfabric provider exports extensions for additional VIC,
97 usNIC, and Ethernet capabilities not provided by the standard libfabric
98 interface.
99 These extensions are available via the "fi_ext_usnic.h" header file.
101 Fabric Extension: getinfo
103 Version 2 of the "fabric getinfo" extension was introduced in Libfabric
104 release v1.3.0 and can be used to retrieve IP and SR-IOV information
105 about a usNIC device obtained from the fi_getinfo(3) function.
106 The "fabric getinfo" extension is obtained by calling fi_open_ops and
108 requesting FI_USNIC_FABRIC_OPS_1 to get the usNIC fabric extension
109 operations. The getinfo function accepts a version parameter that can
110 be used to select different versions of the extension. The information
111 returned by the "fabric getinfo" extension is accessible through a
112 fi_usnic_info struct that uses a version tagged union. The accessed
113 union member must correspond with the requested version. It is recom‐
114 mended that applications explicitly request a version rather than using
115 the header provided FI_EXT_USNIC_INFO_VERSION. Although there is a
116 version 1 of the extension, its use is discouraged, and it may not be
117 available in future releases.
118 Compatibility issues
120 The addition of version 2 of the extension caused an alignment issue
121 that could lead to invalid data in the v1 portion of the structure.
122 This means that the alignment difference manifests when an application
123 using v1 of the extension is compiled with Libfabric v1.1.x or v1.2.x,
124 but then runs with Libfabric.so that is v1.3.x or higher (and vice
125 versa).
126 The v1.4.0 release of Libfabric introduced a padding field to explic‐
128 itly maintain compatibility with the v1.3.0 release. If the issue is
129 encountered, then it is recommended that you upgrade to a release con‐
130 taining version 2 of the extension, or recompile with a patched version
131 of an older release.
132 #include <rdma/fi_ext_usnic.h>
134 struct fi_usnic_info {
136 uint32_t ui_version;
137 uint8_t ui_pad0[4];
138 union {
139 struct fi_usnic_info_v1 v1;
140 struct fi_usnic_info_v2 v2;
141 } ui;
142 } __attribute__((packed));
143 int getinfo(uint32_t version, struct fid_fabric *fabric,
145 struct fi_usnic_info *info);
146 version : Version of getinfo to be used
148 fabric : Fabric descriptor
150 info : Upon successful return, this parameter will contain information
152 about the fabric.
153 · Version 2
155 struct fi_usnic_cap {
157 const char *uc_capability;
158 int uc_present;
159 } __attribute__((packed));
160 struct fi_usnic_info_v2 {
162 uint32_t ui_link_speed;
163 uint32_t ui_netmask_be;
164 char ui_ifname[IFNAMSIZ];
165 unsigned ui_num_vf;
166 unsigned ui_qp_per_vf;
167 unsigned ui_cq_per_vf;
168 char ui_devname[FI_EXT_USNIC_MAX_DEVNAME];
170 uint8_t ui_mac_addr[6];
171 uint8_t ui_pad0[2];
173 uint32_t ui_ipaddr_be;
175 uint32_t ui_prefixlen;
176 uint32_t ui_mtu;
177 uint8_t ui_link_up;
178 uint8_t ui_pad1[3];
180 uint32_t ui_vendor_id;
182 uint32_t ui_vendor_part_id;
183 uint32_t ui_device_id;
184 char ui_firmware[64];
185 unsigned ui_intr_per_vf;
187 unsigned ui_max_cq;
188 unsigned ui_max_qp;
189 unsigned ui_max_cqe;
191 unsigned ui_max_send_credits;
192 unsigned ui_max_recv_credits;
193 const char *ui_nicname;
195 const char *ui_pid;
196 struct fi_usnic_cap **ui_caps;
198 } __attribute__((packed));
199 · Version 1
201 struct fi_usnic_info_v1 {
203 uint32_t ui_link_speed;
204 uint32_t ui_netmask_be;
205 char ui_ifname[IFNAMSIZ];
206 uint32_t ui_num_vf;
208 uint32_t ui_qp_per_vf;
209 uint32_t ui_cq_per_vf;
210 } __attribute__((packed));
211 Version 1 of the "fabric getinfo" extension can be used by explicitly
213 requesting it in the call to getinfo and accessing the v1 portion of
214 the fi_usnic_info.ui union. Use of version 1 is not recommended and it
215 may be removed from future releases.
216 The following is an example of how to utilize version 2 of the usnic
218 "fabric getinfo" extension.
219 #include <stdio.h>
221 #include <rdma/fabric.h>
222 /* The usNIC extensions are all in the
224 rdma/fi_ext_usnic.h header */
225 #include <rdma/fi_ext_usnic.h>
226 int main(int argc, char *argv[]) {
228 struct fi_info *info;
229 struct fi_info *info_list;
230 struct fi_info hints = {0};
231 struct fi_ep_attr ep_attr = {0};
232 struct fi_fabric_attr fabric_attr = {0};
233 fabric_attr.prov_name = "usnic";
235 ep_attr.type = FI_EP_DGRAM;
236 hints.caps = FI_MSG;
238 hints.mode = FI_LOCAL_MR | FI_MSG_PREFIX;
239 hints.addr_format = FI_SOCKADDR;
240 hints.ep_attr = &ep_attr;
241 hints.fabric_attr = &fabric_attr;
242 /* Find all usnic providers */
244 fi_getinfo(FI_VERSION(1, 0), NULL, 0, 0, &hints, &info_list);
245 for (info = info_list; NULL != info; info = info->next) {
247 /* Open the fabric on the interface */
248 struct fid_fabric *fabric;
249 fi_fabric(info->fabric_attr, &fabric, NULL);
250 /* Pass FI_USNIC_FABRIC_OPS_1 to get usnic ops
252 on the fabric */
253 struct fi_usnic_ops_fabric *usnic_fabric_ops;
254 fi_open_ops(&fabric->fid, FI_USNIC_FABRIC_OPS_1, 0,
255 (void **) &usnic_fabric_ops, NULL);
256 /* Now use the returned usnic ops structure to call
258 usnic extensions. The following extension queries
259 some IP and SR-IOV characteristics about the
260 usNIC device. */
261 struct fi_usnic_info usnic_info;
262 /* Explicitly request version 2. */
264 usnic_fabric_ops->getinfo(2, fabric, &usnic_info);
265 printf("Fabric interface %s is %s:\n"
267 "\tNetmask: 0x%08x\n\tLink speed: %d\n"
268 "\tSR-IOV VFs: %d\n\tQPs per SR-IOV VF: %d\n"
269 "\tCQs per SR-IOV VF: %d\n",
270 info->fabric_attr->name,
271 usnic_info.ui.v2.ui_ifname,
272 usnic_info.ui.v2.ui_netmask_be,
273 usnic_info.ui.v2.ui_link_speed,
274 usnic_info.ui.v2.ui_num_vf,
275 usnic_info.ui.v2.ui_qp_per_vf,
276 usnic_info.ui.v2.ui_cq_per_vf);
277 fi_close(&fabric->fid);
279 }
280 fi_freeinfo(info_list);
282 return 0;
283 }
284 Adress Vector Extension: get_distance
286 The "address vector get_distance" extension was introduced in Libfabric
287 release v1.0.0 and can be used to retrieve the network distance of an
288 address.
289 The "get_distance" extension is obtained by calling fi_open_ops and
291 requesting FI_USNIC_AV_OPS_1 to get the usNIC address vector extension
292 operations.
293 int get_distance(struct fid_av *av, void *addr, int *metric);
295 av : Address vector
297 addr : Destination address
299 metric : On output this will contain -1 if the destination host is
301 unreachable, 0 is the destination host is locally connected, and 1 oth‐
302 erwise.
303 See fi_ext_usnic.h for more details.
305
307 New naming convention for fabric/domain starting with libfabric v1.4
308 The release of libfabric v1.4 introduced a new naming convention for
309 fabric and domain. However the usNIC provider remains backward compat‐
310 ible with applications supporting the old scheme and decides which one
311 to use based on the version passed to fi_getinfo:
312 · When FI_VERSION(1,4) or higher is used:
314 · fabric name is the network address with the CIDR notation (i.e.,
316 a.b.c.d/e)
317 · domain name is the usNIC Linux interface name (i.e., usnic_X)
319 · When a lower version number is used, like FI_VERSION(1, 3), it fol‐
321 lows the same behavior the usNIC provider exhibited in libfabric <=
322 v1.3:
323 · fabric name is the usNIC Linux interface name (i.e., usnic_X)
325 · domain name is NULL
327
329 fabric(7), fi_open_ops(3), fi_provider(7),
330
332 OpenFabrics.
333Libfabric Programmer's Manual 2017-12-01 fi_usnic(7)