1fi_usnic(7)                    Libfabric v1.17.0                   fi_usnic(7)
2
3
4

NAME

6       fi_usnic - The usNIC Fabric Provider
7

OVERVIEW

9       The  usnic  provider is designed to run over the Cisco VIC (virtualized
10       NIC) hardware on Cisco  UCS  servers.   It  utilizes  the  Cisco  usNIC
11       (userspace NIC) capabilities of the VIC to enable ultra low latency and
12       other offload capabilities on Ethernet networks.
13

RELEASE NOTES

15       • The usnic libfabric provider requires the use of the “libnl” library.
16
17         • There are two versions of libnl generally available: v1 and v3; the
18           usnic provider can use either version.
19
20         • If  you  are building libfabric/the usnic provider from source, you
21           will need to have the libnl header files available  (e.g.,  if  you
22           are  installing  libnl  from RPM or other packaging system, install
23           the “-devel” versions of the package).
24
25         • If you have libnl (either v1 or v3) installed in a non-standard lo‐
26           cation  (e.g., not in /usr/lib or /usr/lib64), you may need to tell
27           libfabric’s configure where to find libnl via the  --with-libnl=DIR
28           command  line  option  (where DIR is the installation prefix of the
29           libnl package).
30
31       • The most common way to use the libfabric usnic provider is via an MPI
32         implementation that uses libfabric (and the usnic provider) as a low‐
33         er layer transport.  MPI applications do not need  to  know  anything
34         about  libfabric  or  usnic in this use case – the MPI implementation
35         hides all these details from the application.
36
37       • If you are writing applications directly to the libfabric API:
38
39FI_EP_DGRAM endpoints are the best supported  method  of  utilizing
40           the  usNIC  interface.  Specifically, the FI_EP_DGRAM endpoint type
41           has been extensively tested as the underlying layer for Open  MPI’s
42           usnic BTL.
43
44FI_EP_MSG  and  FI_EP_RDM  endpoints  are implemented, but are only
45           lightly tested.  It is likely that there are  still  some  bugs  in
46           these  endpoint  types.  In particular, there are known bugs in RDM
47           support in the presence of congestion or packet loss (issue  1621).
48           RMA is not yet supported.
49
50fi_provider(7) lists requirements for all providers.  The following
51           limitations exist in the usnic provider:
52
53           • multicast  operations  are  not  supported  on  FI_EP_DGRAM   and
54             FI_EP_RDM endpoints.
55
56FI_EP_MSG  endpoints only support connect, accept, and getname CM
57             operations.
58
59           • Passive endpoints only support listen, setname,  and  getname  CM
60             operations.
61
62FI_EP_DGRAM  endpoints support fi_sendmsg() and fi_recvmsg(), but
63             some flags are  ignored.   fi_sendmsg()  supports  FI_INJECT  and
64             FI_COMPLETION.  fi_recvmsg() supports FI_MORE.
65
66           • Address vectors only support FI_AV_MAP.
67
68           • No counters are supported.
69
70           • The tag matching interface is not supported.
71
72FI_MSG_PREFIX  is only supported on FI_EP_DGRAM and usage is lim‐
73             ited to releases 1.1 and beyond.
74
75           • fi_control with FI_GETWAIT may only be used on CQs that have been
76             bound to an endpoint.  If fi_control is used on an unbound CQ, it
77             will return -FI_EOPBADSTATE.
78
79           • There is limited support for data returned as part  of  an  erro‐
80             neous  asynchronous operation.  EQs will return error data for CM
81             operations, CQs do not support returning error data.
82
83           • As of 1.5, usNIC supports fi_mr_regv, and fi_mr_regattr.  Support
84             is limited to a single iov.
85
86           • Atomic operations are not supported.
87
88         • Resource management is not supported.  The application is responsi‐
89           ble for resource protection.
90
91         • The usnic libfabric provider supports extensions that  provide  in‐
92           formation  and  functionality  beyond the standard libfabric inter‐
93           face.  See the “USNIC EXTENSIONS” section, below.
94

USNIC EXTENSIONS

96       The usnic libfabric provider exports extensions for additional VIC, us‐
97       NIC,  and  Ethernet capabilities not provided by the standard libfabric
98       interface.
99
100       These extensions are available via the “fi_ext_usnic.h” header file.
101
102   Fabric Extension: getinfo
103       Version 2 of the “fabric getinfo” extension was introduced in Libfabric
104       release  v1.3.0  and  can be used to retrieve IP and SR-IOV information
105       about a usNIC device obtained from the fi_getinfo(3) function.
106
107       The “fabric getinfo” extension is obtained by calling  fi_open_ops  and
108       requesting  FI_USNIC_FABRIC_OPS_1 to get the usNIC fabric extension op‐
109       erations.  The getinfo function accepts a version parameter that can be
110       used  to  select  different versions of the extension.  The information
111       returned by the “fabric getinfo”  extension  is  accessible  through  a
112       fi_usnic_info  struct  that  uses a version tagged union.  The accessed
113       union member must correspond with the requested version.  It is  recom‐
114       mended that applications explicitly request a version rather than using
115       the header provided FI_EXT_USNIC_INFO_VERSION.   Although  there  is  a
116       version  1  of the extension, its use is discouraged, and it may not be
117       available in future releases.
118
119   Compatibility issues
120       The addition of version 2 of the extension caused  an  alignment  issue
121       that  could  lead  to  invalid data in the v1 portion of the structure.
122       This means that the alignment difference manifests when an  application
123       using  v1 of the extension is compiled with Libfabric v1.1.x or v1.2.x,
124       but then runs with Libfabric.so that is v1.3.x or higher (and vice ver‐
125       sa).
126
127       The v1.4.0 release of Libfabric introduced a padding field to explicit‐
128       ly maintain compatibility with the v1.3.0 release.  If the issue is en‐
129       countered,  then  it  is recommended that you upgrade to a release con‐
130       taining version 2 of the extension, or recompile with a patched version
131       of an older release.
132
133              #include <rdma/fi_ext_usnic.h>
134
135              struct fi_usnic_info {
136                  uint32_t ui_version;
137                  uint8_t ui_pad0[4];
138                  union {
139                      struct fi_usnic_info_v1 v1;
140                      struct fi_usnic_info_v2 v2;
141                  } ui;
142              } __attribute__((packed));
143
144              int getinfo(uint32_t version, struct fid_fabric *fabric,
145                      struct fi_usnic_info *info);
146
147       version
148              Version of getinfo to be used
149
150       fabric Fabric descriptor
151
152       info   Upon  successful return, this parameter will contain information
153              about the fabric.
154
155       • Version 2
156
157         struct fi_usnic_cap {
158             const char *uc_capability;
159             int uc_present;
160         } __attribute__((packed));
161
162         struct fi_usnic_info_v2 {
163             uint32_t        ui_link_speed;
164             uint32_t        ui_netmask_be;
165             char            ui_ifname[IFNAMSIZ];
166             unsigned        ui_num_vf;
167             unsigned        ui_qp_per_vf;
168             unsigned        ui_cq_per_vf;
169
170             char            ui_devname[FI_EXT_USNIC_MAX_DEVNAME];
171             uint8_t         ui_mac_addr[6];
172
173             uint8_t         ui_pad0[2];
174
175             uint32_t        ui_ipaddr_be;
176             uint32_t        ui_prefixlen;
177             uint32_t        ui_mtu;
178             uint8_t         ui_link_up;
179
180             uint8_t         ui_pad1[3];
181
182             uint32_t        ui_vendor_id;
183             uint32_t        ui_vendor_part_id;
184             uint32_t        ui_device_id;
185             char            ui_firmware[64];
186
187             unsigned        ui_intr_per_vf;
188             unsigned        ui_max_cq;
189             unsigned        ui_max_qp;
190
191             unsigned        ui_max_cqe;
192             unsigned        ui_max_send_credits;
193             unsigned        ui_max_recv_credits;
194
195             const char      *ui_nicname;
196             const char      *ui_pid;
197
198             struct fi_usnic_cap **ui_caps;
199         } __attribute__((packed));
200
201       • Version 1
202
203         struct fi_usnic_info_v1 {
204             uint32_t ui_link_speed;
205             uint32_t ui_netmask_be;
206             char ui_ifname[IFNAMSIZ];
207
208             uint32_t ui_num_vf;
209             uint32_t ui_qp_per_vf;
210             uint32_t ui_cq_per_vf;
211         } __attribute__((packed));
212
213       Version 1 of the “fabric getinfo” extension can be used  by  explicitly
214       requesting  it  in  the call to getinfo and accessing the v1 portion of
215       the fi_usnic_info.ui union.  Use of version 1 is not recommended and it
216       may be removed from future releases.
217
218       The  following  is  an example of how to utilize version 2 of the usnic
219       “fabric getinfo” extension.
220
221              #include <stdio.h>
222              #include <rdma/fabric.h>
223
224              /* The usNIC extensions are all in the
225                 rdma/fi_ext_usnic.h header */
226              #include <rdma/fi_ext_usnic.h>
227
228              int main(int argc, char *argv[]) {
229                  struct fi_info *info;
230                  struct fi_info *info_list;
231                  struct fi_info hints = {0};
232                  struct fi_ep_attr ep_attr = {0};
233                  struct fi_fabric_attr fabric_attr = {0};
234
235                  fabric_attr.prov_name = "usnic";
236                  ep_attr.type = FI_EP_DGRAM;
237
238                  hints.caps = FI_MSG;
239                  hints.mode = FI_LOCAL_MR | FI_MSG_PREFIX;
240                  hints.addr_format = FI_SOCKADDR;
241                  hints.ep_attr = &ep_attr;
242                  hints.fabric_attr = &fabric_attr;
243
244                  /* Find all usnic providers */
245                  fi_getinfo(FI_VERSION(1, 0), NULL, 0, 0, &hints, &info_list);
246
247                  for (info = info_list; NULL != info; info = info->next) {
248                      /* Open the fabric on the interface */
249                      struct fid_fabric *fabric;
250                      fi_fabric(info->fabric_attr, &fabric, NULL);
251
252                      /* Pass FI_USNIC_FABRIC_OPS_1 to get usnic ops
253                         on the fabric */
254                      struct fi_usnic_ops_fabric *usnic_fabric_ops;
255                      fi_open_ops(&fabric->fid, FI_USNIC_FABRIC_OPS_1, 0,
256                              (void **) &usnic_fabric_ops, NULL);
257
258                      /* Now use the returned usnic ops structure to call
259                         usnic extensions.  The following extension queries
260                         some IP and SR-IOV characteristics about the
261                         usNIC device. */
262                      struct fi_usnic_info usnic_info;
263
264                      /* Explicitly request version 2. */
265                      usnic_fabric_ops->getinfo(2, fabric, &usnic_info);
266
267                      printf("Fabric interface %s is %s:\n"
268                             "\tNetmask:  0x%08x\n\tLink speed: %d\n"
269                             "\tSR-IOV VFs: %d\n\tQPs per SR-IOV VF: %d\n"
270                             "\tCQs per SR-IOV VF: %d\n",
271                             info->fabric_attr->name,
272                             usnic_info.ui.v2.ui_ifname,
273                             usnic_info.ui.v2.ui_netmask_be,
274                             usnic_info.ui.v2.ui_link_speed,
275                             usnic_info.ui.v2.ui_num_vf,
276                             usnic_info.ui.v2.ui_qp_per_vf,
277                             usnic_info.ui.v2.ui_cq_per_vf);
278
279                      fi_close(&fabric->fid);
280                  }
281
282                  fi_freeinfo(info_list);
283                  return 0;
284              }
285
286   Adress Vector Extension: get_distance
287       The “address vector get_distance” extension was introduced in Libfabric
288       release  v1.0.0  and can be used to retrieve the network distance of an
289       address.
290
291       The “get_distance” extension is obtained by calling fi_open_ops and re‐
292       questing  FI_USNIC_AV_OPS_1  to  get the usNIC address vector extension
293       operations.
294
295              int get_distance(struct fid_av *av, void *addr, int *metric);
296
297       av     Address vector
298
299       addr   Destination address
300
301       metric On output this will contain -1 if the destination  host  is  un‐
302              reachable, 0 is the destination host is locally connected, and 1
303              otherwise.
304
305       See fi_ext_usnic.h for more details.
306

VERSION DIFFERENCES

308   New naming convention for fabric/domain starting with libfabric v1.4
309       The release of libfabric v1.4 introduced a new  naming  convention  for
310       fabric and domain.  However the usNIC provider remains backward compat‐
311       ible with applications supporting the old scheme and decides which  one
312       to use based on the version passed to fi_getinfo:
313
314       • When FI_VERSION(1,4) or higher is used:
315
316         • fabric  name  is  the network address with the CIDR notation (i.e.,
317           a.b.c.d/e)
318
319         • domain name is the usNIC Linux interface name (i.e., usnic_X)
320
321       • When a lower version number is used, like FI_VERSION(1, 3),  it  fol‐
322         lows  the  same behavior the usNIC provider exhibited in libfabric <=
323         v1.3:
324
325         • fabric name is the usNIC Linux interface name (i.e., usnic_X)
326
327         • domain name is NULL
328

SEE ALSO

330       fabric(7), fi_open_ops(3), fi_provider(7),
331

AUTHORS

333       OpenFabrics.
334
335
336
337Libfabric Programmer’s Manual     2022-12-11                       fi_usnic(7)
Impressum