1fi_provider(7) Libfabric v1.17.0 fi_provider(7)
2
6 fi_provider - Fabric Interface Providers
7
9 Conceptually, a fabric provider may be viewed as a local hardware NIC
10 driver, though a provider is not limited by this definition. The first
11 component of libfabric is a general purpose framework that is capable
12 of handling different types of fabric hardware. All fabric hardware
13 devices and their software drivers are required to support this frame‐
14 work. Devices and the drivers that plug into the libfabric framework
15 are referred to as fabric providers, or simply providers.
16 This distribution of libfabric contains the following providers (al‐
18 though more may be available via run-time plug-ins):
19 Core providers
21 GNI A provider for the Aries interconnect in Cray XC(TM) systems
22 utilizing the user-space Generic Networking Interface. See
23 fi_gni(7) for more information.
24 PSM High-speed InfiniBand networking from Intel. See fi_psm(7) for
26 more information.
27 PSM2 High-speed Omni-Path networking from Intel. See fi_psm2(7) for
29 more information.
30 PSM3 High-speed Ethernet networking from Intel. See fi_psm3(7) for
32 more information.
33 OPX High-speed Omni-Path networking from Cornelis Networks. See
35 fi_opx(7) for more information.
36 Sockets
38 A general purpose provider that can be used on any network that
39 supports TCP/UDP sockets. This provider is not intended to pro‐
40 vide performance improvements over regular TCP/UDP sockets, but
41 rather to allow developers to write, test, and debug application
42 code even on platforms that do not have high-speed networking.
43 See fi_sockets(7) for more information.
44 usNIC Ultra low latency Ethernet networking over Cisco userspace VIC
46 adapters. See fi_usnic(7) for more information.
47 Verbs This provider uses the Linux Verbs API for network transport.
49 Application performance is, obviously expected to be similar to
50 that of the native Linux Verbs API. Analogous to the Sockets
51 provider, the Verbs provider is intended to enable developers to
52 write, test, and debug application code on platforms that only
53 have Linux Verbs-based networking. See fi_verbs(7) for more in‐
54 formation.
55 Blue Gene/Q
57 See fi_bgq(7) for more information.
58 EFA A provider for the Amazon EC2 Elastic Fabric Adapter (EFA)
60 (https://aws.amazon.com/hpc/efa/), a custom-built OS bypass
61 hardware interface for inter-instance communication on EC2. See
62 fi_efa(7) for more information.
63 SHM A provider for intranode communication using shared memory. The
65 provider makes use of the Linux kernel feature Cross Memory At‐
66 tach (CMA) which allows processes to have full access to another
67 process’ address space. See fi_shm(7) for more information.
68 Utility providers
70 RxM The RxM provider (ofi_rxm) is an utility provider that supports
71 RDM endpoints emulated over MSG endpoints of a core provider.
72 See fi_rxm(7) for more information.
73 RxD The RxD provider (ofi_rxd) is a utility provider that supports
75 RDM endpoints emulated over DGRAM endpoints of a core provider.
76 See fi_rxd(7) for more information.
77 Special providers
79 Hook The hook provider is a special type of provider that can layer
80 over any other provider, unless FI_FABRIC_DIRECT is used. The
81 hook provider is always available, but has no impact unless en‐
82 abled. When enabled, the hook provider will intercept all calls
83 to the underlying core or utility provider(s). The hook
84 provider is useful for capturing performance data or providing
85 debugging information, even in release builds of the library.
86 See fi_hook(7) for more information.
87
89 Core providers implement the libfabric interfaces directly over low-
90 level hardware and software interfaces. They are designed to support a
91 specific class of hardware, and may be limited to supporting a single
92 NIC. Core providers often only support libfabric features and inter‐
93 faces that map efficiently to their underlying hardware.
94 Utility providers are distinct from core providers in that they are not
96 associated with specific classes of devices. They instead work with
97 core providers to expand their features, and interact with core
98 providers through libfabric interfaces internally. Utility providers
99 are often used to support a specific endpoint type over a simpler end‐
100 point type. For example, the RXD provider implements reliability over
101 unreliable datagram endpoints. The utility providers will not layer
102 over the sockets provider unless it is explicitly requested.
103 Utility providers show up as a component in the core provider’s compo‐
105 nent list. See fi_fabric(3). Utility providers are enabled automati‐
106 cally for core providers that do not support the feature set requested
107 by an application.
108
110 Libfabric provides a general framework for supporting multiple types of
111 fabric objects and their related interfaces. Fabric providers have a
112 large amount of flexibility in selecting which components they are able
113 and willing to support, based on specific hardware constraints.
114 Provider developers should refer to docs/provider for information on
115 functionality supplied by the framework to assist in provider implemen‐
116 tation. To assist in the development of applications, libfabric speci‐
117 fies the following requirements that must be met by any fabric
118 provider, if requested by an application.
119 Note that the instantiation of a specific fabric object is subject to
121 application configuration parameters and need not meet these require‐
122 ments.
123 • A fabric provider must support at least one endpoint type.
125 • All endpoints must support the message queue data transfer interface
127 (fi_ops_msg).
128 • An endpoint that advertises support for a specific endpoint capabili‐
130 ty must support the corresponding data transfer interface.
131 • FI_ATOMIC - fi_ops_atomic
133 • FI_RMA - fi_ops_rma
135 • FI_TAGGED - fi_ops_tagged
137 • Endpoints must support all transmit and receive operations for any
139 data transfer interface that they support.
140 • Exception: If an operation is only usable for an operation that the
142 provider does not support, and support for that operation is con‐
143 veyed using some other mechanism, the operation may return
144 • FI_ENOSYS. For example, if the provider does not support inject‐
146 ed data, it can set the attribute inject_size = 0, and fail all
147 fi_inject operations.
148 • The framework supplies wrappers around the `msg' operations that
150 can be used. For example, the framework implements the sendv() msg
151 operation by calling sendmsg(). Providers may reference the gener‐
152 al operation, and supply on the sendmsg() implementation.
153 • Providers must set all operations to an implementation. Function
155 pointers may not be left NULL or uninitialized. The framework sup‐
156 plies empty functions that return -FI_ENOSYS which can be used for
157 this purpose.
158 • Endpoints must support the CM interface as follows:
160 • FI_EP_MSG endpoints must support all CM operations.
162 • FI_EP_DGRAM endpoints must support CM getname and setname.
164 • FI_EP_RDM endpoints must support CM getname and setname.
166 • Providers that support connectionless endpoints must support all AV
168 operations (fi_ops_av).
169 • Providers that support memory registration, must support all MR oper‐
171 ations (fi_ops_mr).
172 • Providers should support both completion queues and counters.
174 • If FI_RMA_EVENT is not supported, counter support is limited to lo‐
176 cal events only.
177 • Completion queues must support the FI_CQ_FORMAT_CONTEXT and
179 FI_CQ_FORMAT_MSG.
180 • Providers that support FI_REMOTE_CQ_DATA shall support FI_CQ_FOR‐
182 MAT_DATA.
183 • Providers that support FI_TAGGED shall support FI_CQ_FORMAT_TAGGED.
185 • A provider is expected to be forward compatible, and must be able to
187 be compiled against expanded fi_xxx_ops structures that define new
188 functions added after the provider was written. Any unknown func‐
189 tions must be set to NULL.
190 • Providers shall document in their man page which features they sup‐
192 port, and any missing requirements.
193 Future versions of libfabric will automatically enable a more complete
195 set of features for providers that focus their implementation on a nar‐
196 row subset of libfabric capabilities.
197
199 Logging is performed using the FI_ERR, FI_LOG, and FI_DEBUG macros.
200 DEFINITIONS
202 #define FI_ERR(prov_name, subsystem, ...)
203 #define FI_LOG(prov_name, prov, level, subsystem, ...)
205 #define FI_DEBUG(prov_name, subsystem, ...)
207 ARGUMENTS
209 prov_name
210 String representing the provider name.
211 prov Provider context structure.
213 level Log level associated with log statement.
215 subsystem
217 Subsystem being logged from.
218 DESCRIPTION
220 FI_ERR Always logged.
221 FI_LOG Logged if the intended provider, log level, and subsystem param‐
223 eters match the user supplied values.
224 FI_DEBUG
226 Logged if configured with the –enable-debug flag.
227
229 fi_gni(7), fi_hook(7), fi_psm(7), fi_sockets(7), fi_usnic(7),
230 fi_verbs(7), fi_bgq(7), fi_opx(7),
231
233 OpenFabrics.
234Libfabric Programmer’s Manual 2022-12-11 fi_provider(7)