1fabric(7) Libfabric v1.6.1 fabric(7)
2
6 fabric - Fabric Interface Library
7
9 #include <rdma/fabric.h>
10 Libfabric is a high-performance fabric software library designed to
12 provide low-latency interfaces to fabric hardware.
13
15 Libfabric provides 'process direct I/O' to application software commu‐
16 nicating across fabric software and hardware. Process direct I/O, his‐
17 torically referred to as RDMA, allows an application to directly access
18 network resources without operating system interventions. Data trans‐
19 fers can occur directly to and from application memory.
20 There are two components to the libfabric software:
22 Fabric Providers : Conceptually, a fabric provider may be viewed as a
24 local hardware NIC driver, though a provider is not limited by this
25 definition. The first component of libfabric is a general purpose
26 framework that is capable of handling different types of fabric hard‐
27 ware. All fabric hardware devices and their software drivers are
28 required to support this framework. Devices and the drivers that plug
29 into the libfabric framework are referred to as fabric providers, or
30 simply providers. Provider details may be found in fi_provider(7).
31 Fabric Interfaces : The second component is a set of communication
33 operations. Libfabric defines several sets of communication functions
34 that providers can support. It is not required that providers imple‐
35 ment all the interfaces that are defined; however, providers clearly
36 indicate which interfaces they do support.
37
39 The fabric interfaces are designed such that they are cohesive and not
40 simply a union of disjoint interfaces. The interfaces are logically
41 divided into two groups: control interfaces and communication opera‐
42 tions. The control interfaces are a common set of operations that pro‐
43 vide access to local communication resources, such as address vectors
44 and event queues. The communication operations expose particular mod‐
45 els of communication and fabric functionality, such as message queues,
46 remote memory access, and atomic operations. Communication operations
47 are associated with fabric endpoints.
48 Applications will typically use the control interfaces to discover
50 local capabilities and allocate necessary resources. They will then
51 allocate and configure a communication endpoint to send and receive
52 data, or perform other types of data transfers, with remote endpoints.
53
55 The control interfaces APIs provide applications access to network
56 resources. This involves listing all the interfaces available, obtain‐
57 ing the capabilities of the interfaces and opening a provider.
58 fi_getinfo - Fabric Information : The fi_getinfo call is the base call
60 used to discover and request fabric services offered by the system.
61 Applications can use this call to indicate the type of communication
62 that they desire. The results from fi_getinfo, fi_info, are used to
63 reserve and configure fabric resources.
64 fi_getinfo returns a list of fi_info structures. Each structure refer‐
66 ences a single fabric provider, indicating the interfaces that the
67 provider supports, along with a named set of resources. A fabric
68 provider may include multiple fi_info structures in the returned list.
69 fi_fabric - Fabric Domain : A fabric domain represents a collection of
71 hardware and software resources that access a single physical or vir‐
72 tual network. All network ports on a system that can communicate with
73 each other through the fabric belong to the same fabric domain. A fab‐
74 ric domain shares network addresses and can span multiple providers.
75 libfabric supports systems connected to multiple fabrics.
76 fi_domain - Access Domains : An access domain represents a single logi‐
78 cal connection into a fabric. It may map to a single physical or vir‐
79 tual NIC or a port. An access domain defines the boundary across which
80 fabric resources may be associated. Each access domain belongs to a
81 single fabric domain.
82 fi_endpoint - Fabric Endpoint : A fabric endpoint is a communication
84 portal. An endpoint may be either active or passive. Passive end‐
85 points are used to listen for connection requests. Active endpoints
86 can perform data transfers. Endpoints are configured with specific
87 communication capabilities and data transfer interfaces.
88 fi_eq - Event Queue : Event queues, are used to collect and report the
90 completion of asynchronous operations and events. Event queues report
91 events that are not directly associated with data transfer operations.
92 fi_cq - Completion Queue : Completion queues are high-performance event
94 queues used to report the completion of data transfer operations.
95 fi_cntr - Event Counters : Event counters are used to report the number
97 of completed asynchronous operations. Event counters are considered
98 light-weight, in that a completion simply increments a counter, rather
99 than placing an entry into an event queue.
100 fi_mr - Memory Region : Memory regions describe application local mem‐
102 ory buffers. In order for fabric resources to access application mem‐
103 ory, the application must first grant permission to the fabric provider
104 by constructing a memory region. Memory regions are required for spe‐
105 cific types of data transfer operations, such as RMA transfers (see
106 below).
107 fi_av - Address Vector : Address vectors are used to map higher level
109 addresses, such as IP addresses, which may be more natural for an
110 application to use, into fabric specific addresses. The use of address
111 vectors allows providers to reduce the amount of memory required to
112 maintain large address look-up tables, and eliminate expensive address
113 resolution and look-up methods during data transfer operations.
114
116 Fabric endpoints are associated with multiple data transfer interfaces.
117 Each interface set is designed to support a specific style of communi‐
118 cation, with an endpoint allowing the different interfaces to be used
119 in conjunction. The following data transfer interfaces are defined by
120 libfabric.
121 fi_msg - Message Queue : Message queues expose a simple, message-based
123 FIFO queue interface to the application. Message data transfers allow
124 applications to send and receive data with message boundaries being
125 maintained.
126 fi_tagged - Tagged Message Queues : Tagged message lists expose
128 send/receive data transfer operations built on the concept of tagged
129 messaging. The tagged message queue is conceptually similar to stan‐
130 dard message queues, but with the addition of 64-bit tags for each mes‐
131 sage. Sent messages are matched with receive buffers that are tagged
132 with a similar value.
133 fi_rma - Remote Memory Access : RMA transfers are one-sided operations
135 that read or write data directly to a remote memory region. Other than
136 defining the appropriate memory region, RMA operations do not require
137 interaction at the target side for the data transfer to complete.
138 fi_atomic - Atomic : Atomic operations can perform one of several oper‐
140 ations on a remote memory region. Atomic operations include well-known
141 functionality, such as atomic-add and compare-and-swap, plus several
142 other pre-defined calls. Unlike other data transfer interfaces, atomic
143 operations are aware of the data formatting at the target memory
144 region.
145
147 Logging can be controlled using the FI_LOG_LEVEL, FI_LOG_PROV, and
148 FI_LOG_SUBSYS environment variables.
149 FI_LOG_LEVEL : FI_LOG_LEVEL controls the amount of logging data that is
151 output. The following log levels are defined.
152 · Warn : Warn is the least verbose setting and is intended for report‐
154 ing errors or warnings.
155 · Trace : Trace is more verbose and is meant to include non-detailed
157 output helpful to tracing program execution.
158 · Info : Info is high traffic and meant for detailed output.
160 · Debug : Debug is high traffic and is likely to impact application
162 performance. Debug output is only available if the library has been
163 compiled with debugging enabled.
164 FI_LOG_PROV : The FI_LOG_PROV environment variable enables or disables
166 logging from specific providers. Providers can be enabled by listing
167 them in a comma separated fashion. If the list begins with the '^'
168 symbol, then the list will be negated. By default all providers are
169 enabled.
170 Example: To enable logging from the psm and sockets provider:
172 FI_LOG_PROV="psm,sockets"
173 Example: To enable logging from providers other than psm:
175 FI_LOG_PROV="^psm"
176 FI_LOG_SUBSYS : The FI_LOG_SUBSYS environment variable enables or dis‐
178 ables logging at the subsystem level. The syntax for enabling or dis‐
179 abling subsystems is similar to that used for FI_LOG_PROV. The follow‐
180 ing subsystems are defined.
181 · core : Provides output related to the core framework and its manage‐
183 ment of providers.
184 · fabric : Provides output specific to interactions associated with the
186 fabric object.
187 · domain : Provides output specific to interactions associated with the
189 domain object.
190 · ep_ctrl : Provides output specific to endpoint non-data transfer
192 operations, such as CM operations.
193 · ep_data : Provides output specific to endpoint data transfer opera‐
195 tions.
196 · av : Provides output specific to address vector operations.
198 · cq : Provides output specific to completion queue operations.
200 · eq : Provides output specific to event queue operations.
202 · mr : Provides output specific to memory registration.
204
206 Because libfabric is designed to provide applications direct access to
207 fabric hardware, there are limits on how libfabric resources may be
208 used in conjunction with system calls. These limitations are notable
209 for developers who may be familiar programming to the sockets inter‐
210 face. Although limits are provider specific, the following restric‐
211 tions apply to many providers and should be adhered to by applications
212 desiring portability across providers.
213 fork : Fabric resources are not guaranteed to be available by child
215 processes. This includes objects, such as endpoints and completion
216 queues, as well as application controlled data buffers which have been
217 assigned to the network. For example, data buffers that have been reg‐
218 istered with a fabric domain may not be available in a child process
219 because of copy on write restrictions.
220
222 fi_provider(7), fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_av(3),
223 fi_eq(3), fi_cq(3), fi_cntr(3), fi_mr(3)
224
226 OpenFabrics.
227Libfabric Programmer's Manual 2017-12-01 fabric(7)