1fi_efa(7) Libfabric v1.8.0 fi_efa(7)
2
6 fi_efa - The Amazon Elastic Fabric Adapter (EFA) Provider
7
9 The EFA provider supports the Elastic Fabric Adapter (EFA) device on
10 Amazon EC2. EFA provides reliable and unreliable datagram send/receive
11 with direct hardware access from userspace (OS bypass).
12
14 The following features are supported:
15 Endpoint types
17 The provider supports endpoint type FI_EP_DGRAM, and FI_EP_RDM
18 on a new Scalable (unordered) Reliable Datagram protocol (SRD).
19 SRD provides support for reliable datagrams and more complete
20 error handling than typically seen with other Reliable Datagram
21 (RD) implementations. The EFA provider provides segmentation,
22 reassembly of out-of-order packets to provide send-after-send
23 ordering guarantees to applications via its FI_EP_RDM endpoint.
24 RDM Endpoint capabilities
26 The following data transfer interfaces are supported via the
27 FI_EP_RDM endpoint: FI_MSG, FI_TAGGED, and FI_RMA. FI_SEND,
28 FI_RECV, FI_DIRECTED_RECV, FI_MULTI_RECV, and FI_SOURCE capabil‐
29 ities are supported. The endpoint provides send-after-send
30 guarantees for data operations. The FI_EP_RDM endpoint does not
31 have a maximum message size.
32 DGRAM Endpoint capabilities
34 The DGRAM endpoint only supports FI_MSG capability with a maxi‐
35 mum message size of the MTU of the underlying hardware (approxi‐
36 mately 8 KiB).
37 Address vectors
39 The provider supports FI_AV_TABLE and FI_AV_MAP address vector
40 types. FI_EVENT is unsupported.
41 Completion events
43 The provider supports FI_CQ_FORMAT_CONTEXT, FI_CQ_FORMAT_MSG,
44 and FI_CQ_FORMAT_DATA. FI_CQ_FORMAT_TAGGED is supported on the
45 RDM endpoint. Wait objects are not currently supported.
46 Modes The provider requires the use of FI_MSG_PREFIX when running over
48 the DGRAM endpoint, and requires FI_MR_LOCAL for all memory reg‐
49 istrations on the DGRAM endpoint.
50 Memory registration modes
52 The RDM endpoint does not require memory registration and the
53 FI_EP_DGRAM endpoint only supports FI_MR_LOCAL.
54 Progress
56 The RDM endpoint supports both FI_PROGRESS_AUTO and
57 FI_PROGRESS_MANUAL, with the default set to auto. However, re‐
58 ceive side data buffers are not modified outside of completion
59 processing routines. The DGRAM endpoint only supports
60 FI_PROGRESS_MANUAL.
61 Threading
63 The RDM endpoint supports FI_THREAD_SAFE, the DGRAM endpoint
64 supports FI_THREAD_DOMAIN, i.e. the provider is not thread safe
65 when using the DGRAM endpoint.
66
68 The provider does not support FI_ATOMIC interfaces. For RMA opera‐
69 tions, completion events for RMA targets (FI_RMA_EVENT) is not support‐
70 ed. The DGRAM endpoint does not fully protect against resource over‐
71 runs, so resource management is disabled for this endpoint (FI_RM_DIS‐
72 ABLED).
73 No support for selective completions.
75 No support for counters.
77 No support for inject.
79
81 FI_EFA_TX_SIZE
82 Maximum number of transmit operations before the provider re‐
83 turns -FI_EAGAIN. For only the RDM endpoint, this parameter
84 will cause transmit operations to be queued when this value is
85 set higher than the default and the transmit queue is full.
86 FI_EFA_RX_SIZE
88 Maximum number of receive operations before the provider returns
89 -FI_EAGAIN.
90 FI_EFA_TX_IOV_LIMIT
92 Maximum number of IOVs for a transmit operation.
93 FI_EFA_RX_IOV_LIMIT
95 Maximum number of IOVs for a receive operation.
96
98 These OFI runtime parameters apply only to the RDM endpoint.
99 FI_EFA_RX_WINDOW_SIZE
101 Maximum number of MTU-sized messages that can be in flight from
102 any single endpoint as part of long message data transfer.
103 FI_EFA_TX_QUEUE_SIZE
105 Depth of transmit queue opened with the NIC. This may not be
106 set to a value greater than what the NIC supports.
107 FI_EFA_RECVWIN_SIZE
109 Size of out of order reorder buffer (in messages). Messages re‐
110 ceived out of this window will result in an error.
111 FI_EFA_CQ_SIZE
113 Size of any cq created, in number of entries.
114 FI_EFA_MR_CACHE_ENABLE
116 Enables using the mr cache and in-line registration instead of a
117 bounce buffer for iov's larger than max_memcpy_size. Defaults
118 to true. When disabled, only uses a bounce buffer
119 FI_EFA_MR_CACHE_MERGE_REGIONS
121 Enables merging overlapping and adjacent memory registration re‐
122 gions. Defaults to true.
123 FI_EFA_MR_MAX_CACHED_COUNT
125 Sets the maximum number of memory registrations that can be
126 cached at any time.
127 FI_EFA_MR_MAX_CACHED_SIZE
129 Sets the maximum amount of memory that cached memory registra‐
130 tions can hold onto at any time.
131 FI_EFA_MAX_MEMCPY_SIZE
133 Threshold size switch between using memory copy into a pre-reg‐
134 istered bounce buffer and memory registration on the user buf‐
135 fer.
136 FI_EFA_MTU_SIZE
138 Overrides the default MTU size of the device.
139 FI_EFA_RX_COPY_UNEXP
141 Enables the use of a separate pool of bounce-buffers to copy un‐
142 expected messages out of the pre-posted receive buffers.
143 FI_EFA_RX_COPY_OOO
145 Enables the use of a separate pool of bounce-buffers to copy
146 out-of-order RTS packets out of the pre-posted receive buffers.
147 FI_EFA_MAX_TIMEOUT
149 Maximum timeout (us) for backoff to a peer after a receiver not
150 ready error.
151 FI_EFA_TIMEOUT_INTERVAL
153 Time interval (us) for the base timeout to use for exponential
154 backoff to a peer after a receiver not ready error.
155
157 fabric(7), fi_provider(7), fi_getinfo(3)
158
160 OpenFabrics.
161Libfabric Programmer's Manual 2019-06-15 fi_efa(7)