1fi_efa(7)                      Libfabric v1.12.1                     fi_efa(7)
2
3
4

NAME

6       fi_efa - The Amazon Elastic Fabric Adapter (EFA) Provider
7

OVERVIEW

9       The  EFA  provider  supports the Elastic Fabric Adapter (EFA) device on
10       Amazon EC2.  EFA provides reliable and unreliable datagram send/receive
11       with direct hardware access from userspace (OS bypass).
12

SUPPORTED FEATURES

14       The following features are supported:
15
16       Endpoint types
17              The  provider  supports endpoint type FI_EP_DGRAM, and FI_EP_RDM
18              on a new Scalable (unordered) Reliable Datagram protocol  (SRD).
19              SRD  provides  support  for reliable datagrams and more complete
20              error handling than typically seen with other Reliable  Datagram
21              (RD)  implementations.   The EFA provider provides segmentation,
22              reassembly of out-of-order packets  to  provide  send-after-send
23              ordering guarantees to applications via its FI_EP_RDM endpoint.
24
25       RDM Endpoint capabilities
26              The  following  data  transfer  interfaces are supported via the
27              FI_EP_RDM endpoint: FI_MSG,  FI_TAGGED,  and  FI_RMA.   FI_SEND,
28              FI_RECV, FI_DIRECTED_RECV, FI_MULTI_RECV, and FI_SOURCE capabil‐
29              ities are  supported.   The  endpoint  provides  send-after-send
30              guarantees for data operations.  The FI_EP_RDM endpoint does not
31              have a maximum message size.
32
33       DGRAM Endpoint capabilities
34              The DGRAM endpoint only supports FI_MSG capability with a  maxi‐
35              mum message size of the MTU of the underlying hardware (approxi‐
36              mately 8 KiB).
37
38       Address vectors
39              The provider supports FI_AV_TABLE and FI_AV_MAP  address  vector
40              types.  FI_EVENT is unsupported.
41
42       Completion events
43              The  provider  supports  FI_CQ_FORMAT_CONTEXT, FI_CQ_FORMAT_MSG,
44              and FI_CQ_FORMAT_DATA.  FI_CQ_FORMAT_TAGGED is supported on  the
45              RDM endpoint.  Wait objects are not currently supported.
46
47       Modes  The provider requires the use of FI_MSG_PREFIX when running over
48              the DGRAM endpoint, and requires FI_MR_LOCAL for all memory reg‐
49              istrations on the DGRAM endpoint.
50
51       Memory registration modes
52              The  RDM  endpoint does not require memory registration for send
53              and receive operations, i.e.  it does not  require  FI_MR_LOCAL.
54              Applications may specify FI_MR_LOCAL in the MR mode flags in or‐
55              der  to  use  descriptors  provided  by  the  application.   The
56              FI_EP_DGRAM endpoint only supports FI_MR_LOCAL.
57
58       Progress
59              The    RDM   endpoint   supports   both   FI_PROGRESS_AUTO   and
60              FI_PROGRESS_MANUAL, with the default set to auto.  However,  re‐
61              ceive  side  data buffers are not modified outside of completion
62              processing  routines.   The   DGRAM   endpoint   only   supports
63              FI_PROGRESS_MANUAL.
64
65       Threading
66              The  RDM  endpoint  supports  FI_THREAD_SAFE, the DGRAM endpoint
67              supports FI_THREAD_DOMAIN, i.e.  the provider is not thread safe
68              when using the DGRAM endpoint.
69

LIMITATIONS

71       The  DGRAM endpoint does not support FI_ATOMIC interfaces.  For RMA op‐
72       erations, completion events for RMA targets (FI_RMA_EVENT) is not  sup‐
73       ported.   The  DGRAM  endpoint  does not fully protect against resource
74       overruns,  so  resource  management  is  disabled  for  this   endpoint
75       (FI_RM_DISABLED).
76
77       No support for selective completions.
78
79       No support for counters for the DGRAM endpoint.
80
81       No support for inject.
82

RUNTIME PARAMETERS

84       FI_EFA_TX_SIZE
85              Maximum  number  of  transmit operations before the provider re‐
86              turns -FI_EAGAIN.  For only the  RDM  endpoint,  this  parameter
87              will  cause  transmit operations to be queued when this value is
88              set higher than the default and the transmit queue is full.
89
90       FI_EFA_RX_SIZE
91              Maximum number of receive operations before the provider returns
92              -FI_EAGAIN.
93
94       FI_EFA_TX_IOV_LIMIT
95              Maximum number of IOVs for a transmit operation.
96
97       FI_EFA_RX_IOV_LIMIT
98              Maximum number of IOVs for a receive operation.
99

RUNTIME PARAMETERS SPECIFIC TO RDM ENDPOINT

101       These OFI runtime parameters apply only to the RDM endpoint.
102
103       FI_EFA_RX_WINDOW_SIZE
104              Maximum  number of MTU-sized messages that can be in flight from
105              any single endpoint as part of long message data transfer.
106
107       FI_EFA_TX_QUEUE_SIZE
108              Depth of transmit queue opened with the NIC.  This  may  not  be
109              set to a value greater than what the NIC supports.
110
111       FI_EFA_RECVWIN_SIZE
112              Size of out of order reorder buffer (in messages).  Messages re‐
113              ceived out of this window will result in an error.
114
115       FI_EFA_CQ_SIZE
116              Size of any cq created, in number of entries.
117
118       FI_EFA_MR_CACHE_ENABLE
119              Enables using the mr cache and in-line registration instead of a
120              bounce  buffer  for iov's larger than max_memcpy_size.  Defaults
121              to true.  When disabled, only uses a bounce buffer
122
123       FI_EFA_MR_MAX_CACHED_COUNT
124              Sets the maximum number of  memory  registrations  that  can  be
125              cached at any time.
126
127       FI_EFA_MR_MAX_CACHED_SIZE
128              Sets  the  maximum amount of memory that cached memory registra‐
129              tions can hold onto at any time.
130
131       FI_EFA_MAX_MEMCPY_SIZE
132              Threshold size switch between using memory copy into a  pre-reg‐
133              istered  bounce  buffer and memory registration on the user buf‐
134              fer.
135
136       FI_EFA_MTU_SIZE
137              Overrides the default MTU size of the device.
138
139       FI_EFA_RX_COPY_UNEXP
140              Enables the use of a separate pool of bounce-buffers to copy un‐
141              expected messages out of the pre-posted receive buffers.
142
143       FI_EFA_RX_COPY_OOO
144              Enables  the  use  of  a separate pool of bounce-buffers to copy
145              out-of-order RTS packets out of the pre-posted receive buffers.
146
147       FI_EFA_MAX_TIMEOUT
148              Maximum timeout (us) for backoff to a peer after a receiver  not
149              ready error.
150
151       FI_EFA_TIMEOUT_INTERVAL
152              Time  interval  (us) for the base timeout to use for exponential
153              backoff to a peer after a receiver not ready error.
154
155       FI_EFA_ENABLE_SHM_TRANSFER
156              Enable SHM provider to provide the communication across all  in‐
157              tra-node  processes.   SHM transfer will be disabled in the case
158              where ptrace protection is turned on.  You can turn  it  off  to
159              enable shm transfer.
160
161       FI_EFA_SHM_AV_SIZE
162              Defines  the maximum number of entries in SHM provider's address
163              vector.
164
165       FI_EFA_SHM_MAX_MEDIUM_SIZE
166              Defines the switch point between small/medium message and  large
167              message.   The  message  larger  than  this switch point will be
168              transferred with large message protocol.
169
170       FI_EFA_INTER_MAX_MEDIUM_MESSAGE_SIZE
171              The maximum size for inter EFA messages  to  be  sent  by  using
172              medium  message  protocol.  Messages which can fit in one packet
173              will be sent as eager message.  Messages whose sizes are smaller
174              than  this  value  will  be  sent using medium message protocol.
175              Other messages will be sent using CTS based long message  proto‐
176              col.
177

SEE ALSO

179       fabric(7), fi_provider(7), fi_getinfo(3)
180

AUTHORS

182       OpenFabrics.
183
184
185
186Libfabric Programmer's Manual     2020-09-01                         fi_efa(7)
Impressum