1fi_verbs(7)                    Libfabric v1.15.1                   fi_verbs(7)
2
3
4

NAME

6       fi_verbs - The Verbs Fabric Provider
7

OVERVIEW

9       The  verbs  provider  enables applications using OFI to be run over any
10       verbs hardware (Infiniband, iWarp, etc).  It uses the Linux  Verbs  API
11       for network transport and provides a translation of OFI calls to appro‐
12       priate verbs API calls.  It uses librdmacm for communication management
13       and libibverbs for other control and data transfer operations.
14

REQUIREMENTS

16       To  successfully build and install verbs provider as part of libfabric,
17       it needs the following packages: * libibverbs * libibverbs-devel *  li‐
18       brdmacm * librdmacm-devel
19
20       You  may  also  want  to look into any OS specific instructions for en‐
21       abling RDMA.  e.g. RHEL has instructions on their documentation for en‐
22       abling RDMA.
23
24       The IPoIB interface should be configured with a valid IP address.  This
25       is a requirement from librdmacm.
26

SUPPORTED FEATURES

28       The verbs provider supports a subset of OFI features.
29
30   Endpoint types
31       FI_EP_MSG, FI_EP_DGRAM (beta), FI_EP_RDM.
32
33       FI_EP_RDM is supported via OFI RxM and RxD utility providers which  are
34       layered  on  top  of verbs.  To the app, the provider name string would
35       appear as “verbs;ofi_rxm” or “verbs;ofi_rxd”.   Please  refer  the  man
36       pages for RxM (fi_rxm.7) and RxD (fi_rxd.7) to know about the capabili‐
37       ties and limitations for the FI_EP_RDM endpoint.
38
39   Endpoint capabilities and features
40   MSG endpoints
41       FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
42
43   DGRAM endpoints
44       FI_MSG
45
46   Modes
47       Verbs provider requires applications to support the following modes:
48
49   FI_EP_MSG endpoint type
50       • FI_LOCAL_MR / FI_MR_LOCAL mr mode.
51
52       • FI_RX_CQ_DATA for applications that want to  use  RMA.   Applications
53         must  take responsibility of posting receives for any incoming CQ da‐
54         ta.
55
56   Addressing Formats
57       Supported addressing formats include * MSG and RDM (internal  -  depre‐
58       cated)   EPs  support:  FI_SOCKADDR,  FI_SOCKADDR_IN,  FI_SOCKADDR_IN6,
59       FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
60
61   Progress
62       Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations  make
63       forward progress automatically.
64
65   Operation flags
66       Verbs  provider  supports  FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
67       FI_TRANSMIT_COMPLETE.
68
69   Msg Ordering
70       Verbs provider support the following message ordering:
71
72       • Read after Read
73
74       • Read after Write
75
76       • Read after Send
77
78       • Write after Write
79
80       • Write after Send
81
82       • Send after Write
83
84       • Send after Send
85
86       and the following completion ordering:
87
88       • TX contexts: FI_ORDER_STRICT
89
90       • RX contexts: FI_ORDER_DATA
91
92   Fork
93       Verbs provider does not provide fork safety by  default.   Fork  safety
94       can  be requested by setting IBV_FORK_SAFE, or RDMAV_FORK_SAFE.  If the
95       system configuration supports the use of huge pages, it is  recommended
96       to  set  RDMAV_HUGEPAGES_SAFE.  See ibv_fork_init(3) for additional de‐
97       tails.
98
99   Memory Registration Cache
100       The verbs provider uses the common memory registration cache  function‐
101       ality  that’s  part  of  libfabric utility code.  This speeds up memory
102       registration calls from applications by caching registrations  of  fre‐
103       quently  used  memory regions.  Please refer to fi_mr(3): Memory Regis‐
104       tration Cache section for more details.
105

LIMITATIONS

107   Memory Regions
108       Only FI_MR_BASIC mode is supported.  Adding regions  via  s/g  list  is
109       supported only up to a s/g list size of 1.  No support for binding mem‐
110       ory regions to a counter.
111
112   Wait objects
113       Only FI_WAIT_FD wait object is supported only  for  FI_EP_MSG  endpoint
114       type.  Wait sets are not supported.
115
116   Resource Management
117       Application  has to make sure CQs are not overrun as this cannot be de‐
118       tected by the provider.
119
120   Unsupported Features
121       The following features are not supported in verbs provider:
122
123   Unsupported Capabilities
124       FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
125
126   Other unsupported features
127       Scalable endpoints, FABRIC_DIRECT
128
129   Unsupported features specific to MSG endpoints
130       • Counters,  FI_SOURCE,  FI_TAGGED,   FI_PEEK,   FI_CLAIM,   fi_cancel,
131         fi_ep_alias, shared TX context, cq_readfrom operations.
132
133       • Completion  flags are not reported if a request posted to an endpoint
134         completes in error.
135
136   Fork
137       The support for fork in the provider has the following limitations:
138
139       • Fabric resources like endpoint, CQ, EQ, etc.  should not be  used  in
140         the forked process.
141
142       • The  memory  registered  using fi_mr_reg has to be page aligned since
143         ibv_reg_mr marks the entire page that a memory region belongs  to  as
144         not to be re-mapped when the process is forked (MADV_DONTFORK).
145
146   XRC Transport
147       The  XRC  transport  is  intended  to be used when layered with the RXM
148       provider  and  requires  the  use  of  shared  receive  contexts.   See
149       fi_rxm(7).   To  enable  XRC,  the following environment variables must
150       usually be set: FI_VERBS_PREFER_XRC and FI_OFI_RXM_USE_SRX.
151

RUNTIME PARAMETERS

153       The verbs provider checks for the following environment variables.
154
155   Common variables:
156       FI_VERBS_TX_SIZE
157              Default maximum tx context size (default: 384)
158
159       FI_VERBS_RX_SIZE
160              Default maximum rx context size (default: 384)
161
162       FI_VERBS_TX_IOV_LIMIT
163              Default maximum tx iov_limit (default: 4).  Note: RDM  (internal
164              - deprecated) EP type supports only 1
165
166       FI_VERBS_RX_IOV_LIMIT
167              Default  maximum rx iov_limit (default: 4).  Note: RDM (internal
168              - deprecated) EP type supports only 1
169
170       FI_VERBS_INLINE_SIZE
171              Default maximum inline size.  Actual  inject  size  returned  in
172              fi_info may be greater (default: 64)
173
174       FI_VERBS_MIN_RNR_TIMER
175              Set min_rnr_timer QP attribute (0 - 31) (default: 12)
176
177       FI_VERBS_CQREAD_BUNCH_SIZE
178              The number of entries to be read from the verbs completion queue
179              at a time (default: 8).
180
181       FI_VERBS_PREFER_XRC
182              Prioritize XRC transport fi_info  before  RC  transport  fi_info
183              (default: 0, RC fi_info will be before XRC fi_info)
184
185       FI_VERBS_GID_IDX
186              The GID index to use (default: 0)
187
188       FI_VERBS_DEVICE_NAME
189              Specify a specific verbs device to use by name
190
191   Variables specific to MSG endpoints
192       FI_VERBS_IFACE
193              The  prefix or the full name of the network interface associated
194              with the verbs device (default: ib)
195
196   Variables specific to DGRAM endpoints
197       FI_VERBS_DGRAM_USE_NAME_SERVER
198              The option that enables/disables OFI Name Server thread.  The NS
199              thread  is used to resolve IP-addresses to provider specific ad‐
200              dresses (default: 1, if  “OMPI_COMM_WORLD_RANK”  and  “PMI_RANK”
201              environment variables aren’t defined)
202
203       FI_VERBS_NAME_SERVER_PORT
204              The  port  on  which Name Server thread listens incoming connec‐
205              tions and requests (default: 5678)
206
207   Environment variables notes
208       The fi_info utility would give the up-to-date information  on  environ‐
209       ment variables: fi_info -p verbs -e
210

Troubleshooting / Known issues

212   fi_getinfo returns -FI_ENODATA
213       • Set  FI_LOG_LEVEL=info  or FI_LOG_LEVEL=debug (if debug build of lib‐
214         fabric is available) and check if there any errors because of  incor‐
215         rect input parameters to fi_getinfo.
216
217       • Check if “fi_info -p verbs” is successful.  If that fails the follow‐
218         ing checklist may help in ensuring that the RDMA verbs stack is func‐
219         tional:
220
221         • If  libfabric  was  compiled,  check  if  verbs provider was built.
222           Building verbs provider would be skipped if its dependencies (list‐
223           ed in requirements) aren’t available on the system.
224
225         • Verify verbs device is functional:
226
227           • Does ibv_rc_pingpong (available in libibverbs) test work?
228
229             • Does ibv_devinfo (available in libibverbs) show the device with
230               PORT_ACTIVE status?
231
232               • Check if Subnet Manager (SM) is running on the switch  or  on
233                 one of the nodes in the cluster.
234
235               • Is the cable connected?
236
237         • Verify librdmacm is functional:
238
239           • Does ucmatose test (available in librdmacm) work?
240
241           • Is  the IPoIB interface (e.g. ib0) up and configured with a valid
242             IP address?
243
244   Other issues
245       When running an app over verbs provider with Valgrind, there may be re‐
246       ports of memory leak in functions from dependent libraries (e.g. libib‐
247       verbs, librdmacm).  These leaks are safe to ignore.
248
249       The provider protects CQ overruns that may happen because more TX oper‐
250       ations  were posted to endpoints than CQ size.  On the receive side, it
251       isn’t expected to overrun the CQ.  In case it happens  the  application
252       developer should take care not to post excess receives without draining
253       the CQ.  CQ overruns can make the MSG endpoints unusable.
254

SEE ALSO

256       fabric(7), fi_provider(7),
257

AUTHORS

259       OpenFabrics.
260
261
262
263Libfabric Programmer’s Manual     2021-03-22                       fi_verbs(7)
Impressum