1fi_verbs(7)                    Libfabric v1.8.0                    fi_verbs(7)
2
3
4

NAME

6       fi_verbs - The Verbs Fabric Provider
7

OVERVIEW

9       The  verbs  provider  enables applications using OFI to be run over any
10       verbs hardware (Infiniband, iWarp, etc).  It uses the Linux  Verbs  API
11       for network transport and provides a translation of OFI calls to appro‐
12       priate verbs API calls.  It uses librdmacm for communication management
13       and libibverbs for other control and data transfer operations.
14

REQUIREMENTS

16       To  successfully build and install verbs provider as part of libfabric,
17       it needs the following packages: * libibverbs * libibverbs-devel *  li‐
18       brdmacm * librdmacm-devel
19
20       You  may  also  want  to look into any OS specific instructions for en‐
21       abling RDMA.  e.g.  RHEL has instructions on  their  documentation  for
22       enabling RDMA.
23
24       The IPoIB interface should be configured with a valid IP address.  This
25       is a requirement from librdmacm.
26

SUPPORTED FEATURES

28       The verbs provider supports a subset of OFI features.
29
30   Endpoint types
31       FI_EP_MSG, FI_EP_DGRAM (beta), FI_EP_RDM.
32
33       FI_EP_RDM is supported via OFI RxM and RxD utility providers which  are
34       layered  on  top  of verbs.  To the app, the provider name string would
35       appear as "verbs;ofi_rxm" or "verbs;ofi_rxd".   Please  refer  the  man
36       pages for RxM (fi_rxm.7) and RxD (fi_rxd.7) to know about the capabili‐
37       ties and limitations for the FI_EP_RDM endpoint.
38
39   Endpoint capabilities and features
40   MSG endpoints
41       FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
42
43   DGRAM endpoints
44       FI_MSG
45
46   Modes
47       Verbs provider requires applications to support the following modes:
48
49   FI_EP_MSG endpoint type
50       · FI_LOCAL_MR / FI_MR_LOCAL mr mode.
51
52       · FI_RX_CQ_DATA for applications that want to  use  RMA.   Applications
53         must  take responsibility of posting receives for any incoming CQ da‐
54         ta.
55
56   Addressing Formats
57       Supported addressing formats include * MSG and RDM (internal  -  depre‐
58       cated)   EPs  support:  FI_SOCKADDR,  FI_SOCKADDR_IN,  FI_SOCKADDR_IN6,
59       FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
60
61   Progress
62       Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations  make
63       forward progress automatically.
64
65   Operation flags
66       Verbs  provider  supports  FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
67       FI_TRANSMIT_COMPLETE.
68
69   Msg Ordering
70       Verbs provider support the following message ordering:
71
72       · Read after Read
73
74       · Read after Write
75
76       · Read after Send
77
78       · Write after Write
79
80       · Write after Send
81
82       · Send after Write
83
84       · Send after Send
85
86       and the following completion ordering:
87
88       · TX contexts: FI_ORDER_STRICT
89
90       · RX contexts: FI_ORDER_DATA
91
92   Fork
93       Verbs provider does not provide fork safety by  default.   Fork  safety
94       can  be requested by setting IBV_FORK_SAFE, or RDMAV_FORK_SAFE.  If the
95       system configuration supports the use of huge pages, it is  recommended
96       to  set  RDMAV_HUGEPAGES_SAFE.  See ibv_fork_init(3) for additional de‐
97       tails.
98
99   Memory Registration Cache
100       The verbs provider uses the common memory registration cache  function‐
101       ality  that's  part  of  libfabric utility code.  This speeds up memory
102       registration calls from applications by caching registrations  of  fre‐
103       quently  used  memory regions.  Please refer to fi_mr(3): Memory Regis‐
104       tration Cache section for more details.
105

LIMITATIONS

107   Memory Regions
108       Only FI_MR_BASIC mode is supported.  Adding regions  via  s/g  list  is
109       supported only up to a s/g list size of 1.  No support for binding mem‐
110       ory regions to a counter.
111
112   Wait objects
113       Only FI_WAIT_FD wait object is supported only  for  FI_EP_MSG  endpoint
114       type.  Wait sets are not supported.
115
116   Resource Management
117       Application  has to make sure CQs are not overrun as this cannot be de‐
118       tected by the provider.
119
120   Unsupported Features
121       The following features are not supported in verbs provider:
122
123   Unsupported Capabilities
124       FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
125
126   Other unsupported features
127       Scalable endpoints, FABRIC_DIRECT
128
129   Unsupported features specific to MSG endpoints
130       · Counters,  FI_SOURCE,  FI_TAGGED,   FI_PEEK,   FI_CLAIM,   fi_cancel,
131         fi_ep_alias, shared TX context, cq_readfrom operations.
132
133       · Completion  flags are not reported if a request posted to an endpoint
134         completes in error.
135
136   Fork
137       The support for fork in the provider has the following limitations:
138
139       · Fabric resources like endpoint, CQ, EQ, etc.  should not be  used  in
140         the forked process.
141
142       · The  memory  registered  using fi_mr_reg has to be page aligned since
143         ibv_reg_mr marks the entire page that a memory region belongs  to  as
144         not to be re-mapped when the process is forked (MADV_DONTFORK).
145
146   XRC Transport
147       The  XRC  transport  is  intended  to be used when layered with the RXM
148       provider  and  requires  the  use  of  shared  receive  contexts.   See
149       fi_rxm(7).
150

RUNTIME PARAMETERS

152       The verbs provider checks for the following environment variables.
153
154   Common variables:
155       FI_VERBS_TX_SIZE
156              Default maximum tx context size (default: 384)
157
158       FI_VERBS_RX_SIZE
159              Default maximum rx context size (default: 384)
160
161       FI_VERBS_TX_IOV_LIMIT
162              Default  maximum tx iov_limit (default: 4).  Note: RDM (internal
163              - deprecated) EP type supports only 1
164
165       FI_VERBS_RX_IOV_LIMIT
166              Default maximum rx iov_limit (default: 4).  Note: RDM  (internal
167              - deprecated) EP type supports only 1
168
169       FI_VERBS_INLINE_SIZE
170              Default  maximum  inline  size.   Actual inject size returned in
171              fi_info may be greater (default: 64)
172
173       FI_VERBS_MIN_RNR_TIMER
174              Set min_rnr_timer QP attribute (0 - 31) (default: 12)
175
176       FI_VERBS_USE_ODP
177              Enable On-Demand-Paging (ODP) experimental feature.  The feature
178              is supported only on Mellanox OFED (default: 0)
179
180       FI_VERBS_CQREAD_BUNCH_SIZE
181              The number of entries to be read from the verbs completion queue
182              at a time (default: 8).
183
184       FI_VERBS_IFACE
185              The prefix or the full name of the network interface  associated
186              with the verbs device (default: ib)
187
188       FI_VERBS_PREFER_XRC
189              Prioritize  XRC  transport  fi_info  before RC transport fi_info
190              (default: 0, RC fi_info will be before XRC fi_info)
191
192   Variables specific to DGRAM endpoints
193       FI_VERBS_DGRAM_USE_NAME_SERVER
194              The option that enables/disables OFI Name Server thread.  The NS
195              thread  is used to resolve IP-addresses to provider specific ad‐
196              dresses (default: 1, if  "OMPI_COMM_WORLD_RANK"  and  "PMI_RANK"
197              environment variables aren't defined)
198
199       FI_VERBS_NAME_SERVER_PORT
200              The  port  on  which Name Server thread listens incoming connec‐
201              tions and requests (default: 5678)
202
203       FI_VERBS_GID_IDX
204              The GID index to use (default: 0)
205
206   Environment variables notes
207       The fi_info utility would give the up-to-date information  on  environ‐
208       ment variables: fi_info -p verbs -e
209

Troubleshooting / Known issues

211   fi_getinfo returns -FI_ENODATA
212       · Set  FI_LOG_LEVEL=info  or FI_LOG_LEVEL=debug (if debug build of lib‐
213         fabric is available) and check if there any errors because of  incor‐
214         rect input parameters to fi_getinfo.
215
216       · Check if "fi_info -p verbs" is successful.  If that fails the follow‐
217         ing chkecklist may help in ensuring that  the  RDMA  verbs  stack  is
218         functional:
219
220       · If libfabric was compiled, check if verbs provider was built.  Build‐
221         ing verbs provider would be skipped if its  dependencies  (listed  in
222         requirements) aren't available on the system.
223
224       · Verify verbs device is functional:
225
226         · Does ibv_rc_pingpong (available in libibverbs) test work?
227
228         · Does  ibv_devinfo  (available  in  libibverbs) show the device with
229           PORT_ACTIVE status?
230
231           · Check if Subnet Manager (SM) is running on the switch or  on  one
232             of the nodes in the cluster.
233
234           · Is the cable connected?
235
236       · Verify librdmacm is functional:
237
238         · Does ucmatose test (available in librdmacm) work?
239
240         · Is  the  IPoIB interface (e.g.  ib0) up and configured with a valid
241           IP address?
242
243   Other issues
244       When running an app over verbs provider with Valgrind, there may be re‐
245       ports  of  memory leak in functions from dependent libraries (e.g.  li‐
246       bibverbs, librdmacm).  These leaks are safe to ignore.
247
248       The provider protects CQ overruns that may happen because more TX oper‐
249       ations  were posted to endpoints than CQ size.  On the receive side, it
250       isn't expected to overrun the CQ.  In case it happens  the  application
251       developer should take care not to post excess receives without draining
252       the CQ.  CQ overruns can make the MSG endpoints unusable.
253

SEE ALSO

255       fabric(7), fi_provider(7),
256

AUTHORS

258       OpenFabrics.
259
260
261
262Libfabric Programmer's Manual     2019-06-21                       fi_verbs(7)
Impressum