1fi_verbs(7) Libfabric v1.8.0 fi_verbs(7)
2
6 fi_verbs - The Verbs Fabric Provider
7
9 The verbs provider enables applications using OFI to be run over any
10 verbs hardware (Infiniband, iWarp, etc). It uses the Linux Verbs API
11 for network transport and provides a translation of OFI calls to appro‐
12 priate verbs API calls. It uses librdmacm for communication management
13 and libibverbs for other control and data transfer operations.
14
16 To successfully build and install verbs provider as part of libfabric,
17 it needs the following packages: * libibverbs * libibverbs-devel * li‐
18 brdmacm * librdmacm-devel
19 You may also want to look into any OS specific instructions for en‐
21 abling RDMA. e.g. RHEL has instructions on their documentation for
22 enabling RDMA.
23 The IPoIB interface should be configured with a valid IP address. This
25 is a requirement from librdmacm.
26
28 The verbs provider supports a subset of OFI features.
29 Endpoint types
31 FI_EP_MSG, FI_EP_DGRAM (beta), FI_EP_RDM.
32 FI_EP_RDM is supported via OFI RxM and RxD utility providers which are
34 layered on top of verbs. To the app, the provider name string would
35 appear as "verbs;ofi_rxm" or "verbs;ofi_rxd". Please refer the man
36 pages for RxM (fi_rxm.7) and RxD (fi_rxd.7) to know about the capabili‐
37 ties and limitations for the FI_EP_RDM endpoint.
38 Endpoint capabilities and features
40 MSG endpoints
41 FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
42 DGRAM endpoints
44 FI_MSG
45 Modes
47 Verbs provider requires applications to support the following modes:
48 FI_EP_MSG endpoint type
50 · FI_LOCAL_MR / FI_MR_LOCAL mr mode.
51 · FI_RX_CQ_DATA for applications that want to use RMA. Applications
53 must take responsibility of posting receives for any incoming CQ da‐
54 ta.
55 Addressing Formats
57 Supported addressing formats include * MSG and RDM (internal - depre‐
58 cated) EPs support: FI_SOCKADDR, FI_SOCKADDR_IN, FI_SOCKADDR_IN6,
59 FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
60 Progress
62 Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations make
63 forward progress automatically.
64 Operation flags
66 Verbs provider supports FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
67 FI_TRANSMIT_COMPLETE.
68 Msg Ordering
70 Verbs provider support the following message ordering:
71 · Read after Read
73 · Read after Write
75 · Read after Send
77 · Write after Write
79 · Write after Send
81 · Send after Write
83 · Send after Send
85 and the following completion ordering:
87 · TX contexts: FI_ORDER_STRICT
89 · RX contexts: FI_ORDER_DATA
91 Fork
93 Verbs provider does not provide fork safety by default. Fork safety
94 can be requested by setting IBV_FORK_SAFE, or RDMAV_FORK_SAFE. If the
95 system configuration supports the use of huge pages, it is recommended
96 to set RDMAV_HUGEPAGES_SAFE. See ibv_fork_init(3) for additional de‐
97 tails.
98 Memory Registration Cache
100 The verbs provider uses the common memory registration cache function‐
101 ality that's part of libfabric utility code. This speeds up memory
102 registration calls from applications by caching registrations of fre‐
103 quently used memory regions. Please refer to fi_mr(3): Memory Regis‐
104 tration Cache section for more details.
105
107 Memory Regions
108 Only FI_MR_BASIC mode is supported. Adding regions via s/g list is
109 supported only up to a s/g list size of 1. No support for binding mem‐
110 ory regions to a counter.
111 Wait objects
113 Only FI_WAIT_FD wait object is supported only for FI_EP_MSG endpoint
114 type. Wait sets are not supported.
115 Resource Management
117 Application has to make sure CQs are not overrun as this cannot be de‐
118 tected by the provider.
119 Unsupported Features
121 The following features are not supported in verbs provider:
122 Unsupported Capabilities
124 FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
125 Other unsupported features
127 Scalable endpoints, FABRIC_DIRECT
128 Unsupported features specific to MSG endpoints
130 · Counters, FI_SOURCE, FI_TAGGED, FI_PEEK, FI_CLAIM, fi_cancel,
131 fi_ep_alias, shared TX context, cq_readfrom operations.
132 · Completion flags are not reported if a request posted to an endpoint
134 completes in error.
135 Fork
137 The support for fork in the provider has the following limitations:
138 · Fabric resources like endpoint, CQ, EQ, etc. should not be used in
140 the forked process.
141 · The memory registered using fi_mr_reg has to be page aligned since
143 ibv_reg_mr marks the entire page that a memory region belongs to as
144 not to be re-mapped when the process is forked (MADV_DONTFORK).
145 XRC Transport
147 The XRC transport is intended to be used when layered with the RXM
148 provider and requires the use of shared receive contexts. See
149 fi_rxm(7).
150
152 The verbs provider checks for the following environment variables.
153 Common variables:
155 FI_VERBS_TX_SIZE
156 Default maximum tx context size (default: 384)
157 FI_VERBS_RX_SIZE
159 Default maximum rx context size (default: 384)
160 FI_VERBS_TX_IOV_LIMIT
162 Default maximum tx iov_limit (default: 4). Note: RDM (internal
163 - deprecated) EP type supports only 1
164 FI_VERBS_RX_IOV_LIMIT
166 Default maximum rx iov_limit (default: 4). Note: RDM (internal
167 - deprecated) EP type supports only 1
168 FI_VERBS_INLINE_SIZE
170 Default maximum inline size. Actual inject size returned in
171 fi_info may be greater (default: 64)
172 FI_VERBS_MIN_RNR_TIMER
174 Set min_rnr_timer QP attribute (0 - 31) (default: 12)
175 FI_VERBS_USE_ODP
177 Enable On-Demand-Paging (ODP) experimental feature. The feature
178 is supported only on Mellanox OFED (default: 0)
179 FI_VERBS_CQREAD_BUNCH_SIZE
181 The number of entries to be read from the verbs completion queue
182 at a time (default: 8).
183 FI_VERBS_IFACE
185 The prefix or the full name of the network interface associated
186 with the verbs device (default: ib)
187 FI_VERBS_PREFER_XRC
189 Prioritize XRC transport fi_info before RC transport fi_info
190 (default: 0, RC fi_info will be before XRC fi_info)
191 Variables specific to DGRAM endpoints
193 FI_VERBS_DGRAM_USE_NAME_SERVER
194 The option that enables/disables OFI Name Server thread. The NS
195 thread is used to resolve IP-addresses to provider specific ad‐
196 dresses (default: 1, if "OMPI_COMM_WORLD_RANK" and "PMI_RANK"
197 environment variables aren't defined)
198 FI_VERBS_NAME_SERVER_PORT
200 The port on which Name Server thread listens incoming connec‐
201 tions and requests (default: 5678)
202 FI_VERBS_GID_IDX
204 The GID index to use (default: 0)
205 Environment variables notes
207 The fi_info utility would give the up-to-date information on environ‐
208 ment variables: fi_info -p verbs -e
209
211 fi_getinfo returns -FI_ENODATA
212 · Set FI_LOG_LEVEL=info or FI_LOG_LEVEL=debug (if debug build of lib‐
213 fabric is available) and check if there any errors because of incor‐
214 rect input parameters to fi_getinfo.
215 · Check if "fi_info -p verbs" is successful. If that fails the follow‐
217 ing chkecklist may help in ensuring that the RDMA verbs stack is
218 functional:
219 · If libfabric was compiled, check if verbs provider was built. Build‐
221 ing verbs provider would be skipped if its dependencies (listed in
222 requirements) aren't available on the system.
223 · Verify verbs device is functional:
225 · Does ibv_rc_pingpong (available in libibverbs) test work?
227 · Does ibv_devinfo (available in libibverbs) show the device with
229 PORT_ACTIVE status?
230 · Check if Subnet Manager (SM) is running on the switch or on one
232 of the nodes in the cluster.
233 · Is the cable connected?
235 · Verify librdmacm is functional:
237 · Does ucmatose test (available in librdmacm) work?
239 · Is the IPoIB interface (e.g. ib0) up and configured with a valid
241 IP address?
242 Other issues
244 When running an app over verbs provider with Valgrind, there may be re‐
245 ports of memory leak in functions from dependent libraries (e.g. li‐
246 bibverbs, librdmacm). These leaks are safe to ignore.
247 The provider protects CQ overruns that may happen because more TX oper‐
249 ations were posted to endpoints than CQ size. On the receive side, it
250 isn't expected to overrun the CQ. In case it happens the application
251 developer should take care not to post excess receives without draining
252 the CQ. CQ overruns can make the MSG endpoints unusable.
253
255 fabric(7), fi_provider(7),
256
258 OpenFabrics.
259Libfabric Programmer's Manual 2019-06-21 fi_verbs(7)