1fi_verbs(7) Libfabric v1.15.1 fi_verbs(7)
2
6 fi_verbs - The Verbs Fabric Provider
7
9 The verbs provider enables applications using OFI to be run over any
10 verbs hardware (Infiniband, iWarp, etc). It uses the Linux Verbs API
11 for network transport and provides a translation of OFI calls to appro‐
12 priate verbs API calls. It uses librdmacm for communication management
13 and libibverbs for other control and data transfer operations.
14
16 To successfully build and install verbs provider as part of libfabric,
17 it needs the following packages: * libibverbs * libibverbs-devel * li‐
18 brdmacm * librdmacm-devel
19 You may also want to look into any OS specific instructions for en‐
21 abling RDMA. e.g. RHEL has instructions on their documentation for en‐
22 abling RDMA.
23 The IPoIB interface should be configured with a valid IP address. This
25 is a requirement from librdmacm.
26
28 The verbs provider supports a subset of OFI features.
29 Endpoint types
31 FI_EP_MSG, FI_EP_DGRAM (beta), FI_EP_RDM.
32 FI_EP_RDM is supported via OFI RxM and RxD utility providers which are
34 layered on top of verbs. To the app, the provider name string would
35 appear as “verbs;ofi_rxm” or “verbs;ofi_rxd”. Please refer the man
36 pages for RxM (fi_rxm.7) and RxD (fi_rxd.7) to know about the capabili‐
37 ties and limitations for the FI_EP_RDM endpoint.
38 Endpoint capabilities and features
40 MSG endpoints
41 FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
42 DGRAM endpoints
44 FI_MSG
45 Modes
47 Verbs provider requires applications to support the following modes:
48 FI_EP_MSG endpoint type
50 • FI_LOCAL_MR / FI_MR_LOCAL mr mode.
51 • FI_RX_CQ_DATA for applications that want to use RMA. Applications
53 must take responsibility of posting receives for any incoming CQ da‐
54 ta.
55 Addressing Formats
57 Supported addressing formats include * MSG and RDM (internal - depre‐
58 cated) EPs support: FI_SOCKADDR, FI_SOCKADDR_IN, FI_SOCKADDR_IN6,
59 FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
60 Progress
62 Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations make
63 forward progress automatically.
64 Operation flags
66 Verbs provider supports FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
67 FI_TRANSMIT_COMPLETE.
68 Msg Ordering
70 Verbs provider support the following message ordering:
71 • Read after Read
73 • Read after Write
75 • Read after Send
77 • Write after Write
79 • Write after Send
81 • Send after Write
83 • Send after Send
85 and the following completion ordering:
87 • TX contexts: FI_ORDER_STRICT
89 • RX contexts: FI_ORDER_DATA
91 Fork
93 Verbs provider does not provide fork safety by default. Fork safety
94 can be requested by setting IBV_FORK_SAFE, or RDMAV_FORK_SAFE. If the
95 system configuration supports the use of huge pages, it is recommended
96 to set RDMAV_HUGEPAGES_SAFE. See ibv_fork_init(3) for additional de‐
97 tails.
98 Memory Registration Cache
100 The verbs provider uses the common memory registration cache function‐
101 ality that’s part of libfabric utility code. This speeds up memory
102 registration calls from applications by caching registrations of fre‐
103 quently used memory regions. Please refer to fi_mr(3): Memory Regis‐
104 tration Cache section for more details.
105
107 Memory Regions
108 Only FI_MR_BASIC mode is supported. Adding regions via s/g list is
109 supported only up to a s/g list size of 1. No support for binding mem‐
110 ory regions to a counter.
111 Wait objects
113 Only FI_WAIT_FD wait object is supported only for FI_EP_MSG endpoint
114 type. Wait sets are not supported.
115 Resource Management
117 Application has to make sure CQs are not overrun as this cannot be de‐
118 tected by the provider.
119 Unsupported Features
121 The following features are not supported in verbs provider:
122 Unsupported Capabilities
124 FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
125 Other unsupported features
127 Scalable endpoints, FABRIC_DIRECT
128 Unsupported features specific to MSG endpoints
130 • Counters, FI_SOURCE, FI_TAGGED, FI_PEEK, FI_CLAIM, fi_cancel,
131 fi_ep_alias, shared TX context, cq_readfrom operations.
132 • Completion flags are not reported if a request posted to an endpoint
134 completes in error.
135 Fork
137 The support for fork in the provider has the following limitations:
138 • Fabric resources like endpoint, CQ, EQ, etc. should not be used in
140 the forked process.
141 • The memory registered using fi_mr_reg has to be page aligned since
143 ibv_reg_mr marks the entire page that a memory region belongs to as
144 not to be re-mapped when the process is forked (MADV_DONTFORK).
145 XRC Transport
147 The XRC transport is intended to be used when layered with the RXM
148 provider and requires the use of shared receive contexts. See
149 fi_rxm(7). To enable XRC, the following environment variables must
150 usually be set: FI_VERBS_PREFER_XRC and FI_OFI_RXM_USE_SRX.
151
153 The verbs provider checks for the following environment variables.
154 Common variables:
156 FI_VERBS_TX_SIZE
157 Default maximum tx context size (default: 384)
158 FI_VERBS_RX_SIZE
160 Default maximum rx context size (default: 384)
161 FI_VERBS_TX_IOV_LIMIT
163 Default maximum tx iov_limit (default: 4). Note: RDM (internal
164 - deprecated) EP type supports only 1
165 FI_VERBS_RX_IOV_LIMIT
167 Default maximum rx iov_limit (default: 4). Note: RDM (internal
168 - deprecated) EP type supports only 1
169 FI_VERBS_INLINE_SIZE
171 Default maximum inline size. Actual inject size returned in
172 fi_info may be greater (default: 64)
173 FI_VERBS_MIN_RNR_TIMER
175 Set min_rnr_timer QP attribute (0 - 31) (default: 12)
176 FI_VERBS_CQREAD_BUNCH_SIZE
178 The number of entries to be read from the verbs completion queue
179 at a time (default: 8).
180 FI_VERBS_PREFER_XRC
182 Prioritize XRC transport fi_info before RC transport fi_info
183 (default: 0, RC fi_info will be before XRC fi_info)
184 FI_VERBS_GID_IDX
186 The GID index to use (default: 0)
187 FI_VERBS_DEVICE_NAME
189 Specify a specific verbs device to use by name
190 Variables specific to MSG endpoints
192 FI_VERBS_IFACE
193 The prefix or the full name of the network interface associated
194 with the verbs device (default: ib)
195 Variables specific to DGRAM endpoints
197 FI_VERBS_DGRAM_USE_NAME_SERVER
198 The option that enables/disables OFI Name Server thread. The NS
199 thread is used to resolve IP-addresses to provider specific ad‐
200 dresses (default: 1, if “OMPI_COMM_WORLD_RANK” and “PMI_RANK”
201 environment variables aren’t defined)
202 FI_VERBS_NAME_SERVER_PORT
204 The port on which Name Server thread listens incoming connec‐
205 tions and requests (default: 5678)
206 Environment variables notes
208 The fi_info utility would give the up-to-date information on environ‐
209 ment variables: fi_info -p verbs -e
210
212 fi_getinfo returns -FI_ENODATA
213 • Set FI_LOG_LEVEL=info or FI_LOG_LEVEL=debug (if debug build of lib‐
214 fabric is available) and check if there any errors because of incor‐
215 rect input parameters to fi_getinfo.
216 • Check if “fi_info -p verbs” is successful. If that fails the follow‐
218 ing checklist may help in ensuring that the RDMA verbs stack is func‐
219 tional:
220 • If libfabric was compiled, check if verbs provider was built.
222 Building verbs provider would be skipped if its dependencies (list‐
223 ed in requirements) aren’t available on the system.
224 • Verify verbs device is functional:
226 • Does ibv_rc_pingpong (available in libibverbs) test work?
228 • Does ibv_devinfo (available in libibverbs) show the device with
230 PORT_ACTIVE status?
231 • Check if Subnet Manager (SM) is running on the switch or on
233 one of the nodes in the cluster.
234 • Is the cable connected?
236 • Verify librdmacm is functional:
238 • Does ucmatose test (available in librdmacm) work?
240 • Is the IPoIB interface (e.g. ib0) up and configured with a valid
242 IP address?
243 Other issues
245 When running an app over verbs provider with Valgrind, there may be re‐
246 ports of memory leak in functions from dependent libraries (e.g. libib‐
247 verbs, librdmacm). These leaks are safe to ignore.
248 The provider protects CQ overruns that may happen because more TX oper‐
250 ations were posted to endpoints than CQ size. On the receive side, it
251 isn’t expected to overrun the CQ. In case it happens the application
252 developer should take care not to post excess receives without draining
253 the CQ. CQ overruns can make the MSG endpoints unusable.
254
256 fabric(7), fi_provider(7),
257
259 OpenFabrics.
260Libfabric Programmer’s Manual 2021-03-22 fi_verbs(7)