1fi_verbs(7) Libfabric v1.7.0 fi_verbs(7)
2
6 fi_verbs - The Verbs Fabric Provider
7
9 The verbs provider enables applications using OFI to be run over any
10 verbs hardware (Infiniband, iWarp, etc). It uses the Linux Verbs API
11 for network transport and provides a translation of OFI calls to appro‐
12 priate verbs API calls. It uses librdmacm for communication management
13 and libibverbs for other control and data transfer operations.
14
16 The verbs provider supports a subset of OFI features.
17 Endpoint types
19 FI_EP_MSG, FI_EP_RDM
20 New change in libfabric v1.6: FI_EP_RDM is supported through the OFI
22 RxM utility provider. This is done automatically when the app requests
23 FI_EP_RDM endpoint. Please refer the man page for RxM provider to
24 learn more. The provider's internal support for RDM endpoints is dep‐
25 recated and would be removed from libfabric v1.7 onwards. Till then
26 apps can explicitly request the internal RDM support by disabling
27 ofi_rxm provider through FI_PROVIDER env variable
28 (FI_PROVIDER=^ofi_rxm).
29 Endpoint capabilities and features
31 MSG endpoints
32 FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
33 RDM endpoints (internal - deprecated)
35 FI_MSG, FI_TAGGED, FI_RMA
36 DGRAM endpoints
38 FI_MSG
39 Modes
41 Verbs provider requires applications to support the following modes:
42 FI_EP_MSG endpoint type
44 · FI_LOCAL_MR / FI_MR_LOCAL mr mode.
45 · FI_RX_CQ_DATA for applications that want to use RMA. Applications
47 must take responsibility of posting receives for any incoming CQ da‐
48 ta.
49 FI_EP_RDM endpoint type (internal - deprecated)
51 · FI_CONTEXT
52 Addressing Formats
54 Supported addressing formats include * MSG and RDM (internal - depre‐
55 cated) EPs support: FI_SOCKADDR, FI_SOCKADDR_IN, FI_SOCKADDR_IN6,
56 FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
57 Progress
59 Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations make
60 forward progress automatically.
61 Operation flags
63 Verbs provider supports FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
64 FI_TRANSMIT_COMPLETE.
65 Msg Ordering
67 Verbs provider support the following message ordering:
68 · Read after Read
70 · Read after Write
72 · Read after Send
74 · Write after Write
76 · Write after Send
78 · Send after Write
80 · Send after Send
82 and the following completion ordering:
84 · TX contexts: FI_ORDER_STRICT
86 · RX contexts: FI_ORDER_DATA
88 Fork
90 Verbs provider supports the fork system call by default. See the limi‐
91 tations section for restrictions. It can be turned off by setting the
92 FI_FORK_UNSAFE environment variable to "yes". This can improve the
93 performance of memory registrations but it also makes the use of fork
94 unsafe.
95 Memory Registration Cache
97 The verbs provider features a memory registration cache. This speeds
98 up memory registration calls from applications by caching registrations
99 of frequently used memory regions. The user can control the maximum
100 combined size of all cache entries and the maximum number of cache en‐
101 tries with the environment variables FI_VERBS_MR_MAX_CACHED_SIZE and
102 FI_VERBS_MR_MAX_CACHED_CNT respectively. Look below in the environment
103 variables section for details.
104 Note: The memory registration cache framework hooks into alloc and free
106 calls to monitor the memory regions. If this doesn't work as expected
107 caching would not be optimal.
108
110 Memory Regions
111 Only FI_MR_BASIC mode is supported. Adding regions via s/g list is
112 supported only up to a s/g list size of 1. No support for binding mem‐
113 ory regions to a counter.
114 Wait objects
116 Only FI_WAIT_FD wait object is supported only for FI_EP_MSG endpoint
117 type. Wait sets are not supported.
118 Resource Management
120 Application has to make sure CQs are not overrun as this cannot be de‐
121 tected by the provider.
122 Unsupported Features
124 The following features are not supported in verbs provider:
125 Unsupported Capabilities
127 FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
128 Other unsupported features
130 Scalable endpoints, FABRIC_DIRECT
131 Unsupported features specific to MSG endpoints
133 · Counters, FI_SOURCE, FI_TAGGED, FI_PEEK, FI_CLAIM, fi_cancel,
134 fi_ep_alias, shared TX context, cq_readfrom operations.
135 · Completion flags are not reported if a request posted to an endpoint
137 completes in error.
138 Unsupported features specific to RDM (internal - deprecated)
140 endpoints
141 The RDM support for verbs have the following limitations:
143 · Supports iovs of only size 1.
145 · Wait objects are not supported.
147 · Not thread safe.
149 Fork
151 The support for fork in the provider has the following limitations:
152 · Fabric resources like endpoint, CQ, EQ, etc. should not be used in
154 the forked process.
155 · The memory registered using fi_mr_reg has to be page aligned since
157 ibv_reg_mr marks the entire page that a memory region belongs to as
158 not to be re-mapped when the process is forked (MADV_DONTFORK).
159 XRC Transport
161 The XRC transport is intended to be used when layered with the RXM
162 provider and requires the use of shared receive contexts. See
163 fi_rxm(7).
164
166 The verbs provider checks for the following environment variables.
167 Common variables:
169 FI_VERBS_TX_SIZE
170 Default maximum tx context size (default: 384)
171 FI_VERBS_RX_SIZE
173 Default maximum rx context size (default: 384)
174 FI_VERBS_TX_IOV_LIMIT
176 Default maximum tx iov_limit (default: 4). Note: RDM (internal
177 - deprecated) EP type supports only 1
178 FI_VERBS_RX_IOV_LIMIT
180 Default maximum rx iov_limit (default: 4). Note: RDM (internal
181 - deprecated) EP type supports only 1
182 FI_VERBS_INLINE_SIZE
184 Default maximum inline size. Actual inject size returned in
185 fi_info may be greater (default: 64)
186 FI_VERBS_MIN_RNR_TIMER
188 Set min_rnr_timer QP attribute (0 - 31) (default: 12)
189 FI_VERBS_USE_ODP
191 Enable On-Demand-Paging (ODP) experimental feature. The feature
192 is supported only on Mellanox OFED (default: 0)
193 FI_VERBS_CQREAD_BUNCH_SIZE
195 The number of entries to be read from the verbs completion queue
196 at a time (default: 8).
197 FI_VERBS_IFACE
199 The prefix or the full name of the network interface associated
200 with the verbs device (default: ib)
201 FI_VERBS_MR_CACHE_ENABLE
203 Enable Memory Registration caching (default: 0)
204 FI_VERBS_MR_MAX_CACHED_CNT
206 Maximum number of cache entries (default: 4096)
207 FI_VERBS_MR_MAX_CACHED_SIZE
209 Maximum total size of cache entries (default: 4 GB)
210 FI_VERBS_PREFER_XRC
212 Prioritize XRC transport fi_info before RC transport fi_info
213 (default: 0, RC fi_info will be before XRC fi_info)
214 Variables specific to RDM (internal - deprecated) endpoints
216 FI_VERBS_RDM_BUFFER_NUM
217 The number of pre-registered buffers for buffered operations be‐
218 tween the endpoints, must be a power of 2 (default: 8).
219 FI_VERBS_RDM_BUFFER_SIZE
221 The maximum size of a buffered operation (bytes) (default: plat‐
222 form specific).
223 FI_VERBS_RDM_RNDV_SEG_SIZE
225 The segment size for zero copy protocols (bytes)(default:
226 1073741824).
227 FI_VERBS_RDM_THREAD_TIMEOUT
229 The wake up timeout of the helper thread (usec) (default: 100).
230 FI_VERBS_RDM_EAGER_SEND_OPCODE
232 The operation code that will be used for eager messaging. Only
233 IBV_WR_SEND and IBV_WR_RDMA_WRITE_WITH_IMM are supported. The
234 last one is not applicable for iWarp. (default: IBV_WR_SEND)
235 FI_VERBS_RDM_CM_THREAD_AFFINITY
237 If specified, bind the CM thread to the indicated range(s) of
238 Linux virtual processor ID(s). This option is currently not
239 supported on OS X. Usage: id_start[-id_end[:stride]][,]
240 Variables specific to DGRAM endpoints
242 FI_VERBS_DGRAM_USE_NAME_SERVER
243 The option that enables/disables OFI Name Server thread. The NS
244 thread is used to resolve IP-addresses to provider specific ad‐
245 dresses (default: 1, if "OMPI_COMM_WORLD_RANK" and "PMI_RANK"
246 environment variables aren't defined)
247 FI_VERBS_NAME_SERVER_PORT
249 The port on which Name Server thread listens incoming connec‐
250 tions and requests (default: 5678)
251 Environment variables notes
253 The fi_info utility would give the up-to-date information on environ‐
254 ment variables: fi_info -p verbs -e
255
257 When running an app over verbs provider with Valgrind, there may be re‐
258 ports of memory leak in functions from dependent libraries (e.g. li‐
259 bibverbs, librdmacm). These leaks are safe to ignore.
260
262 fabric(7), fi_provider(7),
263
265 OpenFabrics.
266Libfabric Programmer's Manual 2018-12-19 fi_verbs(7)