fi_verbs(7)

1fi_verbs(7)                    Libfabric v1.6.1                    fi_verbs(7)
2
3
4

NAME

6       fi_verbs - The Verbs Fabric Provider
7

OVERVIEW

9       The  verbs  provider  enables applications using OFI to be run over any
10       verbs hardware (Infiniband, iWarp, etc).  It uses the Linux  Verbs  API
11       for network transport and provides a translation of OFI calls to appro‐
12       priate verbs API calls.  It uses librdmacm for communication management
13       and libibverbs for other control and data transfer operations.
14

SUPPORTED FEATURES

16       The verbs provider supports a subset of OFI features.
17
18   Endpoint types
19       FI_EP_MSG, FI_EP_RDM
20
21       New  change  in  libfabric v1.6: FI_EP_RDM is supported through the OFI
22       RxM utility provider.  This is done automatically when the app requests
23       FI_EP_RDM  endpoint.   Please  refer  the  man page for RxM provider to
24       learn more.  The provider's internal support for RDM endpoints is  dep‐
25       recated  and  would  be removed from libfabric v1.7 onwards.  Till then
26       apps can explicitly request  the  internal  RDM  support  by  disabling
27       ofi_rxm      provider      through     FI_PROVIDER     env     variable
28       (FI_PROVIDER=^ofi_rxm).
29
30   Endpoint capabilities and features
31   MSG endpoints
32       FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
33
34   RDM endpoints (internal - deprecated)
35       FI_MSG, FI_TAGGED, FI_RMA
36
37   DGRAM endpoints
38       FI_MSG
39
40   Modes
41       Verbs provider requires applications to support the following modes:
42
43   FI_EP_MSG endpoint type
44       · FI_LOCAL_MR / FI_MR_LOCAL mr mode.
45
46       · FI_RX_CQ_DATA for applications that want to  use  RMA.   Applications
47         must  take  responsibility  of  posting  receives for any incoming CQ
48         data.
49
50   FI_EP_RDM endpoint type (internal - deprecated)
51       · FI_CONTEXT
52
53   Addressing Formats
54       Supported addressing formats include * MSG and RDM (internal  -  depre‐
55       cated)   EPs  support:  FI_SOCKADDR,  FI_SOCKADDR_IN,  FI_SOCKADDR_IN6,
56       FI_SOCKADDR_IB * DGRAM supports: FI_ADDR_IB_UD
57
58   Progress
59       Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations  make
60       forward progress automatically.
61
62   Operation flags
63       Verbs  provider  supports  FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA,
64       FI_TRANSMIT_COMPLETE.
65
66   Msg Ordering
67       Verbs provider support the following message ordering:
68
69       · Read after Read
70
71       · Read after Write
72
73       · Read after Send
74
75       · Write after Write
76
77       · Write after Send
78
79       · Send after Write
80
81       · Send after Send
82
83       and the following completion ordering:
84
85       · TX contexts: FI_ORDER_STRICT
86
87       · RX contexts: FI_ORDER_DATA
88
89   Fork
90       Verbs provider supports the fork system call by default.  See the limi‐
91       tations  section for restrictions.  It can be turned off by setting the
92       FI_FORK_UNSAFE environment variable to "yes".   This  can  improve  the
93       performance  of  memory registrations but it also makes the use of fork
94       unsafe.
95
96   Memory Registration Cache
97       The verbs provider features a memory registration cache.   This  speeds
98       up memory registration calls from applications by caching registrations
99       of frequently used memory regions.  The user can  control  the  maximum
100       combined  size  of  all  cache  entries and the maximum number of cache
101       entries with the environment variables FI_VERBS_MR_MAX_CACHED_SIZE  and
102       FI_VERBS_MR_MAX_CACHED_CNT respectively.  Look below in the environment
103       variables section for details.
104
105       Note: The memory registration cache framework hooks into alloc and free
106       calls  to monitor the memory regions.  If this doesn't work as expected
107       caching would not be optimal.
108

LIMITATIONS

110   Memory Regions
111       Only FI_MR_BASIC mode is supported.  Adding regions  via  s/g  list  is
112       supported only up to a s/g list size of 1.  No support for binding mem‐
113       ory regions to a counter.
114
115   Wait objects
116       Only FI_WAIT_FD wait object is supported only  for  FI_EP_MSG  endpoint
117       type.  Wait sets are not supported.
118
119   Resource Management
120       Application  has  to  make  sure  CQs are not overrun as this cannot be
121       detected by the provider.
122
123   Unsupported Features
124       The following features are not supported in verbs provider:
125
126   Unsupported Capabilities
127       FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_RMA_EVENT
128
129   Other unsupported features
130       Scalable endpoints, FABRIC_DIRECT
131
132   Unsupported features specific to MSG endpoints
133       · Counters,  FI_SOURCE,  FI_TAGGED,   FI_PEEK,   FI_CLAIM,   fi_cancel,
134         fi_ep_alias, shared TX context, cq_readfrom operations.
135
136       · Completion  flags are not reported if a request posted to an endpoint
137         completes in error.
138
139   Unsupported features specific to RDM (internal - deprecated)
140       endpoints
141
142       The RDM support for verbs have the following limitations:
143
144       · Supports iovs of only size 1.
145
146       · Wait objects are not supported.
147
148       · Not thread safe.
149
150   Fork
151       The support for fork in the provider has the following limitations:
152
153       · Fabric resources like endpoint, CQ, EQ, etc.  should not be  used  in
154         the forked process.
155
156       · The  memory  registered  using fi_mr_reg has to be page aligned since
157         ibv_reg_mr marks the entire page that a memory region belongs  to  as
158         not to be re-mapped when the process is forked (MADV_DONTFORK).
159

RUNTIME PARAMETERS

161       The verbs provider checks for the following environment variables.
162
163   Common variables:
164       FI_VERBS_TX_SIZE : Default maximum tx context size (default: 384)
165
166       FI_VERBS_RX_SIZE : Default maximum rx context size (default: 384)
167
168       FI_VERBS_TX_IOV_LIMIT  :  Default  maximum  tx  iov_limit (default: 4).
169       Note: RDM (internal - deprecated) EP type supports only 1
170
171       FI_VERBS_RX_IOV_LIMIT : Default  maximum  rx  iov_limit  (default:  4).
172       Note: RDM (internal - deprecated) EP type supports only 1
173
174       FI_VERBS_INLINE_SIZE : Default maximum inline size.  Actual inject size
175       returned in fi_info may be greater (default: 64)
176
177       FI_VERBS_MIN_RNR_TIMER :  Set  min_rnr_timer  QP  attribute  (0  -  31)
178       (default: 12)
179
180       FI_VERBS_USE_ODP  : Enable On-Demand-Paging (ODP) experimental feature.
181       The feature is supported only on Mellanox OFED (default: 0)
182
183       FI_VERBS_CQREAD_BUNCH_SIZE : The number of entries to be read from  the
184       verbs completion queue at a time (default: 8).
185
186       FI_VERBS_IFACE  :  The prefix or the full name of the network interface
187       associated with the verbs device (default: ib)
188
189       FI_VERBS_MR_CACHE_ENABLE : Enable Memory Registration caching (default:
190       0)
191
192       FI_VERBS_MR_MAX_CACHED_CNT  : Maximum number of cache entries (default:
193       4096)
194
195       FI_VERBS_MR_MAX_CACHED_SIZE :  Maximum  total  size  of  cache  entries
196       (default: 4 GB)
197
198   Variables specific to RDM (internal - deprecated) endpoints
199       FI_VERBS_RDM_BUFFER_NUM  :  The  number  of  pre-registered buffers for
200       buffered operations between  the  endpoints,  must  be  a  power  of  2
201       (default: 8).
202
203       FI_VERBS_RDM_BUFFER_SIZE  :  The  maximum  size of a buffered operation
204       (bytes) (default: platform specific).
205
206       FI_VERBS_RDM_RNDV_SEG_SIZE : The segment size for zero  copy  protocols
207       (bytes)(default: 1073741824).
208
209       FI_VERBS_RDM_THREAD_TIMEOUT  : The wake up timeout of the helper thread
210       (usec) (default: 100).
211
212       FI_VERBS_RDM_EAGER_SEND_OPCODE : The operation code that will  be  used
213       for  eager  messaging.  Only IBV_WR_SEND and IBV_WR_RDMA_WRITE_WITH_IMM
214       are supported.  The last one is not applicable  for  iWarp.   (default:
215       IBV_WR_SEND)
216
217       FI_VERBS_RDM_CM_THREAD_AFFINITY  :  If specified, bind the CM thread to
218       the indicated range(s) of Linux virtual processor ID(s).   This  option
219       is     currently     not     supported     on     OS     X.      Usage:
220       id_start[-id_end[:stride]][,]
221
222   Variables specific to DGRAM endpoints
223       FI_VERBS_DGRAM_USE_NAME_SERVER : The option that  enables/disables  OFI
224       Name  Server  thread.  The NS thread is used to resolve IP-addresses to
225       provider specific addresses (default: 1, if "OMPI_COMM_WORLD_RANK"  and
226       "PMI_RANK" environment variables aren't defined)
227
228       FI_VERBS_NAME_SERVER_PORT  :  The port on which Name Server thread lis‐
229       tens incoming connections and requests (default: 5678)
230
231   Environment variables notes
232       The fi_info utility would give the up-to-date information  on  environ‐
233       ment variables: fi_info -p verbs -e
234

Troubleshooting / Known issues

236       When  running  an  app  over verbs provider with Valgrind, there may be
237       reports of memory leak in  functions  from  dependent  libraries  (e.g.
238       libibverbs, librdmacm).  These leaks are safe to ignore.
239

AUTHORS

244       OpenFabrics.
245
246
247
248Libfabric Programmer's Manual     2018-03-01                       fi_verbs(7)