1fi_mr(3) Libfabric v1.6.1 fi_mr(3)
2
6 fi_mr - Memory region operations
7 fi_mr_reg / fi_mr_regv / fi_mr_regattr : Register local memory buffers
9 for direct fabric access
10 fi_close : Deregister registered memory buffers.
12 fi_mr_desc : Return a local descriptor associated with a registered
14 memory region
15 fi_mr_key : Return the remote key needed to access a registered memory
17 region
18 fi_mr_raw_attr : Return raw memory region attributes.
20 fi_mr_map_raw : Converts a raw memory region key into a key that is
22 usable for data transfer operations.
23 fi_mr_unmap_key : Releases a previously mapped raw memory region key.
25 fi_mr_bind : Associate a registered memory region with a completion
27 counter.
28 fi_mr_refresh : Updates the memory pages associated with a memory
30 region.
31 fi_mr_enable : Enables a memory region for use.
33
35 #include <rdma/fi_domain.h>
36 int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
38 uint64_t access, uint64_t offset, uint64_t requested_key,
39 uint64_t flags, struct fid_mr **mr, void *context);
40 int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
42 size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
43 uint64_t flags, struct fid_mr **mr, void *context);
44 int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
46 uint64_t flags, struct fid_mr **mr);
47 int fi_close(struct fid *mr);
49 void * fi_mr_desc(struct fid_mr *mr);
51 uint64_t fi_mr_key(struct fid_mr *mr);
53 int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
55 uint8_t *raw_key, size_t *key_size, uint64_t flags);
56 int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
58 uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
59 int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
61 int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
63 int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
65 uint64_t flags)
66 int fi_mr_enable(struct fid_mr *mr);
68
70 domain : Resource domain
71 mr : Memory region
73 bfid : Fabric identifier of an associated resource.
75 context : User specified context associated with the memory region.
77 buf : Memory buffer to register with the fabric hardware
79 len : Length of memory buffer to register
81 iov : Vectored memory buffer.
83 count : Count of vectored buffer entries.
85 access : Memory access permissions associated with registration
87 offset : Optional specified offset for accessing specified registered
89 buffers. This parameter is reserved for future use and must be 0.
90 requested_key : Optional requested remote key associated with regis‐
92 tered buffers.
93 attr : Memory region attributes
95 flags : Additional flags to apply to the operation.
97
99 Registered memory regions associate memory buffers with permissions
100 granted for access by fabric resources. A memory buffer must be regis‐
101 tered with a resource domain before it can be used as the target of a
102 remote RMA or atomic data transfer. Additionally, a fabric provider
103 may require that data buffers be registered before being used in local
104 transfers. Memory registration restrictions are controlled using a
105 separate set of mode bits, specified through the domain attributes
106 (mr_mode field).
107 The following apply to memory registration.
109 Scalable Memory Registration : By default, memory registration is con‐
111 sidered scalable. (For library versions 1.4 and earlier, this is indi‐
112 cated by setting mr_mode to FI_MR_SCALABLE, with the fi_info mode bit
113 FI_LOCAL_MR set to 0). For versions 1.5 and later, scalable is implied
114 by the lack of any mr_mode bits being set. The setting of mr_mode bits
115 therefore adjusts application behavior as described below. Default,
116 scalable registration has several properties.
117 In scalable mode, registration occurs on memory address ranges.
119 Because registration refers to memory regions, versus data buffers, the
120 address ranges given for a registration request do not need to map to
121 data buffers allocated by the application at the time the registration
122 call is made. That is, an application can register any range of
123 addresses in their virtual address space, whether or not those
124 addresses are backed by physical pages or have been allocated.
125 The resulting memory regions are accessible by peers starting at a base
127 address of 0. That is, the target address that is specified is a byte
128 offset into the registered region.
129 The application also selects the access key associated with the MR.
131 The key size is restricted to a maximum of 8 bytes.
132 With scalable registration, locally accessed data buffers are not reg‐
134 istered. This includes source buffers for all transmit operations --
135 sends, tagged sends, RMA, and atomics -- as well as buffers posted for
136 receive and tagged receive operations.
137 FI_MR_LOCAL : When the FI_MR_LOCAL mode bit is set, applications must
139 register all data buffers that will be accessed by the local hardware
140 and provide a valid mem_desc parameter into applicable data transfer
141 operations. When FI_MR_LOCAL is zero, applications are not required to
142 register data buffers before using them for local operations (e.g.
143 send and receive data buffers), and the mem_desc parameter into data
144 transfer operations is ignored.
145 A provider may hide local registration requirements from applications
147 by making use of an internal registration cache or similar mechanisms.
148 Such mechanisms, however, may negatively impact performance for some
149 applications, notably those which manage their own network buffers. In
150 order to support as broad range of applications as possible, without
151 unduly affecting their performance, applications that wish to manage
152 their own local memory registrations may do so by using the memory reg‐
153 istration calls.
154 Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
156 bit. When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
157 FI_MR_RAW : Raw memory regions are used to support providers with keys
159 larger than 64-bits or require setup at the peer. When the FI_MR_RAW
160 bit is set, applications must use fi_mr_raw_attr() locally and
161 fi_mr_map_raw() at the peer before targeting a memory region as part of
162 any data transfer request.
163 FI_MR_VIRT_ADDR : The FI_MR_VIRT_ADDR bit indicates that the provider
165 references memory regions by virtual address, rather than a 0-based
166 offset. Peers that target memory regions registered with
167 FI_MR_VIRT_ADDR specify the destination memory buffer using the tar‐
168 get's virtual address, with any offset into the region specified as
169 virtual address + offset. Support of this bit typically implies that
170 peers must exchange addressing data prior to initiating any RMA or
171 atomic operation.
172 FI_MR_ALLOCATED : When set, all registered memory regions must be
174 backed by physical memory pages at the time the registration call is
175 made.
176 FI_MR_PROV_KEY : This memory region mode indicates that the provider
178 does not support application requested MR keys. MR keys are returned
179 by the provider. Applications that support FI_MR_PROV_KEY can obtain
180 the provider key using fi_mr_key(), unless FI_MR_RAW is also set. The
181 returned key should then be exchanged with peers prior to initiating an
182 RMA or atomic operation.
183 FI_MR_MMU_NOTIFY : FI_MR_MMU_NOTIFY is typically set by providers that
185 support memory registration against memory regions that are not neces‐
186 sarily backed by allocated physical pages at the time the memory regis‐
187 tration occurs. (That is, FI_MR_ALLOCATED is typically 0). However,
188 such providers require that applications notify the provider prior to
189 the MR being accessed as part of a data transfer operation. This noti‐
190 fication informs the provider that all necessary physical pages now
191 back the region. The notification is necessary for providers that can‐
192 not hook directly into the operating system page tables or memory man‐
193 agement unit. See fi_mr_refresh() for notification details.
194 FI_MR_RMA_EVENT : This mode bit indicates that the provider must con‐
196 figure memory regions that are associated with RMA events prior to
197 their use. This includes all memory regions that are associated with
198 completion counters. When set, applications must indicate if a memory
199 region will be associated with a completion counter as part of the
200 region's creation. This is done by passing in the FI_RMA_EVENT flag to
201 the memory registration call.
202 Such memory regions will be created in a disabled state and must be
204 associated with all completion counters prior to being enabled. To
205 enable a memory region, the application must call fi_mr_enable().
206 After calling fi_mr_enable(), no further resource bindings may be made
207 to the memory region.
208 FI_MR_ENDPOINT : This mode bit indicates that the provider associates
210 memory regions with endpoints rather than domains. Memory regions that
211 are registered with the provider are created in a disabled state and
212 must be bound to an endpoint prior to being enabled. To bind the MR
213 with an endpoint, the application must use fi_mr_bind(). To enable the
214 memory region, the application must call fi_mr_enable().
215 Basic Memory Registration : Basic memory registration is indicated by
217 the FI_MR_BASIC mr_mode bit. FI_MR_BASIC is maintained for backwards
218 compatibility (libfabric version 1.4 or earlier). The behavior of
219 basic registration is equivalent to setting the following mr_mode bits
220 to one: FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, and FI_MR_PROV_KEY. Addi‐
221 tionally, providers that support basic registration usually required
222 FI_MR_LOCAL. FI_MR_BASIC must either be set alone, or in conjunction
223 with FI_MR_LOCAL. Other mr_mode bit pairings are invalid. Unlike
224 other mr_mode bits, if FI_MR_BASIC is set on input to fi_getinfo(), it
225 will not be cleared by the provider. That is, setting FI_MR_BASIC to
226 one requests basic registration.
227 The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
229 -- are used to register one or more memory regions with fabric
230 resources. The main difference between registration functions are the
231 number and type of parameters that they accept as input. Otherwise,
232 they perform the same general function.
233 By default, memory registration completes synchronously. I.e. the
235 registration call will not return until the registration has completed.
236 Memory registration can complete asynchronous by binding the resource
237 domain to an event queue using the FI_REG_MR flag. See fi_domain_bind.
238 When memory registration is asynchronous, in order to avoid a race con‐
239 dition between the registration call returning and the corresponding
240 reading of the event from the EQ, the mr output parameter will be writ‐
241 ten before any event associated with the operation may be read by the
242 application. An asynchronous event will not be generated unless the
243 registration call returns success (0).
244 fi_mr_reg
246 The fi_mr_reg call registers the user-specified memory buffer with the
247 resource domain. The buffer is enabled for access by the fabric hard‐
248 ware based on the provided access permissions. See the access field
249 description for memory region attributes below.
250 Registered memory is associated with a local memory descriptor and,
252 optionally, a remote memory key. A memory descriptor is a provider
253 specific identifier associated with registered memory. Memory descrip‐
254 tors often map to hardware specific indices or keys associated with the
255 memory region. Remote memory keys provide limited protection against
256 unwanted access by a remote node. Remote accesses to a memory region
257 must provide the key associated with the registration.
258 Because MR keys must be provided by a remote process, an application
260 can use the requested_key parameter to indicate that a specific key
261 value be returned. Support for user requested keys is provider spe‐
262 cific and is determined by the mr_mode domain attribute.
263 Remote RMA and atomic operations indicate the location within a regis‐
265 tered memory region by specifying an address. The location is refer‐
266 enced by adding the offset to either the base virtual address of the
267 buffer or to 0, depending on the mr_mode.
268 The offset parameter is reserved for future use and must be 0.
270 For asynchronous memory registration requests, the result will be
272 reported to the user through an event queue associated with the
273 resource domain. If successful, the allocated memory region structure
274 will be returned to the user through the mr parameter. The mr address
275 must remain valid until the registration operation completes. The con‐
276 text specified with the registration request is returned with the com‐
277 pletion event.
278 fi_mr_regv
280 The fi_mr_regv call adds support for a scatter-gather list to
281 fi_mr_reg. Multiple memory buffers are registered as a single memory
282 region. Otherwise, the operation is the same.
283 fi_mr_regattr
285 The fi_mr_regattr call is a more generic, extensible registration call
286 that allows the user to specify the registration request using a struct
287 fi_mr_attr (defined below).
288 fi_close
290 Fi_close is used to release all resources associated with a registering
291 a memory region. Once unregistered, further access to the registered
292 memory is not guaranteed. Active or queued operations that reference a
293 memory region being closed may fail or result in accesses to invalid
294 memory. Applications are responsible for ensuring that a MR is no
295 longer needed prior to closing it. Note that accesses to a closed MR
296 from a remote peer will result in an error at the peer. The state of
297 the local endpoint will be unaffected.
298 When closing the MR, there must be no opened endpoints or counters
300 associated with the MR. If resources are still associated with the MR
301 when attempting to close, the call will return -FI_EBUSY.
302 fi_mr_desc
304 Obtains the local memory descriptor associated with a MR. The memory
305 registration must have completed successfully before invoking this
306 call.
307 fi_mr_key
309 Returns the remote protection key associated with a MR. The memory
310 registration must have completed successfully before invoking this.
311 The returned key may be used in data transfer operations at a peer. If
312 the FI_RAW_MR mode bit has been set for the domain, then the memory key
313 must be obtained using the fi_mr_raw_key function instead. A return
314 value of FI_KEY_NOTAVAIL will be returned if the registration has not
315 completed or a raw memory key is required.
316 fi_mr_raw_attr
318 Returns the raw, remote protection key and base address associated with
319 a MR. The memory registration must have completed successfully before
320 invoking this routine. Use of this call is required if the FI_RAW_MR
321 mode bit has been set by the provider; however, it is safe to use this
322 call with any memory region.
323 A raw key must be mapped by a peer before it can be used in data trans‐
325 fer operations. See fi_mr_map_raw below.
326 fi_mr_map_raw
328 Raw protection keys must be mapped to a usable key value before they
329 can be used for data transfer operations. The mapping is done by the
330 peer that initiates the RMA or atomic operation. The mapping function
331 takes as input the raw key and its size, and returns the mapped key.
332 Use of the fi_mr_map_raw function is required if the peer has the
333 FI_RAW_MR mode bit set, but this routine may be called on any valid
334 key. All mapped keys must be freed by calling fi_mr_unmap_key when
335 access to the peer memory region is no longer necessary.
336 fi_mr_unmap_key
338 This call releases any resources that may have been allocated as part
339 of mapping a raw memory key. All mapped keys must be freed before the
340 corresponding domain is closed.
341 fi_mr_bind
343 The fi_mr_bind function associates a memory region with a counter or
344 endpoint. Counter bindings are needed by providers that support the
345 generation of completions based on fabric operations. Endpoint bind‐
346 ings are needed if the provider associates memory regions with end‐
347 points (see FI_MR_ENDPOINT).
348 When binding with a counter, the type of events tracked against the
350 memory region is based on the bitwise OR of the following flags.
351 FI_REMOTE_WRITE : Generates an event whenever a remote RMA write or
353 atomic operation modifies the memory region. Use of this flag requires
354 that the endpoint through which the MR is accessed be created with the
355 FI_RMA_EVENT capability.
356 When binding the memory region to an endpoint, flags should be 0.
358 fi_mr_refresh
360 The use of this call is required to notify the provider of any change
361 to the physical pages backing a registered memory region if the
362 FI_MR_MMU_NOTIFY mode bit has been set. This call informs the provider
363 that the page table entries associated with the region may have been
364 modified, and the provider should verify and update the registered
365 region accordingly. The iov parameter is optional and may be used to
366 specify which portions of the registered region requires updating.
367 Providers are only guaranteed to update the specified address ranges.
368 The refresh operation has the effect of disabling and re-enabling
370 access to the registered region. Any operations from peers that
371 attempt to access the region will fail while the refresh is occurring.
372 Additionally, attempts to access the region by the local process
373 through libfabric APIs may result in a page fault or other fatal opera‐
374 tion.
375 The fi_mr_refresh call is only needed if the physical pages might have
377 been updated after the memory region was created.
378 fi_mr_enable
380 The enable call is used with memory registration associated with the
381 FI_MR_RMA_EVENT mode bit. Memory regions created in the disabled state
382 must be explicitly enabled after being fully configured by the applica‐
383 tion. Any resource bindings to the MR must be done prior to enabling
384 the MR.
385
387 Memory regions are created using the following attributes. The struct
388 fi_mr_attr is passed into fi_mr_regattr, but individual fields also
389 apply to other memory registration calls, with the fields passed
390 directly into calls as function parameters.
391 struct fi_mr_attr {
393 const struct iovec *mr_iov;
394 size_t iov_count;
395 uint64_t access;
396 uint64_t requested_key;
397 void *context;
398 size_t auth_key_size;
399 uint8_t *auth_key;
400 };
401 mr_iov
403 This is an IO vector of addresses that will represent a single memory
404 region. The number of entries in the iovec is specified by iov_count.
405 iov_count
407 The number of entries in the mr_iov array. The maximum number of mem‐
408 ory buffers that may be associated with a single memory region is spec‐
409 ified as the mr_iov_limit domain attribute. See fi_domain(3).
410 access
412 Indicates the type of access that the local or a peer endpoint has to
413 the registered memory region. Supported access permissions are the
414 bitwise OR of the following flags:
415 FI_SEND : The memory buffer may be used in outgoing message data trans‐
417 fers. This includes fi_msg and fi_tagged operations.
418 FI_RECV : The memory buffer may be used to receive inbound message
420 transfers. This includes fi_msg and fi_tagged operations.
421 FI_READ : The memory buffer may be used as the result buffer for RMA
423 read and atomic operations on the initiator side.
424 FI_WRITE : The memory buffer may be used as the source buffer for RMA
426 write and atomic operations on the initiator side.
427 FI_REMOTE_READ : The memory buffer may be used as the source buffer of
429 an RMA read operation on the target side.
430 FI_REMOTE_WRITE : The memory buffer may be used as the target buffer of
432 an RMA write or atomic operation.
433 requested_key
435 An application specified access key associated with the memory region.
436 The MR key must be provided by a remote process when performing RMA or
437 atomic operations to a memory region. Applications can use the
438 requested_key field to indicate that a specific key be used by the
439 provider. This allows applications to use well known key values, which
440 can avoid applications needing to exchange and store keys. Support for
441 user requested keys is provider specific and is determined by the
442 mr_mode domain attribute.
443 context
445 Application context associated with asynchronous memory registration
446 operations. This value is returned as part of any asynchronous event
447 associated with the registration. This field is ignored for synchro‐
448 nous registration calls.
449 auth_key_size
451 The size of key referenced by the auth_key field in bytes, or 0 if no
452 authorization key is given. This field is ignored unless the fabric is
453 opened with API version 1.5 or greater.
454 auth_key
456 Indicates the key to associate with this memory registration. Autho‐
457 rization keys are used to limit communication between endpoints. Only
458 peer endpoints that are programmed to use the same authorization key
459 may access the memory region. The domain authorization key will be
460 used if the auth_key_size provided is 0. This field is ignored unless
461 the fabric is opened with API version 1.5 or greater.
462
464 Direct access to an application's memory by a remote peer requires that
465 the application register the targeted memory buffer(s). This is typi‐
466 cally done by calling one of the fi_mr_reg* routines. For
467 FI_MR_PROV_KEY, the provider will return a key that must be used by the
468 peer when accessing the memory region. The application is responsible
469 for transferring this key to the peer. If FI_MR_RAW mode has been set,
470 the key must be retrieved using the fi_mr_raw_attr function.
471 FI_RAW_MR allows support for providers that require more than 8-bytes
473 for their protection keys or need additional setup before a key can be
474 used for transfers. After a raw key has been retrieved, it must be
475 exchanged with the remote peer. The peer must use fi_mr_map_raw to
476 convert the raw key into a usable 64-bit key. The mapping must be done
477 even if the raw key is 64-bits or smaller.
478 The raw key support functions are usable with all registered memory
480 regions, even if FI_MR_RAW has not been set. It is recommended that
481 portable applications target using those interfaces; however, their use
482 does carry extra message and memory footprint overhead, making it less
483 desirable for highly scalable apps.
484
486 The follow flag may be specified to any memory registration call.
487 FI_RMA_EVENT : This flag indicates that the specified memory region
489 will be associated with a completion counter used to count RMA opera‐
490 tions that access the MR.
491 FI_RMA_PMEM : This flag indicates that the underlying memory region is
493 backed by persistent memory and will be used in RMA operations. It
494 must be specified if persistent completion semantics or persistent data
495 transfers are required when accessing the registered region.
496
498 Returns 0 on success. On error, a negative value corresponding to fab‐
499 ric errno is returned.
500 Fabric errno values are defined in rdma/fi_errno.h.
502
504 -FI_ENOKEY : The requested_key is already in use.
505 -FI_EKEYREJECTED : The requested_key is not available. They key may be
507 out of the range supported by the provider, or the provider may not
508 support user-requested memory registration keys.
509 -FI_ENOSYS : Returned by fi_mr_bind if the provider does not support
511 reporting events based on access to registered memory regions.
512 -FI_EBADFLAGS : Returned if the specified flags are not supported by
514 the provider.
515
517 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_rma(3), fi_msg(3),
518 fi_atomic(3)
519
521 OpenFabrics.
522Libfabric Programmer's Manual 2018-02-13 fi_mr(3)