1fi_mr(3) Libfabric v1.8.0 fi_mr(3)
2
6 fi_mr - Memory region operations
7 fi_mr_reg / fi_mr_regv / fi_mr_regattr
9 Register local memory buffers for direct fabric access
10 fi_close
12 Deregister registered memory buffers.
13 fi_mr_desc
15 Return a local descriptor associated with a registered memory
16 region
17 fi_mr_key
19 Return the remote key needed to access a registered memory re‐
20 gion
21 fi_mr_raw_attr
23 Return raw memory region attributes.
24 fi_mr_map_raw
26 Converts a raw memory region key into a key that is usable for
27 data transfer operations.
28 fi_mr_unmap_key
30 Releases a previously mapped raw memory region key.
31 fi_mr_bind
33 Associate a registered memory region with a completion counter.
34 fi_mr_refresh
36 Updates the memory pages associated with a memory region.
37 fi_mr_enable
39 Enables a memory region for use.
40
42 #include <rdma/fi_domain.h>
43 int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
45 uint64_t access, uint64_t offset, uint64_t requested_key,
46 uint64_t flags, struct fid_mr **mr, void *context);
47 int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
49 size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
50 uint64_t flags, struct fid_mr **mr, void *context);
51 int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
53 uint64_t flags, struct fid_mr **mr);
54 int fi_close(struct fid *mr);
56 void * fi_mr_desc(struct fid_mr *mr);
58 uint64_t fi_mr_key(struct fid_mr *mr);
60 int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
62 uint8_t *raw_key, size_t *key_size, uint64_t flags);
63 int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
65 uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
66 int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
68 int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
70 int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
72 uint64_t flags)
73 int fi_mr_enable(struct fid_mr *mr);
75
77 domain Resource domain
78 mr Memory region
80 bfid Fabric identifier of an associated resource.
82 context
84 User specified context associated with the memory region.
85 buf Memory buffer to register with the fabric hardware
87 len Length of memory buffer to register
89 iov Vectored memory buffer.
91 count Count of vectored buffer entries.
93 access Memory access permissions associated with registration
95 offset Optional specified offset for accessing specified registered
97 buffers. This parameter is reserved for future use and must be
98 0.
99 requested_key
101 Optional requested remote key associated with registered buf‐
102 fers.
103 attr Memory region attributes
105 flags Additional flags to apply to the operation.
107
109 Registered memory regions associate memory buffers with permissions
110 granted for access by fabric resources. A memory buffer must be regis‐
111 tered with a resource domain before it can be used as the target of a
112 remote RMA or atomic data transfer. Additionally, a fabric provider
113 may require that data buffers be registered before being used in local
114 transfers. Memory registration restrictions are controlled using a
115 separate set of mode bits, specified through the domain attributes
116 (mr_mode field).
117 The following apply to memory registration.
119 Scalable Memory Registration
121 By default, memory registration is considered scalable. (For
122 library versions 1.4 and earlier, this is indicated by setting
123 mr_mode to FI_MR_SCALABLE, with the fi_info mode bit FI_LOCAL_MR
124 set to 0). For versions 1.5 and later, scalable is implied by
125 the lack of any mr_mode bits being set. The setting of mr_mode
126 bits therefore adjusts application behavior as described below.
127 Default, scalable registration has several properties.
128 In scalable mode, registration occurs on memory address ranges. Be‐
130 cause registration refers to memory regions, versus data buffers, the
131 address ranges given for a registration request do not need to map to
132 data buffers allocated by the application at the time the registration
133 call is made. That is, an application can register any range of ad‐
134 dresses in their virtual address space, whether or not those addresses
135 are backed by physical pages or have been allocated.
136 The resulting memory regions are accessible by peers starting at a base
138 address of 0. That is, the target address that is specified is a byte
139 offset into the registered region.
140 The application also selects the access key associated with the MR.
142 The key size is restricted to a maximum of 8 bytes.
143 With scalable registration, locally accessed data buffers are not reg‐
145 istered. This includes source buffers for all transmit operations --
146 sends, tagged sends, RMA, and atomics -- as well as buffers posted for
147 receive and tagged receive operations.
148 FI_MR_LOCAL
150 When the FI_MR_LOCAL mode bit is set, applications must register
151 all data buffers that will be accessed by the local hardware and
152 provide a valid mem_desc parameter into applicable data transfer
153 operations. When FI_MR_LOCAL is zero, applications are not re‐
154 quired to register data buffers before using them for local op‐
155 erations (e.g. send and receive data buffers), and the mem_desc
156 parameter into data transfer operations is ignored.
157 A provider may hide local registration requirements from applications
159 by making use of an internal registration cache or similar mechanisms.
160 Such mechanisms, however, may negatively impact performance for some
161 applications, notably those which manage their own network buffers. In
162 order to support as broad range of applications as possible, without
163 unduly affecting their performance, applications that wish to manage
164 their own local memory registrations may do so by using the memory reg‐
165 istration calls.
166 Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
168 bit. When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
169 FI_MR_RAW
171 Raw memory regions are used to support providers with keys larg‐
172 er than 64-bits or require setup at the peer. When the
173 FI_MR_RAW bit is set, applications must use fi_mr_raw_attr() lo‐
174 cally and fi_mr_map_raw() at the peer before targeting a memory
175 region as part of any data transfer request.
176 FI_MR_VIRT_ADDR
178 The FI_MR_VIRT_ADDR bit indicates that the provider references
179 memory regions by virtual address, rather than a 0-based offset.
180 Peers that target memory regions registered with FI_MR_VIRT_ADDR
181 specify the destination memory buffer using the target's virtual
182 address, with any offset into the region specified as virtual
183 address + offset. Support of this bit typically implies that
184 peers must exchange addressing data prior to initiating any RMA
185 or atomic operation.
186 FI_MR_ALLOCATED
188 When set, all registered memory regions must be backed by physi‐
189 cal memory pages at the time the registration call is made.
190 FI_MR_PROV_KEY
192 This memory region mode indicates that the provider does not
193 support application requested MR keys. MR keys are returned by
194 the provider. Applications that support FI_MR_PROV_KEY can ob‐
195 tain the provider key using fi_mr_key(), unless FI_MR_RAW is al‐
196 so set. The returned key should then be exchanged with peers
197 prior to initiating an RMA or atomic operation.
198 FI_MR_MMU_NOTIFY
200 FI_MR_MMU_NOTIFY is typically set by providers that support mem‐
201 ory registration against memory regions that are not necessarily
202 backed by allocated physical pages at the time the memory regis‐
203 tration occurs. (That is, FI_MR_ALLOCATED is typically 0).
204 However, such providers require that applications notify the
205 provider prior to the MR being accessed as part of a data trans‐
206 fer operation. This notification informs the provider that all
207 necessary physical pages now back the region. The notification
208 is necessary for providers that cannot hook directly into the
209 operating system page tables or memory management unit. See
210 fi_mr_refresh() for notification details.
211 FI_MR_RMA_EVENT
213 This mode bit indicates that the provider must configure memory
214 regions that are associated with RMA events prior to their use.
215 This includes all memory regions that are associated with com‐
216 pletion counters. When set, applications must indicate if a
217 memory region will be associated with a completion counter as
218 part of the region's creation. This is done by passing in the
219 FI_RMA_EVENT flag to the memory registration call.
220 Such memory regions will be created in a disabled state and must be as‐
222 sociated with all completion counters prior to being enabled. To en‐
223 able a memory region, the application must call fi_mr_enable(). After
224 calling fi_mr_enable(), no further resource bindings may be made to the
225 memory region.
226 FI_MR_ENDPOINT
228 This mode bit indicates that the provider associates memory re‐
229 gions with endpoints rather than domains. Memory regions that
230 are registered with the provider are created in a disabled state
231 and must be bound to an endpoint prior to being enabled. To
232 bind the MR with an endpoint, the application must use
233 fi_mr_bind(). To enable the memory region, the application must
234 call fi_mr_enable().
235 Basic Memory Registration
237 Basic memory registration is indicated by the FI_MR_BASIC
238 mr_mode bit. FI_MR_BASIC is maintained for backwards compati‐
239 bility (libfabric version 1.4 or earlier). The behavior of ba‐
240 sic registration is equivalent to setting the following mr_mode
241 bits to one: FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, and
242 FI_MR_PROV_KEY. Additionally, providers that support basic reg‐
243 istration usually required FI_MR_LOCAL. FI_MR_BASIC must either
244 be set alone, or in conjunction with FI_MR_LOCAL. Other mr_mode
245 bit pairings are invalid. Unlike other mr_mode bits, if
246 FI_MR_BASIC is set on input to fi_getinfo(), it will not be
247 cleared by the provider. That is, setting FI_MR_BASIC to one
248 requests basic registration.
249 The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
251 -- are used to register one or more memory regions with fabric re‐
252 sources. The main difference between registration functions are the
253 number and type of parameters that they accept as input. Otherwise,
254 they perform the same general function.
255 By default, memory registration completes synchronously. I.e. the
257 registration call will not return until the registration has completed.
258 Memory registration can complete asynchronous by binding the resource
259 domain to an event queue using the FI_REG_MR flag. See fi_domain_bind.
260 When memory registration is asynchronous, in order to avoid a race con‐
261 dition between the registration call returning and the corresponding
262 reading of the event from the EQ, the mr output parameter will be writ‐
263 ten before any event associated with the operation may be read by the
264 application. An asynchronous event will not be generated unless the
265 registration call returns success (0).
266 fi_mr_reg
268 The fi_mr_reg call registers the user-specified memory buffer with the
269 resource domain. The buffer is enabled for access by the fabric hard‐
270 ware based on the provided access permissions. See the access field
271 description for memory region attributes below.
272 Registered memory is associated with a local memory descriptor and, op‐
274 tionally, a remote memory key. A memory descriptor is a provider spe‐
275 cific identifier associated with registered memory. Memory descriptors
276 often map to hardware specific indices or keys associated with the mem‐
277 ory region. Remote memory keys provide limited protection against un‐
278 wanted access by a remote node. Remote accesses to a memory region
279 must provide the key associated with the registration.
280 Because MR keys must be provided by a remote process, an application
282 can use the requested_key parameter to indicate that a specific key
283 value be returned. Support for user requested keys is provider specif‐
284 ic and is determined by the mr_mode domain attribute.
285 Remote RMA and atomic operations indicate the location within a regis‐
287 tered memory region by specifying an address. The location is refer‐
288 enced by adding the offset to either the base virtual address of the
289 buffer or to 0, depending on the mr_mode.
290 The offset parameter is reserved for future use and must be 0.
292 For asynchronous memory registration requests, the result will be re‐
294 ported to the user through an event queue associated with the resource
295 domain. If successful, the allocated memory region structure will be
296 returned to the user through the mr parameter. The mr address must re‐
297 main valid until the registration operation completes. The context
298 specified with the registration request is returned with the completion
299 event.
300 fi_mr_regv
302 The fi_mr_regv call adds support for a scatter-gather list to
303 fi_mr_reg. Multiple memory buffers are registered as a single memory
304 region. Otherwise, the operation is the same.
305 fi_mr_regattr
307 The fi_mr_regattr call is a more generic, extensible registration call
308 that allows the user to specify the registration request using a struct
309 fi_mr_attr (defined below).
310 fi_close
312 Fi_close is used to release all resources associated with a registering
313 a memory region. Once unregistered, further access to the registered
314 memory is not guaranteed. Active or queued operations that reference a
315 memory region being closed may fail or result in accesses to invalid
316 memory. Applications are responsible for ensuring that a MR is no
317 longer needed prior to closing it. Note that accesses to a closed MR
318 from a remote peer will result in an error at the peer. The state of
319 the local endpoint will be unaffected.
320 When closing the MR, there must be no opened endpoints or counters as‐
322 sociated with the MR. If resources are still associated with the MR
323 when attempting to close, the call will return -FI_EBUSY.
324 fi_mr_desc
326 Obtains the local memory descriptor associated with a MR. The memory
327 registration must have completed successfully before invoking this
328 call.
329 fi_mr_key
331 Returns the remote protection key associated with a MR. The memory
332 registration must have completed successfully before invoking this.
333 The returned key may be used in data transfer operations at a peer. If
334 the FI_RAW_MR mode bit has been set for the domain, then the memory key
335 must be obtained using the fi_mr_raw_key function instead. A return
336 value of FI_KEY_NOTAVAIL will be returned if the registration has not
337 completed or a raw memory key is required.
338 fi_mr_raw_attr
340 Returns the raw, remote protection key and base address associated with
341 a MR. The memory registration must have completed successfully before
342 invoking this routine. Use of this call is required if the FI_RAW_MR
343 mode bit has been set by the provider; however, it is safe to use this
344 call with any memory region.
345 On input, the key_size parameter should indicate the size of the
347 raw_key buffer. If the actual key is larger than what can fit into the
348 buffer, it will return -FI_ETOOSMALL. On output, key_size is set to
349 the size of the buffer needed to store the key, which may be larger
350 than the input value. The needed key_size can also be obtained through
351 the mr_key_size domain attribute (fi_domain_attr) field.
352 A raw key must be mapped by a peer before it can be used in data trans‐
354 fer operations. See fi_mr_map_raw below.
355 fi_mr_map_raw
357 Raw protection keys must be mapped to a usable key value before they
358 can be used for data transfer operations. The mapping is done by the
359 peer that initiates the RMA or atomic operation. The mapping function
360 takes as input the raw key and its size, and returns the mapped key.
361 Use of the fi_mr_map_raw function is required if the peer has the
362 FI_RAW_MR mode bit set, but this routine may be called on any valid
363 key. All mapped keys must be freed by calling fi_mr_unmap_key when ac‐
364 cess to the peer memory region is no longer necessary.
365 fi_mr_unmap_key
367 This call releases any resources that may have been allocated as part
368 of mapping a raw memory key. All mapped keys must be freed before the
369 corresponding domain is closed.
370 fi_mr_bind
372 The fi_mr_bind function associates a memory region with a counter or
373 endpoint. Counter bindings are needed by providers that support the
374 generation of completions based on fabric operations. Endpoint bind‐
375 ings are needed if the provider associates memory regions with end‐
376 points (see FI_MR_ENDPOINT).
377 When binding with a counter, the type of events tracked against the
379 memory region is based on the bitwise OR of the following flags.
380 FI_REMOTE_WRITE
382 Generates an event whenever a remote RMA write or atomic opera‐
383 tion modifies the memory region. Use of this flag requires that
384 the endpoint through which the MR is accessed be created with
385 the FI_RMA_EVENT capability.
386 When binding the memory region to an endpoint, flags should be 0.
388 fi_mr_refresh
390 The use of this call is required to notify the provider of any change
391 to the physical pages backing a registered memory region if the
392 FI_MR_MMU_NOTIFY mode bit has been set. This call informs the provider
393 that the page table entries associated with the region may have been
394 modified, and the provider should verify and update the registered re‐
395 gion accordingly. The iov parameter is optional and may be used to
396 specify which portions of the registered region requires updating.
397 Providers are only guaranteed to update the specified address ranges.
398 The refresh operation has the effect of disabling and re-enabling ac‐
400 cess to the registered region. Any operations from peers that attempt
401 to access the region will fail while the refresh is occurring. Addi‐
402 tionally, attempts to access the region by the local process through
403 libfabric APIs may result in a page fault or other fatal operation.
404 The fi_mr_refresh call is only needed if the physical pages might have
406 been updated after the memory region was created.
407 fi_mr_enable
409 The enable call is used with memory registration associated with the
410 FI_MR_RMA_EVENT mode bit. Memory regions created in the disabled state
411 must be explicitly enabled after being fully configured by the applica‐
412 tion. Any resource bindings to the MR must be done prior to enabling
413 the MR.
414
416 Memory regions are created using the following attributes. The struct
417 fi_mr_attr is passed into fi_mr_regattr, but individual fields also ap‐
418 ply to other memory registration calls, with the fields passed directly
419 into calls as function parameters.
420 struct fi_mr_attr {
422 const struct iovec *mr_iov;
423 size_t iov_count;
424 uint64_t access;
425 uint64_t offset;
426 uint64_t requested_key;
427 void *context;
428 size_t auth_key_size;
429 uint8_t *auth_key;
430 };
431 mr_iov
433 This is an IO vector of addresses that will represent a single memory
434 region. The number of entries in the iovec is specified by iov_count.
435 iov_count
437 The number of entries in the mr_iov array. The maximum number of memo‐
438 ry buffers that may be associated with a single memory region is speci‐
439 fied as the mr_iov_limit domain attribute. See fi_domain(3).
440 access
442 Indicates the type of operations that the local or a peer endpoint may
443 perform on registered memory region. Supported access permissions are
444 the bitwise OR of the following flags:
445 FI_SEND
447 The memory buffer may be used in outgoing message data trans‐
448 fers. This includes fi_msg and fi_tagged send operations.
449 FI_RECV
451 The memory buffer may be used to receive inbound message trans‐
452 fers. This includes fi_msg and fi_tagged receive operations.
453 FI_READ
455 The memory buffer may be used as the result buffer for RMA read
456 and atomic operations on the initiator side. Note that from the
457 viewpoint of the application, the memory buffer is being written
458 into by the network.
459 FI_WRITE
461 The memory buffer may be used as the source buffer for RMA write
462 and atomic operations on the initiator side. Note that from the
463 viewpoint of the application, the endpoint is reading from the
464 memory buffer and copying the data onto the network.
465 FI_REMOTE_READ
467 The memory buffer may be used as the source buffer of an RMA
468 read operation on the target side. The contents of the memory
469 buffer are not modified by such operations.
470 FI_REMOTE_WRITE
472 The memory buffer may be used as the target buffer of an RMA
473 write or atomic operation. The contents of the memory buffer
474 may be modified as a result of such operations.
475 Note that some providers may not enforce fine grained access permis‐
477 sions. For example, a memory region registered for FI_WRITE access may
478 also behave as if FI_SEND were specified as well. Relaxed enforcement
479 of such access is permitted, though not guaranteed, provided security
480 is maintained.
481 offset
483 The offset field is reserved for future use and must be 0.
484 requested_key
486 An application specified access key associated with the memory region.
487 The MR key must be provided by a remote process when performing RMA or
488 atomic operations to a memory region. Applications can use the re‐
489 quested_key field to indicate that a specific key be used by the
490 provider. This allows applications to use well known key values, which
491 can avoid applications needing to exchange and store keys. Support for
492 user requested keys is provider specific and is determined by the
493 mr_mode domain attribute.
494 context
496 Application context associated with asynchronous memory registration
497 operations. This value is returned as part of any asynchronous event
498 associated with the registration. This field is ignored for synchro‐
499 nous registration calls.
500 auth_key_size
502 The size of key referenced by the auth_key field in bytes, or 0 if no
503 authorization key is given. This field is ignored unless the fabric is
504 opened with API version 1.5 or greater.
505 auth_key
507 Indicates the key to associate with this memory registration. Autho‐
508 rization keys are used to limit communication between endpoints. Only
509 peer endpoints that are programmed to use the same authorization key
510 may access the memory region. The domain authorization key will be
511 used if the auth_key_size provided is 0. This field is ignored unless
512 the fabric is opened with API version 1.5 or greater.
513
515 Direct access to an application's memory by a remote peer requires that
516 the application register the targeted memory buffer(s). This is typi‐
517 cally done by calling one of the fi_mr_reg* routines. For
518 FI_MR_PROV_KEY, the provider will return a key that must be used by the
519 peer when accessing the memory region. The application is responsible
520 for transferring this key to the peer. If FI_MR_RAW mode has been set,
521 the key must be retrieved using the fi_mr_raw_attr function.
522 FI_RAW_MR allows support for providers that require more than 8-bytes
524 for their protection keys or need additional setup before a key can be
525 used for transfers. After a raw key has been retrieved, it must be ex‐
526 changed with the remote peer. The peer must use fi_mr_map_raw to con‐
527 vert the raw key into a usable 64-bit key. The mapping must be done
528 even if the raw key is 64-bits or smaller.
529 The raw key support functions are usable with all registered memory re‐
531 gions, even if FI_MR_RAW has not been set. It is recommended that por‐
532 table applications target using those interfaces; however, their use
533 does carry extra message and memory footprint overhead, making it less
534 desirable for highly scalable apps.
535
537 The follow flag may be specified to any memory registration call.
538 FI_RMA_EVENT
540 This flag indicates that the specified memory region will be as‐
541 sociated with a completion counter used to count RMA operations
542 that access the MR.
543 FI_RMA_PMEM
545 This flag indicates that the underlying memory region is backed
546 by persistent memory and will be used in RMA operations. It
547 must be specified if persistent completion semantics or persis‐
548 tent data transfers are required when accessing the registered
549 region.
550
552 Returns 0 on success. On error, a negative value corresponding to fab‐
553 ric errno is returned.
554 Fabric errno values are defined in rdma/fi_errno.h.
556
558 -FI_ENOKEY
559 The requested_key is already in use.
560 -FI_EKEYREJECTED
562 The requested_key is not available. They key may be out of the
563 range supported by the provider, or the provider may not support
564 user-requested memory registration keys.
565 -FI_ENOSYS
567 Returned by fi_mr_bind if the provider does not support report‐
568 ing events based on access to registered memory regions.
569 -FI_EBADFLAGS
571 Returned if the specified flags are not supported by the
572 provider.
573
575 Many hardware NICs accessed by libfabric require that data buffers be
576 registered with the hardware while the hardware accesses it. This en‐
577 sures that the virtual to physical address mappings for those buffers
578 do not change while the transfer is ocurring. The performance impact
579 of registering memory regions can be significant. As a result, some
580 providers make use of a registration cache, particularly when working
581 with applications that are unable to manage their own network buffers.
582 A registration cache avoids the overhead of registering and unregister‐
583 ing a data buffer with each transfer.
584 As a general rule, if hardware requires the FI_MR_LOCAL mode bit de‐
586 scribed above, but this is not supported by the application, a memory
587 registration cache may be in use. The following environment variables
588 may be used to configure registration caches.
589 FI_MR_CACHE_MAX_SIZE
591 This defines the total number of bytes for all memory regions
592 that may be tracked by the cache. If not set, the cache has no
593 limit on how many bytes may be registered and cached. Setting
594 this will reduce the amount of memory that is not actively being
595 used as part of a data transfer that is registered with a
596 provider. By default, the cache size is unlimited.
597 FI_MR_CACHE_MAX_COUNT
599 This defines the total number of memory regions that may be reg‐
600 istered with the cache. If not set, a default limit is chosen.
601 Setting this will reduce the number of regions that are regis‐
602 tered, regardless of their size, which are not actively being
603 used as part of a data transfer. Setting this to zero will dis‐
604 able registration caching.
605 FI_MR_CACHE_MERGE_REGIONS
607 If this variable is set to true, yes, or 1, then memory regions
608 that are adjacent or overlapping will be merged into a single
609 larger region. Merging regions reduces the total cache size and
610 the number of regions managed by the cache. However, merging
611 regions can have a negative impact on performance if a large
612 number of adjacent regions are sent as separate data transfers
613 (such as sending elements of an array to peer(s)), and the larg‐
614 er region is access infrequently. By default merging regions is
615 disabled.
616
618 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_rma(3), fi_msg(3),
619 fi_atomic(3)
620
622 OpenFabrics.
623Libfabric Programmer's Manual 2019-05-04 fi_mr(3)