1fi_mr(3) Libfabric v1.10.0 fi_mr(3)
2
6 fi_mr - Memory region operations
7 fi_mr_reg / fi_mr_regv / fi_mr_regattr
9 Register local memory buffers for direct fabric access
10 fi_close
12 Deregister registered memory buffers.
13 fi_mr_desc
15 Return a local descriptor associated with a registered memory
16 region
17 fi_mr_key
19 Return the remote key needed to access a registered memory re‐
20 gion
21 fi_mr_raw_attr
23 Return raw memory region attributes.
24 fi_mr_map_raw
26 Converts a raw memory region key into a key that is usable for
27 data transfer operations.
28 fi_mr_unmap_key
30 Releases a previously mapped raw memory region key.
31 fi_mr_bind
33 Associate a registered memory region with a completion counter
34 or an endpoint.
35 fi_mr_refresh
37 Updates the memory pages associated with a memory region.
38 fi_mr_enable
40 Enables a memory region for use.
41
43 #include <rdma/fi_domain.h>
44 int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
46 uint64_t access, uint64_t offset, uint64_t requested_key,
47 uint64_t flags, struct fid_mr **mr, void *context);
48 int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
50 size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
51 uint64_t flags, struct fid_mr **mr, void *context);
52 int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
54 uint64_t flags, struct fid_mr **mr);
55 int fi_close(struct fid *mr);
57 void * fi_mr_desc(struct fid_mr *mr);
59 uint64_t fi_mr_key(struct fid_mr *mr);
61 int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
63 uint8_t *raw_key, size_t *key_size, uint64_t flags);
64 int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
66 uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
67 int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
69 int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
71 int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
73 uint64_t flags)
74 int fi_mr_enable(struct fid_mr *mr);
76
78 domain Resource domain
79 mr Memory region
81 bfid Fabric identifier of an associated resource.
83 context
85 User specified context associated with the memory region.
86 buf Memory buffer to register with the fabric hardware
88 len Length of memory buffer to register
90 iov Vectored memory buffer.
92 count Count of vectored buffer entries.
94 access Memory access permissions associated with registration
96 offset Optional specified offset for accessing specified registered
98 buffers. This parameter is reserved for future use and must be
99 0.
100 requested_key
102 Optional requested remote key associated with registered buf‐
103 fers.
104 attr Memory region attributes
106 flags Additional flags to apply to the operation.
108
110 Registered memory regions associate memory buffers with permissions
111 granted for access by fabric resources. A memory buffer must be regis‐
112 tered with a resource domain before it can be used as the target of a
113 remote RMA or atomic data transfer. Additionally, a fabric provider
114 may require that data buffers be registered before being used in local
115 transfers. Memory registration restrictions are controlled using a
116 separate set of mode bits, specified through the domain attributes
117 (mr_mode field).
118 The following apply to memory registration.
120 Scalable Memory Registration
122 By default, memory registration is considered scalable. (For
123 library versions 1.4 and earlier, this is indicated by setting
124 mr_mode to FI_MR_SCALABLE, with the fi_info mode bit FI_LOCAL_MR
125 set to 0). For versions 1.5 and later, scalable is implied by
126 the lack of any mr_mode bits being set. The setting of mr_mode
127 bits therefore adjusts application behavior as described below.
128 Default, scalable registration has several properties.
129 In scalable mode, registration occurs on memory address ranges. Be‐
131 cause registration refers to memory regions, versus data buffers, the
132 address ranges given for a registration request do not need to map to
133 data buffers allocated by the application at the time the registration
134 call is made. That is, an application can register any range of ad‐
135 dresses in their virtual address space, whether or not those addresses
136 are backed by physical pages or have been allocated.
137 The resulting memory regions are accessible by peers starting at a base
139 address of 0. That is, the target address that is specified is a byte
140 offset into the registered region.
141 The application also selects the access key associated with the MR.
143 The key size is restricted to a maximum of 8 bytes.
144 With scalable registration, locally accessed data buffers are not reg‐
146 istered. This includes source buffers for all transmit operations --
147 sends, tagged sends, RMA, and atomics -- as well as buffers posted for
148 receive and tagged receive operations.
149 FI_MR_LOCAL
151 When the FI_MR_LOCAL mode bit is set, applications must register
152 all data buffers that will be accessed by the local hardware and
153 provide a valid desc parameter into applicable data transfer op‐
154 erations. When FI_MR_LOCAL is zero, applications are not re‐
155 quired to register data buffers before using them for local op‐
156 erations (e.g. send and receive data buffers). The desc param‐
157 eter into data transfer operations will be ignored in this case,
158 unless otherwise required (e.g. se FI_MR_HMEM). It is recom‐
159 mended that applications pass in NULL for desc when not re‐
160 quired.
161 A provider may hide local registration requirements from applications
163 by making use of an internal registration cache or similar mechanisms.
164 Such mechanisms, however, may negatively impact performance for some
165 applications, notably those which manage their own network buffers. In
166 order to support as broad range of applications as possible, without
167 unduly affecting their performance, applications that wish to manage
168 their own local memory registrations may do so by using the memory reg‐
169 istration calls.
170 Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
172 bit. When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
173 FI_MR_RAW
175 Raw memory regions are used to support providers with keys larg‐
176 er than 64-bits or require setup at the peer. When the
177 FI_MR_RAW bit is set, applications must use fi_mr_raw_attr() lo‐
178 cally and fi_mr_map_raw() at the peer before targeting a memory
179 region as part of any data transfer request.
180 FI_MR_VIRT_ADDR
182 The FI_MR_VIRT_ADDR bit indicates that the provider references
183 memory regions by virtual address, rather than a 0-based offset.
184 Peers that target memory regions registered with FI_MR_VIRT_ADDR
185 specify the destination memory buffer using the target's virtual
186 address, with any offset into the region specified as virtual
187 address + offset. Support of this bit typically implies that
188 peers must exchange addressing data prior to initiating any RMA
189 or atomic operation.
190 FI_MR_ALLOCATED
192 When set, all registered memory regions must be backed by physi‐
193 cal memory pages at the time the registration call is made.
194 FI_MR_PROV_KEY
196 This memory region mode indicates that the provider does not
197 support application requested MR keys. MR keys are returned by
198 the provider. Applications that support FI_MR_PROV_KEY can ob‐
199 tain the provider key using fi_mr_key(), unless FI_MR_RAW is al‐
200 so set. The returned key should then be exchanged with peers
201 prior to initiating an RMA or atomic operation.
202 FI_MR_MMU_NOTIFY
204 FI_MR_MMU_NOTIFY is typically set by providers that support mem‐
205 ory registration against memory regions that are not necessarily
206 backed by allocated physical pages at the time the memory regis‐
207 tration occurs. (That is, FI_MR_ALLOCATED is typically 0).
208 However, such providers require that applications notify the
209 provider prior to the MR being accessed as part of a data trans‐
210 fer operation. This notification informs the provider that all
211 necessary physical pages now back the region. The notification
212 is necessary for providers that cannot hook directly into the
213 operating system page tables or memory management unit. See
214 fi_mr_refresh() for notification details.
215 FI_MR_RMA_EVENT
217 This mode bit indicates that the provider must configure memory
218 regions that are associated with RMA events prior to their use.
219 This includes all memory regions that are associated with com‐
220 pletion counters. When set, applications must indicate if a
221 memory region will be associated with a completion counter as
222 part of the region's creation. This is done by passing in the
223 FI_RMA_EVENT flag to the memory registration call.
224 Such memory regions will be created in a disabled state and must be as‐
226 sociated with all completion counters prior to being enabled. To en‐
227 able a memory region, the application must call fi_mr_enable(). After
228 calling fi_mr_enable(), no further resource bindings may be made to the
229 memory region.
230 FI_MR_ENDPOINT
232 This mode bit indicates that the provider associates memory re‐
233 gions with endpoints rather than domains. Memory regions that
234 are registered with the provider are created in a disabled state
235 and must be bound to an endpoint prior to being enabled. To
236 bind the MR with an endpoint, the application must use
237 fi_mr_bind(). To enable the memory region, the application must
238 call fi_mr_enable().
239 FI_MR_HMEM
241 This mode bit is associated with the FI_HMEM capability. If
242 FI_MR_HMEM is set, the application must register buffers that
243 were allocated using a device call and provide a valid desc pa‐
244 rameter into applicable data transfer operations even if they
245 are only used for local operations (e.g. send and receive data
246 buffers). Device memory must be registered using the fi_mr_re‐
247 gattr call, with the iface and device fields filled out.
248 If FI_MR_HMEM is set, but FI_MR_LOCAL is unset, only device buffers
250 must be registered when used locally. In this case, the desc parameter
251 passed into data transfer operations must either be valid or NULL.
252 Similarly, if FI_MR_LOCAL is set, but FI_MR_HMEM is not, the desc pa‐
253 rameter must either be valid or NULL.
254 Basic Memory Registration
256 Basic memory registration is indicated by the FI_MR_BASIC
257 mr_mode bit. FI_MR_BASIC is maintained for backwards compati‐
258 bility (libfabric version 1.4 or earlier). The behavior of ba‐
259 sic registration is equivalent to setting the following mr_mode
260 bits to one: FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, and
261 FI_MR_PROV_KEY. Additionally, providers that support basic reg‐
262 istration usually required the fi_info mode bit FI_LOCAL_MR. As
263 a result, it is recommended that applications migrating from
264 libfabric 1.4 or earlier or wanting to support basic memory reg‐
265 istration set the mr_mode to FI_MR_VIRT_ADDR | FI_MR_ALLOCATED |
266 FI_MR_PROV_KEY | FI_MR_LOCAL. FI_MR_BASIC must be set alone.
267 Other mr_mode bit pairings are invalid. Unlike other mr_mode
268 bits, if FI_MR_BASIC is set on input to fi_getinfo(), it will
269 not be cleared by the provider. That is, setting FI_MR_BASIC to
270 one requests basic registration.
271 The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
273 -- are used to register one or more memory regions with fabric re‐
274 sources. The main difference between registration functions are the
275 number and type of parameters that they accept as input. Otherwise,
276 they perform the same general function.
277 By default, memory registration completes synchronously. I.e. the
279 registration call will not return until the registration has completed.
280 Memory registration can complete asynchronous by binding the resource
281 domain to an event queue using the FI_REG_MR flag. See fi_domain_bind.
282 When memory registration is asynchronous, in order to avoid a race con‐
283 dition between the registration call returning and the corresponding
284 reading of the event from the EQ, the mr output parameter will be writ‐
285 ten before any event associated with the operation may be read by the
286 application. An asynchronous event will not be generated unless the
287 registration call returns success (0).
288 fi_mr_reg
290 The fi_mr_reg call registers the user-specified memory buffer with the
291 resource domain. The buffer is enabled for access by the fabric hard‐
292 ware based on the provided access permissions. See the access field
293 description for memory region attributes below.
294 Registered memory is associated with a local memory descriptor and, op‐
296 tionally, a remote memory key. A memory descriptor is a provider spe‐
297 cific identifier associated with registered memory. Memory descriptors
298 often map to hardware specific indices or keys associated with the mem‐
299 ory region. Remote memory keys provide limited protection against un‐
300 wanted access by a remote node. Remote accesses to a memory region
301 must provide the key associated with the registration.
302 Because MR keys must be provided by a remote process, an application
304 can use the requested_key parameter to indicate that a specific key
305 value be returned. Support for user requested keys is provider specif‐
306 ic and is determined by the mr_mode domain attribute.
307 Remote RMA and atomic operations indicate the location within a regis‐
309 tered memory region by specifying an address. The location is refer‐
310 enced by adding the offset to either the base virtual address of the
311 buffer or to 0, depending on the mr_mode.
312 The offset parameter is reserved for future use and must be 0.
314 For asynchronous memory registration requests, the result will be re‐
316 ported to the user through an event queue associated with the resource
317 domain. If successful, the allocated memory region structure will be
318 returned to the user through the mr parameter. The mr address must re‐
319 main valid until the registration operation completes. The context
320 specified with the registration request is returned with the completion
321 event.
322 fi_mr_regv
324 The fi_mr_regv call adds support for a scatter-gather list to
325 fi_mr_reg. Multiple memory buffers are registered as a single memory
326 region. Otherwise, the operation is the same.
327 fi_mr_regattr
329 The fi_mr_regattr call is a more generic, extensible registration call
330 that allows the user to specify the registration request using a struct
331 fi_mr_attr (defined below).
332 fi_close
334 Fi_close is used to release all resources associated with a registering
335 a memory region. Once unregistered, further access to the registered
336 memory is not guaranteed. Active or queued operations that reference a
337 memory region being closed may fail or result in accesses to invalid
338 memory. Applications are responsible for ensuring that a MR is no
339 longer needed prior to closing it. Note that accesses to a closed MR
340 from a remote peer will result in an error at the peer. The state of
341 the local endpoint will be unaffected.
342 When closing the MR, there must be no opened endpoints or counters as‐
344 sociated with the MR. If resources are still associated with the MR
345 when attempting to close, the call will return -FI_EBUSY.
346 fi_mr_desc
348 Obtains the local memory descriptor associated with a MR. The memory
349 registration must have completed successfully before invoking this
350 call.
351 fi_mr_key
353 Returns the remote protection key associated with a MR. The memory
354 registration must have completed successfully before invoking this.
355 The returned key may be used in data transfer operations at a peer. If
356 the FI_RAW_MR mode bit has been set for the domain, then the memory key
357 must be obtained using the fi_mr_raw_key function instead. A return
358 value of FI_KEY_NOTAVAIL will be returned if the registration has not
359 completed or a raw memory key is required.
360 fi_mr_raw_attr
362 Returns the raw, remote protection key and base address associated with
363 a MR. The memory registration must have completed successfully before
364 invoking this routine. Use of this call is required if the FI_RAW_MR
365 mode bit has been set by the provider; however, it is safe to use this
366 call with any memory region.
367 On input, the key_size parameter should indicate the size of the
369 raw_key buffer. If the actual key is larger than what can fit into the
370 buffer, it will return -FI_ETOOSMALL. On output, key_size is set to
371 the size of the buffer needed to store the key, which may be larger
372 than the input value. The needed key_size can also be obtained through
373 the mr_key_size domain attribute (fi_domain_attr) field.
374 A raw key must be mapped by a peer before it can be used in data trans‐
376 fer operations. See fi_mr_map_raw below.
377 fi_mr_map_raw
379 Raw protection keys must be mapped to a usable key value before they
380 can be used for data transfer operations. The mapping is done by the
381 peer that initiates the RMA or atomic operation. The mapping function
382 takes as input the raw key and its size, and returns the mapped key.
383 Use of the fi_mr_map_raw function is required if the peer has the
384 FI_RAW_MR mode bit set, but this routine may be called on any valid
385 key. All mapped keys must be freed by calling fi_mr_unmap_key when ac‐
386 cess to the peer memory region is no longer necessary.
387 fi_mr_unmap_key
389 This call releases any resources that may have been allocated as part
390 of mapping a raw memory key. All mapped keys must be freed before the
391 corresponding domain is closed.
392 fi_mr_bind
394 The fi_mr_bind function associates a memory region with a counter or
395 endpoint. Counter bindings are needed by providers that support the
396 generation of completions based on fabric operations. Endpoint bind‐
397 ings are needed if the provider associates memory regions with end‐
398 points (see FI_MR_ENDPOINT).
399 When binding with a counter, the type of events tracked against the
401 memory region is based on the bitwise OR of the following flags.
402 FI_REMOTE_WRITE
404 Generates an event whenever a remote RMA write or atomic opera‐
405 tion modifies the memory region. Use of this flag requires that
406 the endpoint through which the MR is accessed be created with
407 the FI_RMA_EVENT capability.
408 When binding the memory region to an endpoint, flags should be 0.
410 fi_mr_refresh
412 The use of this call is required to notify the provider of any change
413 to the physical pages backing a registered memory region if the
414 FI_MR_MMU_NOTIFY mode bit has been set. This call informs the provider
415 that the page table entries associated with the region may have been
416 modified, and the provider should verify and update the registered re‐
417 gion accordingly. The iov parameter is optional and may be used to
418 specify which portions of the registered region requires updating.
419 Providers are only guaranteed to update the specified address ranges.
420 The refresh operation has the effect of disabling and re-enabling ac‐
422 cess to the registered region. Any operations from peers that attempt
423 to access the region will fail while the refresh is occurring. Addi‐
424 tionally, attempts to access the region by the local process through
425 libfabric APIs may result in a page fault or other fatal operation.
426 The fi_mr_refresh call is only needed if the physical pages might have
428 been updated after the memory region was created.
429 fi_mr_enable
431 The enable call is used with memory registration associated with the
432 FI_MR_RMA_EVENT mode bit. Memory regions created in the disabled state
433 must be explicitly enabled after being fully configured by the applica‐
434 tion. Any resource bindings to the MR must be done prior to enabling
435 the MR.
436
438 Memory regions are created using the following attributes. The struct
439 fi_mr_attr is passed into fi_mr_regattr, but individual fields also ap‐
440 ply to other memory registration calls, with the fields passed directly
441 into calls as function parameters.
442 struct fi_mr_attr {
444 const struct iovec *mr_iov;
445 size_t iov_count;
446 uint64_t access;
447 uint64_t offset;
448 uint64_t requested_key;
449 void *context;
450 size_t auth_key_size;
451 uint8_t *auth_key;
452 enum fi_hmem_iface iface;
453 union {
454 uint64_t reserved;
455 int cuda;
456 } device;
457 };
458 mr_iov
460 This is an IO vector of addresses that will represent a single memory
461 region. The number of entries in the iovec is specified by iov_count.
462 iov_count
464 The number of entries in the mr_iov array. The maximum number of memo‐
465 ry buffers that may be associated with a single memory region is speci‐
466 fied as the mr_iov_limit domain attribute. See fi_domain(3).
467 access
469 Indicates the type of operations that the local or a peer endpoint may
470 perform on registered memory region. Supported access permissions are
471 the bitwise OR of the following flags:
472 FI_SEND
474 The memory buffer may be used in outgoing message data trans‐
475 fers. This includes fi_msg and fi_tagged send operations.
476 FI_RECV
478 The memory buffer may be used to receive inbound message trans‐
479 fers. This includes fi_msg and fi_tagged receive operations.
480 FI_READ
482 The memory buffer may be used as the result buffer for RMA read
483 and atomic operations on the initiator side. Note that from the
484 viewpoint of the application, the memory buffer is being written
485 into by the network.
486 FI_WRITE
488 The memory buffer may be used as the source buffer for RMA write
489 and atomic operations on the initiator side. Note that from the
490 viewpoint of the application, the endpoint is reading from the
491 memory buffer and copying the data onto the network.
492 FI_REMOTE_READ
494 The memory buffer may be used as the source buffer of an RMA
495 read operation on the target side. The contents of the memory
496 buffer are not modified by such operations.
497 FI_REMOTE_WRITE
499 The memory buffer may be used as the target buffer of an RMA
500 write or atomic operation. The contents of the memory buffer
501 may be modified as a result of such operations.
502 Note that some providers may not enforce fine grained access permis‐
504 sions. For example, a memory region registered for FI_WRITE access may
505 also behave as if FI_SEND were specified as well. Relaxed enforcement
506 of such access is permitted, though not guaranteed, provided security
507 is maintained.
508 offset
510 The offset field is reserved for future use and must be 0.
511 requested_key
513 An application specified access key associated with the memory region.
514 The MR key must be provided by a remote process when performing RMA or
515 atomic operations to a memory region. Applications can use the re‐
516 quested_key field to indicate that a specific key be used by the
517 provider. This allows applications to use well known key values, which
518 can avoid applications needing to exchange and store keys. Support for
519 user requested keys is provider specific and is determined by the
520 mr_mode domain attribute.
521 context
523 Application context associated with asynchronous memory registration
524 operations. This value is returned as part of any asynchronous event
525 associated with the registration. This field is ignored for synchro‐
526 nous registration calls.
527 auth_key_size
529 The size of key referenced by the auth_key field in bytes, or 0 if no
530 authorization key is given. This field is ignored unless the fabric is
531 opened with API version 1.5 or greater.
532 auth_key
534 Indicates the key to associate with this memory registration. Autho‐
535 rization keys are used to limit communication between endpoints. Only
536 peer endpoints that are programmed to use the same authorization key
537 may access the memory region. The domain authorization key will be
538 used if the auth_key_size provided is 0. This field is ignored unless
539 the fabric is opened with API version 1.5 or greater.
540 iface
542 Indicates the software interfaces used by the application to allocate
543 and manage the memory region. This field is ignored unless the appli‐
544 cation has requested the FI_HMEM capability.
545 FI_HMEM_SYSTEM
547 Uses standard operating system calls and libraries, such as mal‐
548 loc, calloc, realloc, mmap, and free.
549 FI_HMEM_CUDA
551 Uses Nvidia CUDA interfaces such as cuMemAlloc, cuMemAllocHost,
552 cuMemAllocManaged, cuMemFree, cudaMalloc, cudaFree.
553 device
555 Reserved 64 bits for device identifier if using non-standard HMEM in‐
556 terface. This field is ignore unless the iface field is valid.
557 cuda For FI_HMEM_CUDA, this is equivalent to CUdevice (int).
559
561 Direct access to an application's memory by a remote peer requires that
562 the application register the targeted memory buffer(s). This is typi‐
563 cally done by calling one of the fi_mr_reg* routines. For
564 FI_MR_PROV_KEY, the provider will return a key that must be used by the
565 peer when accessing the memory region. The application is responsible
566 for transferring this key to the peer. If FI_MR_RAW mode has been set,
567 the key must be retrieved using the fi_mr_raw_attr function.
568 FI_RAW_MR allows support for providers that require more than 8-bytes
570 for their protection keys or need additional setup before a key can be
571 used for transfers. After a raw key has been retrieved, it must be ex‐
572 changed with the remote peer. The peer must use fi_mr_map_raw to con‐
573 vert the raw key into a usable 64-bit key. The mapping must be done
574 even if the raw key is 64-bits or smaller.
575 The raw key support functions are usable with all registered memory re‐
577 gions, even if FI_MR_RAW has not been set. It is recommended that por‐
578 table applications target using those interfaces; however, their use
579 does carry extra message and memory footprint overhead, making it less
580 desirable for highly scalable apps.
581
583 The follow flag may be specified to any memory registration call.
584 FI_RMA_EVENT
586 This flag indicates that the specified memory region will be as‐
587 sociated with a completion counter used to count RMA operations
588 that access the MR.
589 FI_RMA_PMEM
591 This flag indicates that the underlying memory region is backed
592 by persistent memory and will be used in RMA operations. It
593 must be specified if persistent completion semantics or persis‐
594 tent data transfers are required when accessing the registered
595 region.
596
598 Returns 0 on success. On error, a negative value corresponding to fab‐
599 ric errno is returned.
600 Fabric errno values are defined in rdma/fi_errno.h.
602
604 -FI_ENOKEY
605 The requested_key is already in use.
606 -FI_EKEYREJECTED
608 The requested_key is not available. They key may be out of the
609 range supported by the provider, or the provider may not support
610 user-requested memory registration keys.
611 -FI_ENOSYS
613 Returned by fi_mr_bind if the provider does not support report‐
614 ing events based on access to registered memory regions.
615 -FI_EBADFLAGS
617 Returned if the specified flags are not supported by the
618 provider.
619
621 Many hardware NICs accessed by libfabric require that data buffers be
622 registered with the hardware while the hardware accesses it. This en‐
623 sures that the virtual to physical address mappings for those buffers
624 do not change while the transfer is occurring. The performance impact
625 of registering memory regions can be significant. As a result, some
626 providers make use of a registration cache, particularly when working
627 with applications that are unable to manage their own network buffers.
628 A registration cache avoids the overhead of registering and unregister‐
629 ing a data buffer with each transfer.
630 As a general rule, if hardware requires the FI_MR_LOCAL mode bit de‐
632 scribed above, but this is not supported by the application, a memory
633 registration cache may be in use. The following environment variables
634 may be used to configure registration caches.
635 FI_MR_CACHE_MAX_SIZE
637 This defines the total number of bytes for all memory regions
638 that may be tracked by the cache. If not set, the cache has no
639 limit on how many bytes may be registered and cached. Setting
640 this will reduce the amount of memory that is not actively being
641 used as part of a data transfer that is registered with a
642 provider. By default, the cache size is unlimited.
643 FI_MR_CACHE_MAX_COUNT
645 This defines the total number of memory regions that may be reg‐
646 istered with the cache. If not set, a default limit is chosen.
647 Setting this will reduce the number of regions that are regis‐
648 tered, regardless of their size, which are not actively being
649 used as part of a data transfer. Setting this to zero will dis‐
650 able registration caching.
651 FI_MR_CACHE_MONITOR
653 The cache monitor is responsible for detecting changes made be‐
654 tween the virtual addresses used by an application and the un‐
655 derlying physical pages. Valid monitor options are: userfault‐
656 fd, memhooks, and disabled. Selecting disabled will turn off
657 the registration cache. Userfaultfd is a Linux kernel feature
658 used to report virtual to physical address mapping changes to
659 user space. Memhooks operates by intercepting relevant memory
660 allocation and deallocation calls which may result in the map‐
661 pings changing, such as malloc, mmap, free, etc. Note that
662 memhooks operates at the elf linker layer, and does not use
663 glibc memory hooks.
664
666 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_rma(3), fi_msg(3),
667 fi_atomic(3)
668
670 OpenFabrics.
671Libfabric Programmer's Manual 2020-04-14 fi_mr(3)