1fi_mr(3) Libfabric v1.12.1 fi_mr(3)
2
6 fi_mr - Memory region operations
7 fi_mr_reg / fi_mr_regv / fi_mr_regattr
9 Register local memory buffers for direct fabric access
10 fi_close
12 Deregister registered memory buffers.
13 fi_mr_desc
15 Return a local descriptor associated with a registered memory
16 region
17 fi_mr_key
19 Return the remote key needed to access a registered memory re‐
20 gion
21 fi_mr_raw_attr
23 Return raw memory region attributes.
24 fi_mr_map_raw
26 Converts a raw memory region key into a key that is usable for
27 data transfer operations.
28 fi_mr_unmap_key
30 Releases a previously mapped raw memory region key.
31 fi_mr_bind
33 Associate a registered memory region with a completion counter
34 or an endpoint.
35 fi_mr_refresh
37 Updates the memory pages associated with a memory region.
38 fi_mr_enable
40 Enables a memory region for use.
41
43 #include <rdma/fi_domain.h>
44 int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
46 uint64_t access, uint64_t offset, uint64_t requested_key,
47 uint64_t flags, struct fid_mr **mr, void *context);
48 int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
50 size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
51 uint64_t flags, struct fid_mr **mr, void *context);
52 int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
54 uint64_t flags, struct fid_mr **mr);
55 int fi_close(struct fid *mr);
57 void * fi_mr_desc(struct fid_mr *mr);
59 uint64_t fi_mr_key(struct fid_mr *mr);
61 int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
63 uint8_t *raw_key, size_t *key_size, uint64_t flags);
64 int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
66 uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
67 int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
69 int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
71 int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
73 uint64_t flags)
74 int fi_mr_enable(struct fid_mr *mr);
76
78 domain Resource domain
79 mr Memory region
81 bfid Fabric identifier of an associated resource.
83 context
85 User specified context associated with the memory region.
86 buf Memory buffer to register with the fabric hardware.
88 len Length of memory buffer to register. Must be > 0.
90 iov Vectored memory buffer.
92 count Count of vectored buffer entries.
94 access Memory access permissions associated with registration
96 offset Optional specified offset for accessing specified registered
98 buffers. This parameter is reserved for future use and must be
99 0.
100 requested_key
102 Optional requested remote key associated with registered buf‐
103 fers.
104 attr Memory region attributes
106 flags Additional flags to apply to the operation.
108
110 Registered memory regions associate memory buffers with permissions
111 granted for access by fabric resources. A memory buffer must be regis‐
112 tered with a resource domain before it can be used as the target of a
113 remote RMA or atomic data transfer. Additionally, a fabric provider
114 may require that data buffers be registered before being used in local
115 transfers. Memory registration restrictions are controlled using a
116 separate set of mode bits, specified through the domain attributes
117 (mr_mode field). Each mr_mode bit requires that an application take
118 specific steps in order to use memory buffers with libfabric inter‐
119 faces.
120 The following apply to memory registration.
122 Default Memory Registration
124 If no mr_mode bits are set, the default behaviors describe below
125 are followed. Historically, these defaults were collectively
126 referred to as scalable memory registration. The default re‐
127 quirements are outlined below, followed by definitions of how
128 each mr_mode bit alters the definition.
129 Compatibility: For library versions 1.4 and earlier, this was indicated
131 by setting mr_mode to FI_MR_SCALABLE and the fi_info mode bit FI_LO‐
132 CAL_MR to 0. FI_MR_SCALABLE and FI_LOCAL_MR were deprecated in libfab‐
133 ric version 1.5, though they are supported for backwards compatibility
134 purposes.
135 For security, memory registration is required for data buffers that are
137 accessed directly by a peer process. For example, registration is re‐
138 quired for RMA target buffers (read or written to), and those accessed
139 by atomic or collective operations.
140 By default, registration occurs on virtual address ranges. Because
142 registration refers to address ranges, rather than allocated data buf‐
143 fers, the address ranges do not need to map to data buffers allocated
144 by the application at the time the registration call is made. That is,
145 an application can register any range of addresses in their virtual ad‐
146 dress space, whether or not those addresses are backed by physical
147 pages or have been allocated.
148 Note that physical pages must back addresses prior to the addresses be‐
150 ing accessed as part of a data transfer operation, or the data trans‐
151 fers will fail. Additionally, depending on the operation, this could
152 result in the local process receiving a segmentation fault for access‐
153 ing invalid memory.
154 Once registered, the resulting memory regions are accessible by peers
156 starting at a base address of 0. That is, the target address that is
157 specified is a byte offset into the registered region.
158 The application also selects the access key associated with the MR.
160 The key size is restricted to a maximum of 8 bytes.
161 With scalable registration, locally accessed data buffers are not reg‐
163 istered. This includes source buffers for all transmit operations --
164 sends, tagged sends, RMA, and atomics -- as well as buffers posted for
165 receive and tagged receive operations.
166 Although the default memory registration behavior is convenient for ap‐
168 plication developers, it is difficult to implement in hardware. At‐
169 tempts to hide the hardware requirements from the application often re‐
170 sults in significant and unacceptable impacts to performance. The fol‐
171 lowing mr_mode bits are provided as input into fi_getinfo. If a
172 provider requires the behavior defined for an mr_mode bit, it will
173 leave the bit set on output to fi_getinfo. Otherwise, the provider can
174 clear the bit to indicate that the behavior is not needed.
175 By setting an mr_mode bit, the application has agreed to adjust its be‐
177 havior as indicated. Importantly, applications that choose to support
178 an mr_mode must be prepared to handle the case where the mr_mode is not
179 required. A provider will clear an mr_mode bit if it is not needed.
180 FI_MR_LOCAL
182 When the FI_MR_LOCAL mode bit is set, applications must register
183 all data buffers that will be accessed by the local hardware and
184 provide a valid desc parameter into applicable data transfer op‐
185 erations. When FI_MR_LOCAL is zero, applications are not re‐
186 quired to register data buffers before using them for local op‐
187 erations (e.g. send and receive data buffers). The desc param‐
188 eter into data transfer operations will be ignored in this case,
189 unless otherwise required (e.g. se FI_MR_HMEM). It is recom‐
190 mended that applications pass in NULL for desc when not re‐
191 quired.
192 A provider may hide local registration requirements from applications
194 by making use of an internal registration cache or similar mechanisms.
195 Such mechanisms, however, may negatively impact performance for some
196 applications, notably those which manage their own network buffers. In
197 order to support as broad range of applications as possible, without
198 unduly affecting their performance, applications that wish to manage
199 their own local memory registrations may do so by using the memory reg‐
200 istration calls.
201 Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
203 bit. When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
204 FI_MR_RAW
206 Raw memory regions are used to support providers with keys larg‐
207 er than 64-bits or require setup at the peer. When the
208 FI_MR_RAW bit is set, applications must use fi_mr_raw_attr() lo‐
209 cally and fi_mr_map_raw() at the peer before targeting a memory
210 region as part of any data transfer request.
211 FI_MR_VIRT_ADDR
213 The FI_MR_VIRT_ADDR bit indicates that the provider references
214 memory regions by virtual address, rather than a 0-based offset.
215 Peers that target memory regions registered with FI_MR_VIRT_ADDR
216 specify the destination memory buffer using the target's virtual
217 address, with any offset into the region specified as virtual
218 address + offset. Support of this bit typically implies that
219 peers must exchange addressing data prior to initiating any RMA
220 or atomic operation.
221 FI_MR_ALLOCATED
223 When set, all registered memory regions must be backed by physi‐
224 cal memory pages at the time the registration call is made.
225 FI_MR_PROV_KEY
227 This memory region mode indicates that the provider does not
228 support application requested MR keys. MR keys are returned by
229 the provider. Applications that support FI_MR_PROV_KEY can ob‐
230 tain the provider key using fi_mr_key(), unless FI_MR_RAW is al‐
231 so set. The returned key should then be exchanged with peers
232 prior to initiating an RMA or atomic operation.
233 FI_MR_MMU_NOTIFY
235 FI_MR_MMU_NOTIFY is typically set by providers that support mem‐
236 ory registration against memory regions that are not necessarily
237 backed by allocated physical pages at the time the memory regis‐
238 tration occurs. (That is, FI_MR_ALLOCATED is typically 0).
239 However, such providers require that applications notify the
240 provider prior to the MR being accessed as part of a data trans‐
241 fer operation. This notification informs the provider that all
242 necessary physical pages now back the region. The notification
243 is necessary for providers that cannot hook directly into the
244 operating system page tables or memory management unit. See
245 fi_mr_refresh() for notification details.
246 FI_MR_RMA_EVENT
248 This mode bit indicates that the provider must configure memory
249 regions that are associated with RMA events prior to their use.
250 This includes all memory regions that are associated with com‐
251 pletion counters. When set, applications must indicate if a
252 memory region will be associated with a completion counter as
253 part of the region's creation. This is done by passing in the
254 FI_RMA_EVENT flag to the memory registration call.
255 Such memory regions will be created in a disabled state and must be as‐
257 sociated with all completion counters prior to being enabled. To en‐
258 able a memory region, the application must call fi_mr_enable(). After
259 calling fi_mr_enable(), no further resource bindings may be made to the
260 memory region.
261 FI_MR_ENDPOINT
263 This mode bit indicates that the provider associates memory re‐
264 gions with endpoints rather than domains. Memory regions that
265 are registered with the provider are created in a disabled state
266 and must be bound to an endpoint prior to being enabled. To
267 bind the MR with an endpoint, the application must use
268 fi_mr_bind(). To enable the memory region, the application must
269 call fi_mr_enable().
270 FI_MR_HMEM
272 This mode bit is associated with the FI_HMEM capability. If
273 FI_MR_HMEM is set, the application must register buffers that
274 were allocated using a device call and provide a valid desc pa‐
275 rameter into applicable data transfer operations even if they
276 are only used for local operations (e.g. send and receive data
277 buffers). Device memory must be registered using the fi_mr_re‐
278 gattr call, with the iface and device fields filled out.
279 If FI_MR_HMEM is set, but FI_MR_LOCAL is unset, only device buffers
281 must be registered when used locally. In this case, the desc parameter
282 passed into data transfer operations must either be valid or NULL.
283 Similarly, if FI_MR_LOCAL is set, but FI_MR_HMEM is not, the desc pa‐
284 rameter must either be valid or NULL.
285 Basic Memory Registration
287 Basic memory registration was deprecated in libfabric version
288 1.5, but is supported for backwards compatibility. Basic memory
289 registration is indicated by setting mr_mode equal to FI_MR_BA‐
290 SIC. FI_MR_BASIC must be set alone and not paired with mr_mode
291 bits. Unlike other mr_mode bits, if FI_MR_BASIC is set on input
292 to fi_getinfo(), it will not be cleared by the provider. That
293 is, setting mr_mode equal to FI_MR_BASIC forces basic registra‐
294 tion if the provider supports it.
295 The behavior of basic registration is equivalent to requiring the fol‐
297 lowing mr_mode bits: FI_MR_VIRT_ADDR, FI_MR_ALLOCATED, and
298 FI_MR_PROV_KEY. Additionally, providers that support basic registra‐
299 tion usually require the (deprecated) fi_info mode bit FI_LOCAL_MR,
300 which was incorporated into the FI_MR_LOCAL mr_mode bit.
301 The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
303 -- are used to register one or more memory regions with fabric re‐
304 sources. The main difference between registration functions are the
305 number and type of parameters that they accept as input. Otherwise,
306 they perform the same general function.
307 By default, memory registration completes synchronously. I.e. the
309 registration call will not return until the registration has completed.
310 Memory registration can complete asynchronous by binding the resource
311 domain to an event queue using the FI_REG_MR flag. See fi_domain_bind.
312 When memory registration is asynchronous, in order to avoid a race con‐
313 dition between the registration call returning and the corresponding
314 reading of the event from the EQ, the mr output parameter will be writ‐
315 ten before any event associated with the operation may be read by the
316 application. An asynchronous event will not be generated unless the
317 registration call returns success (0).
318 fi_mr_reg
320 The fi_mr_reg call registers the user-specified memory buffer with the
321 resource domain. The buffer is enabled for access by the fabric hard‐
322 ware based on the provided access permissions. See the access field
323 description for memory region attributes below.
324 Registered memory is associated with a local memory descriptor and, op‐
326 tionally, a remote memory key. A memory descriptor is a provider spe‐
327 cific identifier associated with registered memory. Memory descriptors
328 often map to hardware specific indices or keys associated with the mem‐
329 ory region. Remote memory keys provide limited protection against un‐
330 wanted access by a remote node. Remote accesses to a memory region
331 must provide the key associated with the registration.
332 Because MR keys must be provided by a remote process, an application
334 can use the requested_key parameter to indicate that a specific key
335 value be returned. Support for user requested keys is provider specif‐
336 ic and is determined by the mr_mode domain attribute.
337 Remote RMA and atomic operations indicate the location within a regis‐
339 tered memory region by specifying an address. The location is refer‐
340 enced by adding the offset to either the base virtual address of the
341 buffer or to 0, depending on the mr_mode.
342 The offset parameter is reserved for future use and must be 0.
344 For asynchronous memory registration requests, the result will be re‐
346 ported to the user through an event queue associated with the resource
347 domain. If successful, the allocated memory region structure will be
348 returned to the user through the mr parameter. The mr address must re‐
349 main valid until the registration operation completes. The context
350 specified with the registration request is returned with the completion
351 event.
352 fi_mr_regv
354 The fi_mr_regv call adds support for a scatter-gather list to
355 fi_mr_reg. Multiple memory buffers are registered as a single memory
356 region. Otherwise, the operation is the same.
357 fi_mr_regattr
359 The fi_mr_regattr call is a more generic, extensible registration call
360 that allows the user to specify the registration request using a struct
361 fi_mr_attr (defined below).
362 fi_close
364 Fi_close is used to release all resources associated with a registering
365 a memory region. Once unregistered, further access to the registered
366 memory is not guaranteed. Active or queued operations that reference a
367 memory region being closed may fail or result in accesses to invalid
368 memory. Applications are responsible for ensuring that a MR is no
369 longer needed prior to closing it. Note that accesses to a closed MR
370 from a remote peer will result in an error at the peer. The state of
371 the local endpoint will be unaffected.
372 When closing the MR, there must be no opened endpoints or counters as‐
374 sociated with the MR. If resources are still associated with the MR
375 when attempting to close, the call will return -FI_EBUSY.
376 fi_mr_desc
378 Obtains the local memory descriptor associated with a MR. The memory
379 registration must have completed successfully before invoking this
380 call.
381 fi_mr_key
383 Returns the remote protection key associated with a MR. The memory
384 registration must have completed successfully before invoking this.
385 The returned key may be used in data transfer operations at a peer. If
386 the FI_RAW_MR mode bit has been set for the domain, then the memory key
387 must be obtained using the fi_mr_raw_key function instead. A return
388 value of FI_KEY_NOTAVAIL will be returned if the registration has not
389 completed or a raw memory key is required.
390 fi_mr_raw_attr
392 Returns the raw, remote protection key and base address associated with
393 a MR. The memory registration must have completed successfully before
394 invoking this routine. Use of this call is required if the FI_RAW_MR
395 mode bit has been set by the provider; however, it is safe to use this
396 call with any memory region.
397 On input, the key_size parameter should indicate the size of the
399 raw_key buffer. If the actual key is larger than what can fit into the
400 buffer, it will return -FI_ETOOSMALL. On output, key_size is set to
401 the size of the buffer needed to store the key, which may be larger
402 than the input value. The needed key_size can also be obtained through
403 the mr_key_size domain attribute (fi_domain_attr) field.
404 A raw key must be mapped by a peer before it can be used in data trans‐
406 fer operations. See fi_mr_map_raw below.
407 fi_mr_map_raw
409 Raw protection keys must be mapped to a usable key value before they
410 can be used for data transfer operations. The mapping is done by the
411 peer that initiates the RMA or atomic operation. The mapping function
412 takes as input the raw key and its size, and returns the mapped key.
413 Use of the fi_mr_map_raw function is required if the peer has the
414 FI_RAW_MR mode bit set, but this routine may be called on any valid
415 key. All mapped keys must be freed by calling fi_mr_unmap_key when ac‐
416 cess to the peer memory region is no longer necessary.
417 fi_mr_unmap_key
419 This call releases any resources that may have been allocated as part
420 of mapping a raw memory key. All mapped keys must be freed before the
421 corresponding domain is closed.
422 fi_mr_bind
424 The fi_mr_bind function associates a memory region with a counter or
425 endpoint. Counter bindings are needed by providers that support the
426 generation of completions based on fabric operations. Endpoint bind‐
427 ings are needed if the provider associates memory regions with end‐
428 points (see FI_MR_ENDPOINT).
429 When binding with a counter, the type of events tracked against the
431 memory region is based on the bitwise OR of the following flags.
432 FI_REMOTE_WRITE
434 Generates an event whenever a remote RMA write or atomic opera‐
435 tion modifies the memory region. Use of this flag requires that
436 the endpoint through which the MR is accessed be created with
437 the FI_RMA_EVENT capability.
438 When binding the memory region to an endpoint, flags should be 0.
440 fi_mr_refresh
442 The use of this call is required to notify the provider of any change
443 to the physical pages backing a registered memory region if the
444 FI_MR_MMU_NOTIFY mode bit has been set. This call informs the provider
445 that the page table entries associated with the region may have been
446 modified, and the provider should verify and update the registered re‐
447 gion accordingly. The iov parameter is optional and may be used to
448 specify which portions of the registered region requires updating.
449 Providers are only guaranteed to update the specified address ranges.
450 The refresh operation has the effect of disabling and re-enabling ac‐
452 cess to the registered region. Any operations from peers that attempt
453 to access the region will fail while the refresh is occurring. Addi‐
454 tionally, attempts to access the region by the local process through
455 libfabric APIs may result in a page fault or other fatal operation.
456 The fi_mr_refresh call is only needed if the physical pages might have
458 been updated after the memory region was created.
459 fi_mr_enable
461 The enable call is used with memory registration associated with the
462 FI_MR_RMA_EVENT mode bit. Memory regions created in the disabled state
463 must be explicitly enabled after being fully configured by the applica‐
464 tion. Any resource bindings to the MR must be done prior to enabling
465 the MR.
466
468 Memory regions are created using the following attributes. The struct
469 fi_mr_attr is passed into fi_mr_regattr, but individual fields also ap‐
470 ply to other memory registration calls, with the fields passed directly
471 into calls as function parameters.
472 struct fi_mr_attr {
474 const struct iovec *mr_iov;
475 size_t iov_count;
476 uint64_t access;
477 uint64_t offset;
478 uint64_t requested_key;
479 void *context;
480 size_t auth_key_size;
481 uint8_t *auth_key;
482 enum fi_hmem_iface iface;
483 union {
484 uint64_t reserved;
485 int cuda;
486 int ze
487 } device;
488 };
489 mr_iov
491 This is an IO vector of addresses that will represent a single memory
492 region. The number of entries in the iovec is specified by iov_count.
493 iov_count
495 The number of entries in the mr_iov array. The maximum number of memo‐
496 ry buffers that may be associated with a single memory region is speci‐
497 fied as the mr_iov_limit domain attribute. See fi_domain(3).
498 access
500 Indicates the type of operations that the local or a peer endpoint may
501 perform on registered memory region. Supported access permissions are
502 the bitwise OR of the following flags:
503 FI_SEND
505 The memory buffer may be used in outgoing message data trans‐
506 fers. This includes fi_msg and fi_tagged send operations.
507 FI_RECV
509 The memory buffer may be used to receive inbound message trans‐
510 fers. This includes fi_msg and fi_tagged receive operations.
511 FI_READ
513 The memory buffer may be used as the result buffer for RMA read
514 and atomic operations on the initiator side. Note that from the
515 viewpoint of the application, the memory buffer is being written
516 into by the network.
517 FI_WRITE
519 The memory buffer may be used as the source buffer for RMA write
520 and atomic operations on the initiator side. Note that from the
521 viewpoint of the application, the endpoint is reading from the
522 memory buffer and copying the data onto the network.
523 FI_REMOTE_READ
525 The memory buffer may be used as the source buffer of an RMA
526 read operation on the target side. The contents of the memory
527 buffer are not modified by such operations.
528 FI_REMOTE_WRITE
530 The memory buffer may be used as the target buffer of an RMA
531 write or atomic operation. The contents of the memory buffer
532 may be modified as a result of such operations.
533 Note that some providers may not enforce fine grained access permis‐
535 sions. For example, a memory region registered for FI_WRITE access may
536 also behave as if FI_SEND were specified as well. Relaxed enforcement
537 of such access is permitted, though not guaranteed, provided security
538 is maintained.
539 offset
541 The offset field is reserved for future use and must be 0.
542 requested_key
544 An application specified access key associated with the memory region.
545 The MR key must be provided by a remote process when performing RMA or
546 atomic operations to a memory region. Applications can use the re‐
547 quested_key field to indicate that a specific key be used by the
548 provider. This allows applications to use well known key values, which
549 can avoid applications needing to exchange and store keys. Support for
550 user requested keys is provider specific and is determined by the
551 mr_mode domain attribute.
552 context
554 Application context associated with asynchronous memory registration
555 operations. This value is returned as part of any asynchronous event
556 associated with the registration. This field is ignored for synchro‐
557 nous registration calls.
558 auth_key_size
560 The size of key referenced by the auth_key field in bytes, or 0 if no
561 authorization key is given. This field is ignored unless the fabric is
562 opened with API version 1.5 or greater.
563 auth_key
565 Indicates the key to associate with this memory registration. Autho‐
566 rization keys are used to limit communication between endpoints. Only
567 peer endpoints that are programmed to use the same authorization key
568 may access the memory region. The domain authorization key will be
569 used if the auth_key_size provided is 0. This field is ignored unless
570 the fabric is opened with API version 1.5 or greater.
571 iface
573 Indicates the software interfaces used by the application to allocate
574 and manage the memory region. This field is ignored unless the appli‐
575 cation has requested the FI_HMEM capability.
576 FI_HMEM_SYSTEM
578 Uses standard operating system calls and libraries, such as mal‐
579 loc, calloc, realloc, mmap, and free.
580 FI_HMEM_CUDA
582 Uses Nvidia CUDA interfaces such as cuMemAlloc, cuMemAllocHost,
583 cuMemAllocManaged, cuMemFree, cudaMalloc, cudaFree.
584 FI_HMEM_ROCR
586 Uses AMD ROCR interfaces such as hsa_memory_allocate and
587 hsa_memory_free.
588 FI_HMEM_ZE
590 Uses Intel L0 ZE interfaces such as zeDriverAllocSharedMem,
591 zeDriverFreeMem.
592 device
594 Reserved 64 bits for device identifier if using non-standard HMEM in‐
595 terface. This field is ignore unless the iface field is valid.
596 cuda For FI_HMEM_CUDA, this is equivalent to CUdevice (int).
598 ze For FI_HMEM_ZE, this is equivalent to the ze_device_handle_t in‐
600 dex (int).
601
603 Direct access to an application's memory by a remote peer requires that
604 the application register the targeted memory buffer(s). This is typi‐
605 cally done by calling one of the fi_mr_reg* routines. For
606 FI_MR_PROV_KEY, the provider will return a key that must be used by the
607 peer when accessing the memory region. The application is responsible
608 for transferring this key to the peer. If FI_MR_RAW mode has been set,
609 the key must be retrieved using the fi_mr_raw_attr function.
610 FI_RAW_MR allows support for providers that require more than 8-bytes
612 for their protection keys or need additional setup before a key can be
613 used for transfers. After a raw key has been retrieved, it must be ex‐
614 changed with the remote peer. The peer must use fi_mr_map_raw to con‐
615 vert the raw key into a usable 64-bit key. The mapping must be done
616 even if the raw key is 64-bits or smaller.
617 The raw key support functions are usable with all registered memory re‐
619 gions, even if FI_MR_RAW has not been set. It is recommended that por‐
620 table applications target using those interfaces; however, their use
621 does carry extra message and memory footprint overhead, making it less
622 desirable for highly scalable apps.
623
625 The follow flag may be specified to any memory registration call.
626 FI_RMA_EVENT
628 This flag indicates that the specified memory region will be as‐
629 sociated with a completion counter used to count RMA operations
630 that access the MR.
631 FI_RMA_PMEM
633 This flag indicates that the underlying memory region is backed
634 by persistent memory and will be used in RMA operations. It
635 must be specified if persistent completion semantics or persis‐
636 tent data transfers are required when accessing the registered
637 region.
638
640 Returns 0 on success. On error, a negative value corresponding to fab‐
641 ric errno is returned.
642 Fabric errno values are defined in rdma/fi_errno.h.
644
646 -FI_ENOKEY
647 The requested_key is already in use.
648 -FI_EKEYREJECTED
650 The requested_key is not available. They key may be out of the
651 range supported by the provider, or the provider may not support
652 user-requested memory registration keys.
653 -FI_ENOSYS
655 Returned by fi_mr_bind if the provider does not support report‐
656 ing events based on access to registered memory regions.
657 -FI_EBADFLAGS
659 Returned if the specified flags are not supported by the
660 provider.
661
663 Many hardware NICs accessed by libfabric require that data buffers be
664 registered with the hardware while the hardware accesses it. This en‐
665 sures that the virtual to physical address mappings for those buffers
666 do not change while the transfer is occurring. The performance impact
667 of registering memory regions can be significant. As a result, some
668 providers make use of a registration cache, particularly when working
669 with applications that are unable to manage their own network buffers.
670 A registration cache avoids the overhead of registering and unregister‐
671 ing a data buffer with each transfer.
672 If a registration cache is going to be used for host and device memory,
674 the device must support unified virtual addressing. If the device does
675 not support unified virtual addressing, either an additional registra‐
676 tion cache is required to track this device memory, or device memory
677 cannot be cached.
678 As a general rule, if hardware requires the FI_MR_LOCAL mode bit de‐
680 scribed above, but this is not supported by the application, a memory
681 registration cache may be in use. The following environment variables
682 may be used to configure registration caches.
683 FI_MR_CACHE_MAX_SIZE
685 This defines the total number of bytes for all memory regions
686 that may be tracked by the cache. If not set, the cache has no
687 limit on how many bytes may be registered and cached. Setting
688 this will reduce the amount of memory that is not actively being
689 used as part of a data transfer that is registered with a
690 provider. By default, the cache size is unlimited.
691 FI_MR_CACHE_MAX_COUNT
693 This defines the total number of memory regions that may be reg‐
694 istered with the cache. If not set, a default limit is chosen.
695 Setting this will reduce the number of regions that are regis‐
696 tered, regardless of their size, which are not actively being
697 used as part of a data transfer. Setting this to zero will dis‐
698 able registration caching.
699 FI_MR_CACHE_MONITOR
701 The cache monitor is responsible for detecting system memory
702 (FI_HMEM_SYSTEM) changes made between the virtual addresses used
703 by an application and the underlying physical pages. Valid mon‐
704 itor options are: userfaultfd, memhooks, and disabled. Select‐
705 ing disabled will turn off the registration cache. Userfaultfd
706 is a Linux kernel feature used to report virtual to physical ad‐
707 dress mapping changes to user space. Memhooks operates by in‐
708 tercepting relevant memory allocation and deallocation calls
709 which may result in the mappings changing, such as malloc, mmap,
710 free, etc. Note that memhooks operates at the elf linker layer,
711 and does not use glibc memory hooks.
712 FI_MR_CUDA_CACHE_MONITOR_ENABLED
714 The CUDA cache monitor is responsible for detecting CUDA device
715 memory (FI_HMEM_CUDA) changes made between the device virtual
716 addresses used by an application and the underlying device phys‐
717 ical pages. Valid monitor options are: 0 or 1. Note that the
718 CUDA memory monitor requires a CUDA toolkit version with unified
719 virtual addressing enabled.
720 FI_MR_ROCR_CACHE_MONITOR_ENABLED
722 The ROCR cache monitor is responsible for detecting ROCR device
723 memory (FI_HMEM_ROCR) changes made between the device virtual
724 addresses used by an application and the underlying device phys‐
725 ical pages. Valid monitor options are: 0 or 1. Note that the
726 ROCR memory monitor requires a ROCR version with unified virtual
727 addressing enabled.
728
730 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_rma(3), fi_msg(3),
731 fi_atomic(3)
732
734 OpenFabrics.
735Libfabric Programmer's Manual 2020-10-01 fi_mr(3)