1fi_mr(3)                       Libfabric v1.10.0                      fi_mr(3)
2
3
4

NAME

6       fi_mr - Memory region operations
7
8       fi_mr_reg / fi_mr_regv / fi_mr_regattr
9              Register local memory buffers for direct fabric access
10
11       fi_close
12              Deregister registered memory buffers.
13
14       fi_mr_desc
15              Return  a  local  descriptor associated with a registered memory
16              region
17
18       fi_mr_key
19              Return the remote key needed to access a registered  memory  re‐
20              gion
21
22       fi_mr_raw_attr
23              Return raw memory region attributes.
24
25       fi_mr_map_raw
26              Converts  a  raw memory region key into a key that is usable for
27              data transfer operations.
28
29       fi_mr_unmap_key
30              Releases a previously mapped raw memory region key.
31
32       fi_mr_bind
33              Associate a registered memory region with a  completion  counter
34              or an endpoint.
35
36       fi_mr_refresh
37              Updates the memory pages associated with a memory region.
38
39       fi_mr_enable
40              Enables a memory region for use.
41

SYNOPSIS

43              #include <rdma/fi_domain.h>
44
45              int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
46                  uint64_t access, uint64_t offset, uint64_t requested_key,
47                  uint64_t flags, struct fid_mr **mr, void *context);
48
49              int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
50                  size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
51                  uint64_t flags, struct fid_mr **mr, void *context);
52
53              int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
54                  uint64_t flags, struct fid_mr **mr);
55
56              int fi_close(struct fid *mr);
57
58              void * fi_mr_desc(struct fid_mr *mr);
59
60              uint64_t fi_mr_key(struct fid_mr *mr);
61
62              int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
63                  uint8_t *raw_key, size_t *key_size, uint64_t flags);
64
65              int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
66                  uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
67
68              int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
69
70              int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
71
72              int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
73                  uint64_t flags)
74
75              int fi_mr_enable(struct fid_mr *mr);
76

ARGUMENTS

78       domain Resource domain
79
80       mr     Memory region
81
82       bfid   Fabric identifier of an associated resource.
83
84       context
85              User specified context associated with the memory region.
86
87       buf    Memory buffer to register with the fabric hardware
88
89       len    Length of memory buffer to register
90
91       iov    Vectored memory buffer.
92
93       count  Count of vectored buffer entries.
94
95       access Memory access permissions associated with registration
96
97       offset Optional  specified  offset  for  accessing specified registered
98              buffers.  This parameter is reserved for future use and must  be
99              0.
100
101       requested_key
102              Optional  requested  remote  key associated with registered buf‐
103              fers.
104
105       attr   Memory region attributes
106
107       flags  Additional flags to apply to the operation.
108

DESCRIPTION

110       Registered memory regions associate  memory  buffers  with  permissions
111       granted for access by fabric resources.  A memory buffer must be regis‐
112       tered with a resource domain before it can be used as the target  of  a
113       remote  RMA  or  atomic data transfer.  Additionally, a fabric provider
114       may require that data buffers be registered before being used in  local
115       transfers.   Memory  registration  restrictions  are controlled using a
116       separate set of mode bits,  specified  through  the  domain  attributes
117       (mr_mode field).
118
119       The following apply to memory registration.
120
121       Scalable Memory Registration
122              By  default,  memory  registration is considered scalable.  (For
123              library versions 1.4 and earlier, this is indicated  by  setting
124              mr_mode to FI_MR_SCALABLE, with the fi_info mode bit FI_LOCAL_MR
125              set to 0).  For versions 1.5 and later, scalable is  implied  by
126              the  lack of any mr_mode bits being set.  The setting of mr_mode
127              bits therefore adjusts application behavior as described  below.
128              Default, scalable registration has several properties.
129
130       In  scalable  mode,  registration occurs on memory address ranges.  Be‐
131       cause registration refers to memory regions, versus data  buffers,  the
132       address  ranges  given for a registration request do not need to map to
133       data buffers allocated by the application at the time the  registration
134       call  is  made.   That is, an application can register any range of ad‐
135       dresses in their virtual address space, whether or not those  addresses
136       are backed by physical pages or have been allocated.
137
138       The resulting memory regions are accessible by peers starting at a base
139       address of 0.  That is, the target address that is specified is a  byte
140       offset into the registered region.
141
142       The  application  also  selects  the access key associated with the MR.
143       The key size is restricted to a maximum of 8 bytes.
144
145       With scalable registration, locally accessed data buffers are not  reg‐
146       istered.   This  includes source buffers for all transmit operations --
147       sends, tagged sends, RMA, and atomics -- as well as buffers posted  for
148       receive and tagged receive operations.
149
150       FI_MR_LOCAL
151              When the FI_MR_LOCAL mode bit is set, applications must register
152              all data buffers that will be accessed by the local hardware and
153              provide a valid desc parameter into applicable data transfer op‐
154              erations.  When FI_MR_LOCAL is zero, applications  are  not  re‐
155              quired  to register data buffers before using them for local op‐
156              erations (e.g.  send and receive data buffers).  The desc param‐
157              eter into data transfer operations will be ignored in this case,
158              unless otherwise required (e.g.  se FI_MR_HMEM).  It  is  recom‐
159              mended  that  applications  pass  in  NULL for desc when not re‐
160              quired.
161
162       A provider may hide local registration requirements  from  applications
163       by  making use of an internal registration cache or similar mechanisms.
164       Such mechanisms, however, may negatively impact  performance  for  some
165       applications, notably those which manage their own network buffers.  In
166       order to support as broad range of applications  as  possible,  without
167       unduly  affecting  their  performance, applications that wish to manage
168       their own local memory registrations may do so by using the memory reg‐
169       istration calls.
170
171       Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
172       bit.  When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
173
174       FI_MR_RAW
175              Raw memory regions are used to support providers with keys larg‐
176              er  than  64-bits  or  require  setup  at  the  peer.   When the
177              FI_MR_RAW bit is set, applications must use fi_mr_raw_attr() lo‐
178              cally  and fi_mr_map_raw() at the peer before targeting a memory
179              region as part of any data transfer request.
180
181       FI_MR_VIRT_ADDR
182              The FI_MR_VIRT_ADDR bit indicates that the  provider  references
183              memory regions by virtual address, rather than a 0-based offset.
184              Peers that target memory regions registered with FI_MR_VIRT_ADDR
185              specify the destination memory buffer using the target's virtual
186              address, with any offset into the region  specified  as  virtual
187              address  +  offset.   Support of this bit typically implies that
188              peers must exchange addressing data prior to initiating any  RMA
189              or atomic operation.
190
191       FI_MR_ALLOCATED
192              When set, all registered memory regions must be backed by physi‐
193              cal memory pages at the time the registration call is made.
194
195       FI_MR_PROV_KEY
196              This memory region mode indicates that  the  provider  does  not
197              support  application requested MR keys.  MR keys are returned by
198              the provider.  Applications that support FI_MR_PROV_KEY can  ob‐
199              tain the provider key using fi_mr_key(), unless FI_MR_RAW is al‐
200              so set.  The returned key should then be  exchanged  with  peers
201              prior to initiating an RMA or atomic operation.
202
203       FI_MR_MMU_NOTIFY
204              FI_MR_MMU_NOTIFY is typically set by providers that support mem‐
205              ory registration against memory regions that are not necessarily
206              backed by allocated physical pages at the time the memory regis‐
207              tration occurs.  (That  is,  FI_MR_ALLOCATED  is  typically  0).
208              However,  such  providers  require  that applications notify the
209              provider prior to the MR being accessed as part of a data trans‐
210              fer  operation.  This notification informs the provider that all
211              necessary physical pages now back the region.  The  notification
212              is  necessary  for  providers that cannot hook directly into the
213              operating system page tables or  memory  management  unit.   See
214              fi_mr_refresh() for notification details.
215
216       FI_MR_RMA_EVENT
217              This  mode bit indicates that the provider must configure memory
218              regions that are associated with RMA events prior to their  use.
219              This  includes  all memory regions that are associated with com‐
220              pletion counters.  When set, applications  must  indicate  if  a
221              memory  region  will  be associated with a completion counter as
222              part of the region's creation.  This is done by passing  in  the
223              FI_RMA_EVENT flag to the memory registration call.
224
225       Such memory regions will be created in a disabled state and must be as‐
226       sociated with all completion counters prior to being enabled.   To  en‐
227       able  a memory region, the application must call fi_mr_enable().  After
228       calling fi_mr_enable(), no further resource bindings may be made to the
229       memory region.
230
231       FI_MR_ENDPOINT
232              This  mode bit indicates that the provider associates memory re‐
233              gions with endpoints rather than domains.  Memory  regions  that
234              are registered with the provider are created in a disabled state
235              and must be bound to an endpoint prior  to  being  enabled.   To
236              bind   the  MR  with  an  endpoint,  the  application  must  use
237              fi_mr_bind().  To enable the memory region, the application must
238              call fi_mr_enable().
239
240       FI_MR_HMEM
241              This  mode  bit  is  associated with the FI_HMEM capability.  If
242              FI_MR_HMEM is set, the application must  register  buffers  that
243              were  allocated using a device call and provide a valid desc pa‐
244              rameter into applicable data transfer operations  even  if  they
245              are  only used for local operations (e.g.  send and receive data
246              buffers).  Device memory must be registered using the  fi_mr_re‐
247              gattr call, with the iface and device fields filled out.
248
249       If  FI_MR_HMEM  is  set,  but FI_MR_LOCAL is unset, only device buffers
250       must be registered when used locally.  In this case, the desc parameter
251       passed  into  data  transfer  operations  must either be valid or NULL.
252       Similarly, if FI_MR_LOCAL is set, but FI_MR_HMEM is not, the  desc  pa‐
253       rameter must either be valid or NULL.
254
255       Basic Memory Registration
256              Basic  memory  registration  is  indicated  by  the  FI_MR_BASIC
257              mr_mode bit.  FI_MR_BASIC is maintained for  backwards  compati‐
258              bility  (libfabric version 1.4 or earlier).  The behavior of ba‐
259              sic registration is equivalent to setting the following  mr_mode
260              bits    to    one:    FI_MR_VIRT_ADDR,    FI_MR_ALLOCATED,   and
261              FI_MR_PROV_KEY.  Additionally, providers that support basic reg‐
262              istration usually required the fi_info mode bit FI_LOCAL_MR.  As
263              a result, it is recommended  that  applications  migrating  from
264              libfabric 1.4 or earlier or wanting to support basic memory reg‐
265              istration set the mr_mode to FI_MR_VIRT_ADDR | FI_MR_ALLOCATED |
266              FI_MR_PROV_KEY  |  FI_MR_LOCAL.   FI_MR_BASIC must be set alone.
267              Other mr_mode bit pairings are invalid.   Unlike  other  mr_mode
268              bits,  if  FI_MR_BASIC  is set on input to fi_getinfo(), it will
269              not be cleared by the provider.  That is, setting FI_MR_BASIC to
270              one requests basic registration.
271
272       The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
273       -- are used to register one or more  memory  regions  with  fabric  re‐
274       sources.   The  main  difference between registration functions are the
275       number and type of parameters that they accept  as  input.   Otherwise,
276       they perform the same general function.
277
278       By  default,  memory  registration  completes synchronously.  I.e.  the
279       registration call will not return until the registration has completed.
280       Memory  registration  can complete asynchronous by binding the resource
281       domain to an event queue using the FI_REG_MR flag.  See fi_domain_bind.
282       When memory registration is asynchronous, in order to avoid a race con‐
283       dition between the registration call returning  and  the  corresponding
284       reading of the event from the EQ, the mr output parameter will be writ‐
285       ten before any event associated with the operation may be read  by  the
286       application.   An  asynchronous  event will not be generated unless the
287       registration call returns success (0).
288
289   fi_mr_reg
290       The fi_mr_reg call registers the user-specified memory buffer with  the
291       resource  domain.  The buffer is enabled for access by the fabric hard‐
292       ware based on the provided access permissions.  See  the  access  field
293       description for memory region attributes below.
294
295       Registered memory is associated with a local memory descriptor and, op‐
296       tionally, a remote memory key.  A memory descriptor is a provider  spe‐
297       cific identifier associated with registered memory.  Memory descriptors
298       often map to hardware specific indices or keys associated with the mem‐
299       ory  region.  Remote memory keys provide limited protection against un‐
300       wanted access by a remote node.  Remote accesses  to  a  memory  region
301       must provide the key associated with the registration.
302
303       Because  MR  keys  must be provided by a remote process, an application
304       can use the requested_key parameter to indicate  that  a  specific  key
305       value be returned.  Support for user requested keys is provider specif‐
306       ic and is determined by the mr_mode domain attribute.
307
308       Remote RMA and atomic operations indicate the location within a  regis‐
309       tered  memory  region by specifying an address.  The location is refer‐
310       enced by adding the offset to either the base virtual  address  of  the
311       buffer or to 0, depending on the mr_mode.
312
313       The offset parameter is reserved for future use and must be 0.
314
315       For  asynchronous  memory registration requests, the result will be re‐
316       ported to the user through an event queue associated with the  resource
317       domain.   If  successful, the allocated memory region structure will be
318       returned to the user through the mr parameter.  The mr address must re‐
319       main  valid  until  the  registration operation completes.  The context
320       specified with the registration request is returned with the completion
321       event.
322
323   fi_mr_regv
324       The   fi_mr_regv  call  adds  support  for  a  scatter-gather  list  to
325       fi_mr_reg.  Multiple memory buffers are registered as a  single  memory
326       region.  Otherwise, the operation is the same.
327
328   fi_mr_regattr
329       The  fi_mr_regattr call is a more generic, extensible registration call
330       that allows the user to specify the registration request using a struct
331       fi_mr_attr (defined below).
332
333   fi_close
334       Fi_close is used to release all resources associated with a registering
335       a memory region.  Once unregistered, further access to  the  registered
336       memory is not guaranteed.  Active or queued operations that reference a
337       memory region being closed may fail or result in  accesses  to  invalid
338       memory.   Applications  are  responsible  for  ensuring that a MR is no
339       longer needed prior to closing it.  Note that accesses to a  closed  MR
340       from  a  remote peer will result in an error at the peer.  The state of
341       the local endpoint will be unaffected.
342
343       When closing the MR, there must be no opened endpoints or counters  as‐
344       sociated  with  the  MR.  If resources are still associated with the MR
345       when attempting to close, the call will return -FI_EBUSY.
346
347   fi_mr_desc
348       Obtains the local memory descriptor associated with a MR.   The  memory
349       registration  must  have  completed  successfully  before invoking this
350       call.
351
352   fi_mr_key
353       Returns the remote protection key associated with  a  MR.   The  memory
354       registration  must  have  completed  successfully before invoking this.
355       The returned key may be used in data transfer operations at a peer.  If
356       the FI_RAW_MR mode bit has been set for the domain, then the memory key
357       must be obtained using the fi_mr_raw_key function  instead.   A  return
358       value  of  FI_KEY_NOTAVAIL will be returned if the registration has not
359       completed or a raw memory key is required.
360
361   fi_mr_raw_attr
362       Returns the raw, remote protection key and base address associated with
363       a  MR.  The memory registration must have completed successfully before
364       invoking this routine.  Use of this call is required if  the  FI_RAW_MR
365       mode  bit has been set by the provider; however, it is safe to use this
366       call with any memory region.
367
368       On input, the key_size  parameter  should  indicate  the  size  of  the
369       raw_key buffer.  If the actual key is larger than what can fit into the
370       buffer, it will return -FI_ETOOSMALL.  On output, key_size  is  set  to
371       the  size  of  the  buffer needed to store the key, which may be larger
372       than the input value.  The needed key_size can also be obtained through
373       the mr_key_size domain attribute (fi_domain_attr) field.
374
375       A raw key must be mapped by a peer before it can be used in data trans‐
376       fer operations.  See fi_mr_map_raw below.
377
378   fi_mr_map_raw
379       Raw protection keys must be mapped to a usable key  value  before  they
380       can  be  used for data transfer operations.  The mapping is done by the
381       peer that initiates the RMA or atomic operation.  The mapping  function
382       takes  as  input  the raw key and its size, and returns the mapped key.
383       Use of the fi_mr_map_raw function is  required  if  the  peer  has  the
384       FI_RAW_MR  mode  bit  set,  but this routine may be called on any valid
385       key.  All mapped keys must be freed by calling fi_mr_unmap_key when ac‐
386       cess to the peer memory region is no longer necessary.
387
388   fi_mr_unmap_key
389       This  call  releases any resources that may have been allocated as part
390       of mapping a raw memory key.  All mapped keys must be freed before  the
391       corresponding domain is closed.
392
393   fi_mr_bind
394       The  fi_mr_bind  function  associates a memory region with a counter or
395       endpoint.  Counter bindings are needed by providers  that  support  the
396       generation  of  completions based on fabric operations.  Endpoint bind‐
397       ings are needed if the provider associates  memory  regions  with  end‐
398       points (see FI_MR_ENDPOINT).
399
400       When  binding  with  a  counter, the type of events tracked against the
401       memory region is based on the bitwise OR of the following flags.
402
403       FI_REMOTE_WRITE
404              Generates an event whenever a remote RMA write or atomic  opera‐
405              tion modifies the memory region.  Use of this flag requires that
406              the endpoint through which the MR is accessed  be  created  with
407              the FI_RMA_EVENT capability.
408
409       When binding the memory region to an endpoint, flags should be 0.
410
411   fi_mr_refresh
412       The  use  of this call is required to notify the provider of any change
413       to the physical  pages  backing  a  registered  memory  region  if  the
414       FI_MR_MMU_NOTIFY mode bit has been set.  This call informs the provider
415       that the page table entries associated with the region  may  have  been
416       modified,  and the provider should verify and update the registered re‐
417       gion accordingly.  The iov parameter is optional and  may  be  used  to
418       specify  which  portions  of  the  registered region requires updating.
419       Providers are only guaranteed to update the specified address ranges.
420
421       The refresh operation has the effect of disabling and  re-enabling  ac‐
422       cess  to the registered region.  Any operations from peers that attempt
423       to access the region will fail while the refresh is  occurring.   Addi‐
424       tionally,  attempts  to  access the region by the local process through
425       libfabric APIs may result in a page fault or other fatal operation.
426
427       The fi_mr_refresh call is only needed if the physical pages might  have
428       been updated after the memory region was created.
429
430   fi_mr_enable
431       The  enable  call  is used with memory registration associated with the
432       FI_MR_RMA_EVENT mode bit.  Memory regions created in the disabled state
433       must be explicitly enabled after being fully configured by the applica‐
434       tion.  Any resource bindings to the MR must be done prior  to  enabling
435       the MR.
436

MEMORY REGION ATTRIBUTES

438       Memory  regions are created using the following attributes.  The struct
439       fi_mr_attr is passed into fi_mr_regattr, but individual fields also ap‐
440       ply to other memory registration calls, with the fields passed directly
441       into calls as function parameters.
442
443              struct fi_mr_attr {
444                  const struct iovec *mr_iov;
445                  size_t             iov_count;
446                  uint64_t           access;
447                  uint64_t           offset;
448                  uint64_t           requested_key;
449                  void               *context;
450                  size_t             auth_key_size;
451                  uint8_t            *auth_key;
452                  enum fi_hmem_iface iface;
453                  union {
454                      uint64_t         reserved;
455                      int              cuda;
456                  } device;
457              };
458
459   mr_iov
460       This is an IO vector of addresses that will represent a  single  memory
461       region.  The number of entries in the iovec is specified by iov_count.
462
463   iov_count
464       The number of entries in the mr_iov array.  The maximum number of memo‐
465       ry buffers that may be associated with a single memory region is speci‐
466       fied as the mr_iov_limit domain attribute.  See fi_domain(3).
467
468   access
469       Indicates  the type of operations that the local or a peer endpoint may
470       perform on registered memory region.  Supported access permissions  are
471       the bitwise OR of the following flags:
472
473       FI_SEND
474              The  memory  buffer  may be used in outgoing message data trans‐
475              fers.  This includes fi_msg and fi_tagged send operations.
476
477       FI_RECV
478              The memory buffer may be used to receive inbound message  trans‐
479              fers.  This includes fi_msg and fi_tagged receive operations.
480
481       FI_READ
482              The  memory buffer may be used as the result buffer for RMA read
483              and atomic operations on the initiator side.  Note that from the
484              viewpoint of the application, the memory buffer is being written
485              into by the network.
486
487       FI_WRITE
488              The memory buffer may be used as the source buffer for RMA write
489              and atomic operations on the initiator side.  Note that from the
490              viewpoint of the application, the endpoint is reading  from  the
491              memory buffer and copying the data onto the network.
492
493       FI_REMOTE_READ
494              The  memory  buffer  may  be used as the source buffer of an RMA
495              read operation on the target side.  The contents of  the  memory
496              buffer are not modified by such operations.
497
498       FI_REMOTE_WRITE
499              The  memory  buffer  may  be used as the target buffer of an RMA
500              write or atomic operation.  The contents of  the  memory  buffer
501              may be modified as a result of such operations.
502
503       Note  that  some  providers may not enforce fine grained access permis‐
504       sions.  For example, a memory region registered for FI_WRITE access may
505       also  behave as if FI_SEND were specified as well.  Relaxed enforcement
506       of such access is permitted, though not guaranteed,  provided  security
507       is maintained.
508
509   offset
510       The offset field is reserved for future use and must be 0.
511
512   requested_key
513       An  application specified access key associated with the memory region.
514       The MR key must be provided by a remote process when performing RMA  or
515       atomic  operations  to  a  memory region.  Applications can use the re‐
516       quested_key field to indicate that  a  specific  key  be  used  by  the
517       provider.  This allows applications to use well known key values, which
518       can avoid applications needing to exchange and store keys.  Support for
519       user  requested  keys  is  provider  specific  and is determined by the
520       mr_mode domain attribute.
521
522   context
523       Application context associated with  asynchronous  memory  registration
524       operations.   This  value is returned as part of any asynchronous event
525       associated with the registration.  This field is ignored  for  synchro‐
526       nous registration calls.
527
528   auth_key_size
529       The  size  of key referenced by the auth_key field in bytes, or 0 if no
530       authorization key is given.  This field is ignored unless the fabric is
531       opened with API version 1.5 or greater.
532
533   auth_key
534       Indicates  the  key to associate with this memory registration.  Autho‐
535       rization keys are used to limit communication between endpoints.   Only
536       peer  endpoints  that  are programmed to use the same authorization key
537       may access the memory region.  The domain  authorization  key  will  be
538       used  if the auth_key_size provided is 0.  This field is ignored unless
539       the fabric is opened with API version 1.5 or greater.
540
541   iface
542       Indicates the software interfaces used by the application  to  allocate
543       and  manage the memory region.  This field is ignored unless the appli‐
544       cation has requested the FI_HMEM capability.
545
546       FI_HMEM_SYSTEM
547              Uses standard operating system calls and libraries, such as mal‐
548              loc, calloc, realloc, mmap, and free.
549
550       FI_HMEM_CUDA
551              Uses  Nvidia CUDA interfaces such as cuMemAlloc, cuMemAllocHost,
552              cuMemAllocManaged, cuMemFree, cudaMalloc, cudaFree.
553
554   device
555       Reserved 64 bits for device identifier if using non-standard  HMEM  in‐
556       terface.  This field is ignore unless the iface field is valid.
557
558       cuda   For FI_HMEM_CUDA, this is equivalent to CUdevice (int).
559

NOTES

561       Direct access to an application's memory by a remote peer requires that
562       the application register the targeted memory buffer(s).  This is  typi‐
563       cally   done   by   calling   one  of  the  fi_mr_reg*  routines.   For
564       FI_MR_PROV_KEY, the provider will return a key that must be used by the
565       peer  when accessing the memory region.  The application is responsible
566       for transferring this key to the peer.  If FI_MR_RAW mode has been set,
567       the key must be retrieved using the fi_mr_raw_attr function.
568
569       FI_RAW_MR  allows  support for providers that require more than 8-bytes
570       for their protection keys or need additional setup before a key can  be
571       used for transfers.  After a raw key has been retrieved, it must be ex‐
572       changed with the remote peer.  The peer must use fi_mr_map_raw to  con‐
573       vert  the  raw  key into a usable 64-bit key.  The mapping must be done
574       even if the raw key is 64-bits or smaller.
575
576       The raw key support functions are usable with all registered memory re‐
577       gions, even if FI_MR_RAW has not been set.  It is recommended that por‐
578       table applications target using those interfaces;  however,  their  use
579       does  carry extra message and memory footprint overhead, making it less
580       desirable for highly scalable apps.
581

FLAGS

583       The follow flag may be specified to any memory registration call.
584
585       FI_RMA_EVENT
586              This flag indicates that the specified memory region will be as‐
587              sociated  with a completion counter used to count RMA operations
588              that access the MR.
589
590       FI_RMA_PMEM
591              This flag indicates that the underlying memory region is  backed
592              by  persistent  memory  and  will be used in RMA operations.  It
593              must be specified if persistent completion semantics or  persis‐
594              tent  data  transfers are required when accessing the registered
595              region.
596

RETURN VALUES

598       Returns 0 on success.  On error, a negative value corresponding to fab‐
599       ric errno is returned.
600
601       Fabric errno values are defined in rdma/fi_errno.h.
602

ERRORS

604       -FI_ENOKEY
605              The requested_key is already in use.
606
607       -FI_EKEYREJECTED
608              The  requested_key is not available.  They key may be out of the
609              range supported by the provider, or the provider may not support
610              user-requested memory registration keys.
611
612       -FI_ENOSYS
613              Returned  by fi_mr_bind if the provider does not support report‐
614              ing events based on access to registered memory regions.
615
616       -FI_EBADFLAGS
617              Returned if  the  specified  flags  are  not  supported  by  the
618              provider.
619

MEMORY REGISTRATION CACHE

621       Many  hardware  NICs accessed by libfabric require that data buffers be
622       registered with the hardware while the hardware accesses it.  This  en‐
623       sures  that  the virtual to physical address mappings for those buffers
624       do not change while the transfer is occurring.  The performance  impact
625       of  registering  memory  regions can be significant.  As a result, some
626       providers make use of a registration cache, particularly  when  working
627       with  applications that are unable to manage their own network buffers.
628       A registration cache avoids the overhead of registering and unregister‐
629       ing a data buffer with each transfer.
630
631       As  a  general  rule, if hardware requires the FI_MR_LOCAL mode bit de‐
632       scribed above, but this is not supported by the application,  a  memory
633       registration  cache may be in use.  The following environment variables
634       may be used to configure registration caches.
635
636       FI_MR_CACHE_MAX_SIZE
637              This defines the total number of bytes for  all  memory  regions
638              that  may be tracked by the cache.  If not set, the cache has no
639              limit on how many bytes may be registered and  cached.   Setting
640              this will reduce the amount of memory that is not actively being
641              used as part of a  data  transfer  that  is  registered  with  a
642              provider.  By default, the cache size is unlimited.
643
644       FI_MR_CACHE_MAX_COUNT
645              This defines the total number of memory regions that may be reg‐
646              istered with the cache.  If not set, a default limit is  chosen.
647              Setting  this  will reduce the number of regions that are regis‐
648              tered, regardless of their size, which are  not  actively  being
649              used as part of a data transfer.  Setting this to zero will dis‐
650              able registration caching.
651
652       FI_MR_CACHE_MONITOR
653              The cache monitor is responsible for detecting changes made  be‐
654              tween  the  virtual addresses used by an application and the un‐
655              derlying physical pages.  Valid monitor options are:  userfault‐
656              fd,  memhooks,  and  disabled.  Selecting disabled will turn off
657              the registration cache.  Userfaultfd is a Linux  kernel  feature
658              used  to  report  virtual to physical address mapping changes to
659              user space.  Memhooks operates by intercepting  relevant  memory
660              allocation  and  deallocation calls which may result in the map‐
661              pings changing, such as malloc,  mmap,  free,  etc.   Note  that
662              memhooks  operates  at  the  elf  linker layer, and does not use
663              glibc memory hooks.
664

SEE ALSO

666       fi_getinfo(3),  fi_endpoint(3),  fi_domain(3),  fi_rma(3),   fi_msg(3),
667       fi_atomic(3)
668

AUTHORS

670       OpenFabrics.
671
672
673
674Libfabric Programmer's Manual     2020-04-14                          fi_mr(3)
Impressum