1fi_mr(3)                     Libfabric v1.12.0rc1                     fi_mr(3)
2
3
4

NAME

6       fi_mr - Memory region operations
7
8       fi_mr_reg / fi_mr_regv / fi_mr_regattr
9              Register local memory buffers for direct fabric access
10
11       fi_close
12              Deregister registered memory buffers.
13
14       fi_mr_desc
15              Return  a  local  descriptor associated with a registered memory
16              region
17
18       fi_mr_key
19              Return the remote key needed to access a registered  memory  re‐
20              gion
21
22       fi_mr_raw_attr
23              Return raw memory region attributes.
24
25       fi_mr_map_raw
26              Converts  a  raw memory region key into a key that is usable for
27              data transfer operations.
28
29       fi_mr_unmap_key
30              Releases a previously mapped raw memory region key.
31
32       fi_mr_bind
33              Associate a registered memory region with a  completion  counter
34              or an endpoint.
35
36       fi_mr_refresh
37              Updates the memory pages associated with a memory region.
38
39       fi_mr_enable
40              Enables a memory region for use.
41

SYNOPSIS

43              #include <rdma/fi_domain.h>
44
45              int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
46                  uint64_t access, uint64_t offset, uint64_t requested_key,
47                  uint64_t flags, struct fid_mr **mr, void *context);
48
49              int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
50                  size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
51                  uint64_t flags, struct fid_mr **mr, void *context);
52
53              int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
54                  uint64_t flags, struct fid_mr **mr);
55
56              int fi_close(struct fid *mr);
57
58              void * fi_mr_desc(struct fid_mr *mr);
59
60              uint64_t fi_mr_key(struct fid_mr *mr);
61
62              int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
63                  uint8_t *raw_key, size_t *key_size, uint64_t flags);
64
65              int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
66                  uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
67
68              int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
69
70              int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
71
72              int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
73                  uint64_t flags)
74
75              int fi_mr_enable(struct fid_mr *mr);
76

ARGUMENTS

78       domain Resource domain
79
80       mr     Memory region
81
82       bfid   Fabric identifier of an associated resource.
83
84       context
85              User specified context associated with the memory region.
86
87       buf    Memory buffer to register with the fabric hardware.
88
89       len    Length of memory buffer to register.  Must be > 0.
90
91       iov    Vectored memory buffer.
92
93       count  Count of vectored buffer entries.
94
95       access Memory access permissions associated with registration
96
97       offset Optional  specified  offset  for  accessing specified registered
98              buffers.  This parameter is reserved for future use and must  be
99              0.
100
101       requested_key
102              Optional  requested  remote  key associated with registered buf‐
103              fers.
104
105       attr   Memory region attributes
106
107       flags  Additional flags to apply to the operation.
108

DESCRIPTION

110       Registered memory regions associate  memory  buffers  with  permissions
111       granted for access by fabric resources.  A memory buffer must be regis‐
112       tered with a resource domain before it can be used as the target  of  a
113       remote  RMA  or  atomic data transfer.  Additionally, a fabric provider
114       may require that data buffers be registered before being used in  local
115       transfers.   Memory  registration  restrictions  are controlled using a
116       separate set of mode bits,  specified  through  the  domain  attributes
117       (mr_mode  field).   Each  mr_mode bit requires that an application take
118       specific steps in order to use memory  buffers  with  libfabric  inter‐
119       faces.
120
121       The following apply to memory registration.
122
123       Default Memory Registration
124              If no mr_mode bits are set, the default behaviors describe below
125              are followed.  Historically, these  defaults  were  collectively
126              referred  to  as  scalable memory registration.  The default re‐
127              quirements are outlined below, followed by  definitions  of  how
128              each mr_mode bit alters the definition.
129
130       Compatibility: For library versions 1.4 and earlier, this was indicated
131       by setting mr_mode to FI_MR_SCALABLE and the fi_info  mode  bit  FI_LO‐
132       CAL_MR to 0.  FI_MR_SCALABLE and FI_LOCAL_MR were deprecated in libfab‐
133       ric version 1.5, though they are supported for backwards  compatibility
134       purposes.
135
136       For security, memory registration is required for data buffers that are
137       accessed directly by a peer process.  For example, registration is  re‐
138       quired  for RMA target buffers (read or written to), and those accessed
139       by atomic or collective operations.
140
141       By default, registration occurs on  virtual  address  ranges.   Because
142       registration  refers to address ranges, rather than allocated data buf‐
143       fers, the address ranges do not need to map to data  buffers  allocated
144       by the application at the time the registration call is made.  That is,
145       an application can register any range of addresses in their virtual ad‐
146       dress  space,  whether  or  not  those addresses are backed by physical
147       pages or have been allocated.
148
149       Note that physical pages must back addresses prior to the addresses be‐
150       ing  accessed  as part of a data transfer operation, or the data trans‐
151       fers will fail.  Additionally, depending on the operation,  this  could
152       result  in the local process receiving a segmentation fault for access‐
153       ing invalid memory.
154
155       Once registered, the resulting memory regions are accessible  by  peers
156       starting  at  a base address of 0.  That is, the target address that is
157       specified is a byte offset into the registered region.
158
159       The application also selects the access key  associated  with  the  MR.
160       The key size is restricted to a maximum of 8 bytes.
161
162       With  scalable registration, locally accessed data buffers are not reg‐
163       istered.  This includes source buffers for all transmit  operations  --
164       sends,  tagged sends, RMA, and atomics -- as well as buffers posted for
165       receive and tagged receive operations.
166
167       Although the default memory registration behavior is convenient for ap‐
168       plication  developers,  it  is difficult to implement in hardware.  At‐
169       tempts to hide the hardware requirements from the application often re‐
170       sults in significant and unacceptable impacts to performance.  The fol‐
171       lowing mr_mode bits are  provided  as  input  into  fi_getinfo.   If  a
172       provider  requires  the  behavior  defined  for an mr_mode bit, it will
173       leave the bit set on output to fi_getinfo.  Otherwise, the provider can
174       clear the bit to indicate that the behavior is not needed.
175
176       By setting an mr_mode bit, the application has agreed to adjust its be‐
177       havior as indicated.  Importantly, applications that choose to  support
178       an mr_mode must be prepared to handle the case where the mr_mode is not
179       required.  A provider will clear an mr_mode bit if it is not needed.
180
181       FI_MR_LOCAL
182              When the FI_MR_LOCAL mode bit is set, applications must register
183              all data buffers that will be accessed by the local hardware and
184              provide a valid desc parameter into applicable data transfer op‐
185              erations.   When  FI_MR_LOCAL  is zero, applications are not re‐
186              quired to register data buffers before using them for local  op‐
187              erations (e.g.  send and receive data buffers).  The desc param‐
188              eter into data transfer operations will be ignored in this case,
189              unless  otherwise  required (e.g.  se FI_MR_HMEM).  It is recom‐
190              mended that applications pass in NULL  for  desc  when  not  re‐
191              quired.
192
193       A  provider  may hide local registration requirements from applications
194       by making use of an internal registration cache or similar  mechanisms.
195       Such  mechanisms,  however,  may negatively impact performance for some
196       applications, notably those which manage their own network buffers.  In
197       order  to  support  as broad range of applications as possible, without
198       unduly affecting their performance, applications that  wish  to  manage
199       their own local memory registrations may do so by using the memory reg‐
200       istration calls.
201
202       Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
203       bit.  When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
204
205       FI_MR_RAW
206              Raw memory regions are used to support providers with keys larg‐
207              er than  64-bits  or  require  setup  at  the  peer.   When  the
208              FI_MR_RAW bit is set, applications must use fi_mr_raw_attr() lo‐
209              cally and fi_mr_map_raw() at the peer before targeting a  memory
210              region as part of any data transfer request.
211
212       FI_MR_VIRT_ADDR
213              The  FI_MR_VIRT_ADDR  bit indicates that the provider references
214              memory regions by virtual address, rather than a 0-based offset.
215              Peers that target memory regions registered with FI_MR_VIRT_ADDR
216              specify the destination memory buffer using the target's virtual
217              address,  with  any  offset into the region specified as virtual
218              address + offset.  Support of this bit  typically  implies  that
219              peers  must exchange addressing data prior to initiating any RMA
220              or atomic operation.
221
222       FI_MR_ALLOCATED
223              When set, all registered memory regions must be backed by physi‐
224              cal memory pages at the time the registration call is made.
225
226       FI_MR_PROV_KEY
227              This  memory  region  mode  indicates that the provider does not
228              support application requested MR keys.  MR keys are returned  by
229              the  provider.  Applications that support FI_MR_PROV_KEY can ob‐
230              tain the provider key using fi_mr_key(), unless FI_MR_RAW is al‐
231              so  set.   The  returned key should then be exchanged with peers
232              prior to initiating an RMA or atomic operation.
233
234       FI_MR_MMU_NOTIFY
235              FI_MR_MMU_NOTIFY is typically set by providers that support mem‐
236              ory registration against memory regions that are not necessarily
237              backed by allocated physical pages at the time the memory regis‐
238              tration  occurs.   (That  is,  FI_MR_ALLOCATED  is typically 0).
239              However, such providers require  that  applications  notify  the
240              provider prior to the MR being accessed as part of a data trans‐
241              fer operation.  This notification informs the provider that  all
242              necessary  physical pages now back the region.  The notification
243              is necessary for providers that cannot hook  directly  into  the
244              operating  system  page  tables  or memory management unit.  See
245              fi_mr_refresh() for notification details.
246
247       FI_MR_RMA_EVENT
248              This mode bit indicates that the provider must configure  memory
249              regions  that are associated with RMA events prior to their use.
250              This includes all memory regions that are associated  with  com‐
251              pletion  counters.   When  set,  applications must indicate if a
252              memory region will be associated with a  completion  counter  as
253              part  of  the region's creation.  This is done by passing in the
254              FI_RMA_EVENT flag to the memory registration call.
255
256       Such memory regions will be created in a disabled state and must be as‐
257       sociated  with  all completion counters prior to being enabled.  To en‐
258       able a memory region, the application must call fi_mr_enable().   After
259       calling fi_mr_enable(), no further resource bindings may be made to the
260       memory region.
261
262       FI_MR_ENDPOINT
263              This mode bit indicates that the provider associates memory  re‐
264              gions  with  endpoints rather than domains.  Memory regions that
265              are registered with the provider are created in a disabled state
266              and  must  be  bound  to an endpoint prior to being enabled.  To
267              bind  the  MR  with  an  endpoint,  the  application  must   use
268              fi_mr_bind().  To enable the memory region, the application must
269              call fi_mr_enable().
270
271       FI_MR_HMEM
272              This mode bit is associated with  the  FI_HMEM  capability.   If
273              FI_MR_HMEM  is  set,  the application must register buffers that
274              were allocated using a device call and provide a valid desc  pa‐
275              rameter  into  applicable  data transfer operations even if they
276              are only used for local operations (e.g.  send and receive  data
277              buffers).   Device memory must be registered using the fi_mr_re‐
278              gattr call, with the iface and device fields filled out.
279
280       If FI_MR_HMEM is set, but FI_MR_LOCAL is  unset,  only  device  buffers
281       must be registered when used locally.  In this case, the desc parameter
282       passed into data transfer operations must  either  be  valid  or  NULL.
283       Similarly,  if  FI_MR_LOCAL is set, but FI_MR_HMEM is not, the desc pa‐
284       rameter must either be valid or NULL.
285
286       Basic Memory Registration
287              Basic memory registration was deprecated  in  libfabric  version
288              1.5, but is supported for backwards compatibility.  Basic memory
289              registration is indicated by setting mr_mode equal to  FI_MR_BA‐
290              SIC.   FI_MR_BASIC must be set alone and not paired with mr_mode
291              bits.  Unlike other mr_mode bits, if FI_MR_BASIC is set on input
292              to  fi_getinfo(),  it will not be cleared by the provider.  That
293              is, setting mr_mode equal to FI_MR_BASIC forces basic  registra‐
294              tion if the provider supports it.
295
296       The  behavior of basic registration is equivalent to requiring the fol‐
297       lowing   mr_mode   bits:    FI_MR_VIRT_ADDR,    FI_MR_ALLOCATED,    and
298       FI_MR_PROV_KEY.   Additionally,  providers that support basic registra‐
299       tion usually require the (deprecated)  fi_info  mode  bit  FI_LOCAL_MR,
300       which was incorporated into the FI_MR_LOCAL mr_mode bit.
301
302       The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
303       -- are used to register one or more  memory  regions  with  fabric  re‐
304       sources.   The  main  difference between registration functions are the
305       number and type of parameters that they accept  as  input.   Otherwise,
306       they perform the same general function.
307
308       By  default,  memory  registration  completes synchronously.  I.e.  the
309       registration call will not return until the registration has completed.
310       Memory  registration  can complete asynchronous by binding the resource
311       domain to an event queue using the FI_REG_MR flag.  See fi_domain_bind.
312       When memory registration is asynchronous, in order to avoid a race con‐
313       dition between the registration call returning  and  the  corresponding
314       reading of the event from the EQ, the mr output parameter will be writ‐
315       ten before any event associated with the operation may be read  by  the
316       application.   An  asynchronous  event will not be generated unless the
317       registration call returns success (0).
318
319   fi_mr_reg
320       The fi_mr_reg call registers the user-specified memory buffer with  the
321       resource  domain.  The buffer is enabled for access by the fabric hard‐
322       ware based on the provided access permissions.  See  the  access  field
323       description for memory region attributes below.
324
325       Registered memory is associated with a local memory descriptor and, op‐
326       tionally, a remote memory key.  A memory descriptor is a provider  spe‐
327       cific identifier associated with registered memory.  Memory descriptors
328       often map to hardware specific indices or keys associated with the mem‐
329       ory  region.  Remote memory keys provide limited protection against un‐
330       wanted access by a remote node.  Remote accesses  to  a  memory  region
331       must provide the key associated with the registration.
332
333       Because  MR  keys  must be provided by a remote process, an application
334       can use the requested_key parameter to indicate  that  a  specific  key
335       value be returned.  Support for user requested keys is provider specif‐
336       ic and is determined by the mr_mode domain attribute.
337
338       Remote RMA and atomic operations indicate the location within a  regis‐
339       tered  memory  region by specifying an address.  The location is refer‐
340       enced by adding the offset to either the base virtual  address  of  the
341       buffer or to 0, depending on the mr_mode.
342
343       The offset parameter is reserved for future use and must be 0.
344
345       For  asynchronous  memory registration requests, the result will be re‐
346       ported to the user through an event queue associated with the  resource
347       domain.   If  successful, the allocated memory region structure will be
348       returned to the user through the mr parameter.  The mr address must re‐
349       main  valid  until  the  registration operation completes.  The context
350       specified with the registration request is returned with the completion
351       event.
352
353   fi_mr_regv
354       The   fi_mr_regv  call  adds  support  for  a  scatter-gather  list  to
355       fi_mr_reg.  Multiple memory buffers are registered as a  single  memory
356       region.  Otherwise, the operation is the same.
357
358   fi_mr_regattr
359       The  fi_mr_regattr call is a more generic, extensible registration call
360       that allows the user to specify the registration request using a struct
361       fi_mr_attr (defined below).
362
363   fi_close
364       Fi_close is used to release all resources associated with a registering
365       a memory region.  Once unregistered, further access to  the  registered
366       memory is not guaranteed.  Active or queued operations that reference a
367       memory region being closed may fail or result in  accesses  to  invalid
368       memory.   Applications  are  responsible  for  ensuring that a MR is no
369       longer needed prior to closing it.  Note that accesses to a  closed  MR
370       from  a  remote peer will result in an error at the peer.  The state of
371       the local endpoint will be unaffected.
372
373       When closing the MR, there must be no opened endpoints or counters  as‐
374       sociated  with  the  MR.  If resources are still associated with the MR
375       when attempting to close, the call will return -FI_EBUSY.
376
377   fi_mr_desc
378       Obtains the local memory descriptor associated with a MR.   The  memory
379       registration  must  have  completed  successfully  before invoking this
380       call.
381
382   fi_mr_key
383       Returns the remote protection key associated with  a  MR.   The  memory
384       registration  must  have  completed  successfully before invoking this.
385       The returned key may be used in data transfer operations at a peer.  If
386       the FI_RAW_MR mode bit has been set for the domain, then the memory key
387       must be obtained using the fi_mr_raw_key function  instead.   A  return
388       value  of  FI_KEY_NOTAVAIL will be returned if the registration has not
389       completed or a raw memory key is required.
390
391   fi_mr_raw_attr
392       Returns the raw, remote protection key and base address associated with
393       a  MR.  The memory registration must have completed successfully before
394       invoking this routine.  Use of this call is required if  the  FI_RAW_MR
395       mode  bit has been set by the provider; however, it is safe to use this
396       call with any memory region.
397
398       On input, the key_size  parameter  should  indicate  the  size  of  the
399       raw_key buffer.  If the actual key is larger than what can fit into the
400       buffer, it will return -FI_ETOOSMALL.  On output, key_size  is  set  to
401       the  size  of  the  buffer needed to store the key, which may be larger
402       than the input value.  The needed key_size can also be obtained through
403       the mr_key_size domain attribute (fi_domain_attr) field.
404
405       A raw key must be mapped by a peer before it can be used in data trans‐
406       fer operations.  See fi_mr_map_raw below.
407
408   fi_mr_map_raw
409       Raw protection keys must be mapped to a usable key  value  before  they
410       can  be  used for data transfer operations.  The mapping is done by the
411       peer that initiates the RMA or atomic operation.  The mapping  function
412       takes  as  input  the raw key and its size, and returns the mapped key.
413       Use of the fi_mr_map_raw function is  required  if  the  peer  has  the
414       FI_RAW_MR  mode  bit  set,  but this routine may be called on any valid
415       key.  All mapped keys must be freed by calling fi_mr_unmap_key when ac‐
416       cess to the peer memory region is no longer necessary.
417
418   fi_mr_unmap_key
419       This  call  releases any resources that may have been allocated as part
420       of mapping a raw memory key.  All mapped keys must be freed before  the
421       corresponding domain is closed.
422
423   fi_mr_bind
424       The  fi_mr_bind  function  associates a memory region with a counter or
425       endpoint.  Counter bindings are needed by providers  that  support  the
426       generation  of  completions based on fabric operations.  Endpoint bind‐
427       ings are needed if the provider associates  memory  regions  with  end‐
428       points (see FI_MR_ENDPOINT).
429
430       When  binding  with  a  counter, the type of events tracked against the
431       memory region is based on the bitwise OR of the following flags.
432
433       FI_REMOTE_WRITE
434              Generates an event whenever a remote RMA write or atomic  opera‐
435              tion modifies the memory region.  Use of this flag requires that
436              the endpoint through which the MR is accessed  be  created  with
437              the FI_RMA_EVENT capability.
438
439       When binding the memory region to an endpoint, flags should be 0.
440
441   fi_mr_refresh
442       The  use  of this call is required to notify the provider of any change
443       to the physical  pages  backing  a  registered  memory  region  if  the
444       FI_MR_MMU_NOTIFY mode bit has been set.  This call informs the provider
445       that the page table entries associated with the region  may  have  been
446       modified,  and the provider should verify and update the registered re‐
447       gion accordingly.  The iov parameter is optional and  may  be  used  to
448       specify  which  portions  of  the  registered region requires updating.
449       Providers are only guaranteed to update the specified address ranges.
450
451       The refresh operation has the effect of disabling and  re-enabling  ac‐
452       cess  to the registered region.  Any operations from peers that attempt
453       to access the region will fail while the refresh is  occurring.   Addi‐
454       tionally,  attempts  to  access the region by the local process through
455       libfabric APIs may result in a page fault or other fatal operation.
456
457       The fi_mr_refresh call is only needed if the physical pages might  have
458       been updated after the memory region was created.
459
460   fi_mr_enable
461       The  enable  call  is used with memory registration associated with the
462       FI_MR_RMA_EVENT mode bit.  Memory regions created in the disabled state
463       must be explicitly enabled after being fully configured by the applica‐
464       tion.  Any resource bindings to the MR must be done prior  to  enabling
465       the MR.
466

MEMORY REGION ATTRIBUTES

468       Memory  regions are created using the following attributes.  The struct
469       fi_mr_attr is passed into fi_mr_regattr, but individual fields also ap‐
470       ply to other memory registration calls, with the fields passed directly
471       into calls as function parameters.
472
473              struct fi_mr_attr {
474                  const struct iovec *mr_iov;
475                  size_t             iov_count;
476                  uint64_t           access;
477                  uint64_t           offset;
478                  uint64_t           requested_key;
479                  void               *context;
480                  size_t             auth_key_size;
481                  uint8_t            *auth_key;
482                  enum fi_hmem_iface iface;
483                  union {
484                      uint64_t         reserved;
485                      int              cuda;
486                      int      ze
487                  } device;
488              };
489
490   mr_iov
491       This is an IO vector of addresses that will represent a  single  memory
492       region.  The number of entries in the iovec is specified by iov_count.
493
494   iov_count
495       The number of entries in the mr_iov array.  The maximum number of memo‐
496       ry buffers that may be associated with a single memory region is speci‐
497       fied as the mr_iov_limit domain attribute.  See fi_domain(3).
498
499   access
500       Indicates  the type of operations that the local or a peer endpoint may
501       perform on registered memory region.  Supported access permissions  are
502       the bitwise OR of the following flags:
503
504       FI_SEND
505              The  memory  buffer  may be used in outgoing message data trans‐
506              fers.  This includes fi_msg and fi_tagged send operations.
507
508       FI_RECV
509              The memory buffer may be used to receive inbound message  trans‐
510              fers.  This includes fi_msg and fi_tagged receive operations.
511
512       FI_READ
513              The  memory buffer may be used as the result buffer for RMA read
514              and atomic operations on the initiator side.  Note that from the
515              viewpoint of the application, the memory buffer is being written
516              into by the network.
517
518       FI_WRITE
519              The memory buffer may be used as the source buffer for RMA write
520              and atomic operations on the initiator side.  Note that from the
521              viewpoint of the application, the endpoint is reading  from  the
522              memory buffer and copying the data onto the network.
523
524       FI_REMOTE_READ
525              The  memory  buffer  may  be used as the source buffer of an RMA
526              read operation on the target side.  The contents of  the  memory
527              buffer are not modified by such operations.
528
529       FI_REMOTE_WRITE
530              The  memory  buffer  may  be used as the target buffer of an RMA
531              write or atomic operation.  The contents of  the  memory  buffer
532              may be modified as a result of such operations.
533
534       Note  that  some  providers may not enforce fine grained access permis‐
535       sions.  For example, a memory region registered for FI_WRITE access may
536       also  behave as if FI_SEND were specified as well.  Relaxed enforcement
537       of such access is permitted, though not guaranteed,  provided  security
538       is maintained.
539
540   offset
541       The offset field is reserved for future use and must be 0.
542
543   requested_key
544       An  application specified access key associated with the memory region.
545       The MR key must be provided by a remote process when performing RMA  or
546       atomic  operations  to  a  memory region.  Applications can use the re‐
547       quested_key field to indicate that  a  specific  key  be  used  by  the
548       provider.  This allows applications to use well known key values, which
549       can avoid applications needing to exchange and store keys.  Support for
550       user  requested  keys  is  provider  specific  and is determined by the
551       mr_mode domain attribute.
552
553   context
554       Application context associated with  asynchronous  memory  registration
555       operations.   This  value is returned as part of any asynchronous event
556       associated with the registration.  This field is ignored  for  synchro‐
557       nous registration calls.
558
559   auth_key_size
560       The  size  of key referenced by the auth_key field in bytes, or 0 if no
561       authorization key is given.  This field is ignored unless the fabric is
562       opened with API version 1.5 or greater.
563
564   auth_key
565       Indicates  the  key to associate with this memory registration.  Autho‐
566       rization keys are used to limit communication between endpoints.   Only
567       peer  endpoints  that  are programmed to use the same authorization key
568       may access the memory region.  The domain  authorization  key  will  be
569       used  if the auth_key_size provided is 0.  This field is ignored unless
570       the fabric is opened with API version 1.5 or greater.
571
572   iface
573       Indicates the software interfaces used by the application  to  allocate
574       and  manage the memory region.  This field is ignored unless the appli‐
575       cation has requested the FI_HMEM capability.
576
577       FI_HMEM_SYSTEM
578              Uses standard operating system calls and libraries, such as mal‐
579              loc, calloc, realloc, mmap, and free.
580
581       FI_HMEM_CUDA
582              Uses  Nvidia CUDA interfaces such as cuMemAlloc, cuMemAllocHost,
583              cuMemAllocManaged, cuMemFree, cudaMalloc, cudaFree.
584
585       FI_HMEM_ROCR
586              Uses  AMD  ROCR  interfaces  such  as  hsa_memory_allocate   and
587              hsa_memory_free.
588
589       FI_HMEM_ZE
590              Uses  Intel  L0  ZE  interfaces  such as zeDriverAllocSharedMem,
591              zeDriverFreeMem.
592
593   device
594       Reserved 64 bits for device identifier if using non-standard  HMEM  in‐
595       terface.  This field is ignore unless the iface field is valid.
596
597       cuda   For FI_HMEM_CUDA, this is equivalent to CUdevice (int).
598
599       ze     For FI_HMEM_ZE, this is equivalent to the ze_device_handle_t in‐
600              dex (int).
601

NOTES

603       Direct access to an application's memory by a remote peer requires that
604       the  application register the targeted memory buffer(s).  This is typi‐
605       cally  done  by  calling  one  of   the   fi_mr_reg*   routines.    For
606       FI_MR_PROV_KEY, the provider will return a key that must be used by the
607       peer when accessing the memory region.  The application is  responsible
608       for transferring this key to the peer.  If FI_MR_RAW mode has been set,
609       the key must be retrieved using the fi_mr_raw_attr function.
610
611       FI_RAW_MR allows support for providers that require more  than  8-bytes
612       for  their protection keys or need additional setup before a key can be
613       used for transfers.  After a raw key has been retrieved, it must be ex‐
614       changed  with the remote peer.  The peer must use fi_mr_map_raw to con‐
615       vert the raw key into a usable 64-bit key.  The mapping  must  be  done
616       even if the raw key is 64-bits or smaller.
617
618       The raw key support functions are usable with all registered memory re‐
619       gions, even if FI_MR_RAW has not been set.  It is recommended that por‐
620       table  applications  target  using those interfaces; however, their use
621       does carry extra message and memory footprint overhead, making it  less
622       desirable for highly scalable apps.
623

FLAGS

625       The follow flag may be specified to any memory registration call.
626
627       FI_RMA_EVENT
628              This flag indicates that the specified memory region will be as‐
629              sociated with a completion counter used to count RMA  operations
630              that access the MR.
631
632       FI_RMA_PMEM
633              This  flag indicates that the underlying memory region is backed
634              by persistent memory and will be used  in  RMA  operations.   It
635              must  be specified if persistent completion semantics or persis‐
636              tent data transfers are required when accessing  the  registered
637              region.
638

RETURN VALUES

640       Returns 0 on success.  On error, a negative value corresponding to fab‐
641       ric errno is returned.
642
643       Fabric errno values are defined in rdma/fi_errno.h.
644

ERRORS

646       -FI_ENOKEY
647              The requested_key is already in use.
648
649       -FI_EKEYREJECTED
650              The requested_key is not available.  They key may be out of  the
651              range supported by the provider, or the provider may not support
652              user-requested memory registration keys.
653
654       -FI_ENOSYS
655              Returned by fi_mr_bind if the provider does not support  report‐
656              ing events based on access to registered memory regions.
657
658       -FI_EBADFLAGS
659              Returned  if  the  specified  flags  are  not  supported  by the
660              provider.
661

MEMORY REGISTRATION CACHE

663       Many hardware NICs accessed by libfabric require that data  buffers  be
664       registered  with the hardware while the hardware accesses it.  This en‐
665       sures that the virtual to physical address mappings for  those  buffers
666       do  not change while the transfer is occurring.  The performance impact
667       of registering memory regions can be significant.  As  a  result,  some
668       providers  make  use of a registration cache, particularly when working
669       with applications that are unable to manage their own network  buffers.
670       A registration cache avoids the overhead of registering and unregister‐
671       ing a data buffer with each transfer.
672
673       If a registration cache is going to be used for host and device memory,
674       the device must support unified virtual addressing.  If the device does
675       not support unified virtual addressing, either an additional  registra‐
676       tion  cache  is  required to track this device memory, or device memory
677       cannot be cached.
678
679       As a general rule, if hardware requires the FI_MR_LOCAL  mode  bit  de‐
680       scribed  above,  but this is not supported by the application, a memory
681       registration cache may be in use.  The following environment  variables
682       may be used to configure registration caches.
683
684       FI_MR_CACHE_MAX_SIZE
685              This  defines  the  total number of bytes for all memory regions
686              that may be tracked by the cache.  If not set, the cache has  no
687              limit  on  how many bytes may be registered and cached.  Setting
688              this will reduce the amount of memory that is not actively being
689              used  as  part  of  a  data  transfer  that is registered with a
690              provider.  By default, the cache size is unlimited.
691
692       FI_MR_CACHE_MAX_COUNT
693              This defines the total number of memory regions that may be reg‐
694              istered  with the cache.  If not set, a default limit is chosen.
695              Setting this will reduce the number of regions that  are  regis‐
696              tered,  regardless  of  their size, which are not actively being
697              used as part of a data transfer.  Setting this to zero will dis‐
698              able registration caching.
699
700       FI_MR_CACHE_MONITOR
701              The  cache  monitor  is  responsible for detecting system memory
702              (FI_HMEM_SYSTEM) changes made between the virtual addresses used
703              by an application and the underlying physical pages.  Valid mon‐
704              itor options are: userfaultfd, memhooks, and disabled.   Select‐
705              ing  disabled will turn off the registration cache.  Userfaultfd
706              is a Linux kernel feature used to report virtual to physical ad‐
707              dress  mapping  changes to user space.  Memhooks operates by in‐
708              tercepting relevant memory  allocation  and  deallocation  calls
709              which may result in the mappings changing, such as malloc, mmap,
710              free, etc.  Note that memhooks operates at the elf linker layer,
711              and does not use glibc memory hooks.
712
713       FI_MR_CUDA_CACHE_MONITOR_ENABLED
714              The  CUDA cache monitor is responsible for detecting CUDA device
715              memory (FI_HMEM_CUDA) changes made between  the  device  virtual
716              addresses used by an application and the underlying device phys‐
717              ical pages.  Valid monitor options are: 0 or 1.  Note  that  the
718              CUDA memory monitor requires a CUDA toolkit version with unified
719              virtual addressing enabled.
720
721       FI_MR_ROCR_CACHE_MONITOR_ENABLED
722              The ROCR cache monitor is responsible for detecting ROCR  device
723              memory  (FI_HMEM_ROCR)  changes  made between the device virtual
724              addresses used by an application and the underlying device phys‐
725              ical  pages.   Valid monitor options are: 0 or 1.  Note that the
726              ROCR memory monitor requires a ROCR version with unified virtual
727              addressing enabled.
728

SEE ALSO

730       fi_getinfo(3),   fi_endpoint(3),  fi_domain(3),  fi_rma(3),  fi_msg(3),
731       fi_atomic(3)
732

AUTHORS

734       OpenFabrics.
735
736
737
738Libfabric Programmer's Manual     2020-10-01                          fi_mr(3)
Impressum