1fi_mr(3)                       Libfabric v1.6.1                       fi_mr(3)
2
3
4

NAME

6       fi_mr - Memory region operations
7
8       fi_mr_reg  / fi_mr_regv / fi_mr_regattr : Register local memory buffers
9       for direct fabric access
10
11       fi_close : Deregister registered memory buffers.
12
13       fi_mr_desc : Return a local descriptor  associated  with  a  registered
14       memory region
15
16       fi_mr_key  : Return the remote key needed to access a registered memory
17       region
18
19       fi_mr_raw_attr : Return raw memory region attributes.
20
21       fi_mr_map_raw : Converts a raw memory region key into  a  key  that  is
22       usable for data transfer operations.
23
24       fi_mr_unmap_key : Releases a previously mapped raw memory region key.
25
26       fi_mr_bind  :  Associate  a  registered memory region with a completion
27       counter.
28
29       fi_mr_refresh : Updates the  memory  pages  associated  with  a  memory
30       region.
31
32       fi_mr_enable : Enables a memory region for use.
33

SYNOPSIS

35              #include <rdma/fi_domain.h>
36
37              int fi_mr_reg(struct fid_domain *domain, const void *buf, size_t len,
38                  uint64_t access, uint64_t offset, uint64_t requested_key,
39                  uint64_t flags, struct fid_mr **mr, void *context);
40
41              int fi_mr_regv(struct fid_domain *domain, const struct iovec * iov,
42                  size_t count, uint64_t access, uint64_t offset, uint64_t requested_key,
43                  uint64_t flags, struct fid_mr **mr, void *context);
44
45              int fi_mr_regattr(struct fid_domain *domain, const struct fi_mr_attr *attr,
46                  uint64_t flags, struct fid_mr **mr);
47
48              int fi_close(struct fid *mr);
49
50              void * fi_mr_desc(struct fid_mr *mr);
51
52              uint64_t fi_mr_key(struct fid_mr *mr);
53
54              int fi_mr_raw_attr(struct fid_mr *mr, uint64_t *base_addr,
55                  uint8_t *raw_key, size_t *key_size, uint64_t flags);
56
57              int fi_mr_map_raw(struct fid_domain *domain, uint64_t base_addr,
58                  uint8_t *raw_key, size_t key_size, uint64_t *key, uint64_t flags);
59
60              int fi_mr_unmap_key(struct fid_domain *domain, uint64_t key);
61
62              int fi_mr_bind(struct fid_mr *mr, struct fid *bfid, uint64_t flags);
63
64              int fi_mr_refresh(struct fid_mr *mr, const struct iovec *iov, size, count,
65                  uint64_t flags)
66
67              int fi_mr_enable(struct fid_mr *mr);
68

ARGUMENTS

70       domain : Resource domain
71
72       mr : Memory region
73
74       bfid : Fabric identifier of an associated resource.
75
76       context : User specified context associated with the memory region.
77
78       buf : Memory buffer to register with the fabric hardware
79
80       len : Length of memory buffer to register
81
82       iov : Vectored memory buffer.
83
84       count : Count of vectored buffer entries.
85
86       access : Memory access permissions associated with registration
87
88       offset  :  Optional specified offset for accessing specified registered
89       buffers.  This parameter is reserved for future use and must be 0.
90
91       requested_key : Optional requested remote key  associated  with  regis‐
92       tered buffers.
93
94       attr : Memory region attributes
95
96       flags : Additional flags to apply to the operation.
97

DESCRIPTION

99       Registered  memory  regions  associate  memory buffers with permissions
100       granted for access by fabric resources.  A memory buffer must be regis‐
101       tered  with  a resource domain before it can be used as the target of a
102       remote RMA or atomic data transfer.  Additionally,  a  fabric  provider
103       may  require that data buffers be registered before being used in local
104       transfers.  Memory registration restrictions  are  controlled  using  a
105       separate  set  of  mode  bits,  specified through the domain attributes
106       (mr_mode field).
107
108       The following apply to memory registration.
109
110       Scalable Memory Registration : By default, memory registration is  con‐
111       sidered scalable.  (For library versions 1.4 and earlier, this is indi‐
112       cated by setting mr_mode to FI_MR_SCALABLE, with the fi_info  mode  bit
113       FI_LOCAL_MR set to 0).  For versions 1.5 and later, scalable is implied
114       by the lack of any mr_mode bits being set.  The setting of mr_mode bits
115       therefore  adjusts  application  behavior as described below.  Default,
116       scalable registration has several properties.
117
118       In  scalable  mode,  registration  occurs  on  memory  address  ranges.
119       Because registration refers to memory regions, versus data buffers, the
120       address ranges given for a registration request do not need to  map  to
121       data  buffers allocated by the application at the time the registration
122       call is made.  That is,  an  application  can  register  any  range  of
123       addresses  in  their  virtual  address  space,  whether  or  not  those
124       addresses are backed by physical pages or have been allocated.
125
126       The resulting memory regions are accessible by peers starting at a base
127       address  of 0.  That is, the target address that is specified is a byte
128       offset into the registered region.
129
130       The application also selects the access key  associated  with  the  MR.
131       The key size is restricted to a maximum of 8 bytes.
132
133       With  scalable registration, locally accessed data buffers are not reg‐
134       istered.  This includes source buffers for all transmit  operations  --
135       sends,  tagged sends, RMA, and atomics -- as well as buffers posted for
136       receive and tagged receive operations.
137
138       FI_MR_LOCAL : When the FI_MR_LOCAL mode bit is set,  applications  must
139       register  all  data buffers that will be accessed by the local hardware
140       and provide a valid mem_desc parameter into  applicable  data  transfer
141       operations.  When FI_MR_LOCAL is zero, applications are not required to
142       register data buffers before using  them  for  local  operations  (e.g.
143       send  and  receive  data buffers), and the mem_desc parameter into data
144       transfer operations is ignored.
145
146       A provider may hide local registration requirements  from  applications
147       by  making use of an internal registration cache or similar mechanisms.
148       Such mechanisms, however, may negatively impact  performance  for  some
149       applications, notably those which manage their own network buffers.  In
150       order to support as broad range of applications  as  possible,  without
151       unduly  affecting  their  performance, applications that wish to manage
152       their own local memory registrations may do so by using the memory reg‐
153       istration calls.
154
155       Note: the FI_MR_LOCAL mr_mode bit replaces the FI_LOCAL_MR fi_info mode
156       bit.  When FI_MR_LOCAL is set, FI_LOCAL_MR is ignored.
157
158       FI_MR_RAW : Raw memory regions are used to support providers with  keys
159       larger  than  64-bits or require setup at the peer.  When the FI_MR_RAW
160       bit  is  set,  applications  must  use  fi_mr_raw_attr()  locally   and
161       fi_mr_map_raw() at the peer before targeting a memory region as part of
162       any data transfer request.
163
164       FI_MR_VIRT_ADDR : The FI_MR_VIRT_ADDR bit indicates that  the  provider
165       references  memory  regions  by  virtual address, rather than a 0-based
166       offset.    Peers   that   target   memory   regions   registered   with
167       FI_MR_VIRT_ADDR  specify  the  destination memory buffer using the tar‐
168       get's virtual address, with any offset into  the  region  specified  as
169       virtual  address  + offset.  Support of this bit typically implies that
170       peers must exchange addressing data prior  to  initiating  any  RMA  or
171       atomic operation.
172
173       FI_MR_ALLOCATED  :  When  set,  all  registered  memory regions must be
174       backed by physical memory pages at the time the  registration  call  is
175       made.
176
177       FI_MR_PROV_KEY  :  This  memory region mode indicates that the provider
178       does not support application requested MR keys.  MR keys  are  returned
179       by  the  provider.  Applications that support FI_MR_PROV_KEY can obtain
180       the provider key using fi_mr_key(), unless FI_MR_RAW is also set.   The
181       returned key should then be exchanged with peers prior to initiating an
182       RMA or atomic operation.
183
184       FI_MR_MMU_NOTIFY : FI_MR_MMU_NOTIFY is typically set by providers  that
185       support  memory registration against memory regions that are not neces‐
186       sarily backed by allocated physical pages at the time the memory regis‐
187       tration  occurs.   (That is, FI_MR_ALLOCATED is typically 0).  However,
188       such providers require that applications notify the provider  prior  to
189       the MR being accessed as part of a data transfer operation.  This noti‐
190       fication informs the provider that all  necessary  physical  pages  now
191       back the region.  The notification is necessary for providers that can‐
192       not hook directly into the operating system page tables or memory  man‐
193       agement unit.  See fi_mr_refresh() for notification details.
194
195       FI_MR_RMA_EVENT  :  This mode bit indicates that the provider must con‐
196       figure memory regions that are associated  with  RMA  events  prior  to
197       their  use.   This includes all memory regions that are associated with
198       completion counters.  When set, applications must indicate if a  memory
199       region  will  be  associated  with  a completion counter as part of the
200       region's creation.  This is done by passing in the FI_RMA_EVENT flag to
201       the memory registration call.
202
203       Such  memory  regions  will  be created in a disabled state and must be
204       associated with all completion counters prior  to  being  enabled.   To
205       enable  a  memory  region,  the  application  must call fi_mr_enable().
206       After calling fi_mr_enable(), no further resource bindings may be  made
207       to the memory region.
208
209       FI_MR_ENDPOINT  :  This mode bit indicates that the provider associates
210       memory regions with endpoints rather than domains.  Memory regions that
211       are  registered  with  the provider are created in a disabled state and
212       must be bound to an endpoint prior to being enabled.  To  bind  the  MR
213       with an endpoint, the application must use fi_mr_bind().  To enable the
214       memory region, the application must call fi_mr_enable().
215
216       Basic Memory Registration : Basic memory registration is  indicated  by
217       the  FI_MR_BASIC  mr_mode bit.  FI_MR_BASIC is maintained for backwards
218       compatibility (libfabric version 1.4  or  earlier).   The  behavior  of
219       basic  registration is equivalent to setting the following mr_mode bits
220       to one: FI_MR_VIRT_ADDR, FI_MR_ALLOCATED,  and  FI_MR_PROV_KEY.   Addi‐
221       tionally,  providers  that  support basic registration usually required
222       FI_MR_LOCAL.  FI_MR_BASIC must either be set alone, or  in  conjunction
223       with  FI_MR_LOCAL.   Other  mr_mode  bit  pairings are invalid.  Unlike
224       other mr_mode bits, if FI_MR_BASIC is set on input to fi_getinfo(),  it
225       will  not  be cleared by the provider.  That is, setting FI_MR_BASIC to
226       one requests basic registration.
227
228       The registrations functions -- fi_mr_reg, fi_mr_regv, and fi_mr_regattr
229       --  are  used  to  register  one  or  more  memory  regions with fabric
230       resources.  The main difference between registration functions are  the
231       number  and  type  of parameters that they accept as input.  Otherwise,
232       they perform the same general function.
233
234       By default, memory registration  completes  synchronously.   I.e.   the
235       registration call will not return until the registration has completed.
236       Memory registration can complete asynchronous by binding  the  resource
237       domain to an event queue using the FI_REG_MR flag.  See fi_domain_bind.
238       When memory registration is asynchronous, in order to avoid a race con‐
239       dition  between  the  registration call returning and the corresponding
240       reading of the event from the EQ, the mr output parameter will be writ‐
241       ten  before  any event associated with the operation may be read by the
242       application.  An asynchronous event will not be  generated  unless  the
243       registration call returns success (0).
244
245   fi_mr_reg
246       The  fi_mr_reg call registers the user-specified memory buffer with the
247       resource domain.  The buffer is enabled for access by the fabric  hard‐
248       ware  based  on  the provided access permissions.  See the access field
249       description for memory region attributes below.
250
251       Registered memory is associated with a  local  memory  descriptor  and,
252       optionally,  a  remote  memory  key.  A memory descriptor is a provider
253       specific identifier associated with registered memory.  Memory descrip‐
254       tors often map to hardware specific indices or keys associated with the
255       memory region.  Remote memory keys provide limited  protection  against
256       unwanted  access  by a remote node.  Remote accesses to a memory region
257       must provide the key associated with the registration.
258
259       Because MR keys must be provided by a remote  process,  an  application
260       can  use  the  requested_key  parameter to indicate that a specific key
261       value be returned.  Support for user requested keys  is  provider  spe‐
262       cific and is determined by the mr_mode domain attribute.
263
264       Remote  RMA and atomic operations indicate the location within a regis‐
265       tered memory region by specifying an address.  The location  is  refer‐
266       enced  by  adding  the offset to either the base virtual address of the
267       buffer or to 0, depending on the mr_mode.
268
269       The offset parameter is reserved for future use and must be 0.
270
271       For asynchronous memory  registration  requests,  the  result  will  be
272       reported  to  the  user  through  an  event  queue  associated with the
273       resource domain.  If successful, the allocated memory region  structure
274       will  be returned to the user through the mr parameter.  The mr address
275       must remain valid until the registration operation completes.  The con‐
276       text  specified with the registration request is returned with the com‐
277       pletion event.
278
279   fi_mr_regv
280       The  fi_mr_regv  call  adds  support  for  a  scatter-gather  list   to
281       fi_mr_reg.   Multiple  memory buffers are registered as a single memory
282       region.  Otherwise, the operation is the same.
283
284   fi_mr_regattr
285       The fi_mr_regattr call is a more generic, extensible registration  call
286       that allows the user to specify the registration request using a struct
287       fi_mr_attr (defined below).
288
289   fi_close
290       Fi_close is used to release all resources associated with a registering
291       a  memory  region.  Once unregistered, further access to the registered
292       memory is not guaranteed.  Active or queued operations that reference a
293       memory  region  being  closed may fail or result in accesses to invalid
294       memory.  Applications are responsible for ensuring  that  a  MR  is  no
295       longer  needed  prior to closing it.  Note that accesses to a closed MR
296       from a remote peer will result in an error at the peer.  The  state  of
297       the local endpoint will be unaffected.
298
299       When  closing  the  MR,  there  must be no opened endpoints or counters
300       associated with the MR.  If resources are still associated with the  MR
301       when attempting to close, the call will return -FI_EBUSY.
302
303   fi_mr_desc
304       Obtains  the  local memory descriptor associated with a MR.  The memory
305       registration must have  completed  successfully  before  invoking  this
306       call.
307
308   fi_mr_key
309       Returns  the  remote  protection  key associated with a MR.  The memory
310       registration must have completed  successfully  before  invoking  this.
311       The returned key may be used in data transfer operations at a peer.  If
312       the FI_RAW_MR mode bit has been set for the domain, then the memory key
313       must  be  obtained  using the fi_mr_raw_key function instead.  A return
314       value of FI_KEY_NOTAVAIL will be returned if the registration  has  not
315       completed or a raw memory key is required.
316
317   fi_mr_raw_attr
318       Returns the raw, remote protection key and base address associated with
319       a MR.  The memory registration must have completed successfully  before
320       invoking  this  routine.  Use of this call is required if the FI_RAW_MR
321       mode bit has been set by the provider; however, it is safe to use  this
322       call with any memory region.
323
324       A raw key must be mapped by a peer before it can be used in data trans‐
325       fer operations.  See fi_mr_map_raw below.
326
327   fi_mr_map_raw
328       Raw protection keys must be mapped to a usable key  value  before  they
329       can  be  used for data transfer operations.  The mapping is done by the
330       peer that initiates the RMA or atomic operation.  The mapping  function
331       takes  as  input  the raw key and its size, and returns the mapped key.
332       Use of the fi_mr_map_raw function is  required  if  the  peer  has  the
333       FI_RAW_MR  mode  bit  set,  but this routine may be called on any valid
334       key.  All mapped keys must be freed  by  calling  fi_mr_unmap_key  when
335       access to the peer memory region is no longer necessary.
336
337   fi_mr_unmap_key
338       This  call  releases any resources that may have been allocated as part
339       of mapping a raw memory key.  All mapped keys must be freed before  the
340       corresponding domain is closed.
341
342   fi_mr_bind
343       The  fi_mr_bind  function  associates a memory region with a counter or
344       endpoint.  Counter bindings are needed by providers  that  support  the
345       generation  of  completions based on fabric operations.  Endpoint bind‐
346       ings are needed if the provider associates  memory  regions  with  end‐
347       points (see FI_MR_ENDPOINT).
348
349       When  binding  with  a  counter, the type of events tracked against the
350       memory region is based on the bitwise OR of the following flags.
351
352       FI_REMOTE_WRITE : Generates an event whenever a  remote  RMA  write  or
353       atomic operation modifies the memory region.  Use of this flag requires
354       that the endpoint through which the MR is accessed be created with  the
355       FI_RMA_EVENT capability.
356
357       When binding the memory region to an endpoint, flags should be 0.
358
359   fi_mr_refresh
360       The  use  of this call is required to notify the provider of any change
361       to the physical  pages  backing  a  registered  memory  region  if  the
362       FI_MR_MMU_NOTIFY mode bit has been set.  This call informs the provider
363       that the page table entries associated with the region  may  have  been
364       modified,  and  the  provider  should  verify and update the registered
365       region accordingly.  The iov parameter is optional and may be  used  to
366       specify  which  portions  of  the  registered region requires updating.
367       Providers are only guaranteed to update the specified address ranges.
368
369       The refresh operation has  the  effect  of  disabling  and  re-enabling
370       access  to  the  registered  region.   Any  operations  from peers that
371       attempt to access the region will fail while the refresh is  occurring.
372       Additionally,  attempts  to  access  the  region  by  the local process
373       through libfabric APIs may result in a page fault or other fatal opera‐
374       tion.
375
376       The  fi_mr_refresh call is only needed if the physical pages might have
377       been updated after the memory region was created.
378
379   fi_mr_enable
380       The enable call is used with memory registration  associated  with  the
381       FI_MR_RMA_EVENT mode bit.  Memory regions created in the disabled state
382       must be explicitly enabled after being fully configured by the applica‐
383       tion.   Any  resource bindings to the MR must be done prior to enabling
384       the MR.
385

MEMORY REGION ATTRIBUTES

387       Memory regions are created using the following attributes.  The  struct
388       fi_mr_attr  is  passed  into  fi_mr_regattr, but individual fields also
389       apply to other  memory  registration  calls,  with  the  fields  passed
390       directly into calls as function parameters.
391
392              struct fi_mr_attr {
393                  const struct iovec *mr_iov;
394                  size_t             iov_count;
395                  uint64_t           access;
396                  uint64_t           requested_key;
397                  void               *context;
398                  size_t             auth_key_size;
399                  uint8_t            *auth_key;
400              };
401
402   mr_iov
403       This  is  an IO vector of addresses that will represent a single memory
404       region.  The number of entries in the iovec is specified by iov_count.
405
406   iov_count
407       The number of entries in the mr_iov array.  The maximum number of  mem‐
408       ory buffers that may be associated with a single memory region is spec‐
409       ified as the mr_iov_limit domain attribute.  See fi_domain(3).
410
411   access
412       Indicates the type of access that the local or a peer endpoint  has  to
413       the  registered  memory  region.   Supported access permissions are the
414       bitwise OR of the following flags:
415
416       FI_SEND : The memory buffer may be used in outgoing message data trans‐
417       fers.  This includes fi_msg and fi_tagged operations.
418
419       FI_RECV  :  The  memory  buffer  may be used to receive inbound message
420       transfers.  This includes fi_msg and fi_tagged operations.
421
422       FI_READ : The memory buffer may be used as the result  buffer  for  RMA
423       read and atomic operations on the initiator side.
424
425       FI_WRITE  :  The memory buffer may be used as the source buffer for RMA
426       write and atomic operations on the initiator side.
427
428       FI_REMOTE_READ : The memory buffer may be used as the source buffer  of
429       an RMA read operation on the target side.
430
431       FI_REMOTE_WRITE : The memory buffer may be used as the target buffer of
432       an RMA write or atomic operation.
433
434   requested_key
435       An application specified access key associated with the memory  region.
436       The  MR key must be provided by a remote process when performing RMA or
437       atomic operations  to  a  memory  region.   Applications  can  use  the
438       requested_key  field  to  indicate  that  a specific key be used by the
439       provider.  This allows applications to use well known key values, which
440       can avoid applications needing to exchange and store keys.  Support for
441       user requested keys is provider  specific  and  is  determined  by  the
442       mr_mode domain attribute.
443
444   context
445       Application  context  associated  with asynchronous memory registration
446       operations.  This value is returned as part of any  asynchronous  event
447       associated  with  the registration.  This field is ignored for synchro‐
448       nous registration calls.
449
450   auth_key_size
451       The size of key referenced by the auth_key field in bytes, or 0  if  no
452       authorization key is given.  This field is ignored unless the fabric is
453       opened with API version 1.5 or greater.
454
455   auth_key
456       Indicates the key to associate with this memory  registration.   Autho‐
457       rization  keys are used to limit communication between endpoints.  Only
458       peer endpoints that are programmed to use the  same  authorization  key
459       may  access  the  memory  region.  The domain authorization key will be
460       used if the auth_key_size provided is 0.  This field is ignored  unless
461       the fabric is opened with API version 1.5 or greater.
462

NOTES

464       Direct access to an application's memory by a remote peer requires that
465       the application register the targeted memory buffer(s).  This is  typi‐
466       cally   done   by   calling   one  of  the  fi_mr_reg*  routines.   For
467       FI_MR_PROV_KEY, the provider will return a key that must be used by the
468       peer  when accessing the memory region.  The application is responsible
469       for transferring this key to the peer.  If FI_MR_RAW mode has been set,
470       the key must be retrieved using the fi_mr_raw_attr function.
471
472       FI_RAW_MR  allows  support for providers that require more than 8-bytes
473       for their protection keys or need additional setup before a key can  be
474       used  for  transfers.   After  a raw key has been retrieved, it must be
475       exchanged with the remote peer.  The peer  must  use  fi_mr_map_raw  to
476       convert the raw key into a usable 64-bit key.  The mapping must be done
477       even if the raw key is 64-bits or smaller.
478
479       The raw key support functions are usable  with  all  registered  memory
480       regions,  even  if  FI_MR_RAW has not been set.  It is recommended that
481       portable applications target using those interfaces; however, their use
482       does  carry extra message and memory footprint overhead, making it less
483       desirable for highly scalable apps.
484

FLAGS

486       The follow flag may be specified to any memory registration call.
487
488       FI_RMA_EVENT : This flag indicates that  the  specified  memory  region
489       will  be  associated with a completion counter used to count RMA opera‐
490       tions that access the MR.
491
492       FI_RMA_PMEM : This flag indicates that the underlying memory region  is
493       backed  by  persistent  memory  and will be used in RMA operations.  It
494       must be specified if persistent completion semantics or persistent data
495       transfers are required when accessing the registered region.
496

RETURN VALUES

498       Returns 0 on success.  On error, a negative value corresponding to fab‐
499       ric errno is returned.
500
501       Fabric errno values are defined in rdma/fi_errno.h.
502

ERRORS

504       -FI_ENOKEY : The requested_key is already in use.
505
506       -FI_EKEYREJECTED : The requested_key is not available.  They key may be
507       out  of  the  range  supported by the provider, or the provider may not
508       support user-requested memory registration keys.
509
510       -FI_ENOSYS : Returned by fi_mr_bind if the provider  does  not  support
511       reporting events based on access to registered memory regions.
512
513       -FI_EBADFLAGS  :  Returned  if the specified flags are not supported by
514       the provider.
515

SEE ALSO

517       fi_getinfo(3),  fi_endpoint(3),  fi_domain(3),  fi_rma(3),   fi_msg(3),
518       fi_atomic(3)
519

AUTHORS

521       OpenFabrics.
522
523
524
525Libfabric Programmer's Manual     2018-02-13                          fi_mr(3)
Impressum