1fi_domain(3)                   Libfabric v1.7.0                   fi_domain(3)
2
3
4

NAME

6       fi_domain - Open a fabric access domain
7

SYNOPSIS

9              #include <rdma/fabric.h>
10
11              #include <rdma/fi_domain.h>
12
13              int fi_domain(struct fid_fabric *fabric, struct fi_info *info,
14                  struct fid_domain **domain, void *context);
15
16              int fi_close(struct fid *domain);
17
18              int fi_domain_bind(struct fid_domain *domain, struct fid *eq,
19                  uint64_t flags);
20
21              int fi_open_ops(struct fid *domain, const char *name, uint64_t flags,
22                  void **ops, void *context);
23

ARGUMENTS

25       fabric Fabric domain
26
27       info   Fabric   information,  including  domain  capabilities  and  at‐
28              tributes.
29
30       domain An opened access domain.
31
32       context
33              User specified context associated with the domain.  This context
34              is  returned  as  part of any asynchronous event associated with
35              the domain.
36
37       eq     Event queue for asynchronous operations initiated on the domain.
38
39       name   Name associated with an interface.
40
41       ops    Fabric interface operations.
42

DESCRIPTION

44       An access domain typically refers to a physical or virtual NIC or hard‐
45       ware  port;  however, a domain may span across multiple hardware compo‐
46       nents for fail-over or data striping purposes.  A  domain  defines  the
47       boundary  for  associating  different  resources  together.  Fabric re‐
48       sources belonging to the same domain may share resources.
49
50   fi_domain
51       Opens a fabric access domain, also referred to as  a  resource  domain.
52       Fabric  domains are identified by a name.  The properties of the opened
53       domain are specified using the info parameter.
54
55   fi_open_ops
56       fi_open_ops is used to open provider specific interfaces.  Provider in‐
57       terfaces  may be used to access low-level resources and operations that
58       are specific to the opened resource domain.  The details of domain  in‐
59       terfaces are outside the scope of this documentation.
60
61   fi_domain_bind
62       Associates  an  event queue with the domain.  An event queue bound to a
63       domain will be the default  EQ  associated  with  asynchronous  control
64       events  that occur on the domain or active endpoints allocated on a do‐
65       main.  This includes CM events.  Endpoints  may  direct  their  control
66       events to alternate EQs by binding directly with the EQ.
67
68       Binding  an  event  queue to a domain with the FI_REG_MR flag indicates
69       that the provider should perform  all  memory  registration  operations
70       asynchronously,  with  the completion reported through the event queue.
71       If an event queue is not bound to the domain with the  FI_REG_MR  flag,
72       then memory registration requests complete synchronously.
73
74       See  fi_av_bind(3),  fi_ep_bind(3),  fi_mr_bind(3), fi_pep_bind(3), and
75       fi_scalable_ep_bind(3) for more information.
76
77   fi_close
78       The fi_close call is used to release all resources  associated  with  a
79       domain  or  interface.   All  objects associated with the opened domain
80       must be released prior to calling fi_close, otherwise the call will re‐
81       turn -FI_EBUSY.
82

DOMAIN ATTRIBUTES

84       The  fi_domain_attr  structure defines the set of attributes associated
85       with a domain.
86
87              struct fi_domain_attr {
88                  struct fid_domain     *domain;
89                  char                  *name;
90                  enum fi_threading     threading;
91                  enum fi_progress      control_progress;
92                  enum fi_progress      data_progress;
93                  enum fi_resource_mgmt resource_mgmt;
94                  enum fi_av_type       av_type;
95                  int                   mr_mode;
96                  size_t                mr_key_size;
97                  size_t                cq_data_size;
98                  size_t                cq_cnt;
99                  size_t                ep_cnt;
100                  size_t                tx_ctx_cnt;
101                  size_t                rx_ctx_cnt;
102                  size_t                max_ep_tx_ctx;
103                  size_t                max_ep_rx_ctx;
104                  size_t                max_ep_stx_ctx;
105                  size_t                max_ep_srx_ctx;
106                  size_t                cntr_cnt;
107                  size_t                mr_iov_limit;
108                  uint64_t              caps;
109                  uint64_t              mode;
110                  uint8_t               *auth_key;
111                  size_t                auth_key_size;
112                  size_t                max_err_data;
113                  size_t                mr_cnt;
114              };
115
116   domain
117       On input to fi_getinfo, a user may set this to  an  opened  domain  in‐
118       stance  to restrict output to the given domain.  On output from fi_get‐
119       info, if no domain was specified, but the user has an  opened  instance
120       of the named domain, this will reference the first opened instance.  If
121       no instance has been opened, this field will be NULL.
122
123   Name
124       The name of the access domain.
125
126   Multi-threading Support (threading)
127       The threading model specifies the level of serialization required of an
128       application when using the libfabric data transfer interfaces.  Control
129       interfaces are always considered thread safe, and may  be  accessed  by
130       multiple  threads.   Applications  which can guarantee serialization in
131       their access of provider allocated resources and interfaces  enables  a
132       provider to eliminate lower-level locks.
133
134       FI_THREAD_UNSPEC
135              This  value  indicates that no threading model has been defined.
136              It may be used on input hints  to  the  fi_getinfo  call.   When
137              specified,  providers  will return a threading model that allows
138              for the greatest level of parallelism.
139
140       FI_THREAD_SAFE
141              A thread safe serialization model allows a multi-threaded appli‐
142              cation  to  access any allocated resources through any interface
143              without restriction.  All  providers  are  required  to  support
144              FI_THREAD_SAFE.
145
146       FI_THREAD_FID
147              A  fabric descriptor (FID) serialization model requires applica‐
148              tions to serialize access to individual fabric resources associ‐
149              ated  with  data  transfer operations and completions.  Multiple
150              threads must be serialized when  accessing  the  same  endpoint,
151              transmit  context,  receive  context, completion queue, counter,
152              wait set, or  poll  set.   Serialization  is  required  only  by
153              threads accessing the same object.
154
155       For  example,  one  thread may be initiating a data transfer on an end‐
156       point, while another thread reads from a  completion  queue  associated
157       with the endpoint.
158
159       Serialization  to  endpoint  access is only required when accessing the
160       same endpoint data flow.  Multiple threads may  initiate  transfers  on
161       different  transmit  contexts of the same endpoint without serializing,
162       and no serialization is required between the submission of data  trans‐
163       mit requests and data receive operations.
164
165       In general, FI_THREAD_FID allows the provider to be implemented without
166       needing internal locking when handling data  transfers.   Conceptually,
167       FI_THREAD_FID  maps well to providers that implement fabric services in
168       hardware and provide separate command queues to different data flows.
169
170       FI_THREAD_ENDPOINT
171              The endpoint threading model is similar  to  FI_THREAD_FID,  but
172              with  the  added restriction that serialization is required when
173              accessing the same endpoint, even if multiple transmit  and  re‐
174              ceive  contexts are used.  Conceptually, FI_THREAD_ENDPOINT maps
175              well to providers that implement fabric services in hardware but
176              use a single command queue to access different data flows.
177
178       FI_THREAD_COMPLETION
179              The  completion  threading  model is intended for providers that
180              make use of manual progress.  Applications must serialize access
181              to  all  objects that are associated through the use of having a
182              shared completion structure.  This includes  endpoint,  transmit
183              context,  receive  context, completion queue, counter, wait set,
184              and poll set objects.
185
186       For example, threads must serialize access to an endpoint and its bound
187       completion  queue(s)  and/or  counters.  Access to endpoints that share
188       the same completion queue must also be serialized.
189
190       The  use  of  FI_THREAD_COMPLETION  can   increase   parallelism   over
191       FI_THREAD_SAFE, but requires the use of isolated resources.
192
193       FI_THREAD_DOMAIN
194              A  domain serialization model requires applications to serialize
195              access to all objects belonging to a domain.
196
197   Progress Models (control_progress / data_progress)
198       Progress is the ability of the underlying  implementation  to  complete
199       processing  of  an asynchronous request.  In many cases, the processing
200       of an asynchronous request requires the use of the host processor.  For
201       example,  a  received  message  may need to be matched with the correct
202       buffer, or a timed out request may need to be retransmitted.  For  per‐
203       formance  reasons, it may be undesirable for the provider to allocate a
204       thread for this  purpose,  which  will  compete  with  the  application
205       threads.
206
207       Control  progress  indicates  the method that the provider uses to make
208       progress on asynchronous control operations.   Control  operations  are
209       functions which do not directly involve the transfer of application da‐
210       ta between endpoints.  They include address  vector,  memory  registra‐
211       tion, and connection management routines.
212
213       Data  progress  indicates  the  method  that  the provider uses to make
214       progress on data transfer operations.   This  includes  message  queue,
215       RMA,  tagged messaging, and atomic operations, along with their comple‐
216       tion processing.
217
218       Progress frequently requires action being taken at both  the  transmit‐
219       ting  and receiving sides of an operation.  This is often a requirement
220       for reliable transfers, as a result of retry and  acknowledgement  pro‐
221       cessing.
222
223       To balance between performance and ease of use, two progress models are
224       defined.
225
226       FI_PROGRESS_UNSPEC
227              This value indicates that no progress model  has  been  defined.
228              It may be used on input hints to the fi_getinfo call.
229
230       FI_PROGRESS_AUTO
231              This  progress  model indicates that the provider will make for‐
232              ward progress on an asynchronous operation without  further  in‐
233              tervention by the application.  When FI_PROGRESS_AUTO is provid‐
234              ed as output to fi_getinfo in the absence of any progress hints,
235              it often indicates that the desired functionality is implemented
236              by the provider hardware or is a standard service of the operat‐
237              ing system.
238
239       All  providers are required to support FI_PROGRESS_AUTO.  However, if a
240       provider does not natively support automatic progress, forcing the  use
241       of  FI_PROGRESS_AUTO  may  result  in threads being allocated below the
242       fabric interfaces.
243
244       FI_PROGRESS_MANUAL
245              This progress model indicates that the provider requires the use
246              of  an  application  thread to complete an asynchronous request.
247              When manual progress is set, the provider will  attempt  to  ad‐
248              vance an asynchronous operation forward when the application at‐
249              tempts to wait on or read an event queue, completion  queue,  or
250              counter   where   the  completed  operation  will  be  reported.
251              Progress also occurs when the application processes  a  poll  or
252              wait  set  that has been associated with the event or completion
253              queue.
254
255       Only wait operations defined by the fabric interface will result in  an
256       operation  progressing.   Operating  system or external wait functions,
257       such as select, poll, or pthread routines, cannot.
258
259       Manual progress requirements not only apply to endpoints that  initiate
260       transmit  operations,  but  also to endpoints that may be the target of
261       such operations.  This holds true even if the target endpoint will  not
262       generate  completion  events  for the operations.  For example, an end‐
263       point that acts purely as the target of RMA or atomic  operations  that
264       uses  manual  progress may still need application assistance to process
265       received operations.
266
267   Resource Management (resource_mgmt)
268       Resource management (RM) is provider and protocol  support  to  protect
269       against  overrunning  local  and remote resources.  This includes local
270       and remote transmit contexts, receive contexts, completion queues,  and
271       source and target data buffers.
272
273       When  enabled,  applications are given some level of protection against
274       overrunning provider queues and local and remote  data  buffers.   Such
275       support  may  be built directly into the hardware and/or network proto‐
276       col, but may also require that checks be enabled in the provider  soft‐
277       ware.  By disabling resource management, an application assumes all re‐
278       sponsibility for preventing queue and buffer overruns, but doing so may
279       allow  a  provider to eliminate internal synchronization calls, such as
280       atomic variables or locks.
281
282       It should be noted that even if resource management  is  disabled,  the
283       provider  implementation  and  protocol may still provide some level of
284       protection against overruns.  However, such protection is  not  guaran‐
285       teed.  The following values for resource management are defined.
286
287       FI_RM_UNSPEC
288              This  value indicates that no resource management model has been
289              defined.  It may be used on input hints to the fi_getinfo call.
290
291       FI_RM_DISABLED
292              The provider is free to select an  implementation  and  protocol
293              that  does  not protect against resource overruns.  The applica‐
294              tion is responsible for resource protection.
295
296       FI_RM_ENABLED
297              Resource management is enabled for this provider domain.
298
299       The behavior of the various  resource  management  options  depends  on
300       whether the endpoint is reliable or unreliable, as well as provider and
301       protocol specific implementation details, as shown in the following ta‐
302       ble.  The table assumes that all peers enable or disable RM the same.
303
304       Resource    DGRAM EP-no RM    DGRAM EP-with RM   RDM/MSG   EP-no   RDM/MSG EP-with
305                                                        RM                RM
306       ───────────────────────────────────────────────────────────────────────────────────
307        Tx Ctx    undefined error         EAGAIN        undefined error        EAGAIN
308        Rx Ctx    undefined error         EAGAIN        undefined error        EAGAIN
309        Tx CQ     undefined error         EAGAIN        undefined error        EAGAIN
310        Rx CQ     undefined error         EAGAIN        undefined error        EAGAIN
311        Target        dropped            dropped         transmit error       retried
312        EP
313       No    Rx       dropped            dropped         transmit error       retried
314       Buffer
315       Rx   Buf   truncate or drop   truncate or drop   truncate or er‐   truncate or er‐
316       Overrun                                          ror               ror
317       Un‐         not applicable     not applicable     transmit error    transmit error
318       matched
319       RMA
320       RMA         not applicable     not applicable     transmit error    transmit error
321       Overrun
322
323       The  resource  column  indicates  the resource being accessed by a data
324       transfer operation.
325
326       Tx Ctx / Rx Ctx
327              Refers to the transmit/receive contexts when a data transfer op‐
328              eration  is submitted.  When RM is enabled, attempting to submit
329              a request will fail if the context is full.  If RM is  disabled,
330              an  undefined error (provider specific) will occur.  Such errors
331              should be considered fatal to the context, and applications must
332              take steps to avoid queue overruns.
333
334       Tx CQ / Rx CQ
335              Refers to the completion queue associated with the Tx or Rx con‐
336              text when a local operation completes.  When RM is disabled, ap‐
337              plications  must  take  care to ensure that completion queues do
338              not get overrun.  When an overrun occurs, an undefined, but  fa‐
339              tal,  error  will  occur affecting all endpoints associated with
340              the CQ.  Overruns can be avoided by sizing the CQs appropriately
341              or  by deferring the posting of a data transfer operation unless
342              CQ space is available to store its completion.  When RM  is  en‐
343              abled,  providers  may  use  different  mechanisms to prevent CQ
344              overruns.  This  includes  failing  (returning  -FI_EAGAIN)  the
345              posting  of  operations that could result in CQ overruns, or in‐
346              ternally retrying requests (which will be hidden from the appli‐
347              cation).   See notes at the end of this section regarding CQ re‐
348              source management restrictions.
349
350       Target EP / No Rx Buffer
351              Target EP refers to resources associated with the endpoint  that
352              is the target of a transmit operation.  This includes the target
353              endpoint's receive queue, posted receive  buffers  (no  Rx  buf‐
354              fers),  the  receive  side  completion  queue, and other related
355              packet processing queues.  The defined behavior is that seen  by
356              the  initiator  of a request.  For FI_EP_DGRAM endpoints, if the
357              target EP queues are unable to  accept  incoming  messages,  re‐
358              ceived  messages will be dropped.  For reliable endpoints, if RM
359              is disabled, the transmit operation will complete in error.   If
360              RM is enabled, the provider will internally retry the operation.
361
362       Rx Buffer Overrun
363              This  refers to buffers posted to receive incoming tagged or un‐
364              tagged messages, with the behavior defined from the viewpoint of
365              the  sender.   The  behavior for handling received messages that
366              are larger than the  buffers  provided  by  the  application  is
367              provider  specific.   Providers  may either truncate the message
368              and report a successful completion, or fail the operation.   For
369              datagram  endpoints, failed sends will result in the message be‐
370              ing dropped.  For reliable endpoints, send operations  may  com‐
371              plete  successfully, yet be truncated at the receive side.  This
372              can occur when the target side buffers received  data  until  an
373              application buffer is made available.  The completion status may
374              also be dependent upon the completion model selected byt the ap‐
375              plication  (e.g.   FI_DELIVERY_COMPLETE  versus FI_TRANSMIT_COM‐
376              PLETE).
377
378       Unmatched RMA / RMA Overrun
379              Unmatched RMA and RMA overruns deal with the processing  of  RMA
380              and  atomic  operations.  Unlike send operations, RMA operations
381              that attempt to access a memory address that is either not  reg‐
382              istered for such operations, or attempt to access outside of the
383              target memory region will fail, resulting in a transmit error.
384
385       When a resource management error occurs on an endpoint, the endpoint is
386       transitioned  into a disabled state.  Any operations which have not al‐
387       ready completed will fail and be discarded.  For unconnected endpoints,
388       the endpoint must be re-enabled before it will accept new data transfer
389       operations.  For connected endpoints, the connection is torn  down  and
390       must be re-established.
391
392       There is one notable restriction on the protections offered by resource
393       management.  This occurs when resource management is enabled on an end‐
394       point  that  has  been bound to completion queue(s) using the FI_SELEC‐
395       TIVE_COMPLETION flag.  Operations posted to such an endpoint may speci‐
396       fy that a successful completion should not generate a entry on the cor‐
397       responding completion queue.  (I.e.  the operation leaves  the  FI_COM‐
398       PLETION  flag unset).  In such situations, the provider is not required
399       to reserve an entry in the completion queue to handle  the  case  where
400       the  operation  fails  and does generate a CQ entry, which would effec‐
401       tively require tracking the operation to completion.  Applications con‐
402       cerned  with  avoiding CQ overruns in the occurrence of errors must en‐
403       sure that there is sufficient space in the CQ to report  failed  opera‐
404       tions.  This can typically be achieved by sizing the CQ to at least the
405       same size as the endpoint queue(s) that are bound to it.
406
407   AV Type (av_type)
408       Specifies the type of address vectors that are usable with this domain.
409       For  additional details on AV type, see fi_av(3).  The following values
410       may be specified.
411
412       FI_AV_UNSPEC
413              Any address vector format is requested and supported.
414
415       FI_AV_MAP
416              Only address vectors of type AV map are requested or supported.
417
418       FI_AV_TABLE
419              Only address vectors of type AV index are requested or  support‐
420              ed.
421
422       Address  vectors  are  only used by connectionless endpoints.  Applica‐
423       tions that require the use of a specific type of address vector  should
424       set  the  domain  attribute av_type to the necessary value when calling
425       fi_getinfo.  The value FI_AV_UNSPEC may be used to  indicate  that  the
426       provider  can  support  either  address vector format.  In this case, a
427       provider may return FI_AV_UNSPEC to indicate that either format is sup‐
428       portable, or may return another AV type to indicate the optimal AV type
429       supported by this domain.
430
431   Memory Registration Mode (mr_mode)
432       Defines memory registration specific mode bits used with  this  domain.
433       Full details on MR mode options are available in fi_mr(3).  The follow‐
434       ing values may be specified.
435
436       FI_MR_LOCAL
437              The provider is optimized around  having  applications  register
438              memory  for locally accessed data buffers.  Data buffers used in
439              send and receive operations and as the source buffer for RMA and
440              atomic  operations must be registered by the application for ac‐
441              cess domains opened with this capability.
442
443       FI_MR_RAW
444              The provider requires additional setup as part of  their  memory
445              registration  process.   This mode is required by providers that
446              use a memory key that is larger than 64-bits.
447
448       FI_MR_VIRT_ADDR
449              Registered memory regions are referenced by peers using the vir‐
450              tual  address  of  the  registered  memory region, rather than a
451              0-based offset.
452
453       FI_MR_ALLOCATED
454              Indicates that memory registration occurs on allocated data buf‐
455              fers,  and  physical pages must back all virtual addresses being
456              registered.
457
458       FI_MR_PROV_KEY
459              Memory registration  keys  are  selected  and  returned  by  the
460              provider.
461
462       FI_MR_MMU_NOTIFY
463              Indicates  that the application is responsible for notifying the
464              provider when the page tables referencing  a  registered  memory
465              region may have been updated.
466
467       FI_MR_RMA_EVENT
468              Indicates  that  the  memory  regions associated with completion
469              counters must be explicitly enabled after  being  bound  to  any
470              counter.
471
472       FI_MR_ENDPOINT
473              Memory  registration  occurs  at the endpoint level, rather than
474              domain.
475
476       FI_MR_UNSPEC
477              Defined for compatibility -- library versions 1.4  and  earlier.
478              Setting  mr_mode  to 0 indicates that FI_MR_BASIC or FI_MR_SCAL‐
479              ABLE are requested and supported.
480
481       FI_MR_BASIC
482              Defined for compatibility -- library versions 1.4  and  earlier.
483              Only  basic memory registration operations are requested or sup‐
484              ported.   This  mode  is  equivalent  to  the   FI_MR_VIRT_ADDR,
485              FI_MR_ALLOCATED, and FI_MR_PROV_KEY flags being set in later li‐
486              brary versions.  This flag may not be used in  conjunction  with
487              other mr_mode bits.
488
489       FI_MR_SCALABLE
490              Defined  for  compatibility -- library versions 1.4 and earlier.
491              Only scalable memory registration operations  are  requested  or
492              supported.   Scalable registration uses offset based addressing,
493              with application selectable memory keys.  For  library  versions
494              1.5  and  later, this is the default if no mr_mode bits are set.
495              This flag may not be used  in  conjunction  with  other  mr_mode
496              bits.
497
498       Buffers  used  in  data  transfer  operations may require notifying the
499       provider of their use before a data transfer can  occur.   The  mr_mode
500       field  indicates  the type of memory registration that is required, and
501       when registration is necessary.  Applications that require the use of a
502       specific  registration  mode should set the domain attribute mr_mode to
503       the necessary value when calling fi_getinfo.   The  value  FI_MR_UNSPEC
504       may be used to indicate support for any registration mode.
505
506   MR Key Size (mr_key_size)
507       Size  of  the  memory region remote access key, in bytes.  Applications
508       that request their own MR key must select  a  value  within  the  range
509       specified  by  this value.  Key sizes larger than 8 bytes require using
510       the FI_RAW_KEY mode bit.
511
512   CQ Data Size (cq_data_size)
513       Applications may include a small message with a data transfer  that  is
514       placed  directly into a remote completion queue as part of a completion
515       event.  This is referred to as remote CQ data (sometimes referred to as
516       immediate  data).   This  field  indicates the number of bytes that the
517       provider supports for remote CQ data.  If supported (non-zero value  is
518       returned), the minimum size of remote CQ data must be at least 4-bytes.
519
520   Completion Queue Count (cq_cnt)
521       The  optimal number of completion queues supported by the domain, rela‐
522       tive to any specified or default CQ attributes.  The cq_cnt  value  may
523       be a fixed value of the maximum number of CQs supported by the underly‐
524       ing hardware, or may be a dynamic  value,  based  on  the  default  at‐
525       tributes of an allocated CQ, such as the CQ size and data format.
526
527   Endpoint Count (ep_cnt)
528       The  total number of endpoints supported by the domain, relative to any
529       specified or default endpoint attributes.  The ep_cnt value  may  be  a
530       fixed  value of the maximum number of endpoints supported by the under‐
531       lying hardware, or may be a dynamic value, based  on  the  default  at‐
532       tributes  of  an  allocated endpoint, such as the endpoint capabilities
533       and size.  The endpoint count is the number  of  addressable  endpoints
534       supported by the provider.
535
536   Transmit Context Count (tx_ctx_cnt)
537       The  number  of  outbound  command  queues  optimally  supported by the
538       provider.  For a low-level provider, this represents the number of com‐
539       mand  queues to the hardware and/or the number of parallel transmit en‐
540       gines effectively supported by the hardware and  caches.   Applications
541       which allocate more transmit contexts than this value will end up shar‐
542       ing underlying resources.  By default, there is a single transmit  con‐
543       text  associated with each endpoint, but in an advanced usage model, an
544       endpoint may be configured with multiple transmit contexts.
545
546   Receive Context Count (rx_ctx_cnt)
547       The number of inbound processing  queues  optimally  supported  by  the
548       provider.   For  a low-level provider, this represents the number hard‐
549       ware queues that can be effectively utilized  for  processing  incoming
550       packets.   Applications  which allocate more receive contexts than this
551       value will end up sharing underlying resources.  By default,  a  single
552       receive  context  is  associated with each endpoint, but in an advanced
553       usage model, an endpoint may be configured with multiple  receive  con‐
554       texts.
555
556   Maximum Endpoint Transmit Context (max_ep_tx_ctx)
557       The  maximum number of transmit contexts that may be associated with an
558       endpoint.
559
560   Maximum Endpoint Receive Context (max_ep_rx_ctx)
561       The maximum number of receive contexts that may be associated  with  an
562       endpoint.
563
564   Maximum Sharing of Transmit Context (max_ep_stx_ctx)
565       The  maximum  number  of endpoints that may be associated with a shared
566       transmit context.
567
568   Maximum Sharing of Receive Context (max_ep_srx_ctx)
569       The maximum number of endpoints that may be associated  with  a  shared
570       receive context.
571
572   Counter Count (cntr_cnt)
573       The optimal number of completion counters supported by the domain.  The
574       cq_cnt value may be a fixed value of the  maximum  number  of  counters
575       supported  by the underlying hardware, or may be a dynamic value, based
576       on the default attributes of the domain.
577
578   MR IOV Limit (mr_iov_limit)
579       This is the maximum number of IO vectors (scatter-gather elements) that
580       a single memory registration operation may reference.
581
582   Capabilities (caps)
583       Domain  level  capabilities.  Domain capabilities indicate domain level
584       features that are supported by the provider.
585
586       FI_LOCAL_COMM
587              At a conceptual level, this field indicates that the  underlying
588              device supports loopback communication.  More specifically, this
589              field indicates that an endpoint may communicate with other end‐
590              points that are allocated from the same underlying named domain.
591              If this field is not set, an application may need to use an  al‐
592              ternate domain or mechanism (e.g.  shared memory) to communicate
593              with peers that execute on the same node.
594
595       FI_REMOTE_COMM
596              This field indicates that the underlying provider supports  com‐
597              munication  with  nodes that are reachable over the network.  If
598              this field is not set, then the provider only supports  communi‐
599              cation  between  processes  that  execute  on the same node -- a
600              shared memory provider, for example.
601
602       FI_SHARED_AV
603              Indicates that the domain supports the ability to share  address
604              vectors  among multiple processes using the named address vector
605              feature.
606
607       See fi_getinfo(3) for a discussion on primary versus secondary capabil‐
608       ities.  All domain capabilities are considered secondary capabilities.
609
610   mode
611       The operational mode bit related to using the domain.
612
613       FI_RESTRICTED_COMP
614              This  bit indicates that the domain limits completion queues and
615              counters to only be used with endpoints, transmit contexts,  and
616              receive contexts that have the same set of capability flags.
617
618   Default authorization key (auth_key)
619       The  default  authorization  key  to associate with endpoint and memory
620       registrations created within the domain.  This field is ignored  unless
621       the fabric is opened with API version 1.5 or greater.
622
623   Default authorization key length (auth_key_size)
624       The  length  in  bytes of the default authorization key for the domain.
625       If set to 0, then no authorization key will  be  associated  with  end‐
626       points and memory registrations created within the domain unless speci‐
627       fied in the endpoint or memory registration attributes.  This field  is
628       ignored unless the fabric is opened with API version 1.5 or greater.
629
630   Max Error Data Size (max_err_data)
631       :  The  maximum amount of error data, in bytes, that may be returned as
632       part of a completion or event queue error.  This value  corresponds  to
633       the   err_data_size   field   in   struct  fi_cq_err_entry  and  struct
634       fi_eq_err_entry.
635
636   Memory Regions Count (mr_cnt)
637       The optimal number of memory regions supported by the domain,  or  end‐
638       point if the mr_mode FI_MR_ENDPOINT bit has been set.  The mr_cnt value
639       may be a fixed value of the maximum number of MRs supported by the  un‐
640       derlying  hardware, or may be a dynamic value, based on the default at‐
641       tributes of the domain,  such  as  the  supported  memory  registration
642       modes.   Applications can set the mr_cnt on input to fi_getinfo, in or‐
643       der to indicate their memory registration requirements.  Doing  so  may
644       allow  the provider to optimize any memory registration cache or lookup
645       tables.
646

RETURN VALUE

648       Returns 0 on success.  On error, a negative value corresponding to fab‐
649       ric  errno is returned.  Fabric errno values are defined in rdma/fi_er‐
650       rno.h.
651

NOTES

653       Users should call fi_close to release all resources  allocated  to  the
654       fabric domain.
655
656       The following fabric resources are associated with domains: active end‐
657       points, memory regions, completion event queues, and address vectors.
658
659       Domain attributes reflect the limitations and capabilities of  the  un‐
660       derlying hardware and/or software provider.  They do not reflect system
661       limitations, such as the number of physical pages that  an  application
662       may  pin  or  number of file descriptors that the application may open.
663       As a result, the reported maximums may not be  achievable,  even  on  a
664       lightly loaded systems, without an administrator configuring system re‐
665       sources appropriately for the installed provider(s).
666

SEE ALSO

668       fi_getinfo(3), fi_endpoint(3), fi_av(3), fi_ep(3), fi_eq(3), fi_mr(3)
669

AUTHORS

671       OpenFabrics.
672
673
674
675Libfabric Programmer's Manual     2018-10-05                      fi_domain(3)
Impressum