1fi_domain(3) Libfabric v1.10.0 fi_domain(3)
2
6 fi_domain - Open a fabric access domain
7
9 #include <rdma/fabric.h>
10 #include <rdma/fi_domain.h>
12 int fi_domain(struct fid_fabric *fabric, struct fi_info *info,
14 struct fid_domain **domain, void *context);
15 int fi_close(struct fid *domain);
17 int fi_domain_bind(struct fid_domain *domain, struct fid *eq,
19 uint64_t flags);
20 int fi_open_ops(struct fid *domain, const char *name, uint64_t flags,
22 void **ops, void *context);
23
25 fabric Fabric domain
26 info Fabric information, including domain capabilities and at‐
28 tributes.
29 domain An opened access domain.
31 context
33 User specified context associated with the domain. This context
34 is returned as part of any asynchronous event associated with
35 the domain.
36 eq Event queue for asynchronous operations initiated on the domain.
38 name Name associated with an interface.
40 ops Fabric interface operations.
42
44 An access domain typically refers to a physical or virtual NIC or hard‐
45 ware port; however, a domain may span across multiple hardware compo‐
46 nents for fail-over or data striping purposes. A domain defines the
47 boundary for associating different resources together. Fabric re‐
48 sources belonging to the same domain may share resources.
49 fi_domain
51 Opens a fabric access domain, also referred to as a resource domain.
52 Fabric domains are identified by a name. The properties of the opened
53 domain are specified using the info parameter.
54 fi_open_ops
56 fi_open_ops is used to open provider specific interfaces. Provider in‐
57 terfaces may be used to access low-level resources and operations that
58 are specific to the opened resource domain. The details of domain in‐
59 terfaces are outside the scope of this documentation.
60 fi_domain_bind
62 Associates an event queue with the domain. An event queue bound to a
63 domain will be the default EQ associated with asynchronous control
64 events that occur on the domain or active endpoints allocated on a do‐
65 main. This includes CM events. Endpoints may direct their control
66 events to alternate EQs by binding directly with the EQ.
67 Binding an event queue to a domain with the FI_REG_MR flag indicates
69 that the provider should perform all memory registration operations
70 asynchronously, with the completion reported through the event queue.
71 If an event queue is not bound to the domain with the FI_REG_MR flag,
72 then memory registration requests complete synchronously.
73 See fi_av_bind(3), fi_ep_bind(3), fi_mr_bind(3), fi_pep_bind(3), and
75 fi_scalable_ep_bind(3) for more information.
76 fi_close
78 The fi_close call is used to release all resources associated with a
79 domain or interface. All objects associated with the opened domain
80 must be released prior to calling fi_close, otherwise the call will re‐
81 turn -FI_EBUSY.
82
84 The fi_domain_attr structure defines the set of attributes associated
85 with a domain.
86 struct fi_domain_attr {
88 struct fid_domain *domain;
89 char *name;
90 enum fi_threading threading;
91 enum fi_progress control_progress;
92 enum fi_progress data_progress;
93 enum fi_resource_mgmt resource_mgmt;
94 enum fi_av_type av_type;
95 int mr_mode;
96 size_t mr_key_size;
97 size_t cq_data_size;
98 size_t cq_cnt;
99 size_t ep_cnt;
100 size_t tx_ctx_cnt;
101 size_t rx_ctx_cnt;
102 size_t max_ep_tx_ctx;
103 size_t max_ep_rx_ctx;
104 size_t max_ep_stx_ctx;
105 size_t max_ep_srx_ctx;
106 size_t cntr_cnt;
107 size_t mr_iov_limit;
108 uint64_t caps;
109 uint64_t mode;
110 uint8_t *auth_key;
111 size_t auth_key_size;
112 size_t max_err_data;
113 size_t mr_cnt;
114 uint32_t tclass;
115 };
116 domain
118 On input to fi_getinfo, a user may set this to an opened domain in‐
119 stance to restrict output to the given domain. On output from fi_get‐
120 info, if no domain was specified, but the user has an opened instance
121 of the named domain, this will reference the first opened instance. If
122 no instance has been opened, this field will be NULL.
123 The domain instance returned by fi_getinfo should only be considered
125 valid if the application does not close any domain instances from an‐
126 other thread while fi_getinfo is being processed.
127 Name
129 The name of the access domain.
130 Multi-threading Support (threading)
132 The threading model specifies the level of serialization required of an
133 application when using the libfabric data transfer interfaces. Control
134 interfaces are always considered thread safe, and may be accessed by
135 multiple threads. Applications which can guarantee serialization in
136 their access of provider allocated resources and interfaces enables a
137 provider to eliminate lower-level locks.
138 FI_THREAD_COMPLETION
140 The completion threading model is intended for providers that
141 make use of manual progress. Applications must serialize access
142 to all objects that are associated through the use of having a
143 shared completion structure. This includes endpoint, transmit
144 context, receive context, completion queue, counter, wait set,
145 and poll set objects.
146 For example, threads must serialize access to an endpoint and its bound
148 completion queue(s) and/or counters. Access to endpoints that share
149 the same completion queue must also be serialized.
150 The use of FI_THREAD_COMPLETION can increase parallelism over
152 FI_THREAD_SAFE, but requires the use of isolated resources.
153 FI_THREAD_DOMAIN
155 A domain serialization model requires applications to serialize
156 access to all objects belonging to a domain.
157 FI_THREAD_ENDPOINT
159 The endpoint threading model is similar to FI_THREAD_FID, but
160 with the added restriction that serialization is required when
161 accessing the same endpoint, even if multiple transmit and re‐
162 ceive contexts are used. Conceptually, FI_THREAD_ENDPOINT maps
163 well to providers that implement fabric services in hardware but
164 use a single command queue to access different data flows.
165 FI_THREAD_FID
167 A fabric descriptor (FID) serialization model requires applica‐
168 tions to serialize access to individual fabric resources associ‐
169 ated with data transfer operations and completions. Multiple
170 threads must be serialized when accessing the same endpoint,
171 transmit context, receive context, completion queue, counter,
172 wait set, or poll set. Serialization is required only by
173 threads accessing the same object.
174 For example, one thread may be initiating a data transfer on an end‐
176 point, while another thread reads from a completion queue associated
177 with the endpoint.
178 Serialization to endpoint access is only required when accessing the
180 same endpoint data flow. Multiple threads may initiate transfers on
181 different transmit contexts of the same endpoint without serializing,
182 and no serialization is required between the submission of data trans‐
183 mit requests and data receive operations.
184 In general, FI_THREAD_FID allows the provider to be implemented without
186 needing internal locking when handling data transfers. Conceptually,
187 FI_THREAD_FID maps well to providers that implement fabric services in
188 hardware and provide separate command queues to different data flows.
189 FI_THREAD_SAFE
191 A thread safe serialization model allows a multi-threaded appli‐
192 cation to access any allocated resources through any interface
193 without restriction. All providers are required to support
194 FI_THREAD_SAFE.
195 FI_THREAD_UNSPEC
197 This value indicates that no threading model has been defined.
198 It may be used on input hints to the fi_getinfo call. When
199 specified, providers will return a threading model that allows
200 for the greatest level of parallelism.
201 Progress Models (control_progress / data_progress)
203 Progress is the ability of the underlying implementation to complete
204 processing of an asynchronous request. In many cases, the processing
205 of an asynchronous request requires the use of the host processor. For
206 example, a received message may need to be matched with the correct
207 buffer, or a timed out request may need to be retransmitted. For per‐
208 formance reasons, it may be undesirable for the provider to allocate a
209 thread for this purpose, which will compete with the application
210 threads.
211 Control progress indicates the method that the provider uses to make
213 progress on asynchronous control operations. Control operations are
214 functions which do not directly involve the transfer of application da‐
215 ta between endpoints. They include address vector, memory registra‐
216 tion, and connection management routines.
217 Data progress indicates the method that the provider uses to make
219 progress on data transfer operations. This includes message queue,
220 RMA, tagged messaging, and atomic operations, along with their comple‐
221 tion processing.
222 Progress frequently requires action being taken at both the transmit‐
224 ting and receiving sides of an operation. This is often a requirement
225 for reliable transfers, as a result of retry and acknowledgement pro‐
226 cessing.
227 To balance between performance and ease of use, two progress models are
229 defined.
230 FI_PROGRESS_AUTO
232 This progress model indicates that the provider will make for‐
233 ward progress on an asynchronous operation without further in‐
234 tervention by the application. When FI_PROGRESS_AUTO is provid‐
235 ed as output to fi_getinfo in the absence of any progress hints,
236 it often indicates that the desired functionality is implemented
237 by the provider hardware or is a standard service of the operat‐
238 ing system.
239 All providers are required to support FI_PROGRESS_AUTO. However, if a
241 provider does not natively support automatic progress, forcing the use
242 of FI_PROGRESS_AUTO may result in threads being allocated below the
243 fabric interfaces.
244 FI_PROGRESS_MANUAL
246 This progress model indicates that the provider requires the use
247 of an application thread to complete an asynchronous request.
248 When manual progress is set, the provider will attempt to ad‐
249 vance an asynchronous operation forward when the application at‐
250 tempts to wait on or read an event queue, completion queue, or
251 counter where the completed operation will be reported.
252 Progress also occurs when the application processes a poll or
253 wait set that has been associated with the event or completion
254 queue.
255 Only wait operations defined by the fabric interface will result in an
257 operation progressing. Operating system or external wait functions,
258 such as select, poll, or pthread routines, cannot.
259 Manual progress requirements not only apply to endpoints that initiate
261 transmit operations, but also to endpoints that may be the target of
262 such operations. This holds true even if the target endpoint will not
263 generate completion events for the operations. For example, an end‐
264 point that acts purely as the target of RMA or atomic operations that
265 uses manual progress may still need application assistance to process
266 received operations.
267 FI_PROGRESS_UNSPEC
269 This value indicates that no progress model has been defined.
270 It may be used on input hints to the fi_getinfo call.
271 Resource Management (resource_mgmt)
273 Resource management (RM) is provider and protocol support to protect
274 against overrunning local and remote resources. This includes local
275 and remote transmit contexts, receive contexts, completion queues, and
276 source and target data buffers.
277 When enabled, applications are given some level of protection against
279 overrunning provider queues and local and remote data buffers. Such
280 support may be built directly into the hardware and/or network proto‐
281 col, but may also require that checks be enabled in the provider soft‐
282 ware. By disabling resource management, an application assumes all re‐
283 sponsibility for preventing queue and buffer overruns, but doing so may
284 allow a provider to eliminate internal synchronization calls, such as
285 atomic variables or locks.
286 It should be noted that even if resource management is disabled, the
288 provider implementation and protocol may still provide some level of
289 protection against overruns. However, such protection is not guaran‐
290 teed. The following values for resource management are defined.
291 FI_RM_DISABLED
293 The provider is free to select an implementation and protocol
294 that does not protect against resource overruns. The applica‐
295 tion is responsible for resource protection.
296 FI_RM_ENABLED
298 Resource management is enabled for this provider domain.
299 FI_RM_UNSPEC
301 This value indicates that no resource management model has been
302 defined. It may be used on input hints to the fi_getinfo call.
303 The behavior of the various resource management options depends on
305 whether the endpoint is reliable or unreliable, as well as provider and
306 protocol specific implementation details, as shown in the following ta‐
307 ble. The table assumes that all peers enable or disable RM the same.
308 Resource DGRAM EP-no RM DGRAM EP-with RM RDM/MSG EP-no RDM/MSG EP-with
310 RM RM
311 ───────────────────────────────────────────────────────────────────────────────────
312 Tx Ctx undefined error EAGAIN undefined error EAGAIN
313 Rx Ctx undefined error EAGAIN undefined error EAGAIN
314 Tx CQ undefined error EAGAIN undefined error EAGAIN
315 Rx CQ undefined error EAGAIN undefined error EAGAIN
316 Target dropped dropped transmit error retried
317 EP
318 No Rx dropped dropped transmit error retried
319 Buffer
320 Rx Buf truncate or drop truncate or drop truncate or er‐ truncate or er‐
321 Overrun ror ror
322 Un‐ not applicable not applicable transmit error transmit error
323 matched
324 RMA
325 RMA not applicable not applicable transmit error transmit error
326 Overrun
327 The resource column indicates the resource being accessed by a data
329 transfer operation.
330 Tx Ctx / Rx Ctx
332 Refers to the transmit/receive contexts when a data transfer op‐
333 eration is submitted. When RM is enabled, attempting to submit
334 a request will fail if the context is full. If RM is disabled,
335 an undefined error (provider specific) will occur. Such errors
336 should be considered fatal to the context, and applications must
337 take steps to avoid queue overruns.
338 Tx CQ / Rx CQ
340 Refers to the completion queue associated with the Tx or Rx con‐
341 text when a local operation completes. When RM is disabled, ap‐
342 plications must take care to ensure that completion queues do
343 not get overrun. When an overrun occurs, an undefined, but fa‐
344 tal, error will occur affecting all endpoints associated with
345 the CQ. Overruns can be avoided by sizing the CQs appropriately
346 or by deferring the posting of a data transfer operation unless
347 CQ space is available to store its completion. When RM is en‐
348 abled, providers may use different mechanisms to prevent CQ
349 overruns. This includes failing (returning -FI_EAGAIN) the
350 posting of operations that could result in CQ overruns, or in‐
351 ternally retrying requests (which will be hidden from the appli‐
352 cation). See notes at the end of this section regarding CQ re‐
353 source management restrictions.
354 Target EP / No Rx Buffer
356 Target EP refers to resources associated with the endpoint that
357 is the target of a transmit operation. This includes the target
358 endpoint's receive queue, posted receive buffers (no Rx buf‐
359 fers), the receive side completion queue, and other related
360 packet processing queues. The defined behavior is that seen by
361 the initiator of a request. For FI_EP_DGRAM endpoints, if the
362 target EP queues are unable to accept incoming messages, re‐
363 ceived messages will be dropped. For reliable endpoints, if RM
364 is disabled, the transmit operation will complete in error. If
365 RM is enabled, the provider will internally retry the operation.
366 Rx Buffer Overrun
368 This refers to buffers posted to receive incoming tagged or un‐
369 tagged messages, with the behavior defined from the viewpoint of
370 the sender. The behavior for handling received messages that
371 are larger than the buffers provided by the application is
372 provider specific. Providers may either truncate the message
373 and report a successful completion, or fail the operation. For
374 datagram endpoints, failed sends will result in the message be‐
375 ing dropped. For reliable endpoints, send operations may com‐
376 plete successfully, yet be truncated at the receive side. This
377 can occur when the target side buffers received data until an
378 application buffer is made available. The completion status may
379 also be dependent upon the completion model selected byt the ap‐
380 plication (e.g. FI_DELIVERY_COMPLETE versus FI_TRANSMIT_COM‐
381 PLETE).
382 Unmatched RMA / RMA Overrun
384 Unmatched RMA and RMA overruns deal with the processing of RMA
385 and atomic operations. Unlike send operations, RMA operations
386 that attempt to access a memory address that is either not reg‐
387 istered for such operations, or attempt to access outside of the
388 target memory region will fail, resulting in a transmit error.
389 When a resource management error occurs on an endpoint, the endpoint is
391 transitioned into a disabled state. Any operations which have not al‐
392 ready completed will fail and be discarded. For unconnected endpoints,
393 the endpoint must be re-enabled before it will accept new data transfer
394 operations. For connected endpoints, the connection is torn down and
395 must be re-established.
396 There is one notable restriction on the protections offered by resource
398 management. This occurs when resource management is enabled on an end‐
399 point that has been bound to completion queue(s) using the FI_SELEC‐
400 TIVE_COMPLETION flag. Operations posted to such an endpoint may speci‐
401 fy that a successful completion should not generate a entry on the cor‐
402 responding completion queue. (I.e. the operation leaves the FI_COM‐
403 PLETION flag unset). In such situations, the provider is not required
404 to reserve an entry in the completion queue to handle the case where
405 the operation fails and does generate a CQ entry, which would effec‐
406 tively require tracking the operation to completion. Applications con‐
407 cerned with avoiding CQ overruns in the occurrence of errors must en‐
408 sure that there is sufficient space in the CQ to report failed opera‐
409 tions. This can typically be achieved by sizing the CQ to at least the
410 same size as the endpoint queue(s) that are bound to it.
411 AV Type (av_type)
413 Specifies the type of address vectors that are usable with this domain.
414 For additional details on AV type, see fi_av(3). The following values
415 may be specified.
416 FI_AV_MAP
418 Only address vectors of type AV map are requested or supported.
419 FI_AV_TABLE
421 Only address vectors of type AV index are requested or support‐
422 ed.
423 FI_AV_UNSPEC
425 Any address vector format is requested and supported.
426 Address vectors are only used by connectionless endpoints. Applica‐
428 tions that require the use of a specific type of address vector should
429 set the domain attribute av_type to the necessary value when calling
430 fi_getinfo. The value FI_AV_UNSPEC may be used to indicate that the
431 provider can support either address vector format. In this case, a
432 provider may return FI_AV_UNSPEC to indicate that either format is sup‐
433 portable, or may return another AV type to indicate the optimal AV type
434 supported by this domain.
435 Memory Registration Mode (mr_mode)
437 Defines memory registration specific mode bits used with this domain.
438 Full details on MR mode options are available in fi_mr(3). The follow‐
439 ing values may be specified.
440 FI_MR_ALLOCATED
442 Indicates that memory registration occurs on allocated data buf‐
443 fers, and physical pages must back all virtual addresses being
444 registered.
445 FI_MR_ENDPOINT
447 Memory registration occurs at the endpoint level, rather than
448 domain.
449 FI_MR_LOCAL
451 The provider is optimized around having applications register
452 memory for locally accessed data buffers. Data buffers used in
453 send and receive operations and as the source buffer for RMA and
454 atomic operations must be registered by the application for ac‐
455 cess domains opened with this capability.
456 FI_MR_MMU_NOTIFY
458 Indicates that the application is responsible for notifying the
459 provider when the page tables referencing a registered memory
460 region may have been updated.
461 FI_MR_PROV_KEY
463 Memory registration keys are selected and returned by the
464 provider.
465 FI_MR_RAW
467 The provider requires additional setup as part of their memory
468 registration process. This mode is required by providers that
469 use a memory key that is larger than 64-bits.
470 FI_MR_RMA_EVENT
472 Indicates that the memory regions associated with completion
473 counters must be explicitly enabled after being bound to any
474 counter.
475 FI_MR_UNSPEC
477 Defined for compatibility -- library versions 1.4 and earlier.
478 Setting mr_mode to 0 indicates that FI_MR_BASIC or FI_MR_SCAL‐
479 ABLE are requested and supported.
480 FI_MR_VIRT_ADDR
482 Registered memory regions are referenced by peers using the vir‐
483 tual address of the registered memory region, rather than a
484 0-based offset.
485 FI_MR_BASIC
487 Defined for compatibility -- library versions 1.4 and earlier.
488 Only basic memory registration operations are requested or sup‐
489 ported. This mode is equivalent to the FI_MR_VIRT_ADDR,
490 FI_MR_ALLOCATED, and FI_MR_PROV_KEY flags being set in later li‐
491 brary versions. This flag may not be used in conjunction with
492 other mr_mode bits.
493 FI_MR_SCALABLE
495 Defined for compatibility -- library versions 1.4 and earlier.
496 Only scalable memory registration operations are requested or
497 supported. Scalable registration uses offset based addressing,
498 with application selectable memory keys. For library versions
499 1.5 and later, this is the default if no mr_mode bits are set.
500 This flag may not be used in conjunction with other mr_mode
501 bits.
502 Buffers used in data transfer operations may require notifying the
504 provider of their use before a data transfer can occur. The mr_mode
505 field indicates the type of memory registration that is required, and
506 when registration is necessary. Applications that require the use of a
507 specific registration mode should set the domain attribute mr_mode to
508 the necessary value when calling fi_getinfo. The value FI_MR_UNSPEC
509 may be used to indicate support for any registration mode.
510 MR Key Size (mr_key_size)
512 Size of the memory region remote access key, in bytes. Applications
513 that request their own MR key must select a value within the range
514 specified by this value. Key sizes larger than 8 bytes require using
515 the FI_RAW_KEY mode bit.
516 CQ Data Size (cq_data_size)
518 Applications may include a small message with a data transfer that is
519 placed directly into a remote completion queue as part of a completion
520 event. This is referred to as remote CQ data (sometimes referred to as
521 immediate data). This field indicates the number of bytes that the
522 provider supports for remote CQ data. If supported (non-zero value is
523 returned), the minimum size of remote CQ data must be at least 4-bytes.
524 Completion Queue Count (cq_cnt)
526 The optimal number of completion queues supported by the domain, rela‐
527 tive to any specified or default CQ attributes. The cq_cnt value may
528 be a fixed value of the maximum number of CQs supported by the underly‐
529 ing hardware, or may be a dynamic value, based on the default at‐
530 tributes of an allocated CQ, such as the CQ size and data format.
531 Endpoint Count (ep_cnt)
533 The total number of endpoints supported by the domain, relative to any
534 specified or default endpoint attributes. The ep_cnt value may be a
535 fixed value of the maximum number of endpoints supported by the under‐
536 lying hardware, or may be a dynamic value, based on the default at‐
537 tributes of an allocated endpoint, such as the endpoint capabilities
538 and size. The endpoint count is the number of addressable endpoints
539 supported by the provider.
540 Transmit Context Count (tx_ctx_cnt)
542 The number of outbound command queues optimally supported by the
543 provider. For a low-level provider, this represents the number of com‐
544 mand queues to the hardware and/or the number of parallel transmit en‐
545 gines effectively supported by the hardware and caches. Applications
546 which allocate more transmit contexts than this value will end up shar‐
547 ing underlying resources. By default, there is a single transmit con‐
548 text associated with each endpoint, but in an advanced usage model, an
549 endpoint may be configured with multiple transmit contexts.
550 Receive Context Count (rx_ctx_cnt)
552 The number of inbound processing queues optimally supported by the
553 provider. For a low-level provider, this represents the number hard‐
554 ware queues that can be effectively utilized for processing incoming
555 packets. Applications which allocate more receive contexts than this
556 value will end up sharing underlying resources. By default, a single
557 receive context is associated with each endpoint, but in an advanced
558 usage model, an endpoint may be configured with multiple receive con‐
559 texts.
560 Maximum Endpoint Transmit Context (max_ep_tx_ctx)
562 The maximum number of transmit contexts that may be associated with an
563 endpoint.
564 Maximum Endpoint Receive Context (max_ep_rx_ctx)
566 The maximum number of receive contexts that may be associated with an
567 endpoint.
568 Maximum Sharing of Transmit Context (max_ep_stx_ctx)
570 The maximum number of endpoints that may be associated with a shared
571 transmit context.
572 Maximum Sharing of Receive Context (max_ep_srx_ctx)
574 The maximum number of endpoints that may be associated with a shared
575 receive context.
576 Counter Count (cntr_cnt)
578 The optimal number of completion counters supported by the domain. The
579 cq_cnt value may be a fixed value of the maximum number of counters
580 supported by the underlying hardware, or may be a dynamic value, based
581 on the default attributes of the domain.
582 MR IOV Limit (mr_iov_limit)
584 This is the maximum number of IO vectors (scatter-gather elements) that
585 a single memory registration operation may reference.
586 Capabilities (caps)
588 Domain level capabilities. Domain capabilities indicate domain level
589 features that are supported by the provider.
590 FI_LOCAL_COMM
592 At a conceptual level, this field indicates that the underlying
593 device supports loopback communication. More specifically, this
594 field indicates that an endpoint may communicate with other end‐
595 points that are allocated from the same underlying named domain.
596 If this field is not set, an application may need to use an al‐
597 ternate domain or mechanism (e.g. shared memory) to communicate
598 with peers that execute on the same node.
599 FI_REMOTE_COMM
601 This field indicates that the underlying provider supports com‐
602 munication with nodes that are reachable over the network. If
603 this field is not set, then the provider only supports communi‐
604 cation between processes that execute on the same node -- a
605 shared memory provider, for example.
606 FI_SHARED_AV
608 Indicates that the domain supports the ability to share address
609 vectors among multiple processes using the named address vector
610 feature.
611 See fi_getinfo(3) for a discussion on primary versus secondary capabil‐
613 ities. All domain capabilities are considered secondary capabilities.
614 mode
616 The operational mode bit related to using the domain.
617 FI_RESTRICTED_COMP
619 This bit indicates that the domain limits completion queues and
620 counters to only be used with endpoints, transmit contexts, and
621 receive contexts that have the same set of capability flags.
622 Default authorization key (auth_key)
624 The default authorization key to associate with endpoint and memory
625 registrations created within the domain. This field is ignored unless
626 the fabric is opened with API version 1.5 or greater.
627 Default authorization key length (auth_key_size)
629 The length in bytes of the default authorization key for the domain.
630 If set to 0, then no authorization key will be associated with end‐
631 points and memory registrations created within the domain unless speci‐
632 fied in the endpoint or memory registration attributes. This field is
633 ignored unless the fabric is opened with API version 1.5 or greater.
634 Max Error Data Size (max_err_data)
636 : The maximum amount of error data, in bytes, that may be returned as
637 part of a completion or event queue error. This value corresponds to
638 the err_data_size field in struct fi_cq_err_entry and struct
639 fi_eq_err_entry.
640 Memory Regions Count (mr_cnt)
642 The optimal number of memory regions supported by the domain, or end‐
643 point if the mr_mode FI_MR_ENDPOINT bit has been set. The mr_cnt value
644 may be a fixed value of the maximum number of MRs supported by the un‐
645 derlying hardware, or may be a dynamic value, based on the default at‐
646 tributes of the domain, such as the supported memory registration
647 modes. Applications can set the mr_cnt on input to fi_getinfo, in or‐
648 der to indicate their memory registration requirements. Doing so may
649 allow the provider to optimize any memory registration cache or lookup
650 tables.
651 Traffic Class (tclass)
653 This specifies the default traffic class that will be associated any
654 endpoints created within the domain. See [fi_endpoint(3)](fi_end‐
655 point.3.html for additional information.
656
658 Returns 0 on success. On error, a negative value corresponding to fab‐
659 ric errno is returned. Fabric errno values are defined in rdma/fi_er‐
660 rno.h.
661
663 Users should call fi_close to release all resources allocated to the
664 fabric domain.
665 The following fabric resources are associated with domains: active end‐
667 points, memory regions, completion event queues, and address vectors.
668 Domain attributes reflect the limitations and capabilities of the un‐
670 derlying hardware and/or software provider. They do not reflect system
671 limitations, such as the number of physical pages that an application
672 may pin or number of file descriptors that the application may open.
673 As a result, the reported maximums may not be achievable, even on a
674 lightly loaded systems, without an administrator configuring system re‐
675 sources appropriately for the installed provider(s).
676
678 fi_getinfo(3), fi_endpoint(3), fi_av(3), fi_ep(3), fi_eq(3), fi_mr(3)
679
681 OpenFabrics.
682Libfabric Programmer's Manual 2020-02-07 fi_domain(3)