1fi_endpoint(3) Libfabric v1.8.0 fi_endpoint(3)
2
6 fi_endpoint - Fabric endpoint operations
7 fi_endpoint / fi_scalable_ep / fi_passive_ep / fi_close
9 Allocate or close an endpoint.
10 fi_ep_bind
12 Associate an endpoint with hardware resources, such as event
13 queues, completion queues, counters, address vectors, or shared
14 transmit/receive contexts.
15 fi_scalable_ep_bind
17 Associate a scalable endpoint with an address vector
18 fi_pep_bind
20 Associate a passive endpoint with an event queue
21 fi_enable
23 Transitions an active endpoint into an enabled state.
24 fi_cancel
26 Cancel a pending asynchronous data transfer
27 fi_ep_alias
29 Create an alias to the endpoint
30 fi_control
32 Control endpoint operation.
33 fi_getopt / fi_setopt
35 Get or set endpoint options.
36 fi_rx_context / fi_tx_context / fi_srx_context / fi_stx_context
38 Open a transmit or receive context.
39 fi_rx_size_left / fi_tx_size_left (DEPRECATED)
41 Query the lower bound on how many RX/TX operations may be posted
42 without an operation returning -FI_EAGAIN. This functions have
43 been deprecated and will be removed in a future version of the
44 library.
45
47 #include <rdma/fabric.h>
48 #include <rdma/fi_endpoint.h>
50 int fi_endpoint(struct fid_domain *domain, struct fi_info *info,
52 struct fid_ep **ep, void *context);
53 int fi_scalable_ep(struct fid_domain *domain, struct fi_info *info,
55 struct fid_ep **sep, void *context);
56 int fi_passive_ep(struct fi_fabric *fabric, struct fi_info *info,
58 struct fid_pep **pep, void *context);
59 int fi_tx_context(struct fid_ep *sep, int index,
61 struct fi_tx_attr *attr, struct fid_ep **tx_ep,
62 void *context);
63 int fi_rx_context(struct fid_ep *sep, int index,
65 struct fi_rx_attr *attr, struct fid_ep **rx_ep,
66 void *context);
67 int fi_stx_context(struct fid_domain *domain,
69 struct fi_tx_attr *attr, struct fid_stx **stx,
70 void *context);
71 int fi_srx_context(struct fid_domain *domain,
73 struct fi_rx_attr *attr, struct fid_ep **rx_ep,
74 void *context);
75 int fi_close(struct fid *ep);
77 int fi_ep_bind(struct fid_ep *ep, struct fid *fid, uint64_t flags);
79 int fi_scalable_ep_bind(struct fid_ep *sep, struct fid *fid, uint64_t flags);
81 int fi_pep_bind(struct fid_pep *pep, struct fid *fid, uint64_t flags);
83 int fi_enable(struct fid_ep *ep);
85 int fi_cancel(struct fid_ep *ep, void *context);
87 int fi_ep_alias(struct fid_ep *ep, struct fid_ep **alias_ep, uint64_t flags);
89 int fi_control(struct fid *ep, int command, void *arg);
91 int fi_getopt(struct fid *ep, int level, int optname,
93 void *optval, size_t *optlen);
94 int fi_setopt(struct fid *ep, int level, int optname,
96 const void *optval, size_t optlen);
97 DEPRECATED ssize_t fi_rx_size_left(struct fid_ep *ep);
99 DEPRECATED ssize_t fi_tx_size_left(struct fid_ep *ep);
101
103 fid On creation, specifies a fabric or access domain. On bind,
104 identifies the event queue, completion queue, counter, or ad‐
105 dress vector to bind to the endpoint. In other cases, it's a
106 fabric identifier of an associated resource.
107 info Details about the fabric interface endpoint to be opened, ob‐
109 tained from fi_getinfo.
110 ep A fabric endpoint.
112 sep A scalable fabric endpoint.
114 pep A passive fabric endpoint.
116 context
118 Context associated with the endpoint or asynchronous operation.
119 index Index to retrieve a specific transmit/receive context.
121 attr Transmit or receive context attributes.
123 flags Additional flags to apply to the operation.
125 command
127 Command of control operation to perform on endpoint.
128 arg Optional control argument.
130 level Protocol level at which the desired option resides.
132 optname
134 The protocol option to read or set.
135 optval The option value that was read or to set.
137 optlen The size of the optval buffer.
139
141 Endpoints are transport level communication portals. There are two
142 types of endpoints: active and passive. Passive endpoints belong to a
143 fabric domain and are most often used to listen for incoming connection
144 requests. However, a passive endpoint may be used to reserve a fabric
145 address that can be granted to an active endpoint. Active endpoints
146 belong to access domains and can perform data transfers.
147 Active endpoints may be connection-oriented or connectionless, and may
149 provide data reliability. The data transfer interfaces -- messages
150 (fi_msg), tagged messages (fi_tagged), RMA (fi_rma), and atomics
151 (fi_atomic) -- are associated with active endpoints. In basic configu‐
152 rations, an active endpoint has transmit and receive queues. In gener‐
153 al, operations that generate traffic on the fabric are posted to the
154 transmit queue. This includes all RMA and atomic operations, along
155 with sent messages and sent tagged messages. Operations that post buf‐
156 fers for receiving incoming data are submitted to the receive queue.
157 Active endpoints are created in the disabled state. They must transi‐
159 tion into an enabled state before accepting data transfer operations,
160 including posting of receive buffers. The fi_enable call is used to
161 transition an active endpoint into an enabled state. The fi_connect
162 and fi_accept calls will also transition an endpoint into the enabled
163 state, if it is not already active.
164 In order to transition an endpoint into an enabled state, it must be
166 bound to one or more fabric resources. An endpoint that will generate
167 asynchronous completions, either through data transfer operations or
168 communication establishment events, must be bound to the appropriate
169 completion queues or event queues, respectively, before being enabled.
170 Additionally, endpoints that use manual progress must be associated
171 with relevant completion queues or event queues in order to drive
172 progress. For endpoints that are only used as the target of RMA or
173 atomic operations, this means binding the endpoint to a completion
174 queue associated with receive processing. Unconnected endpoints must
175 be bound to an address vector.
176 Once an endpoint has been activated, it may be associated with an ad‐
178 dress vector. Receive buffers may be posted to it and calls may be
179 made to connection establishment routines. Connectionless endpoints
180 may also perform data transfers.
181 The behavior of an endpoint may be adjusted by setting its control data
183 and protocol options. This allows the underlying provider to redirect
184 function calls to implementations optimized to meet the desired appli‐
185 cation behavior.
186 If an endpoint experiences a critical error, it will transition back
188 into a disabled state. Critical errors are reported through the event
189 queue associated with the EP. In certain cases, a disabled endpoint
190 may be re-enabled. The ability to transition back into an enabled
191 state is provider specific and depends on the type of error that the
192 endpoint experienced. When an endpoint is disabled as a result of a
193 critical error, all pending operations are discarded.
194 fi_endpoint / fi_passive_ep / fi_scalable_ep
196 fi_endpoint allocates a new active endpoint. fi_passive_ep allocates a
197 new passive endpoint. fi_scalable_ep allocates a scalable endpoint.
198 The properties and behavior of the endpoint are defined based on the
199 provided struct fi_info. See fi_getinfo for additional details on
200 fi_info. fi_info flags that control the operation of an endpoint are
201 defined below. See section SCALABLE ENDPOINTS.
202 If an active endpoint is allocated in order to accept a connection re‐
204 quest, the fi_info parameter must be the same as the fi_info structure
205 provided with the connection request (FI_CONNREQ) event.
206 An active endpoint may acquire the properties of a passive endpoint by
208 setting the fi_info handle field to the passive endpoint fabric de‐
209 scriptor. This is useful for applications that need to reserve the
210 fabric address of an endpoint prior to knowing if the endpoint will be
211 used on the active or passive side of a connection. For example, this
212 feature is useful for simulating socket semantics. Once an active end‐
213 point acquires the properties of a passive endpoint, the passive end‐
214 point is no longer bound to any fabric resources and must no longer be
215 used. The user is expected to close the passive endpoint after opening
216 the active endpoint in order to free up any lingering resources that
217 had been used.
218 fi_close
220 Closes an endpoint and release all resources associated with it.
221 When closing a scalable endpoint, there must be no opened transmit con‐
223 texts, or receive contexts associated with the scalable endpoint. If
224 resources are still associated with the scalable endpoint when attempt‐
225 ing to close, the call will return -FI_EBUSY.
226 Outstanding operations posted to the endpoint when fi_close is called
228 will be discarded. Discarded operations will silently be dropped, with
229 no completions reported. Additionally, a provider may discard previ‐
230 ously completed operations from the associated completion queue(s).
231 The behavior to discard completed operations is provider specific.
232 fi_ep_bind
234 fi_ep_bind is used to associate an endpoint with other allocated re‐
235 sources, such as completion queues, counters, address vectors, event
236 queues, shared contexts, and memory regions. The type of objects that
237 must be bound with an endpoint depend on the endpoint type and its con‐
238 figuration.
239 Passive endpoints must be bound with an EQ that supports connection
241 management events. Connectionless endpoints must be bound to a single
242 address vector. If an endpoint is using a shared transmit and/or re‐
243 ceive context, the shared contexts must be bound to the endpoint. CQs,
244 counters, AV, and shared contexts must be bound to endpoints before
245 they are enabled either explicitly or implicitly.
246 An endpoint must be bound with CQs capable of reporting completions for
248 any asynchronous operation initiated on the endpoint. For example, if
249 the endpoint supports any outbound transfers (sends, RMA, atomics,
250 etc.), then it must be bound to a completion queue that can report
251 transmit completions. This is true even if the endpoint is configured
252 to suppress successful completions, in order that operations that com‐
253 plete in error may be reported to the user.
254 An active endpoint may direct asynchronous completions to different
256 CQs, based on the type of operation. This is specified using
257 fi_ep_bind flags. The following flags may be OR'ed together when bind‐
258 ing an endpoint to a completion domain CQ.
259 FI_TRANSMIT
261 Directs the completion of outbound data transfer requests to the
262 specified completion queue. This includes send message, RMA,
263 and atomic operations.
264 FI_RECV
266 Directs the notification of inbound data transfers to the speci‐
267 fied completion queue. This includes received messages. This
268 binding automatically includes FI_REMOTE_WRITE, if applicable to
269 the endpoint.
270 FI_SELECTIVE_COMPLETION
272 By default, data transfer operations write CQ completion entries
273 into the associated completion queue after they have successful‐
274 ly completed. Applications can use this bind flag to selective‐
275 ly enable when completions are generated. If FI_SELECTIVE_COM‐
276 PLETION is specified, data transfer operations will not generate
277 CQ entries for successful completions unless FI_COMPLETION is
278 set as an operational flag for the given operation. Operations
279 that fail asynchronously will still generate completions, even
280 if a completion is not requested. FI_SELECTIVE_COMPLETION must
281 be OR'ed with FI_TRANSMIT and/or FI_RECV flags.
282 When FI_SELECTIVE_COMPLETION is set, the user must determine when a re‐
284 quest that does NOT have FI_COMPLETION set has completed indirectly,
285 usually based on the completion of a subsequent operation or by using
286 completion counters. Use of this flag may improve performance by al‐
287 lowing the provider to avoid writing a CQ completion entry for every
288 operation.
289 See Notes section below for additional information on how this flag in‐
291 teracts with the FI_CONTEXT and FI_CONTEXT2 mode bits.
292 An endpoint may optionally be bound to a completion counter. Associat‐
294 ing an endpoint with a counter is in addition to binding the EP with a
295 CQ. When binding an endpoint to a counter, the following flags may be
296 specified.
297 FI_SEND
299 Increments the specified counter whenever a message transfer
300 initiated over the endpoint has completed successfully or in er‐
301 ror. Sent messages include both tagged and normal message oper‐
302 ations.
303 FI_RECV
305 Increments the specified counter whenever a message is received
306 over the endpoint. Received messages include both tagged and
307 normal message operations.
308 FI_READ
310 Increments the specified counter whenever an RMA read, atomic
311 fetch, or atomic compare operation initiated from the endpoint
312 has completed successfully or in error.
313 FI_WRITE
315 Increments the specified counter whenever an RMA write or base
316 atomic operation initiated from the endpoint has completed suc‐
317 cessfully or in error.
318 FI_REMOTE_READ
320 Increments the specified counter whenever an RMA read, atomic
321 fetch, or atomic compare operation is initiated from a remote
322 endpoint that targets the given endpoint. Use of this flag re‐
323 quires that the endpoint be created using FI_RMA_EVENT.
324 FI_REMOTE_WRITE
326 Increments the specified counter whenever an RMA write or base
327 atomic operation is initiated from a remote endpoint that tar‐
328 gets the given endpoint. Use of this flag requires that the
329 endpoint be created using FI_RMA_EVENT.
330 An endpoint may only be bound to a single CQ or counter for a given
332 type of operation. For example, a EP may not bind to two counters both
333 using FI_WRITE. Furthermore, providers may limit CQ and counter bind‐
334 ings to endpoints of the same endpoint type (DGRAM, MSG, RDM, etc.).
335 fi_scalable_ep_bind
337 fi_scalable_ep_bind is used to associate a scalable endpoint with an
338 address vector. See section on SCALABLE ENDPOINTS. A scalable end‐
339 point has a single transport level address and can support multiple
340 transmit and receive contexts. The transmit and receive contexts share
341 the transport-level address. Address vectors that are bound to scal‐
342 able endpoints are implicitly bound to any transmit or receive contexts
343 created using the scalable endpoint.
344 fi_enable
346 This call transitions the endpoint into an enabled state. An endpoint
347 must be enabled before it may be used to perform data transfers. En‐
348 abling an endpoint typically results in hardware resources being as‐
349 signed to it. Endpoints making use of completion queues, counters,
350 event queues, and/or address vectors must be bound to them before being
351 enabled.
352 Calling connect or accept on an endpoint will implicitly enable an end‐
354 point if it has not already been enabled.
355 fi_enable may also be used to re-enable an endpoint that has been dis‐
357 abled as a result of experiencing a critical error. Applications
358 should check the return value from fi_enable to see if a disabled end‐
359 point has successfully be re-enabled.
360 fi_cancel
362 fi_cancel attempts to cancel an outstanding asynchronous operation.
363 Canceling an operation causes the fabric provider to search for the op‐
364 eration and, if it is still pending, complete it as having been can‐
365 celed. An error queue entry will be available in the associated error
366 queue with error code FI_ECANCELED. On the other hand, if the opera‐
367 tion completed before the call to fi_cancel, then the completion status
368 of that operation will be available in the associated completion queue.
369 No specific entry related to fi_cancel itself will be posted.
370 Cancel uses the context parameter associated with an operation to iden‐
372 tify the request to cancel. Operations posted without a valid context
373 parameter -- either no context parameter is specified or the context
374 value was ignored by the provider -- cannot be canceled. If multiple
375 outstanding operations match the context parameter, only one will be
376 canceled. In this case, the operation which is canceled is provider
377 specific. The cancel operation is asynchronous, but will complete
378 within a bounded period of time.
379 fi_ep_alias
381 This call creates an alias to the specified endpoint. Conceptually, an
382 endpoint alias provides an alternate software path from the application
383 to the underlying provider hardware. An alias EP differs from its par‐
384 ent endpoint only by its default data transfer flags. For example, an
385 alias EP may be configured to use a different completion mode. By de‐
386 fault, an alias EP inherits the same data transfer flags as the parent
387 endpoint. An application can use fi_control to modify the alias EP op‐
388 erational flags.
389 When allocating an alias, an application may configure either the
391 transmit or receive operational flags. This avoids needing a separate
392 call to fi_control to set those flags. The flags passed to fi_ep_alias
393 must include FI_TRANSMIT or FI_RECV (not both) with other operational
394 flags OR'ed in. This will override the transmit or receive flags, re‐
395 spectively, for operations posted through the alias endpoint. All al‐
396 located aliases must be closed for the underlying endpoint to be re‐
397 leased.
398 fi_control
400 The control operation is used to adjust the default behavior of an end‐
401 point. It allows the underlying provider to redirect function calls to
402 implementations optimized to meet the desired application behavior. As
403 a result, calls to fi_ep_control must be serialized against all other
404 calls to an endpoint.
405 The base operation of an endpoint is selected during creation using
407 struct fi_info. The following control commands and arguments may be
408 assigned to an endpoint.
409 **FI_GETOPSFLAG -- uint64_t *flags**
411 Used to retrieve the current value of flags associated with the
412 data transfer operations initiated on the endpoint. The control
413 argument must include FI_TRANSMIT or FI_RECV (not both) flags to
414 indicate the type of data transfer flags to be returned. See
415 below for a list of control flags.
416 **FI_SETOPSFLAG -- uint64_t *flags**
418 Used to change the data transfer operation flags associated with
419 an endpoint. The control argument must include FI_TRANSMIT or
420 FI_RECV (not both) to indicate the type of data transfer that
421 the flags should apply to, with other flags OR'ed in. The given
422 flags will override the previous transmit and receive attributes
423 that were set when the endpoint was created. Valid control
424 flags are defined below.
425 **FI_BACKLOG - int *value**
427 This option only applies to passive endpoints. It is used to
428 set the connection request backlog for listening endpoints.
429 FI_GETWAIT (void **)
431 This command allows the user to retrieve the file descriptor as‐
432 sociated with a socket endpoint. The fi_control arg parameter
433 should be an address where a pointer to the returned file de‐
434 scriptor will be written. See fi_eq.3 for addition details us‐
435 ing fi_control with FI_GETWAIT. The file descriptor may be used
436 for notification that the endpoint is ready to send or receive
437 data.
438 fi_getopt / fi_setopt
440 Endpoint protocol operations may be retrieved using fi_getopt or set
441 using fi_setopt. Applications specify the level that a desired option
442 exists, identify the option, and provide input/output buffers to get or
443 set the option. fi_setopt provides an application a way to adjust
444 low-level protocol and implementation specific details of an endpoint.
445 The following option levels and option names and parameters are de‐
447 fined.
448 FI_OPT_ENDPOINT · .RS 2
450 FI_OPT_MIN_MULTI_RECV - size_t
452 Defines the minimum receive buffer space available when the re‐
453 ceive buffer is released by the provider (see FI_MULTI_RECV).
454 Modifying this value is only guaranteed to set the minimum buf‐
455 fer space needed on receives posted after the value has been
456 changed. It is recommended that applications that want to over‐
457 ride the default MIN_MULTI_RECV value set this option before en‐
458 abling the corresponding endpoint.
459 · .RS 2
460 FI_OPT_CM_DATA_SIZE - size_t
462 Defines the size of available space in CM messages for user-de‐
463 fined data. This value limits the amount of data that applica‐
464 tions can exchange between peer endpoints using the fi_connect,
465 fi_accept, and fi_reject operations. The size returned is de‐
466 pendent upon the properties of the endpoint, except in the case
467 of passive endpoints, in which the size reflects the maximum
468 size of the data that may be present as part of a connection re‐
469 quest event. This option is read only.
470 · .RS 2
471 FI_OPT_BUFFERED_LIMIT - size_t
473 Defines the maximum size of a buffered message that will be re‐
474 ported to users as part of a receive completion when the
475 FI_BUFFERED_RECV mode is enabled on an endpoint.
476 fi_getopt() will return the currently configured threshold, or the
478 provider's default threshold if one has not be set by the application.
479 fi_setopt() allows an application to configure the threshold. If the
480 provider cannot support the requested threshold, it will fail the
481 fi_setopt() call with FI_EMSGSIZE. Calling fi_setopt() with the
482 threshold set to SIZE_MAX will set the threshold to the maximum sup‐
483 ported by the provider. fi_getopt() can then be used to retrieve the
484 set size.
485 In most cases, the sending and receiving endpoints must be configured
487 to use the same threshold value, and the threshold must be set prior to
488 enabling the endpoint. · .RS 2
489 FI_OPT_BUFFERED_MIN - size_t
491 Defines the minimum size of a buffered message that will be re‐
492 ported. Applications would set this to a size that's big enough
493 to decide whether to discard or claim a buffered receive or when
494 to claim a buffered receive on getting a buffered receive com‐
495 pletion. The value is typically used by a provider when sending
496 a rendezvous protocol request where it would send atleast
497 FI_OPT_BUFFERED_MIN bytes of application data along with it. A
498 smaller sized renedezvous protocol message usually results in
499 better latency for the overall transfer of a large message.
500 fi_rx_size_left (DEPRECATED)
502 This function has been deprecated and will be removed in a future ver‐
503 sion of the library. It may not be supported by all providers.
504 The fi_rx_size_left call returns a lower bound on the number of receive
506 operations that may be posted to the given endpoint without that opera‐
507 tion returning -FI_EAGAIN. Depending on the specific details of the
508 subsequently posted receive operations (e.g., number of iov entries,
509 which receive function is called, etc.), it may be possible to post
510 more receive operations than originally indicated by fi_rx_size_left.
511 fi_tx_size_left (DEPRECATED)
513 This function has been deprecated and will be removed in a future ver‐
514 sion of the library. It may not be supported by all providers.
515 The fi_tx_size_left call returns a lower bound on the number of trans‐
517 mit operations that may be posted to the given endpoint without that
518 operation returning -FI_EAGAIN. Depending on the specific details of
519 the subsequently posted transmit operations (e.g., number of iov en‐
520 tries, which transmit function is called, etc.), it may be possible to
521 post more transmit operations than originally indicated by
522 fi_tx_size_left.
523
525 The fi_ep_attr structure defines the set of attributes associated with
526 an endpoint. Endpoint attributes may be further refined using the
527 transmit and receive context attributes as shown below.
528 struct fi_ep_attr {
530 enum fi_ep_type type;
531 uint32_t protocol;
532 uint32_t protocol_version;
533 size_t max_msg_size;
534 size_t msg_prefix_size;
535 size_t max_order_raw_size;
536 size_t max_order_war_size;
537 size_t max_order_waw_size;
538 uint64_t mem_tag_format;
539 size_t tx_ctx_cnt;
540 size_t rx_ctx_cnt;
541 size_t auth_key_size;
542 uint8_t *auth_key;
543 };
544 type - Endpoint Type
546 If specified, indicates the type of fabric interface communication de‐
547 sired. Supported types are:
548 FI_EP_UNSPEC
550 The type of endpoint is not specified. This is usually provided
551 as input, with other attributes of the endpoint or the provider
552 selecting the type.
553 FI_EP_MSG
555 Provides a reliable, connection-oriented data transfer service
556 with flow control that maintains message boundaries.
557 FI_EP_DGRAM
559 Supports a connectionless, unreliable datagram communication.
560 Message boundaries are maintained, but the maximum message size
561 may be limited to the fabric MTU. Flow control is not guaran‐
562 teed.
563 FI_EP_RDM
565 Reliable datagram message. Provides a reliable, unconnected da‐
566 ta transfer service with flow control that maintains message
567 boundaries.
568 FI_EP_SOCK_STREAM
570 Data streaming endpoint with TCP socket-like semantics. Pro‐
571 vides a reliable, connection-oriented data transfer service that
572 does not maintain message boundaries. FI_EP_SOCK_STREAM is most
573 useful for applications designed around using TCP sockets. See
574 the SOCKET ENDPOINT section for additional details and restric‐
575 tions that apply to stream endpoints.
576 FI_EP_SOCK_DGRAM
578 A connectionless, unreliable datagram endpoint with UDP sock‐
579 et-like semantics. FI_EP_SOCK_DGRAM is most useful for applica‐
580 tions designed around using UDP sockets. See the SOCKET END‐
581 POINT section for additional details and restrictions that apply
582 to datagram socket endpoints.
583 Protocol
585 Specifies the low-level end to end protocol employed by the provider.
586 A matching protocol must be used by communicating endpoints to ensure
587 interoperability. The following protocol values are defined. Provider
588 specific protocols are also allowed. Provider specific protocols will
589 be indicated by having the upper bit of the protocol value set to one.
590 FI_PROTO_UNSPEC
592 The protocol is not specified. This is usually provided as in‐
593 put, with other attributes of the socket or the provider select‐
594 ing the actual protocol.
595 FI_PROTO_RDMA_CM_IB_RC
597 The protocol runs over Infiniband reliable-connected queue
598 pairs, using the RDMA CM protocol for connection establishment.
599 FI_PROTO_IWARP
601 The protocol runs over the Internet wide area RDMA protocol
602 transport.
603 FI_PROTO_IB_UD
605 The protocol runs over Infiniband unreliable datagram queue
606 pairs.
607 FI_PROTO_PSMX
609 The protocol is based on an Intel proprietary protocol known as
610 PSM, performance scaled messaging. PSMX is an extended version
611 of the PSM protocol to support the libfabric interfaces.
612 FI_PROTO_UDP
614 The protocol sends and receives UDP datagrams. For example, an
615 endpoint using FI_PROTO_UDP will be able to communicate with a
616 remote peer that is using Berkeley SOCK_DGRAM sockets using IP‐
617 PROTO_UDP.
618 FI_PROTO_SOCK_TCP
620 The protocol is layered over TCP packets.
621 FI_PROTO_IWARP_RDM
623 Reliable-datagram protocol implemented over iWarp reliable-con‐
624 nected queue pairs.
625 FI_PROTO_IB_RDM
627 Reliable-datagram protocol implemented over InfiniBand reli‐
628 able-connected queue pairs.
629 FI_PROTO_GNI
631 Protocol runs over Cray GNI low-level interface.
632 FI_PROTO_RXM
634 Reliable-datagram protocol implemented over message endpoints.
635 RXM is a libfabric utility component that adds RDM endpoint se‐
636 mantics over MSG endpoint semantics.
637 FI_PROTO_RXD
639 Reliable-datagram protocol implemented over datagram endpoints.
640 RXD is a libfabric utility component that adds RDM endpoint se‐
641 mantics over DGRAM endpoint semantics.
642 FI_PROTO_NETWORKDIRECT
644 Protocol runs over Microsoft NetworkDirect service provider in‐
645 terface. This adds reliable-datagram semantics over the Net‐
646 workDirect connection- oriented endpoint semantics.
647 FI_PROTO_PSMX2
649 The protocol is based on an Intel proprietary protocol known as
650 PSM2, performance scaled messaging version 2. PSMX2 is an ex‐
651 tended version of the PSM2 protocol to support the libfabric in‐
652 terfaces.
653 protocol_version - Protocol Version
655 Identifies which version of the protocol is employed by the provider.
656 The protocol version allows providers to extend an existing protocol,
657 by adding support for additional features or functionality for example,
658 in a backward compatible manner. Providers that support different ver‐
659 sions of the same protocol should inter-operate, but only when using
660 the capabilities defined for the lesser version.
661 max_msg_size - Max Message Size
663 Defines the maximum size for an application data transfer as a single
664 operation.
665 msg_prefix_size - Message Prefix Size
667 Specifies the size of any required message prefix buffer space. This
668 field will be 0 unless the FI_MSG_PREFIX mode is enabled. If msg_pre‐
669 fix_size is > 0 the specified value will be a multiple of 8-bytes.
670 Max RMA Ordered Size
672 The maximum ordered size specifies the delivery order of transport data
673 into target memory for RMA and atomic operations. Data ordering is
674 separate, but dependent on message ordering (defined below). Data or‐
675 dering is unspecified where message order is not defined.
676 Data ordering refers to the access of target memory by subsequent oper‐
678 ations. When back to back RMA read or write operations access the same
679 registered memory location, data ordering indicates whether the second
680 operation reads or writes the target memory after the first operation
681 has completed. Because RMA ordering applies between two operations,
682 and not within a single data transfer, ordering is defined per byte-ad‐
683 dressable memory location. I.e. ordering specifies whether location X
684 is accessed by the second operation after the first operation. Nothing
685 is implied about the completion of the first operation before the sec‐
686 ond operation is initiated.
687 In order to support large data transfers being broken into multiple
689 packets and sent using multiple paths through the fabric, data ordering
690 may be limited to transfers of a specific size or less. Providers
691 specify when data ordering is maintained through the following values.
692 Note that even if data ordering is not maintained, message ordering may
693 be.
694 max_order_raw_size
696 Read after write size. If set, an RMA or atomic read operation
697 issued after an RMA or atomic write operation, both of which are
698 smaller than the size, will be ordered. Where the target memory
699 locations overlap, the RMA or atomic read operation will see the
700 results of the previous RMA or atomic write.
701 max_order_war_size
703 Write after read size. If set, an RMA or atomic write operation
704 issued after an RMA or atomic read operation, both of which are
705 smaller than the size, will be ordered. The RMA or atomic read
706 operation will see the initial value of the target memory loca‐
707 tion before a subsequent RMA or atomic write updates the value.
708 max_order_waw_size
710 Write after write size. If set, an RMA or atomic write opera‐
711 tion issued after an RMA or atomic write operation, both of
712 which are smaller than the size, will be ordered. The target
713 memory location will reflect the results of the second RMA or
714 atomic write.
715 An order size value of 0 indicates that ordering is not guaranteed. A
717 value of -1 guarantees ordering for any data size.
718 mem_tag_format - Memory Tag Format
720 The memory tag format is a bit array used to convey the number of
721 tagged bits supported by a provider. Additionally, it may be used to
722 divide the bit array into separate fields. The mem_tag_format option‐
723 ally begins with a series of bits set to 0, to signify bits which are
724 ignored by the provider. Following the initial prefix of ignored bits,
725 the array will consist of alternating groups of bits set to all 1's or
726 all 0's. Each group of bits corresponds to a tagged field. The impli‐
727 cation of defining a tagged field is that when a mask is applied to the
728 tagged bit array, all bits belonging to a single field will either be
729 set to 1 or 0, collectively.
730 For example, a mem_tag_format of 0x30FF indicates support for 14 tagged
732 bits, separated into 3 fields. The first field consists of 2-bits, the
733 second field 4-bits, and the final field 8-bits. Valid masks for such
734 a tagged field would be a bitwise OR'ing of zero or more of the follow‐
735 ing values: 0x3000, 0x0F00, and 0x00FF. The provider may not validate
736 the mask provided by the application for performance reasons.
737 By identifying fields within a tag, a provider may be able to optimize
739 their search routines. An application which requests tag fields must
740 provide tag masks that either set all mask bits corresponding to a
741 field to all 0 or all 1. When negotiating tag fields, an application
742 can request a specific number of fields of a given size. A provider
743 must return a tag format that supports the requested number of fields,
744 with each field being at least the size requested, or fail the request.
745 A provider may increase the size of the fields. When reporting comple‐
746 tions (see FI_CQ_FORMAT_TAGGED), it is not guaranteed that the provider
747 would clear out any unsupported tag bits in the tag field of the com‐
748 pletion entry.
749 It is recommended that field sizes be ordered from smallest to largest.
751 A generic, unstructured tag and mask can be achieved by requesting a
752 bit array consisting of alternating 1's and 0's.
753 tx_ctx_cnt - Transmit Context Count
755 Number of transmit contexts to associate with the endpoint. If not
756 specified (0), 1 context will be assigned if the endpoint supports out‐
757 bound transfers. Transmit contexts are independent transmit queues
758 that may be separately configured. Each transmit context may be bound
759 to a separate CQ, and no ordering is defined between contexts. Addi‐
760 tionally, no synchronization is needed when accessing contexts in par‐
761 allel.
762 If the count is set to the value FI_SHARED_CONTEXT, the endpoint will
764 be configured to use a shared transmit context, if supported by the
765 provider. Providers that do not support shared transmit contexts will
766 fail the request.
767 See the scalable endpoint and shared contexts sections for additional
769 details.
770 rx_ctx_cnt - Receive Context Count
772 Number of receive contexts to associate with the endpoint. If not
773 specified, 1 context will be assigned if the endpoint supports inbound
774 transfers. Receive contexts are independent processing queues that may
775 be separately configured. Each receive context may be bound to a sepa‐
776 rate CQ, and no ordering is defined between contexts. Additionally, no
777 synchronization is needed when accessing contexts in parallel.
778 If the count is set to the value FI_SHARED_CONTEXT, the endpoint will
780 be configured to use a shared receive context, if supported by the
781 provider. Providers that do not support shared receive contexts will
782 fail the request.
783 See the scalable endpoint and shared contexts sections for additional
785 details.
786 auth_key_size - Authorization Key Length
788 The length of the authorization key in bytes. This field will be 0 if
789 authorization keys are not available or used. This field is ignored
790 unless the fabric is opened with API version 1.5 or greater.
791 auth_key - Authorization Key
793 If supported by the fabric, an authorization key (a.k.a. job key) to
794 associate with the endpoint. An authorization key is used to limit
795 communication between endpoints. Only peer endpoints that are pro‐
796 grammed to use the same authorization key may communicate. Authoriza‐
797 tion keys are often used to implement job keys, to ensure that process‐
798 es running in different jobs do not accidentally cross traffic. The
799 domain authorization key will be used if auth_key_size is set to 0.
800 This field is ignored unless the fabric is opened with API version 1.5
801 or greater.
802
804 Attributes specific to the transmit capabilities of an endpoint are
805 specified using struct fi_tx_attr.
806 struct fi_tx_attr {
808 uint64_t caps;
809 uint64_t mode;
810 uint64_t op_flags;
811 uint64_t msg_order;
812 uint64_t comp_order;
813 size_t inject_size;
814 size_t size;
815 size_t iov_limit;
816 size_t rma_iov_limit;
817 };
818 caps - Capabilities
820 The requested capabilities of the context. The capabilities must be a
821 subset of those requested of the associated endpoint. See the CAPABIL‐
822 ITIES section of fi_getinfo(3) for capability details. If the caps
823 field is 0 on input to fi_getinfo(3), the caps value from the fi_info
824 structure will be used.
825 mode
827 The operational mode bits of the context. The mode bits will be a sub‐
828 set of those associated with the endpoint. See the MODE section of
829 fi_getinfo(3) for details. A mode value of 0 will be ignored on input
830 to fi_getinfo(3), with the mode value of the fi_info structure used in‐
831 stead. On return from fi_getinfo(3), the mode will be set only to
832 those constraints specific to transmit operations.
833 op_flags - Default transmit operation flags
835 Flags that control the operation of operations submitted against the
836 context. Applicable flags are listed in the Operation Flags section.
837 msg_order - Message Ordering
839 Message ordering refers to the order in which transport layer headers
840 (as viewed by the application) are identified and processed. Relaxed
841 message order enables data transfers to be sent and received out of or‐
842 der, which may improve performance by utilizing multiple paths through
843 the fabric from the initiating endpoint to a target endpoint. Message
844 order applies only between a single source and destination endpoint
845 pair. Ordering between different target endpoints is not defined.
846 Message order is determined using a set of ordering bits. Each set bit
848 indicates that ordering is maintained between data transfers of the
849 specified type. Message order is defined for [read | write | send] op‐
850 erations submitted by an application after [read | write | send] opera‐
851 tions.
852 Message ordering only applies to the end to end transmission of trans‐
854 port headers. Message ordering is necessary, but does not guarantee,
855 the order in which message data is sent or received by the transport
856 layer. Message ordering requires matching ordering semantics on the
857 receiving side of a data transfer operation in order to guarantee that
858 ordering is met.
859 FI_ORDER_NONE
861 No ordering is specified. This value may be used as input in
862 order to obtain the default message order supported by the
863 provider. FI_ORDER_NONE is an alias for the value 0.
864 FI_ORDER_RAR
866 Read after read. If set, RMA and atomic read operations are
867 transmitted in the order submitted relative to other RMA and
868 atomic read operations. If not set, RMA and atomic reads may be
869 transmitted out of order from their submission.
870 FI_ORDER_RAW
872 Read after write. If set, RMA and atomic read operations are
873 transmitted in the order submitted relative to RMA and atomic
874 write operations. If not set, RMA and atomic reads may be
875 transmitted ahead of RMA and atomic writes.
876 FI_ORDER_RAS
878 Read after send. If set, RMA and atomic read operations are
879 transmitted in the order submitted relative to message send op‐
880 erations, including tagged sends. If not set, RMA and atomic
881 reads may be transmitted ahead of sends.
882 FI_ORDER_WAR
884 Write after read. If set, RMA and atomic write operations are
885 transmitted in the order submitted relative to RMA and atomic
886 read operations. If not set, RMA and atomic writes may be
887 transmitted ahead of RMA and atomic reads.
888 FI_ORDER_WAW
890 Write after write. If set, RMA and atomic write operations are
891 transmitted in the order submitted relative to other RMA and
892 atomic write operations. If not set, RMA and atomic writes may
893 be transmitted out of order from their submission.
894 FI_ORDER_WAS
896 Write after send. If set, RMA and atomic write operations are
897 transmitted in the order submitted relative to message send op‐
898 erations, including tagged sends. If not set, RMA and atomic
899 writes may be transmitted ahead of sends.
900 FI_ORDER_SAR
902 Send after read. If set, message send operations, including
903 tagged sends, are transmitted in order submitted relative to RMA
904 and atomic read operations. If not set, message sends may be
905 transmitted ahead of RMA and atomic reads.
906 FI_ORDER_SAW
908 Send after write. If set, message send operations, including
909 tagged sends, are transmitted in order submitted relative to RMA
910 and atomic write operations. If not set, message sends may be
911 transmitted ahead of RMA and atomic writes.
912 FI_ORDER_SAS
914 Send after send. If set, message send operations, including
915 tagged sends, are transmitted in the order submitted relative to
916 other message send. If not set, message sends may be transmit‐
917 ted out of order from their submission.
918 FI_ORDER_RMA_RAR
920 RMA read after read. If set, RMA read operations are transmit‐
921 ted in the order submitted relative to other RMA read opera‐
922 tions. If not set, RMA reads may be transmitted out of order
923 from their submission.
924 FI_ORDER_RMA_RAW
926 RMA read after write. If set, RMA read operations are transmit‐
927 ted in the order submitted relative to RMA write operations. If
928 not set, RMA reads may be transmitted ahead of RMA writes.
929 FI_ORDER_RMA_WAR
931 RMA write after read. If set, RMA write operations are trans‐
932 mitted in the order submitted relative to RMA read operations.
933 If not set, RMA writes may be transmitted ahead of RMA reads.
934 FI_ORDER_RMA_WAW
936 RMA write after write. If set, RMA write operations are trans‐
937 mitted in the order submitted relative to other RMA write opera‐
938 tions. If not set, RMA writes may be transmitted out of order
939 from their submission.
940 FI_ORDER_ATOMIC_RAR
942 Atomic read after read. If set, atomic fetch operations are
943 transmitted in the order submitted relative to other atomic
944 fetch operations. If not set, atomic fetches may be transmitted
945 out of order from their submission.
946 FI_ORDER_ATOMIC_RAW
948 Atomic read after write. If set, atomic fetch operations are
949 transmitted in the order submitted relative to atomic update op‐
950 erations. If not set, atomic fetches may be transmitted ahead
951 of atomic updates.
952 FI_ORDER_ATOMIC_WAR
954 RMA write after read. If set, atomic update operations are
955 transmitted in the order submitted relative to atomic fetch op‐
956 erations. If not set, atomic updates may be transmitted ahead
957 of atomic fetches.
958 FI_ORDER_ATOMIC_WAW
960 RMA write after write. If set, atomic update operations are
961 transmitted in the order submitted relative to other atomic up‐
962 date operations. If not atomic updates may be transmitted out
963 of order from their submission.
964 comp_order - Completion Ordering
966 Completion ordering refers to the order in which completed requests are
967 written into the completion queue. Completion ordering is similar to
968 message order. Relaxed completion order may enable faster reporting of
969 completed transfers, allow acknowledgments to be sent over different
970 fabric paths, and support more sophisticated retry mechanisms. This
971 can result in lower-latency completions, particularly when using uncon‐
972 nected endpoints. Strict completion ordering may require that
973 providers queue completed operations or limit available optimizations.
974 For transmit requests, completion ordering depends on the endpoint com‐
976 munication type. For unreliable communication, completion ordering ap‐
977 plies to all data transfer requests submitted to an endpoint. For re‐
978 liable communication, completion ordering only applies to requests that
979 target a single destination endpoint. Completion ordering of requests
980 that target different endpoints over a reliable transport is not de‐
981 fined.
982 Applications should specify the completion ordering that they support
984 or require. Providers should return the completion order that they ac‐
985 tually provide, with the constraint that the returned ordering is
986 stricter than that specified by the application. Supported completion
987 order values are:
988 FI_ORDER_NONE
990 No ordering is defined for completed operations. Requests sub‐
991 mitted to the transmit context may complete in any order.
992 FI_ORDER_STRICT
994 Requests complete in the order in which they are submitted to
995 the transmit context.
996 inject_size
998 The requested inject operation size (see the FI_INJECT flag) that the
999 context will support. This is the maximum size data transfer that can
1000 be associated with an inject operation (such as fi_inject) or may be
1001 used with the FI_INJECT data transfer flag.
1002 size
1004 The size of the context. The size is specified as the minimum number
1005 of transmit operations that may be posted to the endpoint without the
1006 operation returning -FI_EAGAIN.
1007 iov_limit
1009 This is the maximum number of IO vectors (scatter-gather elements) that
1010 a single posted operation may reference.
1011 rma_iov_limit
1013 This is the maximum number of RMA IO vectors (scatter-gather elements)
1014 that an RMA or atomic operation may reference. The rma_iov_limit cor‐
1015 responds to the rma_iov_count values in RMA and atomic operations. See
1016 struct fi_msg_rma and struct fi_msg_atomic in fi_rma.3 and fi_atomic.3,
1017 for additional details. This limit applies to both the number of RMA
1018 IO vectors that may be specified when initiating an operation from the
1019 local endpoint, as well as the maximum number of IO vectors that may be
1020 carried in a single request from a remote endpoint.
1021
1023 Attributes specific to the receive capabilities of an endpoint are
1024 specified using struct fi_rx_attr.
1025 struct fi_rx_attr {
1027 uint64_t caps;
1028 uint64_t mode;
1029 uint64_t op_flags;
1030 uint64_t msg_order;
1031 uint64_t comp_order;
1032 size_t total_buffered_recv;
1033 size_t size;
1034 size_t iov_limit;
1035 };
1036 caps - Capabilities
1038 The requested capabilities of the context. The capabilities must be a
1039 subset of those requested of the associated endpoint. See the CAPABIL‐
1040 ITIES section if fi_getinfo(3) for capability details. If the caps
1041 field is 0 on input to fi_getinfo(3), the caps value from the fi_info
1042 structure will be used.
1043 mode
1045 The operational mode bits of the context. The mode bits will be a sub‐
1046 set of those associated with the endpoint. See the MODE section of
1047 fi_getinfo(3) for details. A mode value of 0 will be ignored on input
1048 to fi_getinfo(3), with the mode value of the fi_info structure used in‐
1049 stead. On return from fi_getinfo(3), the mode will be set only to
1050 those constraints specific to receive operations.
1051 op_flags - Default receive operation flags
1053 Flags that control the operation of operations submitted against the
1054 context. Applicable flags are listed in the Operation Flags section.
1055 msg_order - Message Ordering
1057 For a description of message ordering, see the msg_order field in the
1058 Transmit Context Attribute section. Receive context message ordering
1059 defines the order in which received transport message headers are pro‐
1060 cessed when received by an endpoint. When ordering is set, it indi‐
1061 cates that message headers will be processed in order, based on how the
1062 transmit side has identified the messages. Typically, this means that
1063 messages will be handled in order based on a message level sequence
1064 number.
1065 The following ordering flags, as defined for transmit ordering, also
1067 apply to the processing of received operations: FI_ORDER_NONE, FI_OR‐
1068 DER_RAR, FI_ORDER_RAW, FI_ORDER_RAS, FI_ORDER_WAR, FI_ORDER_WAW, FI_OR‐
1069 DER_WAS, FI_ORDER_SAR, FI_ORDER_SAW, FI_ORDER_SAS, FI_ORDER_RMA_RAR,
1070 FI_ORDER_RMA_RAW, FI_ORDER_RMA_WAR, FI_ORDER_RMA_WAW, FI_ORDER_ATOM‐
1071 IC_RAR, FI_ORDER_ATOMIC_RAW, FI_ORDER_ATOMIC_WAR, and FI_ORDER_ATOM‐
1072 IC_WAW.
1073 comp_order - Completion Ordering
1075 For a description of completion ordering, see the comp_order field in
1076 the Transmit Context Attribute section.
1077 FI_ORDER_NONE
1079 No ordering is defined for completed operations. Receive opera‐
1080 tions may complete in any order, regardless of their submission
1081 order.
1082 FI_ORDER_STRICT
1084 Receive operations complete in the order in which they are pro‐
1085 cessed by the receive context, based on the receive side msg_or‐
1086 der attribute.
1087 FI_ORDER_DATA
1089 When set, this bit indicates that received data is written into
1090 memory in order. Data ordering applies to memory accessed as
1091 part of a single operation and between operations if message or‐
1092 dering is guaranteed.
1093 total_buffered_recv
1095 This field is supported for backwards compatibility purposes. It is a
1096 hint to the provider of the total available space that may be needed to
1097 buffer messages that are received for which there is no matching re‐
1098 ceive operation. The provider may adjust or ignore this value. The
1099 allocation of internal network buffering among received message is
1100 provider specific. For instance, a provider may limit the size of mes‐
1101 sages which can be buffered or the amount of buffering allocated to a
1102 single message.
1103 If receive side buffering is disabled (total_buffered_recv = 0) and a
1105 message is received by an endpoint, then the behavior is dependent on
1106 whether resource management has been enabled (FI_RM_ENABLED has be set
1107 or not). See the Resource Management section of fi_domain.3 for fur‐
1108 ther clarification. It is recommended that applications enable re‐
1109 source management if they anticipate receiving unexpected messages,
1110 rather than modifying this value.
1111 size
1113 The size of the context. The size is specified as the minimum number
1114 of receive operations that may be posted to the endpoint without the
1115 operation returning -FI_EAGAIN.
1116 iov_limit
1118 This is the maximum number of IO vectors (scatter-gather elements) that
1119 a single posted operating may reference.
1120
1122 A scalable endpoint is a communication portal that supports multiple
1123 transmit and receive contexts. Scalable endpoints are loosely modeled
1124 after the networking concept of transmit/receive side scaling, also
1125 known as multi-queue. Support for scalable endpoints is domain specif‐
1126 ic. Scalable endpoints may improve the performance of multi-threaded
1127 and parallel applications, by allowing threads to access independent
1128 transmit and receive queues. A scalable endpoint has a single trans‐
1129 port level address, which can reduce the memory requirements needed to
1130 store remote addressing data, versus using standard endpoints. Scal‐
1131 able endpoints cannot be used directly for communication operations,
1132 and require the application to explicitly create transmit and receive
1133 contexts as described below.
1134 fi_tx_context
1136 Transmit contexts are independent transmit queues. Ordering and syn‐
1137 chronization between contexts are not defined. Conceptually a transmit
1138 context behaves similar to a send-only endpoint. A transmit context
1139 may be configured with fewer capabilities than the base endpoint and
1140 with different attributes (such as ordering requirements and inject
1141 size) than other contexts associated with the same scalable endpoint.
1142 Each transmit context has its own completion queue. The number of
1143 transmit contexts associated with an endpoint is specified during end‐
1144 point creation.
1145 The fi_tx_context call is used to retrieve a specific context, identi‐
1147 fied by an index (see above for details on transmit context at‐
1148 tributes). Providers may dynamically allocate contexts when fi_tx_con‐
1149 text is called, or may statically create all contexts when fi_endpoint
1150 is invoked. By default, a transmit context inherits the properties of
1151 its associated endpoint. However, applications may request context
1152 specific attributes through the attr parameter. Support for per trans‐
1153 mit context attributes is provider specific and not guaranteed.
1154 Providers will return the actual attributes assigned to the context
1155 through the attr parameter, if provided.
1156 fi_rx_context
1158 Receive contexts are independent receive queues for receiving incoming
1159 data. Ordering and synchronization between contexts are not guaran‐
1160 teed. Conceptually a receive context behaves similar to a receive-only
1161 endpoint. A receive context may be configured with fewer capabilities
1162 than the base endpoint and with different attributes (such as ordering
1163 requirements and inject size) than other contexts associated with the
1164 same scalable endpoint. Each receive context has its own completion
1165 queue. The number of receive contexts associated with an endpoint is
1166 specified during endpoint creation.
1167 Receive contexts are often associated with steering flows, that specify
1169 which incoming packets targeting a scalable endpoint to process. How‐
1170 ever, receive contexts may be targeted directly by the initiator, if
1171 supported by the underlying protocol. Such contexts are referred to as
1172 'named'. Support for named contexts must be indicated by setting the
1173 caps FI_NAMED_RX_CTX capability when the corresponding endpoint is cre‐
1174 ated. Support for named receive contexts is coordinated with address
1175 vectors. See fi_av(3) and fi_rx_addr(3).
1176 The fi_rx_context call is used to retrieve a specific context, identi‐
1178 fied by an index (see above for details on receive context attributes).
1179 Providers may dynamically allocate contexts when fi_rx_context is
1180 called, or may statically create all contexts when fi_endpoint is in‐
1181 voked. By default, a receive context inherits the properties of its
1182 associated endpoint. However, applications may request context specif‐
1183 ic attributes through the attr parameter. Support for per receive con‐
1184 text attributes is provider specific and not guaranteed. Providers
1185 will return the actual attributes assigned to the context through the
1186 attr parameter, if provided.
1187
1189 Shared contexts are transmit and receive contexts explicitly shared
1190 among one or more endpoints. A shareable context allows an application
1191 to use a single dedicated provider resource among multiple transport
1192 addressable endpoints. This can greatly reduce the resources needed to
1193 manage communication over multiple endpoints by multiplexing transmit
1194 and/or receive processing, with the potential cost of serializing ac‐
1195 cess across multiple endpoints. Support for shareable contexts is do‐
1196 main specific.
1197 Conceptually, shareable transmit contexts are transmit queues that may
1199 be accessed by many endpoints. The use of a shared transmit context is
1200 mostly opaque to an application. Applications must allocate and bind
1201 shared transmit contexts to endpoints, but operations are posted di‐
1202 rectly to the endpoint. Shared transmit contexts are not associated
1203 with completion queues or counters. Completed operations are posted to
1204 the CQs bound to the endpoint. An endpoint may only be associated with
1205 a single shared transmit context.
1206 Unlike shared transmit contexts, applications interact directly with
1208 shared receive contexts. Users post receive buffers directly to a
1209 shared receive context, with the buffers usable by any endpoint bound
1210 to the shared receive context. Shared receive contexts are not associ‐
1211 ated with completion queues or counters. Completed receive operations
1212 are posted to the CQs bound to the endpoint. An endpoint may only be
1213 associated with a single receive context, and all connectionless end‐
1214 points associated with a shared receive context must also share the
1215 same address vector.
1216 Endpoints associated with a shared transmit context may use dedicated
1218 receive contexts, and vice-versa. Or an endpoint may use shared trans‐
1219 mit and receive contexts. And there is no requirement that the same
1220 group of endpoints sharing a context of one type also share the context
1221 of an alternate type. Furthermore, an endpoint may use a shared con‐
1222 text of one type, but a scalable set of contexts of the alternate type.
1223 fi_stx_context
1225 This call is used to open a shareable transmit context (see above for
1226 details on the transmit context attributes). Endpoints associated with
1227 a shared transmit context must use a subset of the transmit context's
1228 attributes. Note that this is the reverse of the requirement for
1229 transmit contexts for scalable endpoints.
1230 fi_srx_context
1232 This allocates a shareable receive context (see above for details on
1233 the receive context attributes). Endpoints associated with a shared
1234 receive context must use a subset of the receive context's attributes.
1235 Note that this is the reverse of the requirement for receive contexts
1236 for scalable endpoints.
1237
1239 The following feature and description should be considered experimen‐
1240 tal. Until the experimental tag is removed, the interfaces, semantics,
1241 and data structures associated with socket endpoints may change between
1242 library versions.
1243 This section applies to endpoints of type FI_EP_SOCK_STREAM and
1245 FI_EP_SOCK_DGRAM, commonly referred to as socket endpoints.
1246 Socket endpoints are defined with semantics that allow them to more
1248 easily be adopted by developers familiar with the UNIX socket API, or
1249 by middleware that exposes the socket API, while still taking advantage
1250 of high-performance hardware features.
1251 The key difference between socket endpoints and other active endpoints
1253 are socket endpoints use synchronous data transfers. Buffers passed
1254 into send and receive operations revert to the control of the applica‐
1255 tion upon returning from the function call. As a result, no data
1256 transfer completions are reported to the application, and socket end‐
1257 points are not associated with completion queues or counters.
1258 Socket endpoints support a subset of message operations: fi_send,
1260 fi_sendv, fi_sendmsg, fi_recv, fi_recvv, fi_recvmsg, and fi_inject.
1261 Because data transfers are synchronous, the return value from send and
1262 receive operations indicate the number of bytes transferred on success,
1263 or a negative value on error, including -FI_EAGAIN if the endpoint can‐
1264 not send or receive any data because of full or empty queues, respec‐
1265 tively.
1266 Socket endpoints are associated with event queues and address vectors,
1268 and process connection management events asynchronously, similar to
1269 other endpoints. Unlike UNIX sockets, socket endpoint must still be
1270 declared as either active or passive.
1271 Socket endpoints behave like non-blocking sockets. In order to support
1273 select and poll semantics, active socket endpoints are associated with
1274 a file descriptor that is signaled whenever the endpoint is ready to
1275 send and/or receive data. The file descriptor may be retrieved using
1276 fi_control.
1277
1279 Operation flags are obtained by OR-ing the following flags together.
1280 Operation flags define the default flags applied to an endpoint's data
1281 transfer operations, where a flags parameter is not available. Data
1282 transfer operations that take flags as input override the op_flags val‐
1283 ue of transmit or receive context attributes of an endpoint.
1284 FI_INJECT
1286 Indicates that all outbound data buffers should be returned to
1287 the user's control immediately after a data transfer call re‐
1288 turns, even if the operation is handled asynchronously. This
1289 may require that the provider copy the data into a local buffer
1290 and transfer out of that buffer. A provider can limit the total
1291 amount of send data that may be buffered and/or the size of a
1292 single send that can use this flag. This limit is indicated us‐
1293 ing inject_size (see inject_size above).
1294 FI_MULTI_RECV
1296 Applies to posted receive operations. This flag allows the user
1297 to post a single buffer that will receive multiple incoming mes‐
1298 sages. Received messages will be packed into the receive buffer
1299 until the buffer has been consumed. Use of this flag may cause
1300 a single posted receive operation to generate multiple comple‐
1301 tions as messages are placed into the buffer. The placement of
1302 received data into the buffer may be subjected to provider spe‐
1303 cific alignment restrictions. The buffer will be released by
1304 the provider when the available buffer space falls below the
1305 specified minimum (see FI_OPT_MIN_MULTI_RECV).
1306 FI_COMPLETION
1308 Indicates that a completion queue entry should be written for
1309 data transfer operations. This flag only applies to operations
1310 issued on an endpoint that was bound to a completion queue with
1311 the FI_SELECTIVE_COMPLETION flag set, otherwise, it is ignored.
1312 See the fi_ep_bind section above for more detail.
1313 FI_INJECT_COMPLETE
1315 Indicates that a completion should be generated when the source
1316 buffer(s) may be reused. See fi_cq(3) for additional details on
1317 completion semantics.
1318 FI_TRANSMIT_COMPLETE
1320 Indicates that a completion should be generated when the trans‐
1321 mit operation has completed relative to the local provider. See
1322 fi_cq(3) for additional details on completion semantics.
1323 FI_DELIVERY_COMPLETE
1325 Indicates that a completion should be generated when the opera‐
1326 tion has been processed by the destination endpoint(s). See
1327 fi_cq(3) for additional details on completion semantics.
1328 FI_COMMIT_COMPLETE
1330 Indicates that a completion should not be generated (locally or
1331 at the peer) until the result of an operation have been made
1332 persistent. See fi_cq(3) for additional details on completion
1333 semantics.
1334 FI_MULTICAST
1336 Indicates that data transfers will target multicast addresses by
1337 default. Any fi_addr_t passed into a data transfer operation
1338 will be treated as a multicast address.
1339
1341 Users should call fi_close to release all resources allocated to the
1342 fabric endpoint.
1343 Endpoints allocated with the FI_CONTEXT or FI_CONTEXT2 mode bits set
1345 must typically provide struct fi_context(2) as their per operation con‐
1346 text parameter. (See fi_getinfo.3 for details.) However, when FI_SE‐
1347 LECTIVE_COMPLETION is enabled to suppress CQ completion entries, and an
1348 operation is initiated without the FI_COMPLETION flag set, then the
1349 context parameter is ignored. An application does not need to pass in
1350 a valid struct fi_context(2) into such data transfers.
1351 Operations that complete in error that are not associated with valid
1353 operational context will use the endpoint context in any error report‐
1354 ing structures.
1355 Although applications typically associate individual completions with
1357 either completion queues or counters, an endpoint can be attached to
1358 both a counter and completion queue. When combined with using selec‐
1359 tive completions, this allows an application to use counters to track
1360 successful completions, with a CQ used to report errors. Operations
1361 that complete with an error increment the error counter and generate a
1362 CQ completion event.
1363 As mentioned in fi_getinfo(3), the ep_attr structure can be used to
1365 query providers that support various endpoint attributes. fi_getinfo
1366 can return provider info structures that can support the minimal set of
1367 requirements (such that the application maintains correctness). Howev‐
1368 er, it can also return provider info structures that exceed application
1369 requirements. As an example, consider an application requesting
1370 msg_order as FI_ORDER_NONE. The resulting output from fi_getinfo may
1371 have all the ordering bits set. The application can reset the ordering
1372 bits it does not require before creating the endpoint. The provider is
1373 free to implement a stricter ordering than is required by the applica‐
1374 tion.
1375
1377 Returns 0 on success. On error, a negative value corresponding to fab‐
1378 ric errno is returned. For fi_cancel, a return value of 0 indicates
1379 that the cancel request was submitted for processing.
1380 Fabric errno values are defined in rdma/fi_errno.h.
1382
1384 -FI_EDOMAIN
1385 A resource domain was not bound to the endpoint or an attempt
1386 was made to bind multiple domains.
1387 -FI_ENOCQ
1389 The endpoint has not been configured with necessary event queue.
1390 -FI_EOPBADSTATE
1392 The endpoint's state does not permit the requested operation.
1393
1395 fi_getinfo(3), fi_domain(3), fi_cq(3) fi_msg(3), fi_tagged(3),
1396 fi_rma(3)
1397
1399 OpenFabrics.
1400Libfabric Programmer's Manual 2019-05-14 fi_endpoint(3)