1fi_cm(3) Libfabric v1.17.0 fi_cm(3)
2
6 fi_cm - Connection management operations
7 fi_connect / fi_listen / fi_accept / fi_reject / fi_shutdown
9 Manage endpoint connection state.
10 fi_setname / fi_getname / fi_getpeer
12 Set local, or return local or peer endpoint address.
13 fi_join / fi_close / fi_mc_addr
15 Join, leave, or retrieve a multicast address.
16
18 #include <rdma/fi_cm.h>
19 int fi_connect(struct fid_ep *ep, const void *addr,
21 const void *param, size_t paramlen);
22 int fi_listen(struct fid_pep *pep);
24 int fi_accept(struct fid_ep *ep, const void *param, size_t paramlen);
26 int fi_reject(struct fid_pep *pep, fid_t handle,
28 const void *param, size_t paramlen);
29 int fi_shutdown(struct fid_ep *ep, uint64_t flags);
31 int fi_setname(fid_t fid, void *addr, size_t addrlen);
33 int fi_getname(fid_t fid, void *addr, size_t *addrlen);
35 int fi_getpeer(struct fid_ep *ep, void *addr, size_t *addrlen);
37 int fi_join(struct fid_ep *ep, const void *addr, uint64_t flags,
39 struct fid_mc **mc, void *context);
40 int fi_close(struct fid *mc);
42 fi_addr_t fi_mc_addr(struct fid_mc *mc);
44
46 ep / pep
47 Fabric endpoint on which to change connection state.
48 fid Active or passive endpoint to get/set address.
50 addr Buffer to address. On a set call, the endpoint will be assigned
52 the specified address. On a get, the local address will be
53 copied into the buffer, up to the space provided. For connect,
54 this parameter indicates the peer address to connect to. The
55 address must be in the same format as that specified using
56 fi_info: addr_format when the endpoint was created.
57 addrlen
59 On input, specifies size of addr buffer. On output, stores num‐
60 ber of bytes written to addr buffer.
61 param User-specified data exchanged as part of the connection ex‐
63 change.
64 paramlen
66 Size of param buffer.
67 info Fabric information associated with a connection request.
69 mc Multicast group associated with an endpoint.
71 flags Additional flags for controlling connection operation.
73 context
75 User context associated with the request.
76
78 Connection management functions are used to connect an connection-ori‐
79 ented endpoint to a peer endpoint.
80 fi_listen
82 The fi_listen call indicates that the specified endpoint should be
83 transitioned into a passive connection state, allowing it to accept in‐
84 coming connection requests. Connection requests against a listening
85 endpoint are reported asynchronously to the user through a bound CM
86 event queue using the FI_CONNREQ event type. The number of outstanding
87 connection requests that can be queued at an endpoint is limited by the
88 listening endpoint’s backlog parameter. The backlog is initialized
89 based on administrative configuration values, but may be adjusted
90 through the fi_control call.
91 fi_connect
93 The fi_connect call initiates a connection request on a connection-ori‐
94 ented endpoint to the destination address. fi_connect may only be
95 called on an endpoint once in its lifetime.
96 fi_accept / fi_reject
98 The fi_accept and fi_reject calls are used on the passive (listening)
99 side of a connection to accept or reject a connection request, respec‐
100 tively. To accept a connection, the listening application first waits
101 for a connection request event (FI_CONNREQ). After receiving such an
102 event, the application allocates a new endpoint to accept the connec‐
103 tion. This endpoint must be allocated using an fi_info structure ref‐
104 erencing the handle from this FI_CONNREQ event. fi_accept is then in‐
105 voked with the newly allocated endpoint. If the listening application
106 wishes to reject a connection request, it calls fi_reject with the lis‐
107 tening endpoint and a reference to the connection request.
108 A successfully accepted connection request will result in the active
110 (connecting) endpoint seeing an FI_CONNECTED event on its associated
111 event queue. A rejected or failed connection request will generate an
112 error event. The error entry will provide additional details describ‐
113 ing the reason for the failed attempt.
114 An FI_CONNECTED event will also be generated on the passive side for
116 the accepting endpoint once the connection has been properly estab‐
117 lished. The fid of the FI_CONNECTED event will be that of the endpoint
118 passed to fi_accept as opposed to the listening passive endpoint. Out‐
119 bound data transfers cannot be initiated on a connection-oriented end‐
120 point until an FI_CONNECTED event has been generated. However, receive
121 buffers may be associated with an endpoint anytime.
122 fi_shutdown
124 The fi_shutdown call is used to gracefully disconnect an endpoint from
125 its peer. The flags parameter is reserved and must be 0.
126 Outstanding operations posted to the endpoint when fi_shutdown is
128 called will be canceled or discarded. Notification of canceled opera‐
129 tions will be reported by the provider to the corresponding completion
130 queue(s). Discarded operations will silently be dropped, with no com‐
131 pletions generated. The choice of canceling, versus discarding opera‐
132 tions, is provider dependent. However, all canceled completions will
133 be written before fi_shutdown returns.
134 When called, fi_shutdown does not affect completions already written to
136 a completion queue. Any queued completions associated with asynchro‐
137 nous operations posted to the endpoint may still be retrieved from the
138 corresponding completion queue(s) after an endpoint has been shutdown.
139 An FI_SHUTDOWN event will be generated for an endpoint when the remote
141 peer issues a disconnect using fi_shutdown or abruptly closes the end‐
142 point. Note that in the abrupt close case, an FI_SHUTDOWN event will
143 only be generated if the peer system is reachable and a service or ker‐
144 nel agent on the peer system is able to notify the local endpoint that
145 the connection has been aborted.
146 fi_close
148 Fi_close is used to disassociate an endpoint from a multicast group and
149 close all resources associated with the group. Fi_close must be called
150 on all multicast groups that an endpoint joins.
151 fi_setname
153 The fi_setname call may be used to modify or assign the address of the
154 local endpoint. It is conceptually similar to the socket bind opera‐
155 tion. An endpoint may be assigned an address on its creation, through
156 the fi_info structure. The fi_setname call allows an endpoint to be
157 created without being associated with a specific service (e.g., port
158 number) and/or node (e.g., network) address, with the addressing as‐
159 signed dynamically. The format of the specified addressing data must
160 match that specified through the fi_info structure when the endpoint
161 was created.
162 If no service address is specified and a service address has not yet
164 been assigned to the endpoint, then the provider will allocate a ser‐
165 vice address and assign it to the endpoint. If a node or service ad‐
166 dress is specified, then, upon successful completion of fi_setname, the
167 endpoint will be assigned the given addressing. If an address cannot
168 be assigned, or the endpoint address cannot be modified, an appropriate
169 fabric error number is returned.
170 fi_getname / fi_getpeer
172 The fi_getname and fi_getpeer calls may be used to retrieve the local
173 or peer endpoint address, respectively. On input, the addrlen parame‐
174 ter should indicate the size of the addr buffer. If the actual address
175 is larger than what can fit into the buffer, it will be truncated and
176 -FI_ETOOSMALL will be returned. On output, addrlen is set to the size
177 of the buffer needed to store the address, which may be larger than the
178 input value.
179 fi_getname is not guaranteed to return a valid source address until af‐
181 ter the specified endpoint has been enabled or has had an address as‐
182 signed. An endpoint may be enabled explicitly through fi_enable, or
183 implicitly, such as through fi_connect or fi_listen. An address may be
184 assigned using fi_setname. fi_getpeer is not guaranteed to return a
185 valid peer address until an endpoint has been completely connected – an
186 FI_CONNECTED event has been generated.
187 fi_join
189 This call attaches an endpoint to a multicast group. By default, the
190 endpoint will join the group based on the data transfer capabilities of
191 the endpoint. For example, if the endpoint has been configured to both
192 send and receive data, then the endpoint will be able to initiate and
193 receive transfers to and from the multicast group. The fi_join flags
194 may be used to restrict access to the multicast group, subject to end‐
195 point capability limitations.
196 Multicast join operations complete asynchronously. An endpoint must be
198 bound to an event queue prior to calling fi_join. The result of the
199 join operation will be reported to the EQ as an FI_JOIN_COMPLETE event.
200 Applications cannot issue multicast transfers until receiving notifica‐
201 tion that the join operation has completed. Note that an endpoint may
202 begin receiving messages from the multicast group as soon as the join
203 completes, which can occur prior to the FI_JOIN_COMPLETE event being
204 generated.
205 Applications must call fi_close on the multicast group to disconnect
207 the endpoint from the group. After a join operation has completed, the
208 fi_mc_addr call may be used to retrieve the address associated with the
209 multicast group.
210 fi_mc_addr
212 Returns the fi_addr_t address associated with a multicast group. This
213 address must be used when transmitting data to a multicast group and
214 paired with the FI_MULTICAST operation flag.
215
217 Except in functions noted below, flags are reserved and must be 0.
218 FI_SEND
220 Applies to fi_join. This flag indicates that the endpoint
221 should join the multicast group as a send only member. The end‐
222 point must be configured for transmit operations to use this
223 flag, or an error will occur.
224 FI_RECV
226 Applies to fi_join. This flag indicates that the endpoint
227 should join the multicast group with receive permissions only.
228 The endpoint must be configured for receive operations to use
229 this flag, or an error will occur.
230
232 Returns 0 on success. On error, a negative value corresponding to fab‐
233 ric errno is returned. Fabric errno values are defined in rdma/fi_er‐
234 rno.h.
235
238 For connection-oriented endpoints, the buffer referenced by param will
239 be sent as part of the connection request or response, subject to the
240 constraints of the underlying connection protocol. Applications may
241 use fi_getopt with the FI_OPT_CM_DATA_SIZE endpoint option to determine
242 the size of application data that may be exchanged as part of a connec‐
243 tion request or response. The fi_connect, fi_accept, and fi_reject
244 calls will silently truncate any application data which cannot fit into
245 underlying protocol messages. User data exchanged as part of the con‐
246 nection process is available as part of the fi_eq_cm_entry structure,
247 for FI_CONNREQ and FI_CONNECTED events, or as additional err_data to
248 fi_eq_err_entry, in the case of a rejected connection.
249
251 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_eq(3)
252
254 OpenFabrics.
255Libfabric Programmer’s Manual 2022-12-11 fi_cm(3)