1fi_tagged(3) Libfabric v1.7.0 fi_tagged(3)
2
6 fi_tagged - Tagged data transfer operations
7 fi_trecv / fi_trecvv / fi_trecvmsg
9 Post a buffer to receive an incoming message
10 fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12 Initiate an operation to send a message
13
15 #include <rdma/fi_tagged.h>
16 ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18 fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19 ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21 size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22 void *context);
23 ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25 uint64_t flags);
26 ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28 void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29 ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31 void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32 void *context);
33 ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35 uint64_t flags);
36 ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38 fi_addr_t dest_addr, uint64_t tag);
39 ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41 void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42 void *context);
43 ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45 uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46
48 fid Fabric endpoint on which to initiate tagged communication opera‐
49 tion.
50 buf Data buffer to send or receive.
52 len Length of data buffer to send or receive.
54 iov Vectored data buffer.
56 count Count of vectored data entries.
58 tag Tag associated with the message.
60 ignore Mask of bits to ignore applied to the tag for receive opera‐
62 tions.
63 desc Memory descriptor associated with the data buffer
65 data Remote CQ data to transfer with the sent data.
67 dest_addr
69 Destination address for connectionless transfers. Ignored for
70 connected endpoints.
71 src_addr
73 Source address to receive from for connectionless transfers.
74 Applies only to connectionless endpoints with the FI_DIRECT‐
75 ED_RECV capability enabled, otherwise this field is ignored. If
76 set to FI_ADDR_UNSPEC, any source address may match.
77 msg Message descriptor for send and receive operations.
79 flags Additional flags to apply for the send or receive operation.
81 context
83 User specified pointer to associate with the operation. This
84 parameter is ignored if the operation will not generate a suc‐
85 cessful completion, unless an op flag specifies the context pa‐
86 rameter be used for required input.
87
89 Tagged messages are data transfers which carry a key or tag with the
90 message buffer. The tag is used at the receiving endpoint to match the
91 incoming message with a corresponding receive buffer. Message tags
92 match when the receive buffer tag is the same as the send buffer tag
93 with the ignored bits masked out. This can be stated as:
94 send_tag & ~ignore == recv_tag & ~ignore
96 In general, message tags are checked against receive buffers in the or‐
98 der in which messages have been posted to the endpoint. See the order‐
99 ing discussion below for more details.
100 The send functions -- fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
102 fi_tsenddata -- are used to transmit a tagged message from one endpoint
103 to another endpoint. The main difference between send functions are
104 the number and type of parameters that they accept as input. Other‐
105 wise, they perform the same general function.
106 The receive functions -- fi_trecv, fi_trecvv, fi_recvmsg -- post a data
108 buffer to an endpoint to receive inbound tagged messages. Similar to
109 the send operations, receive operations operate asynchronously. Users
110 should not touch the posted data buffer(s) until the receive operation
111 has completed. Posted receive buffers are matched with inbound send
112 messages based on the tags associated with the send and receive buf‐
113 fers.
114 An endpoint must be enabled before an application can post send or re‐
116 ceive operations to it. For connected endpoints, receive buffers may
117 be posted prior to connect or accept being called on the endpoint.
118 This ensures that buffers are available to receive incoming data imme‐
119 diately after the connection has been established.
120 Completed message operations are reported to the user through one or
122 more event collectors associated with the endpoint. Users provide con‐
123 text which are associated with each operation, and is returned to the
124 user as part of the event completion. See fi_cq for completion event
125 details.
126 fi_tsend
128 The call fi_tsend transfers the data contained in the user-specified
129 data buffer to a remote endpoint, with message boundaries being main‐
130 tained. The local endpoint must be connected to a remote endpoint or
131 destination before fi_tsend is called. Unless the endpoint has been
132 configured differently, the data buffer passed into fi_tsend must not
133 be touched by the application until the fi_tsend call completes asyn‐
134 chronously.
135 fi_tsendv
137 The fi_tsendv call adds support for a scatter-gather list to fi_tsend.
138 The fi_sendv transfers the set of data buffers referenced by the iov
139 parameter to a remote endpoint as a single message.
140 fi_tsendmsg
142 The fi_tsendmsg call supports data transfers over both connected and
143 unconnected endpoints, with the ability to control the send operation
144 per call through the use of flags. The fi_tsendmsg function takes a
145 struct fi_msg_tagged as input.
146 struct fi_msg_tagged {
148 const struct iovec *msg_iov; /* scatter-gather array */
149 void *desc; /* data descriptor */
150 size_t iov_count;/* # elements in msg_iov */
151 fi_addr_t addr; /* optional endpoint address */
152 uint64_t tag; /* tag associated with message */
153 uint64_t ignore; /* mask applied to tag for receives */
154 void *context; /* user-defined context */
155 uint64_t data; /* optional immediate data */
156 };
157 fi_tinject
159 The tagged inject call is an optimized version of fi_tsend. The
160 fi_tinject function behaves as if the FI_INJECT transfer flag were set,
161 and FI_COMPLETION were not. That is, the data buffer is available for
162 reuse immediately on returning from fi_tinject, and no completion event
163 will be generated for this send. The completion event will be sup‐
164 pressed even if the endpoint has not been configured with FI_SELEC‐
165 TIVE_COMPLETION. See the flags discussion below for more details. The
166 requested message size that can be used with fi_tinject is limited by
167 inject_size.
168 fi_tsenddata
170 The tagged send data call is similar to fi_tsend, but allows for the
171 sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the
172 transfer.
173 fi_tinjectdata
175 The tagged inject data call is similar to fi_tinject, but allows for
176 the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of
177 the transfer.
178 fi_trecv
180 The fi_trecv call posts a data buffer to the receive queue of the cor‐
181 responding endpoint. Posted receives are searched in the order in
182 which they were posted in order to match sends. Message boundaries are
183 maintained. The order in which the receives complete is dependent on
184 the endpoint type and protocol.
185 fi_trecvv
187 The fi_trecvv call adds support for a scatter-gather list to fi_trecv.
188 The fi_trecvv posts the set of data buffers referenced by the iov pa‐
189 rameter to a receive incoming data.
190 fi_trecvmsg
192 The fi_trecvmsg call supports posting buffers over both connected and
193 unconnected endpoints, with the ability to control the receive opera‐
194 tion per call through the use of flags. The fi_trecvmsg function takes
195 a struct fi_msg_tagged as input.
196
198 The fi_trecvmsg and fi_tsendmsg calls allow the user to specify flags
199 which can change the default message handling of the endpoint. Flags
200 specified with fi_trecvmsg / fi_tsendmsg override most flags previously
201 configured with the endpoint, except where noted (see fi_endpoint).
202 The following list of flags are usable with fi_trecvmsg and/or
203 fi_tsendmsg.
204 FI_REMOTE_CQ_DATA
206 Applies to fi_tsendmsg and fi_tsenddata. Indicates that remote
207 CQ data is available and should be sent as part of the request.
208 See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
209 FI_COMPLETION
211 Indicates that a completion entry should be generated for the
212 specified operation. The endpoint must be bound to a completion
213 queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
214 ified operation, or this flag is ignored.
215 FI_MORE
217 Indicates that the user has additional requests that will imme‐
218 diately be posted after the current call returns. Use of this
219 flag may improve performance by enabling the provider to opti‐
220 mize its access to the fabric hardware.
221 FI_INJECT
223 Applies to fi_tsendmsg. Indicates that the outbound data buffer
224 should be returned to user immediately after the send call re‐
225 turns, even if the operation is handled asynchronously. This
226 may require that the underlying provider implementation copy the
227 data into a local buffer and transfer out of that buffer. This
228 flag can only be used with messages smaller than inject_size.
229 FI_INJECT_COMPLETE
231 Applies to fi_tsendmsg. Indicates that a completion should be
232 generated when the source buffer(s) may be reused.
233 FI_TRANSMIT_COMPLETE
235 Applies to fi_tsendmsg. Indicates that a completion should not
236 be generated until the operation has been successfully transmit‐
237 ted and is no longer being tracked by the provider.
238 FI_MATCH_COMPLETE
240 Applies to fi_tsendmsg. Indicates that a completion should be
241 generated only after the message has either been matched with a
242 tagged buffer or was discarded by the target application.
243 FI_FENCE
245 Applies to transmits. Indicates that the requested operation,
246 also known as the fenced operation, and any operation posted af‐
247 ter the fenced operation will be deferred until all previous op‐
248 erations targeting the same peer endpoint have completed. Oper‐
249 ations posted after the fencing will see and/or replace the re‐
250 sults of any operations initiated prior to the fenced operation.
251 The ordering of operations starting at the posting of the fenced opera‐
253 tion (inclusive) to the posting of a subsequent fenced operation (ex‐
254 clusive) is controlled by the endpoint's ordering semantics.
255 The following flags may be used with fi_trecvmsg.
257 FI_PEEK
259 The peek flag may be used to see if a specified message has ar‐
260 rived. A peek request is often useful on endpoints that have
261 provider allocated buffering enabled (see fi_rx_attr to‐
262 tal_buffered_recv). Unlike standard receive operations, a re‐
263 ceive operation with the FI_PEEK flag set does not remain queued
264 with the provider after the peek completes successfully. The
265 peek operation operates asynchronously, and the results of the
266 peek operation are available in the completion queue associated
267 with the endpoint. If no message is found matching the tags
268 specified in the peek request, then a completion queue error en‐
269 try with err field set to FI_ENOMSG will be available.
270 If a peek request locates a matching message, the operation will com‐
272 plete successfully. The returned completion data will indicate the
273 meta-data associated with the message, such as the message length, com‐
274 pletion flags, available CQ data, tag, and source address. The data
275 available is subject to the completion entry format (e.g. struct
276 fi_cq_tagged_entry).
277 An application may supply a buffer if it desires to receive data as a
279 part of the peek operation. In order to receive data as a part of the
280 peek operation, the buf and len fields must be available in the CQ for‐
281 mat. In particular, FI_CQ_FORMAT_CONTEXT and FI_CQ_FORMAT_MSG cannot
282 be used if peek operations desire to obtain a copy of the data. The
283 returned data is limited to the size of the input buffer(s) or the mes‐
284 sage size, if smaller. A provider indicates if data is available by
285 setting the buf field of the CQ entry to the user's first input buffer.
286 If buf is NULL, no data was available to return. A provider may return
287 NULL even if the peek operation completes successfully. Note that the
288 CQ entry len field will reference the size of the message, not neces‐
289 sarily the size of the returned data.
290 FI_CLAIM
292 If this flag is used in conjunction with FI_PEEK, it indicates
293 if the peek request completes successfully -- indicating that a
294 matching message was located -- the message is claimed by call‐
295 er. Claimed messages can only be retrieved using a subsequent,
296 paired receive operation with the FI_CLAIM flag set. A receive
297 operation with the FI_CLAIM flag set, but FI_PEEK not set is
298 used to retrieve a previously claimed message.
299 In order to use the FI_CLAIM flag, an application must supply a struct
301 fi_context structure as the context for the receive operation, or a
302 struct fi_recv_context in the case of buffered receives. The same
303 fi_context structure used for an FI_PEEK + FI_CLAIM operation must be
304 used by the paired FI_CLAIM request.
305 This flag also applies to endpoints configured for FI_BUFFERED_RECV or
307 FI_VARIABLE_MSG. When set, it is used to retrieve a tagged message
308 that was buffered by the provider. See Buffered Tagged Receives sec‐
309 tion for details.
310 FI_DISCARD
312 This flag may be used in conjunction with either FI_PEEK or
313 FI_CLAIM. If this flag is used in conjunction with FI_PEEK, it
314 indicates if the peek request completes successfully -- indicat‐
315 ing that a matching message was located -- the message is dis‐
316 carded by the provider, as the data is not needed by the appli‐
317 cation. This flag may also be used in conjunction with FI_CLAIM
318 in order to discard a message previously claimed using an
319 FI_PEEK + FI_CLAIM request.
320 This flag also applies to endpoints configured for FI_BUFFERED_RECV or
322 FI_VARIABLE_MSG. When set, it indicates that the provider should free
323 a buffered messages. See Buffered Tagged Receives section for details.
324 If this flag is set, the input buffer(s) and length parameters are ig‐
326 nored.
327
329 See fi_msg(3) for an introduction to buffered receives. The handling
330 of buffered receives differs between fi_msg operations and fi_tagged.
331 Although the provider is responsible for allocating and managing net‐
332 work buffers, the application is responsible for identifying the tags
333 that will be used to match incoming messages. The provider handles
334 matching incoming receives to the application specified tags.
335 When FI_BUFFERED_RECV is enabled, the application posts the tags that
337 will be used for matching purposes. Tags are posted using fi_trecv,
338 fi_trecvv, and fi_trecvmsg; however, parameters related to the input
339 buffers are ignored (e.g. buf, len, iov, desc). When a provider re‐
340 ceives a message for which there is a matching tag, it will write an
341 entry to the completion queue associated with the receiving endpoint.
342 For discussion purposes, the completion queue is assumed to be config‐
344 ured for FI_CQ_FORMAT_TAGGED. The op_context field will point to a
345 struct fi_recv_contex.
346 struct fi_recv_context {
348 struct fid_ep *ep;
349 void *context;
350 };
351 The 'ep' field will be NULL. The 'context' field will match the appli‐
353 cation context specified when posting the tag. Other fields are set as
354 defined in fi_msg(3).
355 After being notified that a buffered receive has arrived, applications
357 must either claim or discard the message as described in fi_msg(3).
358
360 Variable length messages are defined in fi_msg(3). The requirements
361 for handling variable length tagged messages is identical to those de‐
362 fined above for buffered tagged receives.
363
365 The tagged send and receive calls return 0 on success. On error, a
366 negative value corresponding to fabric errno is returned. Fabric er‐
367 rno values are defined in fi_errno.h.
368
370 -FI_EAGAIN
371 See fi_msg(3) for a detailed description of handling FI_EAGAIN.
372 -FI_EINVAL
374 Indicates that an invalid argument was supplied by the user.
375 -FI_EOTHER
377 Indicates that an unspecified error occurred.
378
380 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
381
383 OpenFabrics.
384Libfabric Programmer's Manual 2018-11-28 fi_tagged(3)