1fi_tagged(3) Libfabric v1.6.1 fi_tagged(3)
2
6 fi_tagged - Tagged data transfer operations
7 fi_trecv / fi_trecvv / fi_trecvmsg
9 Post a buffer to receive an incoming message
10 fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12 Initiate an operation to send a message
13
15 #include <rdma/fi_tagged.h>
16 ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18 fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19 ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21 size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22 void *context);
23 ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25 uint64_t flags);
26 ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28 void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29 ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31 void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32 void *context);
33 ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35 uint64_t flags);
36 ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38 fi_addr_t dest_addr, uint64_t tag);
39 ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41 void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42 void *context);
43 ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45 uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46
48 fid : Fabric endpoint on which to initiate tagged communication opera‐
49 tion.
50 buf : Data buffer to send or receive.
52 len : Length of data buffer to send or receive.
54 iov : Vectored data buffer.
56 count : Count of vectored data entries.
58 tag : Tag associated with the message.
60 ignore : Mask of bits to ignore applied to the tag for receive opera‐
62 tions.
63 desc : Memory descriptor associated with the data buffer
65 data : Remote CQ data to transfer with the sent data.
67 dest_addr : Destination address for connectionless transfers. Ignored
69 for connected endpoints.
70 src_addr : Source address to receive from for connectionless transfers.
72 Applies only to connectionless endpoints with the FI_DIRECTED_RECV
73 capability enabled, otherwise this field is ignored. If set to
74 FI_ADDR_UNSPEC, any source address may match.
75 msg : Message descriptor for send and receive operations.
77 flags : Additional flags to apply for the send or receive operation.
79 context : User specified pointer to associate with the operation.
81
83 Tagged messages are data transfers which carry a key or tag with the
84 message buffer. The tag is used at the receiving endpoint to match the
85 incoming message with a corresponding receive buffer. Message tags
86 match when the receive buffer tag is the same as the send buffer tag
87 with the ignored bits masked out. This can be stated as:
88 send_tag & ~ignore == recv_tag & ~ignore
90 In general, message tags are checked against receive buffers in the
92 order in which messages have been posted to the endpoint. See the
93 ordering discussion below for more details.
94 The send functions -- fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
96 fi_tsenddata -- are used to transmit a tagged message from one endpoint
97 to another endpoint. The main difference between send functions are
98 the number and type of parameters that they accept as input. Other‐
99 wise, they perform the same general function.
100 The receive functions -- fi_trecv, fi_trecvv, fi_recvmsg -- post a data
102 buffer to an endpoint to receive inbound tagged messages. Similar to
103 the send operations, receive operations operate asynchronously. Users
104 should not touch the posted data buffer(s) until the receive operation
105 has completed. Posted receive buffers are matched with inbound send
106 messages based on the tags associated with the send and receive buf‐
107 fers.
108 An endpoint must be enabled before an application can post send or
110 receive operations to it. For connected endpoints, receive buffers may
111 be posted prior to connect or accept being called on the endpoint.
112 This ensures that buffers are available to receive incoming data imme‐
113 diately after the connection has been established.
114 Completed message operations are reported to the user through one or
116 more event collectors associated with the endpoint. Users provide con‐
117 text which are associated with each operation, and is returned to the
118 user as part of the event completion. See fi_cq for completion event
119 details.
120 fi_tsend
122 The call fi_tsend transfers the data contained in the user-specified
123 data buffer to a remote endpoint, with message boundaries being main‐
124 tained. The local endpoint must be connected to a remote endpoint or
125 destination before fi_tsend is called. Unless the endpoint has been
126 configured differently, the data buffer passed into fi_tsend must not
127 be touched by the application until the fi_tsend call completes asyn‐
128 chronously.
129 fi_tsendv
131 The fi_tsendv call adds support for a scatter-gather list to fi_tsend.
132 The fi_sendv transfers the set of data buffers referenced by the iov
133 parameter to a remote endpoint as a single message.
134 fi_tsendmsg
136 The fi_tsendmsg call supports data transfers over both connected and
137 unconnected endpoints, with the ability to control the send operation
138 per call through the use of flags. The fi_tsendmsg function takes a
139 struct fi_msg_tagged as input.
140 struct fi_msg_tagged {
142 const struct iovec *msg_iov; /* scatter-gather array */
143 void *desc; /* data descriptor */
144 size_t iov_count;/* # elements in msg_iov */
145 fi_addr_t addr; /* optional endpoint address */
146 uint64_t tag; /* tag associated with message */
147 uint64_t ignore; /* mask applied to tag for receives */
148 void *context; /* user-defined context */
149 uint64_t data; /* optional immediate data */
150 };
151 fi_tinject
153 The tagged inject call is an optimized version of fi_tsend. The
154 fi_tinject function behaves as if the FI_INJECT transfer flag were set,
155 and FI_COMPLETION were not. That is, the data buffer is available for
156 reuse immediately on returning from from fi_tinject, and no completion
157 event will be generated for this send. The completion event will be
158 suppressed even if the endpoint has not been configured with FI_SELEC‐
159 TIVE_COMPLETION. See the flags discussion below for more details. The
160 requested message size that can be used with fi_tinject is limited by
161 inject_size.
162 fi_tsenddata
164 The tagged send data call is similar to fi_tsend, but allows for the
165 sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the
166 transfer.
167 fi_tinjectdata
169 The tagged inject data call is similar to fi_tinject, but allows for
170 the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of
171 the transfer.
172 fi_trecv
174 The fi_trecv call posts a data buffer to the receive queue of the cor‐
175 responding endpoint. Posted receives are searched in the order in
176 which they were posted in order to match sends. Message boundaries are
177 maintained. The order in which the receives complete is dependent on
178 the endpoint type and protocol.
179 fi_trecvv
181 The fi_trecvv call adds support for a scatter-gather list to fi_trecv.
182 The fi_trecvv posts the set of data buffers referenced by the iov
183 parameter to a receive incoming data.
184 fi_trecvmsg
186 The fi_trecvmsg call supports posting buffers over both connected and
187 unconnected endpoints, with the ability to control the receive opera‐
188 tion per call through the use of flags. The fi_trecvmsg function takes
189 a struct fi_msg_tagged as input.
190
192 The fi_trecvmsg and fi_tsendmsg calls allow the user to specify flags
193 which can change the default message handling of the endpoint. Flags
194 specified with fi_trecvmsg / fi_tsendmsg override most flags previously
195 configured with the endpoint, except where noted (see fi_endpoint).
196 The following list of flags are usable with fi_trecvmsg and/or
197 fi_tsendmsg.
198 FI_REMOTE_CQ_DATA : Applies to fi_tsendmsg and fi_tsenddata. Indicates
200 that remote CQ data is available and should be sent as part of the
201 request. See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
202 FI_COMPLETION : Indicates that a completion entry should be generated
204 for the specified operation. The endpoint must be bound to a comple‐
205 tion queue with FI_SELECTIVE_COMPLETION that corresponds to the speci‐
206 fied operation, or this flag is ignored.
207 FI_MORE : Indicates that the user has additional requests that will
209 immediately be posted after the current call returns. Use of this flag
210 may improve performance by enabling the provider to optimize its access
211 to the fabric hardware.
212 FI_INJECT : Applies to fi_tsendmsg. Indicates that the outbound data
214 buffer should be returned to user immediately after the send call
215 returns, even if the operation is handled asynchronously. This may
216 require that the underlying provider implementation copy the data into
217 a local buffer and transfer out of that buffer. This flag can only be
218 used with messages smaller than inject_size.
219 FI_INJECT_COMPLETE : Applies to fi_tsendmsg. Indicates that a comple‐
221 tion should be generated when the source buffer(s) may be reused.
222 FI_TRANSMIT_COMPLETE : Applies to fi_tsendmsg. Indicates that a com‐
224 pletion should not be generated until the operation has been success‐
225 fully transmitted and is no longer being tracked by the provider.
226 FI_FENCE : Applies to transmits. Indicates that the requested opera‐
228 tion, also known as the fenced operation, and any operation posted
229 after the fenced operation will be deferred until all previous opera‐
230 tions targeting the same peer endpoint have completed. Operations
231 posted after the fencing will see and/or replace the results of any
232 operations initiated prior to the fenced operation.
233 The ordering of operations starting at the posting of the fenced opera‐
235 tion (inclusive) to the posting of a subsequent fenced operation
236 (exclusive) is controlled by the endpoint's ordering semantics.
237 The following flags may be used with fi_trecvmsg.
239 FI_PEEK : The peek flag may be used to see if a specified message has
241 arrived. A peek request is often useful on endpoints that have
242 provider allocated buffering enabled (see fi_rx_attr
243 total_buffered_recv). Unlike standard receive operations, a receive
244 operation with the FI_PEEK flag set does not remain queued with the
245 provider after the peek completes successfully. The peek operation
246 operates asynchronously, and the results of the peek operation are
247 available in the completion queue associated with the endpoint. If no
248 message is found matching the tags specified in the peek request, then
249 a completion queue error entry with err field set to FI_ENOMSG will be
250 available.
251 If a peek request locates a matching message, the operation will com‐
253 plete successfully. The returned completion data will indicate the
254 meta-data associated with the message, such as the message length, com‐
255 pletion flags, available CQ data, tag, and source address. The data
256 available is subject to the completion entry format (e.g. struct
257 fi_cq_tagged_entry).
258 An application may supply a buffer if it desires to receive data as a
260 part of the peek operation. In order to receive data as a part of the
261 peek operation, the buf and len fields must be available in the CQ for‐
262 mat. In particular, FI_CQ_FORMAT_CONTEXT and FI_CQ_FORMAT_MSG cannot
263 be used if peek operations desire to obtain a copy of the data. The
264 returned data is limited to the size of the input buffer(s) or the mes‐
265 sage size, if smaller. A provider indicates if data is available by
266 setting the buf field of the CQ entry to the user's first input buffer.
267 If buf is NULL, no data was available to return. A provider may return
268 NULL even if the peek operation completes successfully. Note that the
269 CQ entry len field will reference the size of the message, not neces‐
270 sarily the size of the returned data.
271 FI_CLAIM : If this flag is used in conjunction with FI_PEEK, it indi‐
273 cates if the peek request completes successfully -- indicating that a
274 matching message was located -- the message is claimed by caller.
275 Claimed messages can only be retrieved using a subsequent, paired
276 receive operation with the FI_CLAIM flag set. A receive operation with
277 the FI_CLAIM flag set, but FI_PEEK not set is used to retrieve a previ‐
278 ously claimed message.
279 In order to use the FI_CLAIM flag, an application must supply a struct
281 fi_context structure as the context for the receive operation. The
282 same fi_context structure used for an FI_PEEK + FI_CLAIM operation must
283 be used by the paired FI_CLAIM request.
284 FI_DISCARD : This flag must be used in conjunction with either FI_PEEK
286 or FI_CLAIM. If this flag is used in conjunction with FI_PEEK, it
287 indicates if the peek request completes successfully -- indicating that
288 a matching message was located -- the message is discarded by the
289 provider, as the data is not needed by the application. This flag may
290 also be used in conjunction with FI_CLAIM in order to retrieve and dis‐
291 card a message previously claimed using an FI_PEEK + FI_CLAIM request.
292 If this flag is set, the input buffer(s) and length parameters are
294 ignored.
295
297 The tagged send and receive calls return 0 on success. On error, a
298 negative value corresponding to fabric errno is returned. Fabric
299 errno values are defined in fi_errno.h.
300
302 -FI_EAGAIN : See fi_msg(3) for a detailed description of handling
303 FI_EAGAIN.
304 -FI_EINVAL : Indicates that an invalid argument was supplied by the
306 user.
307 -FI_EOTHER : Indicates that an unspecified error occurred.
309
311 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
312
314 OpenFabrics.
315Libfabric Programmer's Manual 2017-10-20 fi_tagged(3)