1fi_tagged(3)                   Libfabric v1.17.0                  fi_tagged(3)
2
3
4

NAME

6       fi_tagged - Tagged data transfer operations
7
8       fi_trecv / fi_trecvv / fi_trecvmsg
9              Post a buffer to receive an incoming message
10
11       fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12              Initiate an operation to send a message
13

SYNOPSIS

15              #include <rdma/fi_tagged.h>
16
17              ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19
20              ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22                  void *context);
23
24              ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25                  uint64_t flags);
26
27              ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29
30              ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31                  void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32                  void *context);
33
34              ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35                  uint64_t flags);
36
37              ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38                  fi_addr_t dest_addr, uint64_t tag);
39
40              ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42                  void *context);
43
44              ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45                  uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46

ARGUMENTS

48       fid    Fabric endpoint on which to initiate tagged communication opera‐
49              tion.
50
51       buf    Data buffer to send or receive.
52
53       len    Length of data buffer to send or receive.
54
55       iov    Vectored data buffer.
56
57       count  Count of vectored data entries.
58
59       tag    Tag associated with the message.
60
61       ignore Mask of bits to ignore applied to the  tag  for  receive  opera‐
62              tions.
63
64       desc   Memory   descriptor   associated  with  the  data  buffer.   See
65              fi_mr(3).
66
67       data   Remote CQ data to transfer with the sent data.
68
69       dest_addr
70              Destination address for connectionless transfers.   Ignored  for
71              connected endpoints.
72
73       src_addr
74              Source  address  to  receive  from for connectionless transfers.
75              Applies only to connectionless  endpoints  with  the  FI_DIRECT‐
76              ED_RECV capability enabled, otherwise this field is ignored.  If
77              set to FI_ADDR_UNSPEC, any source address may match.
78
79       msg    Message descriptor for send and receive operations.
80
81       flags  Additional flags to apply for the send or receive operation.
82
83       context
84              User specified pointer to associate with  the  operation.   This
85              parameter  is  ignored if the operation will not generate a suc‐
86              cessful completion, unless an op flag specifies the context  pa‐
87              rameter be used for required input.
88

DESCRIPTION

90       Tagged  messages  are  data transfers which carry a key or tag with the
91       message buffer.  The tag is used at the receiving endpoint to match the
92       incoming  message  with  a  corresponding receive buffer.  Message tags
93       match when the receive buffer tag is the same as the  send  buffer  tag
94       with the ignored bits masked out.  This can be stated as:
95
96              send_tag & ~ignore == recv_tag & ~ignore
97
98       In general, message tags are checked against receive buffers in the or‐
99       der in which messages have been posted to the endpoint.  See the order‐
100       ing discussion below for more details.
101
102       The  send functions – fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
103       fi_tsenddata – are used to transmit a tagged message from one  endpoint
104       to  another  endpoint.   The main difference between send functions are
105       the number and type of parameters that they accept  as  input.   Other‐
106       wise, they perform the same general function.
107
108       The  receive  functions – fi_trecv, fi_trecvv, fi_recvmsg – post a data
109       buffer to an endpoint to receive inbound tagged messages.   Similar  to
110       the  send operations, receive operations operate asynchronously.  Users
111       should not touch the posted data buffer(s) until the receive  operation
112       has  completed.   Posted  receive buffers are matched with inbound send
113       messages based on the tags associated with the send  and  receive  buf‐
114       fers.
115
116       An  endpoint must be enabled before an application can post send or re‐
117       ceive operations to it.  For connected endpoints, receive  buffers  may
118       be  posted  prior  to  connect  or accept being called on the endpoint.
119       This ensures that buffers are available to receive incoming data  imme‐
120       diately after the connection has been established.
121
122       Completed  message  operations  are reported to the user through one or
123       more event collectors associated with the endpoint.  Users provide con‐
124       text  which  are associated with each operation, and is returned to the
125       user as part of the event completion.  See fi_cq for  completion  event
126       details.
127
128   fi_tsend
129       The  call  fi_tsend  transfers the data contained in the user-specified
130       data buffer to a remote endpoint, with message boundaries  being  main‐
131       tained.   The  local endpoint must be connected to a remote endpoint or
132       destination before fi_tsend is called.  Unless the  endpoint  has  been
133       configured  differently,  the data buffer passed into fi_tsend must not
134       be touched by the application until the fi_tsend call  completes  asyn‐
135       chronously.
136
137   fi_tsendv
138       The  fi_tsendv call adds support for a scatter-gather list to fi_tsend.
139       The fi_sendv transfers the set of data buffers referenced  by  the  iov
140       parameter to a remote endpoint as a single message.
141
142   fi_tsendmsg
143       The  fi_tsendmsg  call  supports data transfers over both connected and
144       connectionless endpoints, with the ability to control the  send  opera‐
145       tion per call through the use of flags.  The fi_tsendmsg function takes
146       a struct fi_msg_tagged as input.
147
148              struct fi_msg_tagged {
149                  const struct iovec *msg_iov; /* scatter-gather array */
150                  void               *desc;    /* data descriptor */
151                  size_t             iov_count;/* # elements in msg_iov */
152                  fi_addr_t          addr;    /* optional endpoint address */
153                  uint64_t           tag;      /* tag associated with message */
154                  uint64_t           ignore;   /* mask applied to tag for receives */
155                  void               *context; /* user-defined context */
156                  uint64_t           data;     /* optional immediate data */
157              };
158
159   fi_tinject
160       The tagged inject call is an optimized version of  fi_tsend.   It  pro‐
161       vides similar completion semantics as fi_inject fi_msg(3).
162
163   fi_tsenddata
164       The  tagged  send  data call is similar to fi_tsend, but allows for the
165       sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part  of  the
166       transfer.
167
168   fi_tinjectdata
169       The  tagged  inject  data call is similar to fi_tinject, but allows for
170       the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as  part  of
171       the transfer.
172
173   fi_trecv
174       The  fi_trecv call posts a data buffer to the receive queue of the cor‐
175       responding endpoint.  Posted receives are  searched  in  the  order  in
176       which they were posted in order to match sends.  Message boundaries are
177       maintained.  The order in which the receives complete is  dependent  on
178       the endpoint type and protocol.
179
180   fi_trecvv
181       The  fi_trecvv call adds support for a scatter-gather list to fi_trecv.
182       The fi_trecvv posts the set of data buffers referenced by the  iov  pa‐
183       rameter to a receive incoming data.
184
185   fi_trecvmsg
186       The  fi_trecvmsg  call supports posting buffers over both connected and
187       connectionless endpoints, with the ability to control the receive oper‐
188       ation  per  call  through  the  use of flags.  The fi_trecvmsg function
189       takes a struct fi_msg_tagged as input.
190

FLAGS

192       The fi_trecvmsg and fi_tsendmsg calls allow the user to  specify  flags
193       which  can  change the default message handling of the endpoint.  Flags
194       specified with fi_trecvmsg / fi_tsendmsg override most flags previously
195       configured  with  the  endpoint,  except where noted (see fi_endpoint).
196       The  following  list  of  flags  are  usable  with  fi_trecvmsg  and/or
197       fi_tsendmsg.
198
199       FI_REMOTE_CQ_DATA
200              Applies  to fi_tsendmsg and fi_tsenddata.  Indicates that remote
201              CQ data is available and should be sent as part of the  request.
202              See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
203
204       FI_COMPLETION
205              Indicates  that  a  completion entry should be generated for the
206              specified operation.  The endpoint must be bound to a completion
207              queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
208              ified operation, or this flag is ignored.
209
210       FI_MORE
211              Indicates that the user has additional requests that will  imme‐
212              diately  be  posted after the current call returns.  Use of this
213              flag may improve performance by enabling the provider  to  opti‐
214              mize its access to the fabric hardware.
215
216       FI_INJECT
217              Applies to fi_tsendmsg.  Indicates that the outbound data buffer
218              should be returned to user immediately after the send  call  re‐
219              turns,  even  if  the operation is handled asynchronously.  This
220              may require that the underlying provider implementation copy the
221              data  into a local buffer and transfer out of that buffer.  This
222              flag can only be used with messages smaller than inject_size.
223
224       FI_INJECT_COMPLETE
225              Applies to fi_tsendmsg.  Indicates that a completion  should  be
226              generated when the source buffer(s) may be reused.
227
228       FI_TRANSMIT_COMPLETE
229              Applies  to fi_tsendmsg.  Indicates that a completion should not
230              be generated until the operation has been successfully transmit‐
231              ted and is no longer being tracked by the provider.
232
233       FI_MATCH_COMPLETE
234              Applies  to  fi_tsendmsg.  Indicates that a completion should be
235              generated only after the message has either been matched with  a
236              tagged buffer or was discarded by the target application.
237
238       FI_FENCE
239              Applies  to  transmits.  Indicates that the requested operation,
240              also known as the fenced operation, and any operation posted af‐
241              ter the fenced operation will be deferred until all previous op‐
242              erations targeting the same peer endpoint have completed.  Oper‐
243              ations  posted after the fencing will see and/or replace the re‐
244              sults of any operations initiated prior to the fenced operation.
245
246       The ordering of operations starting at the posting of the fenced opera‐
247       tion  (inclusive)  to the posting of a subsequent fenced operation (ex‐
248       clusive) is controlled by the endpoint’s ordering semantics.
249
250       The following flags may be used with fi_trecvmsg.
251
252       FI_PEEK
253              The peek flag may be used to see if a specified message has  ar‐
254              rived.   A  peek  request is often useful on endpoints that have
255              provider  allocated  buffering  enabled  (see   fi_rx_attr   to‐
256              tal_buffered_recv).   Unlike  standard receive operations, a re‐
257              ceive operation with the FI_PEEK flag set does not remain queued
258              with  the  provider  after the peek completes successfully.  The
259              peek operation operates asynchronously, and the results  of  the
260              peek  operation are available in the completion queue associated
261              with the endpoint.  If no message is  found  matching  the  tags
262              specified in the peek request, then a completion queue error en‐
263              try with err field set to FI_ENOMSG will be available.
264
265       If a peek request locates a matching message, the operation  will  com‐
266       plete  successfully.   The  returned  completion data will indicate the
267       meta-data associated with the message, such as the message length, com‐
268       pletion  flags,  available  CQ data, tag, and source address.  The data
269       available is  subject  to  the  completion  entry  format  (e.g. struct
270       fi_cq_tagged_entry).
271
272       An  application  may supply a buffer if it desires to receive data as a
273       part of the peek operation.  In order to receive data as a part of  the
274       peek operation, the buf and len fields must be available in the CQ for‐
275       mat.  In particular, FI_CQ_FORMAT_CONTEXT and  FI_CQ_FORMAT_MSG  cannot
276       be  used  if  peek operations desire to obtain a copy of the data.  The
277       returned data is limited to the size of the input buffer(s) or the mes‐
278       sage  size,  if  smaller.  A provider indicates if data is available by
279       setting the buf field of the CQ entry to the user’s first input buffer.
280       If buf is NULL, no data was available to return.  A provider may return
281       NULL even if the peek operation completes successfully.  Note that  the
282       CQ  entry  len field will reference the size of the message, not neces‐
283       sarily the size of the returned data.
284
285       FI_CLAIM
286              If this flag is used in conjunction with FI_PEEK,  it  indicates
287              if  the  peek request completes successfully – indicating that a
288              matching message was located – the message is claimed by caller.
289              Claimed  messages  can  only  be  retrieved  using a subsequent,
290              paired receive operation with the FI_CLAIM flag set.  A  receive
291              operation  with  the  FI_CLAIM  flag set, but FI_PEEK not set is
292              used to retrieve a previously claimed message.
293
294       In order to use the FI_CLAIM flag, an application must supply a  struct
295       fi_context  structure  as  the  context for the receive operation, or a
296       struct fi_recv_context in the case  of  buffered  receives.   The  same
297       fi_context  structure  used for an FI_PEEK + FI_CLAIM operation must be
298       used by the paired FI_CLAIM request.
299
300       This flag also applies to endpoints configured for FI_BUFFERED_RECV  or
301       FI_VARIABLE_MSG.   When  set,  it  is used to retrieve a tagged message
302       that was buffered by the provider.  See Buffered Tagged  Receives  sec‐
303       tion for details.
304
305       FI_DISCARD
306              This  flag  may  be  used  in conjunction with either FI_PEEK or
307              FI_CLAIM.  If this flag is used in conjunction with FI_PEEK,  it
308              indicates  if the peek request completes successfully – indicat‐
309              ing that a matching message was located – the  message  is  dis‐
310              carded  by the provider, as the data is not needed by the appli‐
311              cation.  This flag may also be used in conjunction with FI_CLAIM
312              in  order  to  discard  a  message  previously  claimed using an
313              FI_PEEK + FI_CLAIM request.
314
315       This flag also applies to endpoints configured for FI_BUFFERED_RECV  or
316       FI_VARIABLE_MSG.   When set, it indicates that the provider should free
317       a buffered messages.  See Buffered Tagged Receives section for details.
318
319       If this flag is set, the input buffer(s) and length parameters are  ig‐
320       nored.
321

Buffered Tagged Receives

323       See  fi_msg(3)  for an introduction to buffered receives.  The handling
324       of buffered receives differs between fi_msg operations  and  fi_tagged.
325       Although  the  provider is responsible for allocating and managing net‐
326       work buffers, the application is responsible for identifying  the  tags
327       that  will  be  used  to match incoming messages.  The provider handles
328       matching incoming receives to the application specified tags.
329
330       When FI_BUFFERED_RECV is enabled, the application posts the  tags  that
331       will  be  used  for matching purposes.  Tags are posted using fi_trecv,
332       fi_trecvv, and fi_trecvmsg; however, parameters related  to  the  input
333       buffers  are  ignored  (e.g. buf, len, iov, desc).  When a provider re‐
334       ceives a message for which there is a matching tag, it  will  write  an
335       entry to the completion queue associated with the receiving endpoint.
336
337       For  discussion purposes, the completion queue is assumed to be config‐
338       ured for FI_CQ_FORMAT_TAGGED.  The op_context field  will  point  to  a
339       struct fi_recv_context.
340
341              struct fi_recv_context {
342                  struct fid_ep *ep;
343                  void *context;
344              };
345
346       The `ep' field will be NULL.  The `context' field will match the appli‐
347       cation context specified when posting the tag.  Other fields are set as
348       defined in fi_msg(3).
349
350       After  being notified that a buffered receive has arrived, applications
351       must either claim or discard the message as described in fi_msg(3).
352

Variable Length Tagged Messages

354       Variable length messages are defined in  fi_msg(3).   The  requirements
355       for  handling variable length tagged messages is identical to those de‐
356       fined above for buffered tagged receives.
357

RETURN VALUE

359       The tagged send and receive calls return 0 on  success.   On  error,  a
360       negative  value corresponding to fabric errno  is returned.  Fabric er‐
361       rno values are defined in fi_errno.h.
362

ERRORS

364       -FI_EAGAIN
365              See fi_msg(3) for a detailed description of handling FI_EAGAIN.
366
367       -FI_EINVAL
368              Indicates that an invalid argument was supplied by the user.
369
370       -FI_EOTHER
371              Indicates that an unspecified error occurred.
372

SEE ALSO

374       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
375

AUTHORS

377       OpenFabrics.
378
379
380
381Libfabric Programmer’s Manual     2022-12-11                      fi_tagged(3)
Impressum