1fi_tagged(3)                   Libfabric v1.7.0                   fi_tagged(3)
2
3
4

NAME

6       fi_tagged - Tagged data transfer operations
7
8       fi_trecv / fi_trecvv / fi_trecvmsg
9              Post a buffer to receive an incoming message
10
11       fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12              Initiate an operation to send a message
13

SYNOPSIS

15              #include <rdma/fi_tagged.h>
16
17              ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19
20              ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22                  void *context);
23
24              ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25                  uint64_t flags);
26
27              ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29
30              ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31                  void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32                  void *context);
33
34              ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35                  uint64_t flags);
36
37              ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38                  fi_addr_t dest_addr, uint64_t tag);
39
40              ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42                  void *context);
43
44              ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45                  uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46

ARGUMENTS

48       fid    Fabric endpoint on which to initiate tagged communication opera‐
49              tion.
50
51       buf    Data buffer to send or receive.
52
53       len    Length of data buffer to send or receive.
54
55       iov    Vectored data buffer.
56
57       count  Count of vectored data entries.
58
59       tag    Tag associated with the message.
60
61       ignore Mask of bits to ignore applied to the  tag  for  receive  opera‐
62              tions.
63
64       desc   Memory descriptor associated with the data buffer
65
66       data   Remote CQ data to transfer with the sent data.
67
68       dest_addr
69              Destination  address  for connectionless transfers.  Ignored for
70              connected endpoints.
71
72       src_addr
73              Source address to receive  from  for  connectionless  transfers.
74              Applies  only  to  connectionless  endpoints with the FI_DIRECT‐
75              ED_RECV capability enabled, otherwise this field is ignored.  If
76              set to FI_ADDR_UNSPEC, any source address may match.
77
78       msg    Message descriptor for send and receive operations.
79
80       flags  Additional flags to apply for the send or receive operation.
81
82       context
83              User  specified  pointer  to associate with the operation.  This
84              parameter is ignored if the operation will not generate  a  suc‐
85              cessful  completion, unless an op flag specifies the context pa‐
86              rameter be used for required input.
87

DESCRIPTION

89       Tagged messages are data transfers which carry a key or  tag  with  the
90       message buffer.  The tag is used at the receiving endpoint to match the
91       incoming message with a corresponding  receive  buffer.   Message  tags
92       match  when  the  receive buffer tag is the same as the send buffer tag
93       with the ignored bits masked out.  This can be stated as:
94
95              send_tag & ~ignore == recv_tag & ~ignore
96
97       In general, message tags are checked against receive buffers in the or‐
98       der in which messages have been posted to the endpoint.  See the order‐
99       ing discussion below for more details.
100
101       The send functions -- fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
102       fi_tsenddata -- are used to transmit a tagged message from one endpoint
103       to another endpoint.  The main difference between  send  functions  are
104       the  number  and  type of parameters that they accept as input.  Other‐
105       wise, they perform the same general function.
106
107       The receive functions -- fi_trecv, fi_trecvv, fi_recvmsg -- post a data
108       buffer  to  an endpoint to receive inbound tagged messages.  Similar to
109       the send operations, receive operations operate asynchronously.   Users
110       should  not touch the posted data buffer(s) until the receive operation
111       has completed.  Posted receive buffers are matched  with  inbound  send
112       messages  based  on  the tags associated with the send and receive buf‐
113       fers.
114
115       An endpoint must be enabled before an application can post send or  re‐
116       ceive  operations  to it.  For connected endpoints, receive buffers may
117       be posted prior to connect or accept  being  called  on  the  endpoint.
118       This  ensures that buffers are available to receive incoming data imme‐
119       diately after the connection has been established.
120
121       Completed message operations are reported to the user  through  one  or
122       more event collectors associated with the endpoint.  Users provide con‐
123       text which are associated with each operation, and is returned  to  the
124       user  as  part of the event completion.  See fi_cq for completion event
125       details.
126
127   fi_tsend
128       The call fi_tsend transfers the data contained  in  the  user-specified
129       data  buffer  to a remote endpoint, with message boundaries being main‐
130       tained.  The local endpoint must be connected to a remote  endpoint  or
131       destination  before  fi_tsend  is called.  Unless the endpoint has been
132       configured differently, the data buffer passed into fi_tsend  must  not
133       be  touched  by the application until the fi_tsend call completes asyn‐
134       chronously.
135
136   fi_tsendv
137       The fi_tsendv call adds support for a scatter-gather list to  fi_tsend.
138       The  fi_sendv  transfers  the set of data buffers referenced by the iov
139       parameter to a remote endpoint as a single message.
140
141   fi_tsendmsg
142       The fi_tsendmsg call supports data transfers over  both  connected  and
143       unconnected  endpoints,  with the ability to control the send operation
144       per call through the use of flags.  The fi_tsendmsg  function  takes  a
145       struct fi_msg_tagged as input.
146
147              struct fi_msg_tagged {
148                  const struct iovec *msg_iov; /* scatter-gather array */
149                  void               *desc;    /* data descriptor */
150                  size_t             iov_count;/* # elements in msg_iov */
151                  fi_addr_t          addr;    /* optional endpoint address */
152                  uint64_t           tag;      /* tag associated with message */
153                  uint64_t           ignore;   /* mask applied to tag for receives */
154                  void               *context; /* user-defined context */
155                  uint64_t           data;     /* optional immediate data */
156              };
157
158   fi_tinject
159       The  tagged  inject  call  is  an  optimized  version of fi_tsend.  The
160       fi_tinject function behaves as if the FI_INJECT transfer flag were set,
161       and  FI_COMPLETION were not.  That is, the data buffer is available for
162       reuse immediately on returning from fi_tinject, and no completion event
163       will  be  generated  for  this send.  The completion event will be sup‐
164       pressed even if the endpoint has not  been  configured  with  FI_SELEC‐
165       TIVE_COMPLETION.  See the flags discussion below for more details.  The
166       requested message size that can be used with fi_tinject is  limited  by
167       inject_size.
168
169   fi_tsenddata
170       The  tagged  send  data call is similar to fi_tsend, but allows for the
171       sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part  of  the
172       transfer.
173
174   fi_tinjectdata
175       The  tagged  inject  data call is similar to fi_tinject, but allows for
176       the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as  part  of
177       the transfer.
178
179   fi_trecv
180       The  fi_trecv call posts a data buffer to the receive queue of the cor‐
181       responding endpoint.  Posted receives are  searched  in  the  order  in
182       which they were posted in order to match sends.  Message boundaries are
183       maintained.  The order in which the receives complete is  dependent  on
184       the endpoint type and protocol.
185
186   fi_trecvv
187       The  fi_trecvv call adds support for a scatter-gather list to fi_trecv.
188       The fi_trecvv posts the set of data buffers referenced by the  iov  pa‐
189       rameter to a receive incoming data.
190
191   fi_trecvmsg
192       The  fi_trecvmsg  call supports posting buffers over both connected and
193       unconnected endpoints, with the ability to control the  receive  opera‐
194       tion per call through the use of flags.  The fi_trecvmsg function takes
195       a struct fi_msg_tagged as input.
196

FLAGS

198       The fi_trecvmsg and fi_tsendmsg calls allow the user to  specify  flags
199       which  can  change the default message handling of the endpoint.  Flags
200       specified with fi_trecvmsg / fi_tsendmsg override most flags previously
201       configured  with  the  endpoint,  except where noted (see fi_endpoint).
202       The  following  list  of  flags  are  usable  with  fi_trecvmsg  and/or
203       fi_tsendmsg.
204
205       FI_REMOTE_CQ_DATA
206              Applies  to fi_tsendmsg and fi_tsenddata.  Indicates that remote
207              CQ data is available and should be sent as part of the  request.
208              See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
209
210       FI_COMPLETION
211              Indicates  that  a  completion entry should be generated for the
212              specified operation.  The endpoint must be bound to a completion
213              queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
214              ified operation, or this flag is ignored.
215
216       FI_MORE
217              Indicates that the user has additional requests that will  imme‐
218              diately  be  posted after the current call returns.  Use of this
219              flag may improve performance by enabling the provider  to  opti‐
220              mize its access to the fabric hardware.
221
222       FI_INJECT
223              Applies to fi_tsendmsg.  Indicates that the outbound data buffer
224              should be returned to user immediately after the send  call  re‐
225              turns,  even  if  the operation is handled asynchronously.  This
226              may require that the underlying provider implementation copy the
227              data  into a local buffer and transfer out of that buffer.  This
228              flag can only be used with messages smaller than inject_size.
229
230       FI_INJECT_COMPLETE
231              Applies to fi_tsendmsg.  Indicates that a completion  should  be
232              generated when the source buffer(s) may be reused.
233
234       FI_TRANSMIT_COMPLETE
235              Applies  to fi_tsendmsg.  Indicates that a completion should not
236              be generated until the operation has been successfully transmit‐
237              ted and is no longer being tracked by the provider.
238
239       FI_MATCH_COMPLETE
240              Applies  to  fi_tsendmsg.  Indicates that a completion should be
241              generated only after the message has either been matched with  a
242              tagged buffer or was discarded by the target application.
243
244       FI_FENCE
245              Applies  to  transmits.  Indicates that the requested operation,
246              also known as the fenced operation, and any operation posted af‐
247              ter the fenced operation will be deferred until all previous op‐
248              erations targeting the same peer endpoint have completed.  Oper‐
249              ations  posted after the fencing will see and/or replace the re‐
250              sults of any operations initiated prior to the fenced operation.
251
252       The ordering of operations starting at the posting of the fenced opera‐
253       tion  (inclusive)  to the posting of a subsequent fenced operation (ex‐
254       clusive) is controlled by the endpoint's ordering semantics.
255
256       The following flags may be used with fi_trecvmsg.
257
258       FI_PEEK
259              The peek flag may be used to see if a specified message has  ar‐
260              rived.   A  peek  request is often useful on endpoints that have
261              provider  allocated  buffering  enabled  (see   fi_rx_attr   to‐
262              tal_buffered_recv).   Unlike  standard receive operations, a re‐
263              ceive operation with the FI_PEEK flag set does not remain queued
264              with  the  provider  after the peek completes successfully.  The
265              peek operation operates asynchronously, and the results  of  the
266              peek  operation are available in the completion queue associated
267              with the endpoint.  If no message is  found  matching  the  tags
268              specified in the peek request, then a completion queue error en‐
269              try with err field set to FI_ENOMSG will be available.
270
271       If a peek request locates a matching message, the operation  will  com‐
272       plete  successfully.   The  returned  completion data will indicate the
273       meta-data associated with the message, such as the message length, com‐
274       pletion  flags,  available  CQ data, tag, and source address.  The data
275       available is subject to  the  completion  entry  format  (e.g.   struct
276       fi_cq_tagged_entry).
277
278       An  application  may supply a buffer if it desires to receive data as a
279       part of the peek operation.  In order to receive data as a part of  the
280       peek operation, the buf and len fields must be available in the CQ for‐
281       mat.  In particular, FI_CQ_FORMAT_CONTEXT and  FI_CQ_FORMAT_MSG  cannot
282       be  used  if  peek operations desire to obtain a copy of the data.  The
283       returned data is limited to the size of the input buffer(s) or the mes‐
284       sage  size,  if  smaller.  A provider indicates if data is available by
285       setting the buf field of the CQ entry to the user's first input buffer.
286       If buf is NULL, no data was available to return.  A provider may return
287       NULL even if the peek operation completes successfully.  Note that  the
288       CQ  entry  len field will reference the size of the message, not neces‐
289       sarily the size of the returned data.
290
291       FI_CLAIM
292              If this flag is used in conjunction with FI_PEEK,  it  indicates
293              if  the peek request completes successfully -- indicating that a
294              matching message was located -- the message is claimed by  call‐
295              er.   Claimed messages can only be retrieved using a subsequent,
296              paired receive operation with the FI_CLAIM flag set.  A  receive
297              operation  with  the  FI_CLAIM  flag set, but FI_PEEK not set is
298              used to retrieve a previously claimed message.
299
300       In order to use the FI_CLAIM flag, an application must supply a  struct
301       fi_context  structure  as  the  context for the receive operation, or a
302       struct fi_recv_context in the case  of  buffered  receives.   The  same
303       fi_context  structure  used for an FI_PEEK + FI_CLAIM operation must be
304       used by the paired FI_CLAIM request.
305
306       This flag also applies to endpoints configured for FI_BUFFERED_RECV  or
307       FI_VARIABLE_MSG.   When  set,  it  is used to retrieve a tagged message
308       that was buffered by the provider.  See Buffered Tagged  Receives  sec‐
309       tion for details.
310
311       FI_DISCARD
312              This  flag  may  be  used  in conjunction with either FI_PEEK or
313              FI_CLAIM.  If this flag is used in conjunction with FI_PEEK,  it
314              indicates if the peek request completes successfully -- indicat‐
315              ing that a matching message was located -- the message  is  dis‐
316              carded  by the provider, as the data is not needed by the appli‐
317              cation.  This flag may also be used in conjunction with FI_CLAIM
318              in  order  to  discard  a  message  previously  claimed using an
319              FI_PEEK + FI_CLAIM request.
320
321       This flag also applies to endpoints configured for FI_BUFFERED_RECV  or
322       FI_VARIABLE_MSG.   When set, it indicates that the provider should free
323       a buffered messages.  See Buffered Tagged Receives section for details.
324
325       If this flag is set, the input buffer(s) and length parameters are  ig‐
326       nored.
327

Buffered Tagged Receives

329       See  fi_msg(3)  for an introduction to buffered receives.  The handling
330       of buffered receives differs between fi_msg operations  and  fi_tagged.
331       Although  the  provider is responsible for allocating and managing net‐
332       work buffers, the application is responsible for identifying  the  tags
333       that  will  be  used  to match incoming messages.  The provider handles
334       matching incoming receives to the application specified tags.
335
336       When FI_BUFFERED_RECV is enabled, the application posts the  tags  that
337       will  be  used  for matching purposes.  Tags are posted using fi_trecv,
338       fi_trecvv, and fi_trecvmsg; however, parameters related  to  the  input
339       buffers  are  ignored (e.g.  buf, len, iov, desc).  When a provider re‐
340       ceives a message for which there is a matching tag, it  will  write  an
341       entry to the completion queue associated with the receiving endpoint.
342
343       For  discussion purposes, the completion queue is assumed to be config‐
344       ured for FI_CQ_FORMAT_TAGGED.  The op_context field  will  point  to  a
345       struct fi_recv_contex.
346
347              struct fi_recv_context {
348                  struct fid_ep *ep;
349                  void *context;
350              };
351
352       The 'ep' field will be NULL.  The 'context' field will match the appli‐
353       cation context specified when posting the tag.  Other fields are set as
354       defined in fi_msg(3).
355
356       After  being notified that a buffered receive has arrived, applications
357       must either claim or discard the message as described in fi_msg(3).
358

Variable Length Tagged Messages

360       Variable length messages are defined in  fi_msg(3).   The  requirements
361       for  handling variable length tagged messages is identical to those de‐
362       fined above for buffered tagged receives.
363

RETURN VALUE

365       The tagged send and receive calls return 0 on  success.   On  error,  a
366       negative  value corresponding to fabric errno  is returned.  Fabric er‐
367       rno values are defined in fi_errno.h.
368

ERRORS

370       -FI_EAGAIN
371              See fi_msg(3) for a detailed description of handling FI_EAGAIN.
372
373       -FI_EINVAL
374              Indicates that an invalid argument was supplied by the user.
375
376       -FI_EOTHER
377              Indicates that an unspecified error occurred.
378

SEE ALSO

380       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
381

AUTHORS

383       OpenFabrics.
384
385
386
387Libfabric Programmer's Manual     2018-11-28                      fi_tagged(3)
Impressum