1fi_tagged(3)                   Libfabric v1.8.0                   fi_tagged(3)
2
3
4

NAME

6       fi_tagged - Tagged data transfer operations
7
8       fi_trecv / fi_trecvv / fi_trecvmsg
9              Post a buffer to receive an incoming message
10
11       fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12              Initiate an operation to send a message
13

SYNOPSIS

15              #include <rdma/fi_tagged.h>
16
17              ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19
20              ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22                  void *context);
23
24              ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25                  uint64_t flags);
26
27              ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29
30              ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31                  void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32                  void *context);
33
34              ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35                  uint64_t flags);
36
37              ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38                  fi_addr_t dest_addr, uint64_t tag);
39
40              ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42                  void *context);
43
44              ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45                  uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46

ARGUMENTS

48       fid    Fabric endpoint on which to initiate tagged communication opera‐
49              tion.
50
51       buf    Data buffer to send or receive.
52
53       len    Length of data buffer to send or receive.
54
55       iov    Vectored data buffer.
56
57       count  Count of vectored data entries.
58
59       tag    Tag associated with the message.
60
61       ignore Mask of bits to ignore applied to the  tag  for  receive  opera‐
62              tions.
63
64       desc   Memory descriptor associated with the data buffer
65
66       data   Remote CQ data to transfer with the sent data.
67
68       dest_addr
69              Destination  address  for connectionless transfers.  Ignored for
70              connected endpoints.
71
72       src_addr
73              Source address to receive  from  for  connectionless  transfers.
74              Applies  only  to  connectionless  endpoints with the FI_DIRECT‐
75              ED_RECV capability enabled, otherwise this field is ignored.  If
76              set to FI_ADDR_UNSPEC, any source address may match.
77
78       msg    Message descriptor for send and receive operations.
79
80       flags  Additional flags to apply for the send or receive operation.
81
82       context
83              User  specified  pointer  to associate with the operation.  This
84              parameter is ignored if the operation will not generate  a  suc‐
85              cessful  completion, unless an op flag specifies the context pa‐
86              rameter be used for required input.
87

DESCRIPTION

89       Tagged messages are data transfers which carry a key or  tag  with  the
90       message buffer.  The tag is used at the receiving endpoint to match the
91       incoming message with a corresponding  receive  buffer.   Message  tags
92       match  when  the  receive buffer tag is the same as the send buffer tag
93       with the ignored bits masked out.  This can be stated as:
94
95              send_tag & ~ignore == recv_tag & ~ignore
96
97       In general, message tags are checked against receive buffers in the or‐
98       der in which messages have been posted to the endpoint.  See the order‐
99       ing discussion below for more details.
100
101       The send functions -- fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
102       fi_tsenddata -- are used to transmit a tagged message from one endpoint
103       to another endpoint.  The main difference between  send  functions  are
104       the  number  and  type of parameters that they accept as input.  Other‐
105       wise, they perform the same general function.
106
107       The receive functions -- fi_trecv, fi_trecvv, fi_recvmsg -- post a data
108       buffer  to  an endpoint to receive inbound tagged messages.  Similar to
109       the send operations, receive operations operate asynchronously.   Users
110       should  not touch the posted data buffer(s) until the receive operation
111       has completed.  Posted receive buffers are matched  with  inbound  send
112       messages  based  on  the tags associated with the send and receive buf‐
113       fers.
114
115       An endpoint must be enabled before an application can post send or  re‐
116       ceive  operations  to it.  For connected endpoints, receive buffers may
117       be posted prior to connect or accept  being  called  on  the  endpoint.
118       This  ensures that buffers are available to receive incoming data imme‐
119       diately after the connection has been established.
120
121       Completed message operations are reported to the user  through  one  or
122       more event collectors associated with the endpoint.  Users provide con‐
123       text which are associated with each operation, and is returned  to  the
124       user  as  part of the event completion.  See fi_cq for completion event
125       details.
126
127   fi_tsend
128       The call fi_tsend transfers the data contained  in  the  user-specified
129       data  buffer  to a remote endpoint, with message boundaries being main‐
130       tained.  The local endpoint must be connected to a remote  endpoint  or
131       destination  before  fi_tsend  is called.  Unless the endpoint has been
132       configured differently, the data buffer passed into fi_tsend  must  not
133       be  touched  by the application until the fi_tsend call completes asyn‐
134       chronously.
135
136   fi_tsendv
137       The fi_tsendv call adds support for a scatter-gather list to  fi_tsend.
138       The  fi_sendv  transfers  the set of data buffers referenced by the iov
139       parameter to a remote endpoint as a single message.
140
141   fi_tsendmsg
142       The fi_tsendmsg call supports data transfers over  both  connected  and
143       unconnected  endpoints,  with the ability to control the send operation
144       per call through the use of flags.  The fi_tsendmsg  function  takes  a
145       struct fi_msg_tagged as input.
146
147              struct fi_msg_tagged {
148                  const struct iovec *msg_iov; /* scatter-gather array */
149                  void               *desc;    /* data descriptor */
150                  size_t             iov_count;/* # elements in msg_iov */
151                  fi_addr_t          addr;    /* optional endpoint address */
152                  uint64_t           tag;      /* tag associated with message */
153                  uint64_t           ignore;   /* mask applied to tag for receives */
154                  void               *context; /* user-defined context */
155                  uint64_t           data;     /* optional immediate data */
156              };
157
158   fi_tinject
159       The  tagged  inject  call is an optimized version of fi_tsend.  It pro‐
160       vides similar completion semantics as fi_inject fi_msg(3).
161
162   fi_tsenddata
163       The tagged send data call is similar to fi_tsend, but  allows  for  the
164       sending  of  remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the
165       transfer.
166
167   fi_tinjectdata
168       The tagged inject data call is similar to fi_tinject,  but  allows  for
169       the  sending  of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of
170       the transfer.
171
172   fi_trecv
173       The fi_trecv call posts a data buffer to the receive queue of the  cor‐
174       responding  endpoint.   Posted  receives  are  searched in the order in
175       which they were posted in order to match sends.  Message boundaries are
176       maintained.   The  order in which the receives complete is dependent on
177       the endpoint type and protocol.
178
179   fi_trecvv
180       The fi_trecvv call adds support for a scatter-gather list to  fi_trecv.
181       The  fi_trecvv  posts the set of data buffers referenced by the iov pa‐
182       rameter to a receive incoming data.
183
184   fi_trecvmsg
185       The fi_trecvmsg call supports posting buffers over both  connected  and
186       unconnected  endpoints,  with the ability to control the receive opera‐
187       tion per call through the use of flags.  The fi_trecvmsg function takes
188       a struct fi_msg_tagged as input.
189

FLAGS

191       The  fi_trecvmsg  and fi_tsendmsg calls allow the user to specify flags
192       which can change the default message handling of the  endpoint.   Flags
193       specified with fi_trecvmsg / fi_tsendmsg override most flags previously
194       configured with the endpoint, except  where  noted  (see  fi_endpoint).
195       The  following  list  of  flags  are  usable  with  fi_trecvmsg  and/or
196       fi_tsendmsg.
197
198       FI_REMOTE_CQ_DATA
199              Applies to fi_tsendmsg and fi_tsenddata.  Indicates that  remote
200              CQ  data is available and should be sent as part of the request.
201              See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
202
203       FI_COMPLETION
204              Indicates that a completion entry should be  generated  for  the
205              specified operation.  The endpoint must be bound to a completion
206              queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
207              ified operation, or this flag is ignored.
208
209       FI_MORE
210              Indicates  that the user has additional requests that will imme‐
211              diately be posted after the current call returns.  Use  of  this
212              flag  may  improve performance by enabling the provider to opti‐
213              mize its access to the fabric hardware.
214
215       FI_INJECT
216              Applies to fi_tsendmsg.  Indicates that the outbound data buffer
217              should  be  returned to user immediately after the send call re‐
218              turns, even if the operation is  handled  asynchronously.   This
219              may require that the underlying provider implementation copy the
220              data into a local buffer and transfer out of that buffer.   This
221              flag can only be used with messages smaller than inject_size.
222
223       FI_INJECT_COMPLETE
224              Applies  to  fi_tsendmsg.  Indicates that a completion should be
225              generated when the source buffer(s) may be reused.
226
227       FI_TRANSMIT_COMPLETE
228              Applies to fi_tsendmsg.  Indicates that a completion should  not
229              be generated until the operation has been successfully transmit‐
230              ted and is no longer being tracked by the provider.
231
232       FI_MATCH_COMPLETE
233              Applies to fi_tsendmsg.  Indicates that a completion  should  be
234              generated  only after the message has either been matched with a
235              tagged buffer or was discarded by the target application.
236
237       FI_FENCE
238              Applies to transmits.  Indicates that the  requested  operation,
239              also known as the fenced operation, and any operation posted af‐
240              ter the fenced operation will be deferred until all previous op‐
241              erations targeting the same peer endpoint have completed.  Oper‐
242              ations posted after the fencing will see and/or replace the  re‐
243              sults of any operations initiated prior to the fenced operation.
244
245       The ordering of operations starting at the posting of the fenced opera‐
246       tion (inclusive) to the posting of a subsequent fenced  operation  (ex‐
247       clusive) is controlled by the endpoint's ordering semantics.
248
249       The following flags may be used with fi_trecvmsg.
250
251       FI_PEEK
252              The  peek flag may be used to see if a specified message has ar‐
253              rived.  A peek request is often useful on  endpoints  that  have
254              provider   allocated   buffering  enabled  (see  fi_rx_attr  to‐
255              tal_buffered_recv).  Unlike standard receive operations,  a  re‐
256              ceive operation with the FI_PEEK flag set does not remain queued
257              with the provider after the peek  completes  successfully.   The
258              peek  operation  operates asynchronously, and the results of the
259              peek operation are available in the completion queue  associated
260              with  the  endpoint.   If  no message is found matching the tags
261              specified in the peek request, then a completion queue error en‐
262              try with err field set to FI_ENOMSG will be available.
263
264       If  a  peek request locates a matching message, the operation will com‐
265       plete successfully.  The returned completion  data  will  indicate  the
266       meta-data associated with the message, such as the message length, com‐
267       pletion flags, available CQ data, tag, and source  address.   The  data
268       available  is  subject  to  the  completion  entry format (e.g.  struct
269       fi_cq_tagged_entry).
270
271       An application may supply a buffer if it desires to receive data  as  a
272       part  of the peek operation.  In order to receive data as a part of the
273       peek operation, the buf and len fields must be available in the CQ for‐
274       mat.   In  particular, FI_CQ_FORMAT_CONTEXT and FI_CQ_FORMAT_MSG cannot
275       be used if peek operations desire to obtain a copy of  the  data.   The
276       returned data is limited to the size of the input buffer(s) or the mes‐
277       sage size, if smaller.  A provider indicates if data  is  available  by
278       setting the buf field of the CQ entry to the user's first input buffer.
279       If buf is NULL, no data was available to return.  A provider may return
280       NULL  even if the peek operation completes successfully.  Note that the
281       CQ entry len field will reference the size of the message,  not  neces‐
282       sarily the size of the returned data.
283
284       FI_CLAIM
285              If  this  flag is used in conjunction with FI_PEEK, it indicates
286              if the peek request completes successfully -- indicating that  a
287              matching  message was located -- the message is claimed by call‐
288              er.  Claimed messages can only be retrieved using a  subsequent,
289              paired  receive operation with the FI_CLAIM flag set.  A receive
290              operation with the FI_CLAIM flag set, but  FI_PEEK  not  set  is
291              used to retrieve a previously claimed message.
292
293       In  order to use the FI_CLAIM flag, an application must supply a struct
294       fi_context structure as the context for the  receive  operation,  or  a
295       struct  fi_recv_context  in  the  case  of buffered receives.  The same
296       fi_context structure used for an FI_PEEK + FI_CLAIM operation  must  be
297       used by the paired FI_CLAIM request.
298
299       This  flag also applies to endpoints configured for FI_BUFFERED_RECV or
300       FI_VARIABLE_MSG.  When set, it is used to  retrieve  a  tagged  message
301       that  was  buffered by the provider.  See Buffered Tagged Receives sec‐
302       tion for details.
303
304       FI_DISCARD
305              This flag may be used in  conjunction  with  either  FI_PEEK  or
306              FI_CLAIM.   If this flag is used in conjunction with FI_PEEK, it
307              indicates if the peek request completes successfully -- indicat‐
308              ing  that  a matching message was located -- the message is dis‐
309              carded by the provider, as the data is not needed by the  appli‐
310              cation.  This flag may also be used in conjunction with FI_CLAIM
311              in order to  discard  a  message  previously  claimed  using  an
312              FI_PEEK + FI_CLAIM request.
313
314       This  flag also applies to endpoints configured for FI_BUFFERED_RECV or
315       FI_VARIABLE_MSG.  When set, it indicates that the provider should  free
316       a buffered messages.  See Buffered Tagged Receives section for details.
317
318       If  this flag is set, the input buffer(s) and length parameters are ig‐
319       nored.
320

Buffered Tagged Receives

322       See fi_msg(3) for an introduction to buffered receives.   The  handling
323       of  buffered  receives differs between fi_msg operations and fi_tagged.
324       Although the provider is responsible for allocating and  managing  net‐
325       work  buffers,  the application is responsible for identifying the tags
326       that will be used to match incoming  messages.   The  provider  handles
327       matching incoming receives to the application specified tags.
328
329       When  FI_BUFFERED_RECV  is enabled, the application posts the tags that
330       will be used for matching purposes.  Tags are  posted  using  fi_trecv,
331       fi_trecvv,  and  fi_trecvmsg;  however, parameters related to the input
332       buffers are ignored (e.g.  buf, len, iov, desc).  When a  provider  re‐
333       ceives  a  message  for which there is a matching tag, it will write an
334       entry to the completion queue associated with the receiving endpoint.
335
336       For discussion purposes, the completion queue is assumed to be  config‐
337       ured  for  FI_CQ_FORMAT_TAGGED.   The  op_context field will point to a
338       struct fi_recv_contex.
339
340              struct fi_recv_context {
341                  struct fid_ep *ep;
342                  void *context;
343              };
344
345       The 'ep' field will be NULL.  The 'context' field will match the appli‐
346       cation context specified when posting the tag.  Other fields are set as
347       defined in fi_msg(3).
348
349       After being notified that a buffered receive has arrived,  applications
350       must either claim or discard the message as described in fi_msg(3).
351

Variable Length Tagged Messages

353       Variable  length  messages  are defined in fi_msg(3).  The requirements
354       for handling variable length tagged messages is identical to those  de‐
355       fined above for buffered tagged receives.
356

RETURN VALUE

358       The  tagged  send  and  receive calls return 0 on success.  On error, a
359       negative value corresponding to fabric errno  is returned.  Fabric  er‐
360       rno values are defined in fi_errno.h.
361

ERRORS

363       -FI_EAGAIN
364              See fi_msg(3) for a detailed description of handling FI_EAGAIN.
365
366       -FI_EINVAL
367              Indicates that an invalid argument was supplied by the user.
368
369       -FI_EOTHER
370              Indicates that an unspecified error occurred.
371

SEE ALSO

373       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
374

AUTHORS

376       OpenFabrics.
377
378
379
380Libfabric Programmer's Manual     2019-02-04                      fi_tagged(3)
Impressum