1fi_tagged(3)                   Libfabric v1.6.1                   fi_tagged(3)
2
3
4

NAME

6       fi_tagged - Tagged data transfer operations
7
8       fi_trecv / fi_trecvv / fi_trecvmsg
9              Post a buffer to receive an incoming message
10
11       fi_tsend / fi_tsendv / fi_tsendmsg / fi_tinject / fi_tsenddata
12              Initiate an operation to send a message
13

SYNOPSIS

15              #include <rdma/fi_tagged.h>
16
17              ssize_t fi_trecv(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t tag, uint64_t ignore, void *context);
19
20              ssize_t fi_trecvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t tag, uint64_t ignore,
22                  void *context);
23
24              ssize_t fi_trecvmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
25                  uint64_t flags);
26
27              ssize_t fi_tsend(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t tag, void *context);
29
30              ssize_t fi_tsendv(struct fid_ep *ep, const struct iovec *iov,
31                  void **desc, size_t count, fi_addr_t dest_addr, uint64_t tag,
32                  void *context);
33
34              ssize_t fi_tsendmsg(struct fid_ep *ep, const struct fi_msg_tagged *msg,
35                  uint64_t flags);
36
37              ssize_t fi_tinject(struct fid_ep *ep, const void *buf, size_t len,
38                  fi_addr_t dest_addr, uint64_t tag);
39
40              ssize_t fi_tsenddata(struct fid_ep *ep, const void *buf, size_t len,
41                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t tag,
42                  void *context);
43
44              ssize_t fi_tinjectdata(struct fid_ep *ep, const void *buf, size_t len,
45                  uint64_t data, fi_addr_t dest_addr, uint64_t tag);
46

ARGUMENTS

48       fid  : Fabric endpoint on which to initiate tagged communication opera‐
49       tion.
50
51       buf : Data buffer to send or receive.
52
53       len : Length of data buffer to send or receive.
54
55       iov : Vectored data buffer.
56
57       count : Count of vectored data entries.
58
59       tag : Tag associated with the message.
60
61       ignore : Mask of bits to ignore applied to the tag for  receive  opera‐
62       tions.
63
64       desc : Memory descriptor associated with the data buffer
65
66       data : Remote CQ data to transfer with the sent data.
67
68       dest_addr  : Destination address for connectionless transfers.  Ignored
69       for connected endpoints.
70
71       src_addr : Source address to receive from for connectionless transfers.
72       Applies  only  to  connectionless  endpoints  with the FI_DIRECTED_RECV
73       capability enabled,  otherwise  this  field  is  ignored.   If  set  to
74       FI_ADDR_UNSPEC, any source address may match.
75
76       msg : Message descriptor for send and receive operations.
77
78       flags : Additional flags to apply for the send or receive operation.
79
80       context : User specified pointer to associate with the operation.
81

DESCRIPTION

83       Tagged  messages  are  data transfers which carry a key or tag with the
84       message buffer.  The tag is used at the receiving endpoint to match the
85       incoming  message  with  a  corresponding receive buffer.  Message tags
86       match when the receive buffer tag is the same as the  send  buffer  tag
87       with the ignored bits masked out.  This can be stated as:
88
89              send_tag & ~ignore == recv_tag & ~ignore
90
91       In  general,  message  tags  are checked against receive buffers in the
92       order in which messages have been posted  to  the  endpoint.   See  the
93       ordering discussion below for more details.
94
95       The send functions -- fi_tsend, fi_tsendv, fi_tsendmsg, fi_tinject, and
96       fi_tsenddata -- are used to transmit a tagged message from one endpoint
97       to  another  endpoint.   The main difference between send functions are
98       the number and type of parameters that they accept  as  input.   Other‐
99       wise, they perform the same general function.
100
101       The receive functions -- fi_trecv, fi_trecvv, fi_recvmsg -- post a data
102       buffer to an endpoint to receive inbound tagged messages.   Similar  to
103       the  send operations, receive operations operate asynchronously.  Users
104       should not touch the posted data buffer(s) until the receive  operation
105       has  completed.   Posted  receive buffers are matched with inbound send
106       messages based on the tags associated with the send  and  receive  buf‐
107       fers.
108
109       An  endpoint  must  be  enabled  before an application can post send or
110       receive operations to it.  For connected endpoints, receive buffers may
111       be  posted  prior  to  connect  or accept being called on the endpoint.
112       This ensures that buffers are available to receive incoming data  imme‐
113       diately after the connection has been established.
114
115       Completed  message  operations  are reported to the user through one or
116       more event collectors associated with the endpoint.  Users provide con‐
117       text  which  are associated with each operation, and is returned to the
118       user as part of the event completion.  See fi_cq for  completion  event
119       details.
120
121   fi_tsend
122       The  call  fi_tsend  transfers the data contained in the user-specified
123       data buffer to a remote endpoint, with message boundaries  being  main‐
124       tained.   The  local endpoint must be connected to a remote endpoint or
125       destination before fi_tsend is called.  Unless the  endpoint  has  been
126       configured  differently,  the data buffer passed into fi_tsend must not
127       be touched by the application until the fi_tsend call  completes  asyn‐
128       chronously.
129
130   fi_tsendv
131       The  fi_tsendv call adds support for a scatter-gather list to fi_tsend.
132       The fi_sendv transfers the set of data buffers referenced  by  the  iov
133       parameter to a remote endpoint as a single message.
134
135   fi_tsendmsg
136       The  fi_tsendmsg  call  supports data transfers over both connected and
137       unconnected endpoints, with the ability to control the  send  operation
138       per  call  through  the use of flags.  The fi_tsendmsg function takes a
139       struct fi_msg_tagged as input.
140
141              struct fi_msg_tagged {
142                  const struct iovec *msg_iov; /* scatter-gather array */
143                  void               *desc;    /* data descriptor */
144                  size_t             iov_count;/* # elements in msg_iov */
145                  fi_addr_t          addr;    /* optional endpoint address */
146                  uint64_t           tag;      /* tag associated with message */
147                  uint64_t           ignore;   /* mask applied to tag for receives */
148                  void               *context; /* user-defined context */
149                  uint64_t           data;     /* optional immediate data */
150              };
151
152   fi_tinject
153       The tagged inject call  is  an  optimized  version  of  fi_tsend.   The
154       fi_tinject function behaves as if the FI_INJECT transfer flag were set,
155       and FI_COMPLETION were not.  That is, the data buffer is available  for
156       reuse  immediately on returning from from fi_tinject, and no completion
157       event will be generated for this send.  The completion  event  will  be
158       suppressed  even if the endpoint has not been configured with FI_SELEC‐
159       TIVE_COMPLETION.  See the flags discussion below for more details.  The
160       requested  message  size that can be used with fi_tinject is limited by
161       inject_size.
162
163   fi_tsenddata
164       The tagged send data call is similar to fi_tsend, but  allows  for  the
165       sending  of  remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the
166       transfer.
167
168   fi_tinjectdata
169       The tagged inject data call is similar to fi_tinject,  but  allows  for
170       the  sending  of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of
171       the transfer.
172
173   fi_trecv
174       The fi_trecv call posts a data buffer to the receive queue of the  cor‐
175       responding  endpoint.   Posted  receives  are  searched in the order in
176       which they were posted in order to match sends.  Message boundaries are
177       maintained.   The  order in which the receives complete is dependent on
178       the endpoint type and protocol.
179
180   fi_trecvv
181       The fi_trecvv call adds support for a scatter-gather list to  fi_trecv.
182       The  fi_trecvv  posts  the  set  of  data buffers referenced by the iov
183       parameter to a receive incoming data.
184
185   fi_trecvmsg
186       The fi_trecvmsg call supports posting buffers over both  connected  and
187       unconnected  endpoints,  with the ability to control the receive opera‐
188       tion per call through the use of flags.  The fi_trecvmsg function takes
189       a struct fi_msg_tagged as input.
190

FLAGS

192       The  fi_trecvmsg  and fi_tsendmsg calls allow the user to specify flags
193       which can change the default message handling of the  endpoint.   Flags
194       specified with fi_trecvmsg / fi_tsendmsg override most flags previously
195       configured with the endpoint, except  where  noted  (see  fi_endpoint).
196       The  following  list  of  flags  are  usable  with  fi_trecvmsg  and/or
197       fi_tsendmsg.
198
199       FI_REMOTE_CQ_DATA : Applies to fi_tsendmsg and fi_tsenddata.  Indicates
200       that  remote  CQ  data  is  available and should be sent as part of the
201       request.  See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
202
203       FI_COMPLETION : Indicates that a completion entry should  be  generated
204       for  the  specified operation.  The endpoint must be bound to a comple‐
205       tion queue with FI_SELECTIVE_COMPLETION that corresponds to the  speci‐
206       fied operation, or this flag is ignored.
207
208       FI_MORE  :  Indicates  that  the user has additional requests that will
209       immediately be posted after the current call returns.  Use of this flag
210       may improve performance by enabling the provider to optimize its access
211       to the fabric hardware.
212
213       FI_INJECT : Applies to fi_tsendmsg.  Indicates that the  outbound  data
214       buffer  should  be  returned  to  user  immediately after the send call
215       returns, even if the operation is  handled  asynchronously.   This  may
216       require  that the underlying provider implementation copy the data into
217       a local buffer and transfer out of that buffer.  This flag can only  be
218       used with messages smaller than inject_size.
219
220       FI_INJECT_COMPLETE  : Applies to fi_tsendmsg.  Indicates that a comple‐
221       tion should be generated when the source buffer(s) may be reused.
222
223       FI_TRANSMIT_COMPLETE : Applies to fi_tsendmsg.  Indicates that  a  com‐
224       pletion  should  not be generated until the operation has been success‐
225       fully transmitted and is no longer being tracked by the provider.
226
227       FI_FENCE : Applies to transmits.  Indicates that the  requested  opera‐
228       tion,  also  known  as  the  fenced operation, and any operation posted
229       after the fenced operation will be deferred until all  previous  opera‐
230       tions  targeting  the  same  peer  endpoint have completed.  Operations
231       posted after the fencing will see and/or replace  the  results  of  any
232       operations initiated prior to the fenced operation.
233
234       The ordering of operations starting at the posting of the fenced opera‐
235       tion (inclusive) to  the  posting  of  a  subsequent  fenced  operation
236       (exclusive) is controlled by the endpoint's ordering semantics.
237
238       The following flags may be used with fi_trecvmsg.
239
240       FI_PEEK  :  The peek flag may be used to see if a specified message has
241       arrived.  A peek  request  is  often  useful  on  endpoints  that  have
242       provider     allocated     buffering     enabled     (see    fi_rx_attr
243       total_buffered_recv).  Unlike standard receive  operations,  a  receive
244       operation  with  the  FI_PEEK  flag set does not remain queued with the
245       provider after the peek completes  successfully.   The  peek  operation
246       operates  asynchronously,  and  the  results  of the peek operation are
247       available in the completion queue associated with the endpoint.  If  no
248       message  is found matching the tags specified in the peek request, then
249       a completion queue error entry with err field set to FI_ENOMSG will  be
250       available.
251
252       If  a  peek request locates a matching message, the operation will com‐
253       plete successfully.  The returned completion  data  will  indicate  the
254       meta-data associated with the message, such as the message length, com‐
255       pletion flags, available CQ data, tag, and source  address.   The  data
256       available  is  subject  to  the  completion  entry format (e.g.  struct
257       fi_cq_tagged_entry).
258
259       An application may supply a buffer if it desires to receive data  as  a
260       part  of the peek operation.  In order to receive data as a part of the
261       peek operation, the buf and len fields must be available in the CQ for‐
262       mat.   In  particular, FI_CQ_FORMAT_CONTEXT and FI_CQ_FORMAT_MSG cannot
263       be used if peek operations desire to obtain a copy of  the  data.   The
264       returned data is limited to the size of the input buffer(s) or the mes‐
265       sage size, if smaller.  A provider indicates if data  is  available  by
266       setting the buf field of the CQ entry to the user's first input buffer.
267       If buf is NULL, no data was available to return.  A provider may return
268       NULL  even if the peek operation completes successfully.  Note that the
269       CQ entry len field will reference the size of the message,  not  neces‐
270       sarily the size of the returned data.
271
272       FI_CLAIM  :  If this flag is used in conjunction with FI_PEEK, it indi‐
273       cates if the peek request completes successfully -- indicating  that  a
274       matching  message  was  located  --  the  message is claimed by caller.
275       Claimed messages can only  be  retrieved  using  a  subsequent,  paired
276       receive operation with the FI_CLAIM flag set.  A receive operation with
277       the FI_CLAIM flag set, but FI_PEEK not set is used to retrieve a previ‐
278       ously claimed message.
279
280       In  order to use the FI_CLAIM flag, an application must supply a struct
281       fi_context structure as the context for  the  receive  operation.   The
282       same fi_context structure used for an FI_PEEK + FI_CLAIM operation must
283       be used by the paired FI_CLAIM request.
284
285       FI_DISCARD : This flag must be used in conjunction with either  FI_PEEK
286       or  FI_CLAIM.   If  this  flag  is used in conjunction with FI_PEEK, it
287       indicates if the peek request completes successfully -- indicating that
288       a  matching  message  was  located  --  the message is discarded by the
289       provider, as the data is not needed by the application.  This flag  may
290       also be used in conjunction with FI_CLAIM in order to retrieve and dis‐
291       card a message previously claimed using an FI_PEEK + FI_CLAIM request.
292
293       If this flag is set, the input  buffer(s)  and  length  parameters  are
294       ignored.
295

RETURN VALUE

297       The  tagged  send  and  receive calls return 0 on success.  On error, a
298       negative value corresponding to  fabric  errno   is  returned.   Fabric
299       errno values are defined in fi_errno.h.
300

ERRORS

302       -FI_EAGAIN  :  See  fi_msg(3)  for  a  detailed description of handling
303       FI_EAGAIN.
304
305       -FI_EINVAL : Indicates that an invalid argument  was  supplied  by  the
306       user.
307
308       -FI_EOTHER : Indicates that an unspecified error occurred.
309

SEE ALSO

311       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
312

AUTHORS

314       OpenFabrics.
315
316
317
318Libfabric Programmer's Manual     2017-10-20                      fi_tagged(3)
Impressum