1fi_msg(3)                      Libfabric v1.6.1                      fi_msg(3)
2
3
4

NAME

6       fi_msg - Message data transfer operations
7
8       fi_recv / fi_recvv / fi_recvmsg
9              Post a buffer to receive an incoming message
10
11       fi_send / fi_sendv / fi_sendmsg
12       fi_inject / fi_senddata : Initiate an operation to send a message
13

SYNOPSIS

15              #include <rdma/fi_endpoint.h>
16
17              ssize_t fi_recv(struct fid_ep *ep, void * buf, size_t len,
18                  void *desc, fi_addr_t src_addr, void *context);
19
20              ssize_t fi_recvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, void *context);
22
23              ssize_t fi_recvmsg(struct fid_ep *ep, const struct fi_msg *msg,
24                  uint64_t flags);
25
26              ssize_t fi_send(struct fid_ep *ep, const void *buf, size_t len,
27                  void *desc, fi_addr_t dest_addr, void *context);
28
29              ssize_t fi_sendv(struct fid_ep *ep, const struct iovec *iov,
30                  void **desc, size_t count, fi_addr_t dest_addr, void *context);
31
32              ssize_t fi_sendmsg(struct fid_ep *ep, const struct fi_msg *msg,
33                  uint64_t flags);
34
35              ssize_t fi_inject(struct fid_ep *ep, const void *buf, size_t len,
36                  fi_addr_t dest_addr);
37
38              ssize_t fi_senddata(struct fid_ep *ep, const void *buf, size_t len,
39                  void *desc, uint64_t data, fi_addr_t dest_addr, void *context);
40
41              ssize_t fi_injectdata(struct fid_ep *ep, const void *buf, size_t len,
42                  uint64_t data, fi_addr_t dest_addr);
43

ARGUMENTS

45       ep : Fabric endpoint on which to initiate send or post receive buffer.
46
47       buf : Data buffer to send or receive.
48
49       len  :  Length  of  data buffer to send or receive, specified in bytes.
50       Valid transfers are from 0 bytes up to the endpoint's max_msg_size.
51
52       iov : Vectored data buffer.
53
54       count : Count of vectored data entries.
55
56       desc : Descriptor associated with the data buffer
57
58       data : Remote CQ data to transfer with the sent message.
59
60       dest_addr : Destination address for connectionless transfers.   Ignored
61       for connected endpoints.
62
63       src_addr : Source address to receive from for connectionless transfers.
64       Applies only to  connectionless  endpoints  with  the  FI_DIRECTED_RECV
65       capability  enabled,  otherwise  this  field  is  ignored.   If  set to
66       FI_ADDR_UNSPEC, any source address may match.
67
68       msg : Message descriptor for send and receive operations.
69
70       flags : Additional flags to apply for the send or receive operation.
71
72       context : User specified pointer to associate with the operation.
73

DESCRIPTION

75       The send functions -- fi_send,  fi_sendv,  fi_sendmsg,  fi_inject,  and
76       fi_senddata  --  are  used  to  transmit a message from one endpoint to
77       another endpoint.  The main difference between send functions  are  the
78       number  and  type  of parameters that they accept as input.  Otherwise,
79       they perform the same general function.   Messages  sent  using  fi_msg
80       operations  are  received  by a remote endpoint into a buffer posted to
81       receive such messages.
82
83       The receive functions -- fi_recv, fi_recvv, fi_recvmsg -- post  a  data
84       buffer to an endpoint to receive inbound messages.  Similar to the send
85       operations, receive operations operate  asynchronously.   Users  should
86       not  touch  the  posted  data buffer(s) until the receive operation has
87       completed.
88
89       An endpoint must be enabled before an  application  can  post  send  or
90       receive operations to it.  For connected endpoints, receive buffers may
91       be posted prior to connect or accept  being  called  on  the  endpoint.
92       This  ensures that buffers are available to receive incoming data imme‐
93       diately after the connection has been established.
94
95       Completed message operations are reported to the user  through  one  or
96       more event collectors associated with the endpoint.  Users provide con‐
97       text which are associated with each operation, and is returned  to  the
98       user  as  part of the event completion.  See fi_cq for completion event
99       details.
100
101   fi_send
102       The call fi_send transfers the data  contained  in  the  user-specified
103       data  buffer  to a remote endpoint, with message boundaries being main‐
104       tained.  For connection based endpoints (FI_EP_MSG) the local  endpoint
105       must be connected to a remote endpoint or destination before fi_send is
106       called.  Unless the endpoint has been configured differently, the  data
107       buffer passed into fi_send must not be touched by the application until
108       the fi_send call completes asynchronously.
109
110   fi_sendv
111       The fi_sendv call adds support for a scatter-gather  list  to  fi_send.
112       The  fi_sendv  transfers  the set of data buffers referenced by the iov
113       parameter to a remote endpoint as a single message.
114
115   fi_sendmsg
116       The fi_sendmsg call supports data transfers  over  both  connected  and
117       unconnected  endpoints,  with the ability to control the send operation
118       per call through the use of flags.  The  fi_sendmsg  function  takes  a
119       struct fi_msg as input.
120
121              struct fi_msg {
122                  const struct iovec *msg_iov; /* scatter-gather array */
123                  void               **desc;   /* local request descriptors */
124                  size_t             iov_count;/* # elements in iov */
125                  fi_addr_t          addr;     /* optional endpoint address */
126                  void               *context; /* user-defined context */
127                  uint64_t           data;     /* optional message data */
128              };
129
130   fi_inject
131       The send inject call is an optimized version of fi_send.  The fi_inject
132       function behaves as if  the  FI_INJECT  transfer  flag  were  set,  and
133       FI_COMPLETION  were not.  That is, the data buffer is available for re‐
134       use immediately on returning from from  fi_inject,  and  no  completion
135       event  will  be  generated for this send.  The completion event will be
136       suppressed even if the CQ was bound without FI_SELECTIVE_COMPLETION  or
137       the  endpoint's  op_flags contain FI_COMPLETION.  See the flags discus‐
138       sion below for more details.  The requested message size  that  can  be
139       used with fi_inject is limited by inject_size.
140
141   fi_senddata
142       The send data call is similar to fi_send, but allows for the sending of
143       remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
144
145   fi_injectdata
146       The inject data call is similar to fi_inject, but allows for the  send‐
147       ing  of  remote  CQ  data  (see  FI_REMOTE_CQ_DATA flag) as part of the
148       transfer.
149
150   fi_recv
151       The fi_recv call posts a data buffer to the receive queue of the corre‐
152       sponding  endpoint.  Posted receives are searched in the order in which
153       they were posted in order to match sends.  Message boundaries are main‐
154       tained.   The  order in which the receives complete is dependent on the
155       endpoint type and protocol.  For unconnected  endpoints,  the  src_addr
156       parameter  can  be  used  to indicate that a buffer should be posted to
157       receive incoming data from a specific remote endpoint.
158
159   fi_recvv
160       The fi_recvv call adds support for a scatter-gather  list  to  fi_recv.
161       The fi_recvv posts the set of data buffers referenced by the iov param‐
162       eter to a receive incoming data.
163
164   fi_recvmsg
165       The fi_recvmsg call supports posting buffers over  both  connected  and
166       unconnected  endpoints,  with the ability to control the receive opera‐
167       tion per call through the use of flags.  The fi_recvmsg function  takes
168       a struct fi_msg as input.
169

FLAGS

171       The  fi_recvmsg  and  fi_sendmsg  calls allow the user to specify flags
172       which can change the default message handling of the  endpoint.   Flags
173       specified  with  fi_recvmsg / fi_sendmsg override most flags previously
174       configured with the endpoint, except where noted  (see  fi_endpoint.3).
175       The   following  list  of  flags  are  usable  with  fi_recvmsg  and/or
176       fi_sendmsg.
177
178       FI_REMOTE_CQ_DATA : Applies to fi_sendmsg and  fi_senddata.   Indicates
179       that  remote  CQ  data  is  available and should be sent as part of the
180       request.  See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
181
182       FI_COMPLETION : Indicates that a completion entry should  be  generated
183       for  the  specified operation.  The endpoint must be bound to a comple‐
184       tion queue with FI_SELECTIVE_COMPLETION that corresponds to the  speci‐
185       fied operation, or this flag is ignored.
186
187       FI_MORE  :  Indicates  that  the user has additional requests that will
188       immediately be posted after the current call returns.  Use of this flag
189       may improve performance by enabling the provider to optimize its access
190       to the fabric hardware.
191
192       FI_INJECT : Applies to fi_sendmsg.  Indicates that  the  outbound  data
193       buffer  should  be  returned  to  user  immediately after the send call
194       returns, even if the operation is  handled  asynchronously.   This  may
195       require  that the underlying provider implementation copy the data into
196       a local buffer and transfer out of that buffer.  This flag can only  be
197       used with messages smaller than inject_size.
198
199       FI_MULTI_RECV : Applies to posted receive operations.  This flag allows
200       the user to post a single buffer that will  receive  multiple  incoming
201       messages.   Received  messages  will  be packed into the receive buffer
202       until the buffer has been consumed.  Use of this flag may cause a  sin‐
203       gle  posted  receive  operation to generate multiple events as messages
204       are placed into the buffer.  The placement of received  data  into  the
205       buffer may be subjected to provider specific alignment restrictions.
206
207       The  buffer  will be released by the provider when the available buffer
208       space falls below the specified  minimum  (see  FI_OPT_MIN_MULTI_RECV).
209       Note  that  an  entry  to  the associated receive completion queue will
210       always be generated when the buffer has been consumed,  even  if  other
211       receive completions have been suppressed (i.e.  the Rx context has been
212       configured for FI_SELECTIVE_COMPLETION).  See the FI_MULTI_RECV comple‐
213       tion flag fi_cq(3).
214
215       FI_INJECT_COMPLETE  :  Applies to fi_sendmsg.  Indicates that a comple‐
216       tion should be generated when the source buffer(s) may be reused.
217
218       FI_TRANSMIT_COMPLETE : Applies to fi_sendmsg.  Indicates that a comple‐
219       tion  should not be generated until the operation has been successfully
220       transmitted and is no longer being tracked by the provider.
221
222       FI_DELIVERY_COMPLETE : Applies to fi_sendmsg.  Indicates that a comple‐
223       tion  should  be generated when the operation has been processed by the
224       destination.
225
226       FI_FENCE : Applies to transmits.  Indicates that the  requested  opera‐
227       tion,  also  known  as  the  fenced operation, and any operation posted
228       after the fenced operation will be deferred until all  previous  opera‐
229       tions  targeting  the  same  peer  endpoint have completed.  Operations
230       posted after the fencing will see and/or replace  the  results  of  any
231       operations initiated prior to the fenced operation.
232
233       The ordering of operations starting at the posting of the fenced opera‐
234       tion (inclusive) to  the  posting  of  a  subsequent  fenced  operation
235       (exclusive) is controlled by the endpoint's ordering semantics.
236
237       FI_MULTICAST  :  Applies  to  transmits.   This flag indicates that the
238       address specified as the  data  transfer  destination  is  a  multicast
239       address.   This  flag  must be used in all multicast transfers, in con‐
240       junction with a multicast fi_addr_t.
241

NOTES

243       If an endpoint has been configured with FI_MSG_PREFIX, the  application
244       must  include buffer space of size msg_prefix_size, as specified by the
245       endpoint attributes.  The prefix buffer must occur at the start of  the
246       data  referenced by the buf parameter, or be referenced by the first IO
247       vector.  Message prefix space cannot be split between multiple IO  vec‐
248       tors.   The size of the prefix buffer should be included as part of the
249       total buffer length.
250

RETURN VALUE

252       Returns 0 on success.  On error, a negative value corresponding to fab‐
253       ric   errno   is   returned.    Fabric  errno  values  are  defined  in
254       rdma/fi_errno.h.
255
256       See the discussion below for details handling FI_EAGAIN.
257

ERRORS

259       -FI_EAGAIN : Indicates that the underlying provider currently lacks the
260       resources  needed to initiate the requested operation.  The reasons for
261       a provider returning FI_EAGAIN are  varied.   However,  common  reasons
262       include insufficient internal buffering or full processing queues.
263
264       Insufficient  internal  buffering  is  often associated with operations
265       that use FI_INJECT.  In such cases,  additional  buffering  may  become
266       available as posted operations complete.
267
268       Full  processing  queues may be a temporary state related to local pro‐
269       cessing (for example, a large message is being transferred), or may  be
270       the  result of flow control.  In the latter case, the queues may remain
271       blocked until additional resources are made  available  at  the  remote
272       side of the transfer.
273
274       In  all  cases, the operation may be retried after additional resources
275       become available.  It is strongly recommended that  applications  check
276       for  transmit  and  receive  completions after receiving FI_EAGAIN as a
277       return value, independent of the operation which failed.  This is  par‐
278       ticularly  important  in  cases  where  manual progress is employed, as
279       acknowledgements or flow control messages may need to be  processed  in
280       order to resume execution.
281

SEE ALSO

283       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
284

AUTHORS

286       OpenFabrics.
287
288
289
290Libfabric Programmer's Manual     2018-01-08                         fi_msg(3)
Impressum