1fi_msg(3) Libfabric v1.6.1 fi_msg(3)
2
6 fi_msg - Message data transfer operations
7 fi_recv / fi_recvv / fi_recvmsg
9 Post a buffer to receive an incoming message
10 fi_send / fi_sendv / fi_sendmsg
12 fi_inject / fi_senddata : Initiate an operation to send a message
13
15 #include <rdma/fi_endpoint.h>
16 ssize_t fi_recv(struct fid_ep *ep, void * buf, size_t len,
18 void *desc, fi_addr_t src_addr, void *context);
19 ssize_t fi_recvv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21 size_t count, fi_addr_t src_addr, void *context);
22 ssize_t fi_recvmsg(struct fid_ep *ep, const struct fi_msg *msg,
24 uint64_t flags);
25 ssize_t fi_send(struct fid_ep *ep, const void *buf, size_t len,
27 void *desc, fi_addr_t dest_addr, void *context);
28 ssize_t fi_sendv(struct fid_ep *ep, const struct iovec *iov,
30 void **desc, size_t count, fi_addr_t dest_addr, void *context);
31 ssize_t fi_sendmsg(struct fid_ep *ep, const struct fi_msg *msg,
33 uint64_t flags);
34 ssize_t fi_inject(struct fid_ep *ep, const void *buf, size_t len,
36 fi_addr_t dest_addr);
37 ssize_t fi_senddata(struct fid_ep *ep, const void *buf, size_t len,
39 void *desc, uint64_t data, fi_addr_t dest_addr, void *context);
40 ssize_t fi_injectdata(struct fid_ep *ep, const void *buf, size_t len,
42 uint64_t data, fi_addr_t dest_addr);
43
45 ep : Fabric endpoint on which to initiate send or post receive buffer.
46 buf : Data buffer to send or receive.
48 len : Length of data buffer to send or receive, specified in bytes.
50 Valid transfers are from 0 bytes up to the endpoint's max_msg_size.
51 iov : Vectored data buffer.
53 count : Count of vectored data entries.
55 desc : Descriptor associated with the data buffer
57 data : Remote CQ data to transfer with the sent message.
59 dest_addr : Destination address for connectionless transfers. Ignored
61 for connected endpoints.
62 src_addr : Source address to receive from for connectionless transfers.
64 Applies only to connectionless endpoints with the FI_DIRECTED_RECV
65 capability enabled, otherwise this field is ignored. If set to
66 FI_ADDR_UNSPEC, any source address may match.
67 msg : Message descriptor for send and receive operations.
69 flags : Additional flags to apply for the send or receive operation.
71 context : User specified pointer to associate with the operation.
73
75 The send functions -- fi_send, fi_sendv, fi_sendmsg, fi_inject, and
76 fi_senddata -- are used to transmit a message from one endpoint to
77 another endpoint. The main difference between send functions are the
78 number and type of parameters that they accept as input. Otherwise,
79 they perform the same general function. Messages sent using fi_msg
80 operations are received by a remote endpoint into a buffer posted to
81 receive such messages.
82 The receive functions -- fi_recv, fi_recvv, fi_recvmsg -- post a data
84 buffer to an endpoint to receive inbound messages. Similar to the send
85 operations, receive operations operate asynchronously. Users should
86 not touch the posted data buffer(s) until the receive operation has
87 completed.
88 An endpoint must be enabled before an application can post send or
90 receive operations to it. For connected endpoints, receive buffers may
91 be posted prior to connect or accept being called on the endpoint.
92 This ensures that buffers are available to receive incoming data imme‐
93 diately after the connection has been established.
94 Completed message operations are reported to the user through one or
96 more event collectors associated with the endpoint. Users provide con‐
97 text which are associated with each operation, and is returned to the
98 user as part of the event completion. See fi_cq for completion event
99 details.
100 fi_send
102 The call fi_send transfers the data contained in the user-specified
103 data buffer to a remote endpoint, with message boundaries being main‐
104 tained. For connection based endpoints (FI_EP_MSG) the local endpoint
105 must be connected to a remote endpoint or destination before fi_send is
106 called. Unless the endpoint has been configured differently, the data
107 buffer passed into fi_send must not be touched by the application until
108 the fi_send call completes asynchronously.
109 fi_sendv
111 The fi_sendv call adds support for a scatter-gather list to fi_send.
112 The fi_sendv transfers the set of data buffers referenced by the iov
113 parameter to a remote endpoint as a single message.
114 fi_sendmsg
116 The fi_sendmsg call supports data transfers over both connected and
117 unconnected endpoints, with the ability to control the send operation
118 per call through the use of flags. The fi_sendmsg function takes a
119 struct fi_msg as input.
120 struct fi_msg {
122 const struct iovec *msg_iov; /* scatter-gather array */
123 void **desc; /* local request descriptors */
124 size_t iov_count;/* # elements in iov */
125 fi_addr_t addr; /* optional endpoint address */
126 void *context; /* user-defined context */
127 uint64_t data; /* optional message data */
128 };
129 fi_inject
131 The send inject call is an optimized version of fi_send. The fi_inject
132 function behaves as if the FI_INJECT transfer flag were set, and
133 FI_COMPLETION were not. That is, the data buffer is available for re‐
134 use immediately on returning from from fi_inject, and no completion
135 event will be generated for this send. The completion event will be
136 suppressed even if the CQ was bound without FI_SELECTIVE_COMPLETION or
137 the endpoint's op_flags contain FI_COMPLETION. See the flags discus‐
138 sion below for more details. The requested message size that can be
139 used with fi_inject is limited by inject_size.
140 fi_senddata
142 The send data call is similar to fi_send, but allows for the sending of
143 remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
144 fi_injectdata
146 The inject data call is similar to fi_inject, but allows for the send‐
147 ing of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the
148 transfer.
149 fi_recv
151 The fi_recv call posts a data buffer to the receive queue of the corre‐
152 sponding endpoint. Posted receives are searched in the order in which
153 they were posted in order to match sends. Message boundaries are main‐
154 tained. The order in which the receives complete is dependent on the
155 endpoint type and protocol. For unconnected endpoints, the src_addr
156 parameter can be used to indicate that a buffer should be posted to
157 receive incoming data from a specific remote endpoint.
158 fi_recvv
160 The fi_recvv call adds support for a scatter-gather list to fi_recv.
161 The fi_recvv posts the set of data buffers referenced by the iov param‐
162 eter to a receive incoming data.
163 fi_recvmsg
165 The fi_recvmsg call supports posting buffers over both connected and
166 unconnected endpoints, with the ability to control the receive opera‐
167 tion per call through the use of flags. The fi_recvmsg function takes
168 a struct fi_msg as input.
169
171 The fi_recvmsg and fi_sendmsg calls allow the user to specify flags
172 which can change the default message handling of the endpoint. Flags
173 specified with fi_recvmsg / fi_sendmsg override most flags previously
174 configured with the endpoint, except where noted (see fi_endpoint.3).
175 The following list of flags are usable with fi_recvmsg and/or
176 fi_sendmsg.
177 FI_REMOTE_CQ_DATA : Applies to fi_sendmsg and fi_senddata. Indicates
179 that remote CQ data is available and should be sent as part of the
180 request. See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
181 FI_COMPLETION : Indicates that a completion entry should be generated
183 for the specified operation. The endpoint must be bound to a comple‐
184 tion queue with FI_SELECTIVE_COMPLETION that corresponds to the speci‐
185 fied operation, or this flag is ignored.
186 FI_MORE : Indicates that the user has additional requests that will
188 immediately be posted after the current call returns. Use of this flag
189 may improve performance by enabling the provider to optimize its access
190 to the fabric hardware.
191 FI_INJECT : Applies to fi_sendmsg. Indicates that the outbound data
193 buffer should be returned to user immediately after the send call
194 returns, even if the operation is handled asynchronously. This may
195 require that the underlying provider implementation copy the data into
196 a local buffer and transfer out of that buffer. This flag can only be
197 used with messages smaller than inject_size.
198 FI_MULTI_RECV : Applies to posted receive operations. This flag allows
200 the user to post a single buffer that will receive multiple incoming
201 messages. Received messages will be packed into the receive buffer
202 until the buffer has been consumed. Use of this flag may cause a sin‐
203 gle posted receive operation to generate multiple events as messages
204 are placed into the buffer. The placement of received data into the
205 buffer may be subjected to provider specific alignment restrictions.
206 The buffer will be released by the provider when the available buffer
208 space falls below the specified minimum (see FI_OPT_MIN_MULTI_RECV).
209 Note that an entry to the associated receive completion queue will
210 always be generated when the buffer has been consumed, even if other
211 receive completions have been suppressed (i.e. the Rx context has been
212 configured for FI_SELECTIVE_COMPLETION). See the FI_MULTI_RECV comple‐
213 tion flag fi_cq(3).
214 FI_INJECT_COMPLETE : Applies to fi_sendmsg. Indicates that a comple‐
216 tion should be generated when the source buffer(s) may be reused.
217 FI_TRANSMIT_COMPLETE : Applies to fi_sendmsg. Indicates that a comple‐
219 tion should not be generated until the operation has been successfully
220 transmitted and is no longer being tracked by the provider.
221 FI_DELIVERY_COMPLETE : Applies to fi_sendmsg. Indicates that a comple‐
223 tion should be generated when the operation has been processed by the
224 destination.
225 FI_FENCE : Applies to transmits. Indicates that the requested opera‐
227 tion, also known as the fenced operation, and any operation posted
228 after the fenced operation will be deferred until all previous opera‐
229 tions targeting the same peer endpoint have completed. Operations
230 posted after the fencing will see and/or replace the results of any
231 operations initiated prior to the fenced operation.
232 The ordering of operations starting at the posting of the fenced opera‐
234 tion (inclusive) to the posting of a subsequent fenced operation
235 (exclusive) is controlled by the endpoint's ordering semantics.
236 FI_MULTICAST : Applies to transmits. This flag indicates that the
238 address specified as the data transfer destination is a multicast
239 address. This flag must be used in all multicast transfers, in con‐
240 junction with a multicast fi_addr_t.
241
243 If an endpoint has been configured with FI_MSG_PREFIX, the application
244 must include buffer space of size msg_prefix_size, as specified by the
245 endpoint attributes. The prefix buffer must occur at the start of the
246 data referenced by the buf parameter, or be referenced by the first IO
247 vector. Message prefix space cannot be split between multiple IO vec‐
248 tors. The size of the prefix buffer should be included as part of the
249 total buffer length.
250
252 Returns 0 on success. On error, a negative value corresponding to fab‐
253 ric errno is returned. Fabric errno values are defined in
254 rdma/fi_errno.h.
255 See the discussion below for details handling FI_EAGAIN.
257
259 -FI_EAGAIN : Indicates that the underlying provider currently lacks the
260 resources needed to initiate the requested operation. The reasons for
261 a provider returning FI_EAGAIN are varied. However, common reasons
262 include insufficient internal buffering or full processing queues.
263 Insufficient internal buffering is often associated with operations
265 that use FI_INJECT. In such cases, additional buffering may become
266 available as posted operations complete.
267 Full processing queues may be a temporary state related to local pro‐
269 cessing (for example, a large message is being transferred), or may be
270 the result of flow control. In the latter case, the queues may remain
271 blocked until additional resources are made available at the remote
272 side of the transfer.
273 In all cases, the operation may be retried after additional resources
275 become available. It is strongly recommended that applications check
276 for transmit and receive completions after receiving FI_EAGAIN as a
277 return value, independent of the operation which failed. This is par‐
278 ticularly important in cases where manual progress is employed, as
279 acknowledgements or flow control messages may need to be processed in
280 order to resume execution.
281
283 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
284
286 OpenFabrics.
287Libfabric Programmer's Manual 2018-01-08 fi_msg(3)