1fi_rma(3) Libfabric v1.7.0 fi_rma(3)
2
6 fi_rma - Remote memory access operations
7 fi_read / fi_readv / fi_readmsg
9 Initiates a read from remote memory
10 fi_write / fi_writev / fi_writemsg
12 fi_inject_write / fi_writedata : Initiate a write to remote memory
13
15 #include <rdma/fi_rma.h>
16 ssize_t fi_read(struct fid_ep *ep, void *buf, size_t len, void *desc,
18 fi_addr_t src_addr, uint64_t addr, uint64_t key, void *context);
19 ssize_t fi_readv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21 size_t count, fi_addr_t src_addr, uint64_t addr, uint64_t key,
22 void *context);
23 ssize_t fi_readmsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
25 uint64_t flags);
26 ssize_t fi_write(struct fid_ep *ep, const void *buf, size_t len,
28 void *desc, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
29 void *context);
30 ssize_t fi_writev(struct fid_ep *ep, const struct iovec *iov, void **desc,
32 size_t count, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
33 void *context);
34 ssize_t fi_writemsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
36 uint64_t flags);
37 ssize_t fi_inject_write(struct fid_ep *ep, const void *buf, size_t len,
39 fi_addr_t dest_addr, uint64_t addr, uint64_t key);
40 ssize_t fi_writedata(struct fid_ep *ep, const void *buf, size_t len,
42 void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t addr,
43 uint64_t key, void *context);
44 ssize_t fi_inject_writedata(struct fid_ep *ep, const void *buf, size_t len,
46 uint64_t data, fi_addr_t dest_addr, uint64_t addr, uint64_t key);
47
49 ep Fabric endpoint on which to initiate read or write operation.
50 buf Local data buffer to read into (read target) or write from
52 (write source)
53 len Length of data to read or write, specified in bytes. Valid
55 transfers are from 0 bytes up to the endpoint's max_msg_size.
56 iov Vectored data buffer.
58 count Count of vectored data entries.
60 addr Address of remote memory to access. This will be the virtual
62 address of the remote region in the case of FI_MR_BASIC, or the
63 offset from the starting address in the case of FI_MR_SCALABLE.
64 key Protection key associated with the remote memory.
66 desc Descriptor associated with the local data buffer
68 data Remote CQ data to transfer with the operation.
70 dest_addr
72 Destination address for connectionless write transfers. Ignored
73 for connected endpoints.
74 src_addr
76 Source address to read from for connectionless transfers. Ig‐
77 nored for connected endpoints.
78 msg Message descriptor for read and write operations.
80 flags Additional flags to apply for the read or write operation.
82 context
84 User specified pointer to associate with the operation. This
85 parameter is ignored if the operation will not generate a suc‐
86 cessful completion, unless an op flag specifies the context pa‐
87 rameter be used for required input.
88
90 RMA (remote memory access) operations are used to transfer data direct‐
91 ly between a local data buffer and a remote data buffer. RMA transfers
92 occur on a byte level granularity, and no message boundaries are main‐
93 tained.
94 The write functions -- fi_write, fi_writev, fi_writemsg, fi_in‐
96 ject_write, and fi_writedata -- are used to transmit data into a remote
97 memory buffer. The main difference between write functions are the
98 number and type of parameters that they accept as input. Otherwise,
99 they perform the same general function.
100 The read functions -- fi_read, fi_readv, and fi_readmsg -- are used to
102 transfer data from a remote memory region into local data buffer(s).
103 Similar to the write operations, read operations operate asynchronous‐
104 ly. Users should not touch the posted data buffer(s) until the read
105 operation has completed.
106 Completed RMA operations are reported to the user through one or more
108 completion queues associated with the endpoint. Users provide context
109 which are associated with each operation, and is returned to the user
110 as part of the completion. See fi_cq for completion event details.
111 By default, the remote endpoint does not generate an event or notify
113 the user when a memory region has been accessed by an RMA read or write
114 operation. However, immediate data may be associated with an RMA write
115 operation. RMA writes with immediate data will generate a completion
116 entry at the remote endpoint, so that the immediate data may be deliv‐
117 ered.
118 fi_write
120 The call fi_write transfers the data contained in the user-specified
121 data buffer to a remote memory region. The local endpoint must be con‐
122 nected to a remote endpoint or destination before fi_write is called.
123 Unless the endpoint has been configured differently, the data buffer
124 passed into fi_write must not be touched by the application until the
125 fi_write call completes asynchronously.
126 fi_writev
128 The fi_writev call adds support for a scatter-gather list to fi_write.
129 The fi_writev transfers the set of data buffers referenced by the iov
130 parameter to the remote memory region.
131 fi_writemsg
133 The fi_writemsg call supports data transfers over both connected and
134 unconnected endpoints, with the ability to control the write operation
135 per call through the use of flags. The fi_writemsg function takes a
136 struct fi_msg_rma as input.
137 struct fi_msg_rma {
139 const struct iovec *msg_iov; /* local scatter-gather array */
140 void **desc; /* operation descriptor */
141 size_t iov_count; /* # elements in msg_iov */
142 fi_addr_t addr; /* optional endpoint address */
143 const struct fi_rma_iov *rma_iov;/* remote SGL */
144 size_t rma_iov_count;/* # elements in rma_iov */
145 void *context; /* user-defined context */
146 uint64_t data; /* optional immediate data */
147 };
148 struct fi_rma_iov {
150 uint64_t addr; /* target RMA address */
151 size_t len; /* size of target buffer */
152 uint64_t key; /* access key */
153 };
154 fi_inject_write
156 The write inject call is an optimized version of fi_write. The fi_in‐
157 ject_write function behaves as if the FI_INJECT transfer flag were set,
158 and FI_COMPLETION were not. That is, the data buffer is available for
159 reuse immediately on returning from fi_inject_write, and no completion
160 event will be generated for this write. The completion event will be
161 suppressed even if the endpoint has not been configured with FI_SELEC‐
162 TIVE_COMPLETION. See the flags discussion below for more details. The
163 requested message size that can be used with fi_inject_write is limited
164 by inject_size.
165 fi_writedata
167 The write data call is similar to fi_write, but allows for the sending
168 of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
169 fi_inject_writedata
171 The inject write data call is similar to fi_inject_write, but allows
172 for the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part
173 of the transfer.
174 fi_read
176 The fi_read call requests that the remote endpoint transfer data from
177 the remote memory region into the local data buffer. The local end‐
178 point must be connected to a remote endpoint or destination before
179 fi_read is called.
180 fi_readv
182 The fi_readv call adds support for a scatter-gather list to fi_read.
183 The fi_readv transfers data from the remote memory region into the set
184 of data buffers referenced by the iov parameter.
185 fi_readmsg
187 The fi_readmsg call supports data transfers over both connected and un‐
188 connected endpoints, with the ability to control the read operation per
189 call through the use of flags. The fi_readmsg function takes a struct
190 fi_msg_rma as input.
191
193 The fi_readmsg and fi_writemsg calls allow the user to specify flags
194 which can change the default data transfer operation. Flags specified
195 with fi_readmsg / fi_writemsg override most flags previously configured
196 with the endpoint, except where noted (see fi_endpoint.3). The follow‐
197 ing list of flags are usable with fi_readmsg and/or fi_writemsg.
198 FI_REMOTE_CQ_DATA
200 Applies to fi_writemsg and fi_writedata. Indicates that remote
201 CQ data is available and should be sent as part of the request.
202 See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
203 FI_COMPLETION
205 Indicates that a completion entry should be generated for the
206 specified operation. The endpoint must be bound to a completion
207 queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
208 ified operation, or this flag is ignored.
209 FI_MORE
211 Indicates that the user has additional requests that will imme‐
212 diately be posted after the current call returns. Use of this
213 flag may improve performance by enabling the provider to opti‐
214 mize its access to the fabric hardware.
215 FI_INJECT
217 Applies to fi_writemsg. Indicates that the outbound data buffer
218 should be returned to user immediately after the write call re‐
219 turns, even if the operation is handled asynchronously. This
220 may require that the underlying provider implementation copy the
221 data into a local buffer and transfer out of that buffer. This
222 flag can only be used with messages smaller than inject_size.
223 FI_INJECT_COMPLETE
225 Applies to fi_writemsg. Indicates that a completion should be
226 generated when the source buffer(s) may be reused.
227 FI_TRANSMIT_COMPLETE
229 Applies to fi_writemsg. Indicates that a completion should not
230 be generated until the operation has been successfully transmit‐
231 ted and is no longer being tracked by the provider.
232 FI_DELIVERY_COMPLETE
234 Applies to fi_writemsg. Indicates that a completion should be
235 generated when the operation has been processed by the destina‐
236 tion.
237 FI_COMMIT_COMPLETE
239 Applies to fi_writemsg when targeting persistent memory regions.
240 Indicates that a completion should be generated only after the
241 result of the operation has been made durable.
242 FI_FENCE
244 Applies to transmits. Indicates that the requested operation,
245 also known as the fenced operation, and any operation posted af‐
246 ter the fenced operation will be deferred until all previous op‐
247 erations targeting the same peer endpoint have completed. Oper‐
248 ations posted after the fencing will see and/or replace the re‐
249 sults of any operations initiated prior to the fenced operation.
250 The ordering of operations starting at the posting of the fenced opera‐
252 tion (inclusive) to the posting of a subsequent fenced operation (ex‐
253 clusive) is controlled by the endpoint's ordering semantics.
254
256 Returns 0 on success. On error, a negative value corresponding to fab‐
257 ric errno is returned. Fabric errno values are defined in rdma/fi_er‐
258 rno.h.
259
261 -FI_EAGAIN
262 See fi_msg(3) for a detailed description of handling FI_EAGAIN.
263
265 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
266
268 OpenFabrics.
269Libfabric Programmer's Manual 2018-11-28 fi_rma(3)