1fi_rma(3) Libfabric v1.6.1 fi_rma(3)
2
6 fi_rma - Remote memory access operations
7 fi_read / fi_readv / fi_readmsg
9 Initiates a read from remote memory
10 fi_write / fi_writev / fi_writemsg
12 fi_inject_write / fi_writedata : Initiate a write to remote memory
13
15 #include <rdma/fi_rma.h>
16 ssize_t fi_read(struct fid_ep *ep, void *buf, size_t len, void *desc,
18 fi_addr_t src_addr, uint64_t addr, uint64_t key, void *context);
19 ssize_t fi_readv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21 size_t count, fi_addr_t src_addr, uint64_t addr, uint64_t key,
22 void *context);
23 ssize_t fi_readmsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
25 uint64_t flags);
26 ssize_t fi_write(struct fid_ep *ep, const void *buf, size_t len,
28 void *desc, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
29 void *context);
30 ssize_t fi_writev(struct fid_ep *ep, const struct iovec *iov, void **desc,
32 size_t count, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
33 void *context);
34 ssize_t fi_writemsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
36 uint64_t flags);
37 ssize_t fi_inject_write(struct fid_ep *ep, const void *buf, size_t len,
39 fi_addr_t dest_addr, uint64_t addr, uint64_t key);
40 ssize_t fi_writedata(struct fid_ep *ep, const void *buf, size_t len,
42 void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t addr,
43 uint64_t key, void *context);
44 ssize_t fi_inject_writedata(struct fid_ep *ep, const void *buf, size_t len,
46 uint64_t data, fi_addr_t dest_addr, uint64_t addr, uint64_t key);
47
49 ep : Fabric endpoint on which to initiate read or write operation.
50 buf : Local data buffer to read into (read target) or write from (write
52 source)
53 len : Length of data to read or write, specified in bytes. Valid
55 transfers are from 0 bytes up to the endpoint's max_msg_size.
56 iov : Vectored data buffer.
58 count : Count of vectored data entries.
60 addr : Address of remote memory to access. This will be the virtual
62 address of the remote region in the case of FI_MR_BASIC, or the offset
63 from the starting address in the case of FI_MR_SCALABLE.
64 key : Protection key associated with the remote memory.
66 desc : Descriptor associated with the local data buffer
68 data : Remote CQ data to transfer with the operation.
70 dest_addr : Destination address for connectionless write transfers.
72 Ignored for connected endpoints.
73 src_addr : Source address to read from for connectionless transfers.
75 Ignored for connected endpoints.
76 msg : Message descriptor for read and write operations.
78 flags : Additional flags to apply for the read or write operation.
80 context : User specified pointer to associate with the operation.
82
84 RMA (remote memory access) operations are used to transfer data
85 directly between a local data buffer and a remote data buffer. RMA
86 transfers occur on a byte level granularity, and no message boundaries
87 are maintained.
88 The write functions -- fi_write, fi_writev, fi_writemsg,
90 fi_inject_write, and fi_writedata -- are used to transmit data into a
91 remote memory buffer. The main difference between write functions are
92 the number and type of parameters that they accept as input. Other‐
93 wise, they perform the same general function.
94 The read functions -- fi_read, fi_readv, and fi_readmsg -- are used to
96 transfer data from a remote memory region into local data buffer(s).
97 Similar to the write operations, read operations operate asyn‐
98 chronously. Users should not touch the posted data buffer(s) until the
99 read operation has completed.
100 Completed RMA operations are reported to the user through one or more
102 completion queues associated with the endpoint. Users provide context
103 which are associated with each operation, and is returned to the user
104 as part of the completion. See fi_cq for completion event details.
105 By default, the remote endpoint does not generate an event or notify
107 the user when a memory region has been accessed by an RMA read or write
108 operation. However, immediate data may be associated with an RMA write
109 operation. RMA writes with immediate data will generate a completion
110 entry at the remote endpoint, so that the immediate data may be deliv‐
111 ered.
112 fi_write
114 The call fi_write transfers the data contained in the user-specified
115 data buffer to a remote memory region. The local endpoint must be con‐
116 nected to a remote endpoint or destination before fi_write is called.
117 Unless the endpoint has been configured differently, the data buffer
118 passed into fi_write must not be touched by the application until the
119 fi_write call completes asynchronously.
120 fi_writev
122 The fi_writev call adds support for a scatter-gather list to fi_write.
123 The fi_writev transfers the set of data buffers referenced by the iov
124 parameter to the remote memory region.
125 fi_writemsg
127 The fi_writemsg call supports data transfers over both connected and
128 unconnected endpoints, with the ability to control the write operation
129 per call through the use of flags. The fi_writemsg function takes a
130 struct fi_msg_rma as input.
131 struct fi_msg_rma {
133 const struct iovec *msg_iov; /* local scatter-gather array */
134 void **desc; /* operation descriptor */
135 size_t iov_count; /* # elements in msg_iov */
136 fi_addr_t addr; /* optional endpoint address */
137 const struct fi_rma_iov *rma_iov;/* remote SGL */
138 size_t rma_iov_count;/* # elements in rma_iov */
139 void *context; /* user-defined context */
140 uint64_t data; /* optional immediate data */
141 };
142 struct fi_rma_iov {
144 uint64_t addr; /* target RMA address */
145 size_t len; /* size of target buffer */
146 uint64_t key; /* access key */
147 };
148 fi_inject_write
150 The write inject call is an optimized version of fi_write. The
151 fi_inject_write function behaves as if the FI_INJECT transfer flag were
152 set, and FI_COMPLETION were not. That is, the data buffer is available
153 for reuse immediately on returning from fi_inject_write, and no comple‐
154 tion event will be generated for this write. The completion event will
155 be suppressed even if the endpoint has not been configured with
156 FI_SELECTIVE_COMPLETION. See the flags discussion below for more
157 details. The requested message size that can be used with
158 fi_inject_write is limited by inject_size.
159 fi_writedata
161 The write data call is similar to fi_write, but allows for the sending
162 of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
163 fi_inject_writedata
165 The inject write data call is similar to fi_inject_write, but allows
166 for the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part
167 of the transfer.
168 fi_read
170 The fi_read call requests that the remote endpoint transfer data from
171 the remote memory region into the local data buffer. The local end‐
172 point must be connected to a remote endpoint or destination before
173 fi_read is called.
174 fi_readv
176 The fi_readv call adds support for a scatter-gather list to fi_read.
177 The fi_readv transfers data from the remote memory region into the set
178 of data buffers referenced by the iov parameter.
179 fi_readmsg
181 The fi_readmsg call supports data transfers over both connected and
182 unconnected endpoints, with the ability to control the read operation
183 per call through the use of flags. The fi_readmsg function takes a
184 struct fi_msg_rma as input.
185
187 The fi_readmsg and fi_writemsg calls allow the user to specify flags
188 which can change the default data transfer operation. Flags specified
189 with fi_readmsg / fi_writemsg override most flags previously configured
190 with the endpoint, except where noted (see fi_endpoint.3). The follow‐
191 ing list of flags are usable with fi_readmsg and/or fi_writemsg.
192 FI_REMOTE_CQ_DATA : Applies to fi_writemsg and fi_writedata. Indicates
194 that remote CQ data is available and should be sent as part of the
195 request. See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
196 FI_COMPLETION : Indicates that a completion entry should be generated
198 for the specified operation. The endpoint must be bound to a comple‐
199 tion queue with FI_SELECTIVE_COMPLETION that corresponds to the speci‐
200 fied operation, or this flag is ignored.
201 FI_MORE : Indicates that the user has additional requests that will
203 immediately be posted after the current call returns. Use of this flag
204 may improve performance by enabling the provider to optimize its access
205 to the fabric hardware.
206 FI_INJECT : Applies to fi_writemsg. Indicates that the outbound data
208 buffer should be returned to user immediately after the write call
209 returns, even if the operation is handled asynchronously. This may
210 require that the underlying provider implementation copy the data into
211 a local buffer and transfer out of that buffer. This flag can only be
212 used with messages smaller than inject_size.
213 FI_INJECT_COMPLETE : Applies to fi_writemsg. Indicates that a comple‐
215 tion should be generated when the source buffer(s) may be reused.
216 FI_TRANSMIT_COMPLETE : Applies to fi_writemsg. Indicates that a com‐
218 pletion should not be generated until the operation has been success‐
219 fully transmitted and is no longer being tracked by the provider.
220 FI_DELIVERY_COMPLETE : Applies to fi_writemsg. Indicates that a com‐
222 pletion should be generated when the operation has been processed by
223 the destination.
224 FI_COMMIT_COMPLETE : Applies to fi_writemsg when targeting persistent
226 memory regions. Indicates that a completion should be generated only
227 after the result of the operation has been made durable.
228 FI_FENCE : Applies to transmits. Indicates that the requested opera‐
230 tion, also known as the fenced operation, and any operation posted
231 after the fenced operation will be deferred until all previous opera‐
232 tions targeting the same peer endpoint have completed. Operations
233 posted after the fencing will see and/or replace the results of any
234 operations initiated prior to the fenced operation.
235 The ordering of operations starting at the posting of the fenced opera‐
237 tion (inclusive) to the posting of a subsequent fenced operation
238 (exclusive) is controlled by the endpoint's ordering semantics.
239
241 Returns 0 on success. On error, a negative value corresponding to fab‐
242 ric errno is returned. Fabric errno values are defined in
243 rdma/fi_errno.h.
244
246 -FI_EAGAIN : See fi_msg(3) for a detailed description of handling
247 FI_EAGAIN.
248
250 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
251
253 OpenFabrics.
254Libfabric Programmer's Manual 2018-02-13 fi_rma(3)