1fi_rma(3)                    Libfabric v1.12.0rc1                    fi_rma(3)
2
3
4

NAME

6       fi_rma - Remote memory access operations
7
8       fi_read / fi_readv / fi_readmsg
9              Initiates a read from remote memory
10
11       fi_write  /  fi_writev  /  fi_writemsg fi_inject_write / fi_writedata :
12       Initiate a write to remote memory
13

SYNOPSIS

15              #include <rdma/fi_rma.h>
16
17              ssize_t fi_read(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t addr, uint64_t key, void *context);
19
20              ssize_t fi_readv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t addr, uint64_t key,
22                  void *context);
23
24              ssize_t fi_readmsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
25                  uint64_t flags);
26
27              ssize_t fi_write(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
29                  void *context);
30
31              ssize_t fi_writev(struct fid_ep *ep, const struct iovec *iov, void **desc,
32                  size_t count, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
33                  void *context);
34
35              ssize_t fi_writemsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
36                  uint64_t flags);
37
38              ssize_t fi_inject_write(struct fid_ep *ep, const void *buf, size_t len,
39                  fi_addr_t dest_addr, uint64_t addr, uint64_t key);
40
41              ssize_t fi_writedata(struct fid_ep *ep, const void *buf, size_t len,
42                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t addr,
43                  uint64_t key, void *context);
44
45              ssize_t fi_inject_writedata(struct fid_ep *ep, const void *buf, size_t len,
46                  uint64_t data, fi_addr_t dest_addr, uint64_t addr, uint64_t key);
47

ARGUMENTS

49       ep     Fabric endpoint on which to initiate read or write operation.
50
51       buf    Local data buffer to read  into  (read  target)  or  write  from
52              (write source)
53
54       len    Length  of  data  to  read  or write, specified in bytes.  Valid
55              transfers are from 0 bytes up to the endpoint's max_msg_size.
56
57       iov    Vectored data buffer.
58
59       count  Count of vectored data entries.
60
61       addr   Address of remote memory to access.  This will  be  the  virtual
62              address  of the remote region in the case of FI_MR_BASIC, or the
63              offset from the starting address in the case of FI_MR_SCALABLE.
64
65       key    Protection key associated with the remote memory.
66
67       desc   Descriptor associated with the local data buffer See fi_mr(3).
68
69       data   Remote CQ data to transfer with the operation.
70
71       dest_addr
72              Destination address for connectionless write transfers.  Ignored
73              for connected endpoints.
74
75       src_addr
76              Source  address  to read from for connectionless transfers.  Ig‐
77              nored for connected endpoints.
78
79       msg    Message descriptor for read and write operations.
80
81       flags  Additional flags to apply for the read or write operation.
82
83       context
84              User specified pointer to associate with  the  operation.   This
85              parameter  is  ignored if the operation will not generate a suc‐
86              cessful completion, unless an op flag specifies the context  pa‐
87              rameter be used for required input.
88

DESCRIPTION

90       RMA (remote memory access) operations are used to transfer data direct‐
91       ly between a local data buffer and a remote data buffer.  RMA transfers
92       occur  on a byte level granularity, and no message boundaries are main‐
93       tained.
94
95       The  write  functions  --  fi_write,  fi_writev,  fi_writemsg,   fi_in‐
96       ject_write, and fi_writedata -- are used to transmit data into a remote
97       memory buffer.  The main difference between  write  functions  are  the
98       number  and  type  of parameters that they accept as input.  Otherwise,
99       they perform the same general function.
100
101       The read functions -- fi_read, fi_readv, and fi_readmsg -- are used  to
102       transfer  data  from  a remote memory region into local data buffer(s).
103       Similar to the write operations, read operations operate  asynchronous‐
104       ly.   Users  should  not touch the posted data buffer(s) until the read
105       operation has completed.
106
107       Completed RMA operations are reported to the user through one  or  more
108       completion  queues associated with the endpoint.  Users provide context
109       which are associated with each operation, and is returned to  the  user
110       as part of the completion.  See fi_cq for completion event details.
111
112       By  default,  the  remote endpoint does not generate an event or notify
113       the user when a memory region has been accessed by an RMA read or write
114       operation.  However, immediate data may be associated with an RMA write
115       operation.  RMA writes with immediate data will generate  a  completion
116       entry  at the remote endpoint, so that the immediate data may be deliv‐
117       ered.
118
119   fi_write
120       The call fi_write transfers the data contained  in  the  user-specified
121       data buffer to a remote memory region.
122
123   fi_writev
124       The  fi_writev call adds support for a scatter-gather list to fi_write.
125       The fi_writev transfers the set of data buffers referenced by  the  iov
126       parameter to the remote memory region.
127
128   fi_writemsg
129       The  fi_writemsg  call  supports data transfers over both connected and
130       connectionless endpoints, with the ability to control the write  opera‐
131       tion per call through the use of flags.  The fi_writemsg function takes
132       a struct fi_msg_rma as input.
133
134              struct fi_msg_rma {
135                  const struct iovec *msg_iov;     /* local scatter-gather array */
136                  void               **desc;       /* operation descriptor */
137                  size_t             iov_count;    /* # elements in msg_iov */
138                  fi_addr_t          addr;        /* optional endpoint address */
139                  const struct fi_rma_iov *rma_iov;/* remote SGL */
140                  size_t             rma_iov_count;/* # elements in rma_iov */
141                  void               *context;     /* user-defined context */
142                  uint64_t           data;         /* optional immediate data */
143              };
144
145              struct fi_rma_iov {
146                  uint64_t           addr;         /* target RMA address */
147                  size_t             len;          /* size of target buffer */
148                  uint64_t           key;          /* access key */
149              };
150
151   fi_inject_write
152       The write inject call is an optimized version of fi_write.  It provides
153       similar completion semantics as fi_inject fi_msg(3).
154
155   fi_writedata
156       The  write data call is similar to fi_write, but allows for the sending
157       of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
158
159   fi_inject_writedata
160       The inject write data call is similar to  fi_inject_write,  but  allows
161       for  the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part
162       of the transfer.
163
164   fi_read
165       The fi_read call requests that the remote endpoint transfer  data  from
166       the remote memory region into the local data buffer.
167
168   fi_readv
169       The  fi_readv  call  adds support for a scatter-gather list to fi_read.
170       The fi_readv transfers data from the remote memory region into the  set
171       of data buffers referenced by the iov parameter.
172
173   fi_readmsg
174       The  fi_readmsg  call  supports  data transfers over both connected and
175       connectionless endpoints, with the ability to control the  read  opera‐
176       tion  per call through the use of flags.  The fi_readmsg function takes
177       a struct fi_msg_rma as input.
178

FLAGS

180       The fi_readmsg and fi_writemsg calls allow the user  to  specify  flags
181       which  can change the default data transfer operation.  Flags specified
182       with fi_readmsg / fi_writemsg override most flags previously configured
183       with the endpoint, except where noted (see fi_endpoint.3).  The follow‐
184       ing list of flags are usable with fi_readmsg and/or fi_writemsg.
185
186       FI_REMOTE_CQ_DATA
187              Applies to fi_writemsg and fi_writedata.  Indicates that  remote
188              CQ  data is available and should be sent as part of the request.
189              See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
190
191       FI_COMPLETION
192              Indicates that a completion entry should be  generated  for  the
193              specified operation.  The endpoint must be bound to a completion
194              queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
195              ified operation, or this flag is ignored.
196
197       FI_MORE
198              Indicates  that the user has additional requests that will imme‐
199              diately be posted after the current call returns.  Use  of  this
200              flag  may  improve performance by enabling the provider to opti‐
201              mize its access to the fabric hardware.
202
203       FI_INJECT
204              Applies to fi_writemsg.  Indicates that the outbound data buffer
205              should  be returned to user immediately after the write call re‐
206              turns, even if the operation is  handled  asynchronously.   This
207              may require that the underlying provider implementation copy the
208              data into a local buffer and transfer out of that buffer.   This
209              flag can only be used with messages smaller than inject_size.
210
211       FI_INJECT_COMPLETE
212              Applies  to  fi_writemsg.  Indicates that a completion should be
213              generated when the source buffer(s) may be reused.
214
215       FI_TRANSMIT_COMPLETE
216              Applies to fi_writemsg.  Indicates that a completion should  not
217              be generated until the operation has been successfully transmit‐
218              ted and is no longer being tracked by the provider.
219
220       FI_DELIVERY_COMPLETE
221              Applies to fi_writemsg.  Indicates that a completion  should  be
222              generated  when the operation has been processed by the destina‐
223              tion.
224
225       FI_COMMIT_COMPLETE
226              Applies to fi_writemsg when targeting persistent memory regions.
227              Indicates  that  a completion should be generated only after the
228              result of the operation has been made durable.
229
230       FI_FENCE
231              Applies to transmits.  Indicates that the  requested  operation,
232              also known as the fenced operation, and any operation posted af‐
233              ter the fenced operation will be deferred until all previous op‐
234              erations targeting the same peer endpoint have completed.  Oper‐
235              ations posted after the fencing will see and/or replace the  re‐
236              sults of any operations initiated prior to the fenced operation.
237
238       The ordering of operations starting at the posting of the fenced opera‐
239       tion (inclusive) to the posting of a subsequent fenced  operation  (ex‐
240       clusive) is controlled by the endpoint's ordering semantics.
241

RETURN VALUE

243       Returns 0 on success.  On error, a negative value corresponding to fab‐
244       ric errno is returned.  Fabric errno values are defined in  rdma/fi_er‐
245       rno.h.
246

ERRORS

248       -FI_EAGAIN
249              See fi_msg(3) for a detailed description of handling FI_EAGAIN.
250

SEE ALSO

252       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
253

AUTHORS

255       OpenFabrics.
256
257
258
259Libfabric Programmer's Manual     2020-10-14                         fi_rma(3)
Impressum