1fi_rma(3)                      Libfabric v1.7.0                      fi_rma(3)
2
3
4

NAME

6       fi_rma - Remote memory access operations
7
8       fi_read / fi_readv / fi_readmsg
9              Initiates a read from remote memory
10
11       fi_write / fi_writev / fi_writemsg
12       fi_inject_write / fi_writedata : Initiate a write to remote memory
13

SYNOPSIS

15              #include <rdma/fi_rma.h>
16
17              ssize_t fi_read(struct fid_ep *ep, void *buf, size_t len, void *desc,
18                  fi_addr_t src_addr, uint64_t addr, uint64_t key, void *context);
19
20              ssize_t fi_readv(struct fid_ep *ep, const struct iovec *iov, void **desc,
21                  size_t count, fi_addr_t src_addr, uint64_t addr, uint64_t key,
22                  void *context);
23
24              ssize_t fi_readmsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
25                  uint64_t flags);
26
27              ssize_t fi_write(struct fid_ep *ep, const void *buf, size_t len,
28                  void *desc, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
29                  void *context);
30
31              ssize_t fi_writev(struct fid_ep *ep, const struct iovec *iov, void **desc,
32                  size_t count, fi_addr_t dest_addr, uint64_t addr, uint64_t key,
33                  void *context);
34
35              ssize_t fi_writemsg(struct fid_ep *ep, const struct fi_msg_rma *msg,
36                  uint64_t flags);
37
38              ssize_t fi_inject_write(struct fid_ep *ep, const void *buf, size_t len,
39                  fi_addr_t dest_addr, uint64_t addr, uint64_t key);
40
41              ssize_t fi_writedata(struct fid_ep *ep, const void *buf, size_t len,
42                  void *desc, uint64_t data, fi_addr_t dest_addr, uint64_t addr,
43                  uint64_t key, void *context);
44
45              ssize_t fi_inject_writedata(struct fid_ep *ep, const void *buf, size_t len,
46                  uint64_t data, fi_addr_t dest_addr, uint64_t addr, uint64_t key);
47

ARGUMENTS

49       ep     Fabric endpoint on which to initiate read or write operation.
50
51       buf    Local  data  buffer  to  read  into  (read target) or write from
52              (write source)
53
54       len    Length of data to read or  write,  specified  in  bytes.   Valid
55              transfers are from 0 bytes up to the endpoint's max_msg_size.
56
57       iov    Vectored data buffer.
58
59       count  Count of vectored data entries.
60
61       addr   Address  of  remote  memory to access.  This will be the virtual
62              address of the remote region in the case of FI_MR_BASIC, or  the
63              offset from the starting address in the case of FI_MR_SCALABLE.
64
65       key    Protection key associated with the remote memory.
66
67       desc   Descriptor associated with the local data buffer
68
69       data   Remote CQ data to transfer with the operation.
70
71       dest_addr
72              Destination address for connectionless write transfers.  Ignored
73              for connected endpoints.
74
75       src_addr
76              Source address to read from for connectionless  transfers.   Ig‐
77              nored for connected endpoints.
78
79       msg    Message descriptor for read and write operations.
80
81       flags  Additional flags to apply for the read or write operation.
82
83       context
84              User  specified  pointer  to associate with the operation.  This
85              parameter is ignored if the operation will not generate  a  suc‐
86              cessful  completion, unless an op flag specifies the context pa‐
87              rameter be used for required input.
88

DESCRIPTION

90       RMA (remote memory access) operations are used to transfer data direct‐
91       ly between a local data buffer and a remote data buffer.  RMA transfers
92       occur on a byte level granularity, and no message boundaries are  main‐
93       tained.
94
95       The   write  functions  --  fi_write,  fi_writev,  fi_writemsg,  fi_in‐
96       ject_write, and fi_writedata -- are used to transmit data into a remote
97       memory  buffer.   The  main  difference between write functions are the
98       number and type of parameters that they accept  as  input.   Otherwise,
99       they perform the same general function.
100
101       The  read functions -- fi_read, fi_readv, and fi_readmsg -- are used to
102       transfer data from a remote memory region into  local  data  buffer(s).
103       Similar  to the write operations, read operations operate asynchronous‐
104       ly.  Users should not touch the posted data buffer(s)  until  the  read
105       operation has completed.
106
107       Completed  RMA  operations are reported to the user through one or more
108       completion queues associated with the endpoint.  Users provide  context
109       which  are  associated with each operation, and is returned to the user
110       as part of the completion.  See fi_cq for completion event details.
111
112       By default, the remote endpoint does not generate an  event  or  notify
113       the user when a memory region has been accessed by an RMA read or write
114       operation.  However, immediate data may be associated with an RMA write
115       operation.   RMA  writes with immediate data will generate a completion
116       entry at the remote endpoint, so that the immediate data may be  deliv‐
117       ered.
118
119   fi_write
120       The  call  fi_write  transfers the data contained in the user-specified
121       data buffer to a remote memory region.  The local endpoint must be con‐
122       nected  to  a remote endpoint or destination before fi_write is called.
123       Unless the endpoint has been configured differently,  the  data  buffer
124       passed  into  fi_write must not be touched by the application until the
125       fi_write call completes asynchronously.
126
127   fi_writev
128       The fi_writev call adds support for a scatter-gather list to  fi_write.
129       The  fi_writev  transfers the set of data buffers referenced by the iov
130       parameter to the remote memory region.
131
132   fi_writemsg
133       The fi_writemsg call supports data transfers over  both  connected  and
134       unconnected  endpoints, with the ability to control the write operation
135       per call through the use of flags.  The fi_writemsg  function  takes  a
136       struct fi_msg_rma as input.
137
138              struct fi_msg_rma {
139                  const struct iovec *msg_iov;     /* local scatter-gather array */
140                  void               **desc;       /* operation descriptor */
141                  size_t             iov_count;    /* # elements in msg_iov */
142                  fi_addr_t          addr;        /* optional endpoint address */
143                  const struct fi_rma_iov *rma_iov;/* remote SGL */
144                  size_t             rma_iov_count;/* # elements in rma_iov */
145                  void               *context;     /* user-defined context */
146                  uint64_t           data;         /* optional immediate data */
147              };
148
149              struct fi_rma_iov {
150                  uint64_t           addr;         /* target RMA address */
151                  size_t             len;          /* size of target buffer */
152                  uint64_t           key;          /* access key */
153              };
154
155   fi_inject_write
156       The  write inject call is an optimized version of fi_write.  The fi_in‐
157       ject_write function behaves as if the FI_INJECT transfer flag were set,
158       and  FI_COMPLETION were not.  That is, the data buffer is available for
159       reuse immediately on returning from fi_inject_write, and no  completion
160       event  will  be generated for this write.  The completion event will be
161       suppressed even if the endpoint has not been configured with  FI_SELEC‐
162       TIVE_COMPLETION.  See the flags discussion below for more details.  The
163       requested message size that can be used with fi_inject_write is limited
164       by inject_size.
165
166   fi_writedata
167       The  write data call is similar to fi_write, but allows for the sending
168       of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part of the transfer.
169
170   fi_inject_writedata
171       The inject write data call is similar to  fi_inject_write,  but  allows
172       for  the sending of remote CQ data (see FI_REMOTE_CQ_DATA flag) as part
173       of the transfer.
174
175   fi_read
176       The fi_read call requests that the remote endpoint transfer  data  from
177       the  remote  memory  region into the local data buffer.  The local end‐
178       point must be connected to a  remote  endpoint  or  destination  before
179       fi_read is called.
180
181   fi_readv
182       The  fi_readv  call  adds support for a scatter-gather list to fi_read.
183       The fi_readv transfers data from the remote memory region into the  set
184       of data buffers referenced by the iov parameter.
185
186   fi_readmsg
187       The fi_readmsg call supports data transfers over both connected and un‐
188       connected endpoints, with the ability to control the read operation per
189       call  through the use of flags.  The fi_readmsg function takes a struct
190       fi_msg_rma as input.
191

FLAGS

193       The fi_readmsg and fi_writemsg calls allow the user  to  specify  flags
194       which  can change the default data transfer operation.  Flags specified
195       with fi_readmsg / fi_writemsg override most flags previously configured
196       with the endpoint, except where noted (see fi_endpoint.3).  The follow‐
197       ing list of flags are usable with fi_readmsg and/or fi_writemsg.
198
199       FI_REMOTE_CQ_DATA
200              Applies to fi_writemsg and fi_writedata.  Indicates that  remote
201              CQ  data is available and should be sent as part of the request.
202              See fi_getinfo for additional details on FI_REMOTE_CQ_DATA.
203
204       FI_COMPLETION
205              Indicates that a completion entry should be  generated  for  the
206              specified operation.  The endpoint must be bound to a completion
207              queue with FI_SELECTIVE_COMPLETION that corresponds to the spec‐
208              ified operation, or this flag is ignored.
209
210       FI_MORE
211              Indicates  that the user has additional requests that will imme‐
212              diately be posted after the current call returns.  Use  of  this
213              flag  may  improve performance by enabling the provider to opti‐
214              mize its access to the fabric hardware.
215
216       FI_INJECT
217              Applies to fi_writemsg.  Indicates that the outbound data buffer
218              should  be returned to user immediately after the write call re‐
219              turns, even if the operation is  handled  asynchronously.   This
220              may require that the underlying provider implementation copy the
221              data into a local buffer and transfer out of that buffer.   This
222              flag can only be used with messages smaller than inject_size.
223
224       FI_INJECT_COMPLETE
225              Applies  to  fi_writemsg.  Indicates that a completion should be
226              generated when the source buffer(s) may be reused.
227
228       FI_TRANSMIT_COMPLETE
229              Applies to fi_writemsg.  Indicates that a completion should  not
230              be generated until the operation has been successfully transmit‐
231              ted and is no longer being tracked by the provider.
232
233       FI_DELIVERY_COMPLETE
234              Applies to fi_writemsg.  Indicates that a completion  should  be
235              generated  when the operation has been processed by the destina‐
236              tion.
237
238       FI_COMMIT_COMPLETE
239              Applies to fi_writemsg when targeting persistent memory regions.
240              Indicates  that  a completion should be generated only after the
241              result of the operation has been made durable.
242
243       FI_FENCE
244              Applies to transmits.  Indicates that the  requested  operation,
245              also known as the fenced operation, and any operation posted af‐
246              ter the fenced operation will be deferred until all previous op‐
247              erations targeting the same peer endpoint have completed.  Oper‐
248              ations posted after the fencing will see and/or replace the  re‐
249              sults of any operations initiated prior to the fenced operation.
250
251       The ordering of operations starting at the posting of the fenced opera‐
252       tion (inclusive) to the posting of a subsequent fenced  operation  (ex‐
253       clusive) is controlled by the endpoint's ordering semantics.
254

RETURN VALUE

256       Returns 0 on success.  On error, a negative value corresponding to fab‐
257       ric errno is returned.  Fabric errno values are defined in  rdma/fi_er‐
258       rno.h.
259

ERRORS

261       -FI_EAGAIN
262              See fi_msg(3) for a detailed description of handling FI_EAGAIN.
263

SEE ALSO

265       fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cq(3)
266

AUTHORS

268       OpenFabrics.
269
270
271
272Libfabric Programmer's Manual     2018-11-28                         fi_rma(3)
Impressum