1mlx5dv_wr_mkey_configure(3)                        mlx5dv_wr_mkey_configure(3)
2
3
4

NAME

6       mlx5dv_wr_mkey_configure - Create a work request to configure an MKEY
7
8       mlx5dv_wr_set_mkey_access_flags  - Set the memory protection attributes
9       for an MKEY
10
11       mlx5dv_wr_set_mkey_layout_list - Set a memory layout for an MKEY  based
12       on SGE list
13
14       mlx5dv_wr_set_mkey_layout_interleaved  - Set an interleaved memory lay‐
15       out for an MKEY
16

SYNOPSIS

18              #include <infiniband/mlx5dv.h>
19
20              static inline void mlx5dv_wr_mkey_configure(struct mlx5dv_qp_ex *mqp,
21                                                          struct mlx5dv_mkey *mkey,
22                                                          uint8_t num_setters,
23                                                          struct mlx5dv_mkey_conf_attr *attr);
24
25              static inline void mlx5dv_wr_set_mkey_access_flags(struct mlx5dv_qp_ex *mqp,
26                                                                 uint32_t access_flags);
27
28              static inline void mlx5dv_wr_set_mkey_layout_list(struct mlx5dv_qp_ex *mqp,
29                                                                uint16_t num_sges,
30                                                                const struct ibv_sge *sge);
31
32              static inline void mlx5dv_wr_set_mkey_layout_interleaved(struct mlx5dv_qp_ex *mqp,
33                                                                       uint32_t repeat_count,
34                                                                       uint16_t num_interleaved,
35                                                                       const struct mlx5dv_mr_interleaved *data);
36

DESCRIPTION

38       The   MLX5DV   MKEY   configure   API   and   the    related    setters
39       (mlx5dv_wr_set_mkey*)   are  an  extension  of  IBV  work  request  API
40       (ibv_wr*) with specific features for MLX5DV MKEY.
41
42       MKEYs allow creation of virtually-contiguous address spaces out of non-
43       contiguous  chunks  of memory regions already registered with the hard‐
44       ware.  Additionally it provides access to some advanced  hardware  off‐
45       load features, e.g.  signature offload.
46
47       These  APIs  are intended to be used to access additional functionality
48       beyond what is provided by mlx5dv_wr_mr_list() and  mlx5dv_wr_mr_inter‐
49       leaved().   The  MKEY features can be optionally enabled using the mkey
50       configure setters.  It allows to use different  features  in  the  same
51       MKEY.
52

USAGE

54       To  use these APIs a QP must be created using mlx5dv_create_qp(3) which
55       allows setting the MLX5DV_QP_EX_WITH_MKEY_CONFIGURE in send_ops_flags.
56
57       The  MKEY  configuration   work   request   is   created   by   calling
58       mlx5dv_wr_mkey_configure(), a WR builder function, followed by required
59       setter functions.  num_setters is a number of required setters for  the
60       WR.   All  setters are optional.  num_setters can be zero to apply attr
61       only.  Each setter can be called only once per the WR builder.
62
63       The WR configures mkey and applies attr of  the  builder  function  and
64       setter  functions’ arguments for it.  If mkey is already configured the
65       WR overrides some mkey properties depends on builder and  setter  func‐
66       tions’  arguments (see details in setters’ description).  To clear con‐
67       figuration of mkey, use ibv_post_send() with IBV_WR_LOCAL_INV opcode or
68       ibv_wr_local_inv().
69
70       Current implementation requires the IBV_SEND_INLINE option to be set in
71       wr_flags field of ibv_qp_ex structure prior to builder  function  call.
72       Non-inline payload is currently not supported by this API.  Please note
73       that inlining here is done for MKEY configuration data,  not  for  user
74       data referenced by data layouts.
75
76       Once  MKEY  is  configured,  it may be used in subsequent work requests
77       (SEND, RDMA_READ, RDMA_WRITE, etc).  If these work requests are  posted
78       on the same QP, there is no need to wait for completion of MKEY config‐
79       uration work request.  They can be posted immediately  after  the  last
80       setter  (or  builder  if no setters).  Usually there is no need to even
81       request a completion for MKEY configuration work request.
82
83       If completion is requested for MKEY configuration work request it  will
84       be delivered with the IBV_WC_DRIVER1 opcode.
85
86   Builder function
87       mlx5dv_wr_mkey_configure()
88              Post  a  work  request to configure an existing MKEY.  With this
89              call alone it is possible to configure the MKEY and keep or  re‐
90              set  signature attributes.  This call may be followed by zero or
91              more optional setters.
92
93              mqp    The QP to post the work request on.
94
95              mkey   The MKEY to configure.
96
97              num_setters
98                     The number of setters that  must  be  called  after  this
99                     function.
100
101              attr   The MKEY configuration attributes
102
103   MKEY configuration attributes
104       MKEY  configuration  attributes  are  provided in mlx5dv_mkey_conf_attr
105       structure.
106
107              struct mlx5dv_mkey_conf_attr {
108                      uint32_t conf_flags;
109                      uint64_t comp_mask;
110              };
111
112       conf_flags
113              Bitwise OR of the following flags:
114
115              MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR
116                     Reset the signature attributes of the MKEY.  If not  set,
117                     previously configured signature attributes will be kept.
118
119       comp_mask
120              Reserved for future extension, must be 0 now.
121
122   Generic setters
123       mlx5dv_wr_set_mkey_access_flags()
124              Set  the memory protection attributes for the MKEY.  If the MKEY
125              is configured, the setter overrides the previous value.  For ex‐
126              ample,  two  MKEY  configuration  WRs are posted.  The first one
127              sets IBV_ACCESS_REMOTE_READ.  The second one sets IBV_ACCESS_RE‐
128              MOTE_WRITE.   In  this  case, the second WR overrides the memory
129              protection attributes, and only IBV_ACCESS_REMOTE_WRITE  is  al‐
130              lowed for the MKEY when the WR is completed.
131
132              mqp    The  QP where an MKEY configuration work request was cre‐
133                     ated by mlx5dv_wr_mkey_configure().
134
135              access_flags
136                     The desired memory protection attributes; it is either  0
137                     or the bitwise OR of one or more of flags in enum ibv_ac‐
138                     cess_flags.
139
140   Data layout setters
141       Data layout setters define how data referenced  by  the  MKEY  will  be
142       scattered/gathered in the memory.  In order to use MKEY with RDMA oper‐
143       ations it must be configured with a layout.
144
145       Not more than one data layout setter may follow builder function.  Lay‐
146       out can be updated in the next calls to builder function.
147
148       When MKEY is used in RDMA operations, it should be used in a zero-based
149       mode, i.e. the addr field in ibv_sge structure is an offset in the  to‐
150       tal data.
151
152       mlx5dv_wr_set_mkey_layout_list()
153              Set  a memory layout for an MKEY based on SGE list.  If the MKEY
154              is configured and the data layout was defined by some data  lay‐
155              out  setter  (not  necessary this one), the setter overrides the
156              previous value.
157
158              Default WQE size can fit only 4 SGE entries.  To allow more  the
159              QP  should  be  created  with a larger WQE size that may fit it.
160              This should be  done  using  the  max_inline_data  attribute  of
161              struct ibv_qp_cap upon QP creation.
162
163              mqp    The  QP where an MKEY configuration work request was cre‐
164                     ated by mlx5dv_wr_mkey_configure().
165
166              num_sges
167                     Number of SGEs in the list.
168
169              sge    Pointer to the list of ibv_sge structures.
170
171       mlx5dv_wr_set_mkey_layout_interleaved()
172              Set an interleaved memory layout for an MKEY.  If  the  MKEY  is
173              configured  and  the data layout was defined by some data layout
174              setter (not necessary this one), the setter overrides the previ‐
175              ous value.
176
177              Default  WQE  size can fit only 3 interleaved entries.  To allow
178              more the QP should be created with a larger WQE  size  that  may
179              fit it.  This should be done using the max_inline_data attribute
180              of struct ibv_qp_cap upon QP creation.
181
182              As one entry will be  consumed  for  strided  header,  the  MKEY
183              should  be created with one more entry than the required num_in‐
184              terleaved.
185
186              mqp    The QP where an MKEY configuration work request was  cre‐
187                     ated by mlx5dv_wr_mkey_configure().
188
189              repeat_count
190                     The  data  layout representation is repeated repeat_count
191                     times.
192
193              num_interleaved
194                     Number of entries in the data representation.
195
196              data   Pointer to the list of interleaved data  layout  descrip‐
197                     tions.
198
199              Interleaved  data  layout  is described by mlx5dv_mr_interleaved
200              structure.
201
202                     struct mlx5dv_mr_interleaved {
203                         uint64_t addr;
204                         uint32_t bytes_count;
205                         uint32_t bytes_skip;
206                         uint32_t lkey;
207                     };
208
209              addr   Start address of the local memory buffer.
210
211              bytes_count
212                     Number of data bytes to put into the buffer.
213
214              bytes_skip
215                     Number of bytes to skip in the buffer before the next da‐
216                     ta block.
217
218              lkey   Key of the local Memory Region
219
220   Signature setters
221       The  signature  attributes  of  the  MKEY allow adding/modifying/strip‐
222       ping/validating integrity fields when transmitting data from memory  to
223       network and when receiving data from network to memory.
224
225       Use the signature setters to set/update the signature attributes of the
226       MKEY.  To reset the signature attributes without invalidating the MKEY,
227       use the MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR flag.
228
229       mlx5dv_wr_set_mkey_sig_block()
230              Set  MKEY  block  signature  attributes.  If the MKEY is already
231              configured with the signature attributes, the  setter  overrides
232              the    previous    value.     See   dedicated   man   page   for
233              mlx5dv_wr_set_mkey_sig_block(3).
234
235   Crypto setter
236       The crypto attributes of the MKey allow encryption  and  decryption  of
237       transmitted  data  from  memory to network and when receiving data from
238       network to memory.
239
240       Use the crypto setter to set/update the crypto attributes of the  MKey.
241       When  the  MKey  is  created with MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO it
242       must be configured with crypto attributes before the MKey can be used.
243
244       mlx5dv_wr_set_mkey_crypto()
245              Set MKey crypto attributes.  If the MKey is  already  configured
246              with crypto attributes, the setter overrides the previous value.
247              see dedicated man page for mlx5dv_wr_set_mkey_crypto(3).
248

EXAMPLES

250   Create QP and MKEY
251       Code below creates a QP with MKEY configure operation  support  and  an
252       indirect mkey.
253
254              /* Create QP with MKEY configure support */
255              struct ibv_qp_init_attr_ex attr_ex = {};
256              attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
257              attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
258
259              struct mlx5dv_qp_init_attr attr_dv = {};
260              attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS;
261              attr_dv.send_ops_flags = MLX5DV_QP_EX_WITH_MKEY_CONFIGURE;
262
263              ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
264              ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
265              mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
266
267              mkey_attr.create_flags = MLX5DV_MKEY_INIT_ATTR_FLAGS_INDIRECT;
268              struct mlx5dv_mkey *mkey = mlx5dv_create_mkey(&mkey_attr);
269
270   List data layout configuration
271       Code  below  configures an MKEY which allows remote access for read and
272       write and is based on SGE list layout with two entries.  When this MKEY
273       is  used  in  RDMA  write operation, data will be scattered between two
274       memory regions.  The first 64 bytes will go  to  memory  referenced  by
275       mr1.  The next 4096 bytes will go to memory referenced by mr2.
276
277              ibv_wr_start(qpx);
278              qpx->wr_id = my_wr_id_1;
279              qpx->wr_flags = IBV_SEND_INLINE;
280
281              struct mlx5dv_mkey_conf_attr mkey_attr = {};
282              mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
283              mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
284              struct ibv_sge sgl[2];
285              sgl[0].addr = mr1->addr;
286              sgl[0].length = 64;
287              sgl[0].lkey = mr1->lkey;
288              sgl[1].addr = mr2->addr;
289              sgl[1].length = 4096;
290              sgl[1].lkey = mr2->lkey;
291              mlx5dv_wr_set_mkey_layout_list(mqpx, 2, sgl);
292              ret = ibv_wr_complete(qpx);
293
294   Interleaved data layout configuration
295       Code  below  configures an MKEY which allows remote access for read and
296       write and is based on interleaved data layout with two entries and  re‐
297       peat count of two.  When this MKEY is used in RDMA write operation, da‐
298       ta will be scattered between two memory regions.  The first  512  bytes
299       will go to memory referenced by mr1 at offset 0.  The next 8 bytes will
300       go to memory referenced by mr2 at offset 0.  The next 512 bytes will go
301       to memory referenced by mr1 at offset 516.  The next 8 bytes will go to
302       memory referenced by mr2 at offset 8.
303
304              ibv_wr_start(qpx);
305              qpx->wr_id = my_wr_id_1;
306              qpx->wr_flags = IBV_SEND_INLINE;
307
308              struct mlx5dv_mkey_conf_attr mkey_attr = {};
309              mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
310              mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
311              struct mlx5dv_mr_interleaved data[2];
312              data[0].addr = mr1->addr;
313              data[0].bytes_count = 512;
314              data[0].bytes_skip = 4;
315              data[0].lkey = mr1->lkey;
316              data[1].addr = mr2->addr;
317              data[1].bytes_count = 8;
318              data[1].bytes_skip = 0;
319              data[1].lkey = mr2->lkey;
320              mlx5dv_wr_set_mkey_layout_interleaved(mqpx, 2, 2, &data);
321              ret = ibv_wr_complete(qpx);
322

NOTES

324       A DEVX context should be opened by using mlx5dv_open_device(3).
325

SEE ALSO

327       mlx5dv_create_mkey(3),                             mlx5dv_create_qp(3),
328       mlx5dv_wr_set_mkey_sig_block(3)
329

AUTHORS

331       Oren Duer <oren@nvidia.com>
332
333       Sergey Gorenko <sergeygo@nvidia.com>
334
335       Evgenii Kochetov <evgeniik@nvidia.com>
336
337
338
339                                                   mlx5dv_wr_mkey_configure(3)
Impressum