1mlx5dv_wr_mkey_configure(3) mlx5dv_wr_mkey_configure(3)
2
6 mlx5dv_wr_mkey_configure - Create a work request to configure an MKEY
7 mlx5dv_wr_set_mkey_access_flags - Set the memory protection attributes
9 for an MKEY
10 mlx5dv_wr_set_mkey_layout_list - Set a memory layout for an MKEY based
12 on SGE list
13 mlx5dv_wr_set_mkey_layout_interleaved - Set an interleaved memory lay‐
15 out for an MKEY
16
18 #include <infiniband/mlx5dv.h>
19 static inline void mlx5dv_wr_mkey_configure(struct mlx5dv_qp_ex *mqp,
21 struct mlx5dv_mkey *mkey,
22 uint8_t num_setters,
23 struct mlx5dv_mkey_conf_attr *attr);
24 static inline void mlx5dv_wr_set_mkey_access_flags(struct mlx5dv_qp_ex *mqp,
26 uint32_t access_flags);
27 static inline void mlx5dv_wr_set_mkey_layout_list(struct mlx5dv_qp_ex *mqp,
29 uint16_t num_sges,
30 const struct ibv_sge *sge);
31 static inline void mlx5dv_wr_set_mkey_layout_interleaved(struct mlx5dv_qp_ex *mqp,
33 uint32_t repeat_count,
34 uint16_t num_interleaved,
35 const struct mlx5dv_mr_interleaved *data);
36
38 The MLX5DV MKEY configure API and the related setters
39 (mlx5dv_wr_set_mkey*) are an extension of IBV work request API
40 (ibv_wr*) with specific features for MLX5DV MKEY.
41 MKEYs allow creation of virtually-contiguous address spaces out of non-
43 contiguous chunks of memory regions already registered with the hard‐
44 ware. Additionally it provides access to some advanced hardware off‐
45 load features, e.g. signature offload.
46 These APIs are intended to be used to access additional functionality
48 beyond what is provided by mlx5dv_wr_mr_list() and mlx5dv_wr_mr_inter‐
49 leaved(). The MKEY features can be optionally enabled using the mkey
50 configure setters. It allows using different features in the same
51 MKEY.
52
54 To use these APIs a QP must be created using mlx5dv_create_qp(3) which
55 allows setting the MLX5DV_QP_EX_WITH_MKEY_CONFIGURE in send_ops_flags.
56 The MKEY configuration work request is created by calling
58 mlx5dv_wr_mkey_configure(), a WR builder function, followed by required
59 setter functions. num_setters is a number of required setters for the
60 WR. All setters are optional. num_setters can be zero to apply attr
61 only. Each setter can be called only once per the WR builder.
62 The WR configures mkey and applies attr of the builder function and
64 setter functions’ arguments for it. If mkey is already configured the
65 WR overrides some mkey properties depends on builder and setter func‐
66 tions’ arguments (see details in setters’ description). To clear con‐
67 figuration of mkey, use ibv_post_send() with IBV_WR_LOCAL_INV opcode or
68 ibv_wr_local_inv().
69 Current implementation requires the IBV_SEND_INLINE option to be set in
71 wr_flags field of ibv_qp_ex structure prior to builder function call.
72 Non-inline payload is currently not supported by this API. Please note
73 that inlining here is done for MKEY configuration data, not for user
74 data referenced by data layouts.
75 Once MKEY is configured, it may be used in subsequent work requests
77 (SEND, RDMA_READ, RDMA_WRITE, etc). If these work requests are posted
78 on the same QP, there is no need to wait for completion of MKEY config‐
79 uration work request. They can be posted immediately after the last
80 setter (or builder if no setters). Usually there is no need to even
81 request a completion for MKEY configuration work request.
82 If completion is requested for MKEY configuration work request it will
84 be delivered with the IBV_WC_DRIVER1 opcode.
85 Builder function
87 mlx5dv_wr_mkey_configure()
88 Post a work request to configure an existing MKEY. With this
89 call alone it is possible to configure the MKEY and keep or re‐
90 set signature attributes. This call may be followed by zero or
91 more optional setters.
92 mqp The QP to post the work request on.
94 mkey The MKEY to configure.
96 num_setters
98 The number of setters that must be called after this
99 function.
100 attr The MKEY configuration attributes
102 MKEY configuration attributes
104 MKEY configuration attributes are provided in mlx5dv_mkey_conf_attr
105 structure.
106 struct mlx5dv_mkey_conf_attr {
108 uint32_t conf_flags;
109 uint64_t comp_mask;
110 };
111 conf_flags
113 Bitwise OR of the following flags:
114 MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR
116 Reset the signature attributes of the MKEY. If not set,
117 previously configured signature attributes will be kept.
118 comp_mask
120 Reserved for future extension, must be 0 now.
121 Generic setters
123 mlx5dv_wr_set_mkey_access_flags()
124 Set the memory protection attributes for the MKEY. If the MKEY
125 is configured, the setter overrides the previous value. For ex‐
126 ample, two MKEY configuration WRs are posted. The first one
127 sets IBV_ACCESS_REMOTE_READ. The second one sets IBV_ACCESS_RE‐
128 MOTE_WRITE. In this case, the second WR overrides the memory
129 protection attributes, and only IBV_ACCESS_REMOTE_WRITE is al‐
130 lowed for the MKEY when the WR is completed.
131 mqp The QP where an MKEY configuration work request was cre‐
133 ated by mlx5dv_wr_mkey_configure().
134 access_flags
136 The desired memory protection attributes; it is either 0
137 or the bitwise OR of one or more of flags in enum ibv_ac‐
138 cess_flags.
139 Data layout setters
141 Data layout setters define how data referenced by the MKEY will be
142 scattered/gathered in the memory. In order to use MKEY with RDMA oper‐
143 ations it must be configured with a layout.
144 Not more than one data layout setter may follow builder function. Lay‐
146 out can be updated in the next calls to builder function.
147 When MKEY is used in RDMA operations, it should be used in a zero-based
149 mode, i.e. the addr field in ibv_sge structure is an offset in the to‐
150 tal data.
151 mlx5dv_wr_set_mkey_layout_list()
153 Set a memory layout for an MKEY based on SGE list. If the MKEY
154 is configured and the data layout was defined by some data lay‐
155 out setter (not necessary this one), the setter overrides the
156 previous value.
157 Default WQE size can fit only 4 SGE entries. To allow more the
159 QP should be created with a larger WQE size that may fit it.
160 This should be done using the max_inline_data attribute of
161 struct ibv_qp_cap upon QP creation.
162 mqp The QP where an MKEY configuration work request was cre‐
164 ated by mlx5dv_wr_mkey_configure().
165 num_sges
167 Number of SGEs in the list.
168 sge Pointer to the list of ibv_sge structures.
170 mlx5dv_wr_set_mkey_layout_interleaved()
172 Set an interleaved memory layout for an MKEY. If the MKEY is
173 configured and the data layout was defined by some data layout
174 setter (not necessary this one), the setter overrides the previ‐
175 ous value.
176 Default WQE size can fit only 3 interleaved entries. To allow
178 more the QP should be created with a larger WQE size that may
179 fit it. This should be done using the max_inline_data attribute
180 of struct ibv_qp_cap upon QP creation.
181 As one entry will be consumed for strided header, the MKEY
183 should be created with one more entry than the required num_in‐
184 terleaved.
185 mqp The QP where an MKEY configuration work request was cre‐
187 ated by mlx5dv_wr_mkey_configure().
188 repeat_count
190 The data layout representation is repeated repeat_count
191 times.
192 num_interleaved
194 Number of entries in the data representation.
195 data Pointer to the list of interleaved data layout descrip‐
197 tions.
198 Interleaved data layout is described by mlx5dv_mr_interleaved
200 structure.
201 struct mlx5dv_mr_interleaved {
203 uint64_t addr;
204 uint32_t bytes_count;
205 uint32_t bytes_skip;
206 uint32_t lkey;
207 };
208 addr Start address of the local memory buffer.
210 bytes_count
212 Number of data bytes to put into the buffer.
213 bytes_skip
215 Number of bytes to skip in the buffer before the next da‐
216 ta block.
217 lkey Key of the local Memory Region
219 Signature setters
221 The signature attributes of the MKEY allow adding/modifying/strip‐
222 ping/validating integrity fields when transmitting data from memory to
223 network and when receiving data from network to memory.
224 Use the signature setters to set/update the signature attributes of the
226 MKEY. To reset the signature attributes without invalidating the MKEY,
227 use the MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR flag.
228 mlx5dv_wr_set_mkey_sig_block()
230 Set MKEY block signature attributes. If the MKEY is already
231 configured with the signature attributes, the setter overrides
232 the previous value. See dedicated man page for
233 mlx5dv_wr_set_mkey_sig_block(3).
234 Crypto setter
236 The crypto attributes of the MKey allow encryption and decryption of
237 transmitted data from memory to network and when receiving data from
238 network to memory.
239 Use the crypto setter to set/update the crypto attributes of the MKey.
241 When the MKey is created with MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO it
242 must be configured with crypto attributes before the MKey can be used.
243 mlx5dv_wr_set_mkey_crypto()
245 Set MKey crypto attributes. If the MKey is already configured
246 with crypto attributes, the setter overrides the previous value.
247 see dedicated man page for mlx5dv_wr_set_mkey_crypto(3).
248
250 Create QP and MKEY
251 Code below creates a QP with MKEY configure operation support and an
252 indirect mkey.
253 /* Create QP with MKEY configure support */
255 struct ibv_qp_init_attr_ex attr_ex = {};
256 attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
257 attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
258 struct mlx5dv_qp_init_attr attr_dv = {};
260 attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_SEND_OPS_FLAGS;
261 attr_dv.send_ops_flags = MLX5DV_QP_EX_WITH_MKEY_CONFIGURE;
262 ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
264 ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
265 mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
266 mkey_attr.create_flags = MLX5DV_MKEY_INIT_ATTR_FLAGS_INDIRECT;
268 struct mlx5dv_mkey *mkey = mlx5dv_create_mkey(&mkey_attr);
269 List data layout configuration
271 Code below configures an MKEY which allows remote access for read and
272 write and is based on SGE list layout with two entries. When this MKEY
273 is used in RDMA write operation, data will be scattered between two
274 memory regions. The first 64 bytes will go to memory referenced by
275 mr1. The next 4096 bytes will go to memory referenced by mr2.
276 ibv_wr_start(qpx);
278 qpx->wr_id = my_wr_id_1;
279 qpx->wr_flags = IBV_SEND_INLINE;
280 struct mlx5dv_mkey_conf_attr mkey_attr = {};
282 mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
283 mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
284 struct ibv_sge sgl[2];
285 sgl[0].addr = mr1->addr;
286 sgl[0].length = 64;
287 sgl[0].lkey = mr1->lkey;
288 sgl[1].addr = mr2->addr;
289 sgl[1].length = 4096;
290 sgl[1].lkey = mr2->lkey;
291 mlx5dv_wr_set_mkey_layout_list(mqpx, 2, sgl);
292 ret = ibv_wr_complete(qpx);
293 Interleaved data layout configuration
295 Code below configures an MKEY which allows remote access for read and
296 write and is based on interleaved data layout with two entries and re‐
297 peat count of two. When this MKEY is used in RDMA write operation, da‐
298 ta will be scattered between two memory regions. The first 512 bytes
299 will go to memory referenced by mr1 at offset 0. The next 8 bytes will
300 go to memory referenced by mr2 at offset 0. The next 512 bytes will go
301 to memory referenced by mr1 at offset 516. The next 8 bytes will go to
302 memory referenced by mr2 at offset 8.
303 ibv_wr_start(qpx);
305 qpx->wr_id = my_wr_id_1;
306 qpx->wr_flags = IBV_SEND_INLINE;
307 struct mlx5dv_mkey_conf_attr mkey_attr = {};
309 mlx5dv_wr_mkey_configure(mqpx, mkey, 2, &mkey_attr);
310 mlx5dv_wr_set_mkey_access_flags(mqpx, IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE);
311 struct mlx5dv_mr_interleaved data[2];
312 data[0].addr = mr1->addr;
313 data[0].bytes_count = 512;
314 data[0].bytes_skip = 4;
315 data[0].lkey = mr1->lkey;
316 data[1].addr = mr2->addr;
317 data[1].bytes_count = 8;
318 data[1].bytes_skip = 0;
319 data[1].lkey = mr2->lkey;
320 mlx5dv_wr_set_mkey_layout_interleaved(mqpx, 2, 2, &data);
321 ret = ibv_wr_complete(qpx);
322
324 A DEVX context should be opened by using mlx5dv_open_device(3).
325
327 mlx5dv_create_mkey(3), mlx5dv_create_qp(3),
328 mlx5dv_wr_set_mkey_sig_block(3), mlx5dv_wr_set_mkey_crypto(3)
329
331 Oren Duer <oren@nvidia.com>
332 Sergey Gorenko <sergeygo@nvidia.com>
334 Evgenii Kochetov <evgeniik@nvidia.com>
336 mlx5dv_wr_mkey_configure(3)