1MLX5DV_WR(3) mlx5 Programmer’s Manual MLX5DV_WR(3)
2
6 mlx5dv_wr_set_dc_addr - Attach a DC info to the last work request
7
9 #include <infiniband/mlx5dv.h>
10 static inline void mlx5dv_wr_set_dc_addr(struct mlx5dv_qp_ex *mqp,
12 struct ibv_ah *ah,
13 uint32_t remote_dctn,
14 uint64_t remote_dc_key);
15 struct mlx5dv_mr_interleaved {
17 uint64_t addr;
18 uint32_t bytes_count;
19 uint32_t bytes_skip;
20 uint32_t lkey;
21 };
22 static inline void mlx5dv_wr_mr_interleaved(struct mlx5dv_qp_ex *mqp,
24 struct mlx5dv_mkey *mkey,
25 uint32_t access_flags, /* use enum ibv_access_flags */
26 uint32_t repeat_count,
27 uint16_t num_interleaved,
28 struct mlx5dv_mr_interleaved *data);
29 static inline void mlx5dv_wr_mr_list(struct mlx5dv_qp_ex *mqp,
31 struct mlx5dv_mkey *mkey,
32 uint32_t access_flags, /* use enum ibv_access_flags */
33 uint16_t num_sges,
34 struct ibv_sge *sge);
35
37 The MLX5DV work request APIs (mlx5dv_wr_*) is an extension for IBV work
38 request API (ibv_wr_*) with mlx5 specific features for send work re‐
39 quest. This may be used together with or without ibv_wr_* calls.
40
42 To use these APIs a QP must be created using mlx5dv_create_qp() with
43 send_ops_flags of struct ibv_qp_init_attr_ex set.
44 If the QP does not support all the requested work request types then QP
46 creation will fail.
47 The mlx5dv_qp_ex is extracted from the IBV_QP by ibv_qp_to_qp_ex() and
49 mlx5dv_qp_ex_from_ibv_qp_ex(). This should be used to apply the mlx5
50 specific features on the posted WR.
51 A work request creation requires to use the ibv_qp_ex as described in
53 the man for ibv_wr_post and mlx5dv_qp with its available builders and
54 setters.
55 QP Specific builders
57 RC QPs mlx5dv_wr_mr_interleaved()
58 registers an interleaved memory layout by using an indirect mkey
60 and some interleaved data. The layout of the memory pointed by
61 the mkey after its registration will be the data representation
62 for the num_interleaved entries. This single layout representa‐
63 tion is repeated by repeat_count.
64 The data as described by struct mlx5dv_mr_interleaved will hold
66 real data defined by bytes_count and then a padding of
67 bytes_skip. Post a successful registration, RDMA operations can
68 use this mkey. The hardware will scatter the data according to
69 the pattern. The mkey should be used in a zero-based mode. The
70 addr field in its ibv_sge is an offset in the total data. To
71 create this mkey mlx5dv_create_mkey() should be used.
72 Current implementation requires the IBV_SEND_INLINE option to be
74 on in ibv_qp_ex->wr_flags field. To be able to have more than 3
75 num_interleaved entries, the QP should be created with a larger
76 WQE size that may fit it. This should be done using the max_in‐
77 line_data attribute of struct ibv_qp_cap upon its creation.
78 As one entry will be consumed for strided header, the mkey
80 should be created with one more entry than the required num_in‐
81 terleaved.
82 In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the re‐
84 ported WC opcode will be MLX5DV_WC_UMR. Unregister the mkey to
85 enable another pattern registration should be done via
86 ibv_post_send with IBV_WR_LOCAL_INV opcode.
87 mlx5dv_wr_mr_list()
88 registers a memory layout based on list of ibv_sge. The layout
90 of the memory pointed by the mkey after its registration will be
91 based on the list of sge counted by num_sges. Post a successful
92 registration RDMA operations can use this mkey, the hardware
93 will scatter the data according to the pattern. The mkey should
94 be used in a zero-based mode, the addr field in its ibv_sge is
95 an offset in the total data.
96 Current implementation requires the IBV_SEND_INLINE option to be
98 on in ibv_qp_ex->wr_flags field. To be able to have more than 4
99 num_sge entries, the QP should be created with a larger WQE size
100 that may fit it. This should be done using the max_inline_data
101 attribute of struct ibv_qp_cap upon its creation.
102 In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the re‐
104 ported WC opcode will be MLX5DV_WC_UMR. Unregister the mkey to
105 enable other pattern registration should be done via
106 ibv_post_send with IBV_WR_LOCAL_INV opcode.
107 QP Specific setters
109 DCI QPs
110 mlx5dv_wr_set_dc_addr() must be called to set the DCI WR proper‐
111 ties. The destination address of the work is specified by ah,
112 the remote DCT number is specified by remote_dctn and the DC key
113 is specified by remote_dc_key. This setter is available when
114 the QP transport is DCI and send_ops_flags in struct
115 ibv_qp_init_attr_ex is set. The available builders and setters
116 for DCI QP are the same as RC QP.
117
119 /* create DC QP type and specify the required send opcodes */
120 attr_ex.qp_type = IBV_QPT_DRIVER;
121 attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
122 attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
123 attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_DC;
125 attr_dv.dc_init_attr.dc_type = MLX5DV_DCTYPE_DCI;
126 ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
128 ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
129 mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
130 ibv_wr_start(qpx);
132 /* Use ibv_qp_ex object to set WR generic attributes */
134 qpx->wr_id = my_wr_id_1;
135 qpx->wr_flags = IBV_SEND_SIGNALED;
136 ibv_wr_rdma_write(qpx, rkey, remote_addr_1);
137 ibv_wr_set_sge(qpx, lkey, local_addr_1, length_1);
138 /* Use mlx5 DC setter using mlx5dv_qp_ex object */
140 mlx5dv_wr_set_wr_dc_addr(mqpx, ah, remote_dctn, remote_dc_key);
141 ret = ibv_wr_complete(qpx);
143
145 ibv_post_send(3), ibv_create_qp_ex(3), ibv_wr_post(3), mlx5dv_cre‐
146 ate_mkey(3).
147
149 Guy Levi <guyle@mellanox.com>
150mlx5 2019-02-24 MLX5DV_WR(3)