rdma_cm(7)

1RDMA_CM(7)               Librdmacm Programmer's Manual              RDMA_CM(7)
2
3
4

NAME

6       rdma_cm - RDMA communication manager.
7

SYNOPSIS

9       #include <rdma/rdma_cma.h>
10

DESCRIPTION

12       Used to establish communication over RDMA transports.
13

NOTES

15       The  RDMA  CM  is  a communication manager used to setup reliable, con‐
16       nected and unreliable datagram data transfers.   It  provides  an  RDMA
17       transport neutral interface for establishing connections.  The API con‐
18       cepts are based on sockets, but  adapted  for  queue  pair  (QP)  based
19       semantics:  communication must be over a specific RDMA device, and data
20       transfers are message based.
21
22       The RDMA CM can control both the QP and communication management  (con‐
23       nection setup / teardown) portions of an RDMA API, or only the communi‐
24       cation management piece.  It works in conjunction with  the  verbs  API
25       defined by the libibverbs library.  The libibverbs library provides the
26       underlying interfaces needed to send and receive data.
27
28       The RDMA CM can operate asynchronously or synchronously.  The  mode  of
29       operation  is  controlled  by  the  user through the use of the rdma_cm
30       event channel parameter in specific calls.  If an event channel is pro‐
31       vided,  an  rdma_cm  identifier  will report its event data (results of
32       connecting, for example), on that channel.  If a channel  is  not  pro‐
33       vided,  then all rdma_cm operations for the selected rdma_cm identifier
34       will block until they complete.
35
36       The RDMA CM gives an option to different libibverbs providers to adver‐
37       tise  and  use  various  specific  to  that  provider  QP configuration
38       options. This functionality is called ECE (enhanced  connection  estab‐
39       lishment).
40

RDMA VERBS

42       The  rdma_cm  supports  the  full  range of verbs available through the
43       libibverbs library and interfaces.  However, it also  provides  wrapper
44       functions for some of the more commonly used verbs funcationality.  The
45       full set of abstracted verb calls are:
46
47       rdma_reg_msgs  - register an array of buffers for sending and receiving
48
49       rdma_reg_read  - registers a buffer for RDMA read operations
50
51       rdma_reg_write - registers a buffer for RDMA write operations
52
53       rdma_dereg_mr  - deregisters a memory region
54
55       rdma_post_recv  - post a buffer to receive a message
56
57       rdma_post_send  - post a buffer to send a message
58
59       rdma_post_read  - post an RDMA to read data into a buffer
60
61       rdma_post_write - post an RDMA to send data from a buffer
62
63       rdma_post_recvv  - post a vector of buffers to receive a message
64
65       rdma_post_sendv  - post a vector of buffers to send a message
66
67       rdma_post_readv  - post a vector of buffers to receive an RDMA read
68
69       rdma_post_writev - post a vector of buffers to send an RDMA write
70
71       rdma_post_ud_send - post a buffer to send a message on a UD QP
72
73       rdma_get_send_comp - get completion status for a send or RDMA operation
74
75       rdma_get_recv_comp - get information about a completed receive
76

CLIENT OPERATION

78       This section provides a general overview of the basic operation for the
79       active,  or  client, side of communication.  This flow assume asynchro‐
80       nous operation with low level  call  details  shown.   For  synchronous
81       operation,   calls   to  rdma_create_event_channel,  rdma_get_cm_event,
82       rdma_ack_cm_event, and rdma_destroy_event_channel would be  eliminated.
83       Abstracted  calls,  such as rdma_create_ep encapsulate several of these
84       calls under a single API.  Users may also refer to the example applica‐
85       tions for code samples.  A general connection flow would be:
86
87       rdma_getaddrinfo
88              retrieve address information of the destination
89
90       rdma_create_event_channel
91              create channel to receive events
92
93       rdma_create_id
94              allocate an rdma_cm_id, this is conceptually similar to a socket
95
96       rdma_resolve_addr
97              obtain a local RDMA device to reach the remote address
98
99       rdma_get_cm_event
100              wait for RDMA_CM_EVENT_ADDR_RESOLVED event
101
102       rdma_ack_cm_event
103              ack event
104
105       rdma_create_qp
106              allocate a QP for the communication
107
108       rdma_resolve_route
109              determine the route to the remote address
110
111       rdma_get_cm_event
112              wait for RDMA_CM_EVENT_ROUTE_RESOLVED event
113
114       rdma_ack_cm_event
115              ack event
116
117       rdma_connect
118              connect to the remote server
119
120       rdma_get_cm_event
121              wait for RDMA_CM_EVENT_ESTABLISHED event
122
123       rdma_ack_cm_event
124              ack event
125
126       Perform data transfers over connection
127
128       rdma_disconnect
129              tear-down connection
130
131       rdma_get_cm_event
132              wait for RDMA_CM_EVENT_DISCONNECTED event
133
134       rdma_ack_cm_event
135              ack event
136
137       rdma_destroy_qp
138              destroy the QP
139
140       rdma_destroy_id
141              release the rdma_cm_id
142
143       rdma_destroy_event_channel
144              release the event channel
145
146       rdma_set_local_ece
147              set desired ECE options
148
149       An  almost  identical process is used to setup unreliable datagram (UD)
150       communication between nodes.  No actual connection  is  formed  between
151       QPs however, so disconnection is not needed.
152
153       Although  this  example  shows  the  client  initiating the disconnect,
154       either side of a connection may initiate the disconnect.
155

SERVER OPERATION

157       This section provides a general overview of the basic operation for the
158       passive,  or  server, side of communication.  A general connection flow
159       would be:
160
161       rdma_create_event_channel
162              create channel to receive events
163
164       rdma_create_id
165              allocate an rdma_cm_id, this is conceptually similar to a socket
166
167       rdma_bind_addr
168              set the local port number to listen on
169
170       rdma_listen
171              begin listening for connection requests
172
173       rdma_get_cm_event
174              wait  for  RDMA_CM_EVENT_CONNECT_REQUEST  event   with   a   new
175              rdma_cm_id
176
177       rdma_create_qp
178              allocate a QP for the communication on the new rdma_cm_id
179
180       rdma_accept
181              accept the connection request
182
183       rdma_ack_cm_event
184              ack event
185
186       rdma_get_cm_event
187              wait for RDMA_CM_EVENT_ESTABLISHED event
188
189       rdma_ack_cm_event
190              ack event
191
192       Perform data transfers over connection
193
194       rdma_get_cm_event
195              wait for RDMA_CM_EVENT_DISCONNECTED event
196
197       rdma_ack_cm_event
198              ack event
199
200       rdma_disconnect
201              tear-down connection
202
203       rdma_destroy_qp
204              destroy the QP
205
206       rdma_destroy_id
207              release the connected rdma_cm_id
208
209       rdma_destroy_id
210              release the listening rdma_cm_id
211
212       rdma_destroy_event_channel
213              release the event channel
214
215       rdma_get_remote_ece
216              get ECe options sent by the client
217
218       rdma_set_local_ece
219              set desired ECE options
220

RETURN CODES

222       =  0   success
223
224       = -1   error - see errno for more details
225
226       Most  librdmacm functions return 0 to indicate success, and a -1 return
227       value to indicate failure.  If a function  operates  asynchronously,  a
228       return  value  of  0 means that the operation was successfully started.
229       The operation could still complete in error;  users  should  check  the
230       status  of  the  related  event.  If the return value is -1, then errno
231       will contain additional information regarding the reason for the  fail‐
232       ure.
233
234       Prior versions of the library would return -errno and not set errno for
235       some cases related to ENOMEM, ENODEV,  ENODATA,  EINVAL,  and  EADDRNO‐
236       TAVAIL codes. Applications that want to check these codes and have com‐
237       patibility with prior library versions must manually set errno  to  the
238       negative of the return code if it is < -1.
239