rdma_cm(7)

1RDMA_CM(7)               Librdmacm Programmer's Manual              RDMA_CM(7)
2
3
4

NAME

6       rdma_cm - RDMA communication manager.
7

SYNOPSIS

9       #include <rdma/rdma_cma.h>
10

DESCRIPTION

12       Used to establish communication over RDMA transports.
13

NOTES

15       The  RDMA  CM  is  a communication manager used to setup reliable, con‐
16       nected and unreliable datagram data transfers.   It  provides  an  RDMA
17       transport neutral interface for establishing connections.  The API con‐
18       cepts are based on sockets, but  adapted  for  queue  pair  (QP)  based
19       semantics:  communication must be over a specific RDMA device, and data
20       transfers are message based.
21
22       The RDMA CM can control both the QP and communication management  (con‐
23       nection setup / teardown) portions of an RDMA API, or only the communi‐
24       cation management piece.  It works in conjunction with  the  verbs  API
25       defined by the libibverbs library.  The libibverbs library provides the
26       underlying interfaces needed to send and receive data.
27
28       The RDMA CM can operate asynchronously or synchronously.  The  mode  of
29       operation  is  controlled  by  the  user through the use of the rdma_cm
30       event channel parameter in specific calls.  If an event channel is pro‐
31       vided,  an  rdma_cm  identifier  will report its event data (results of
32       connecting, for example), on that channel.  If a channel  is  not  pro‐
33       vided,  then all rdma_cm operations for the selected rdma_cm identifier
34       will block until they complete.
35

RDMA VERBS

37       The rdma_cm supports the full range  of  verbs  available  through  the
38       libibverbs  library  and interfaces.  However, it also provides wrapper
39       functions for some of the more commonly used verbs funcationality.  The
40       full set of abstracted verb calls are:
41
42       rdma_reg_msgs  - register an array of buffers for sending and receiving
43
44       rdma_reg_read  - registers a buffer for RDMA read operations
45
46       rdma_reg_write - registers a buffer for RDMA write operations
47
48       rdma_dereg_mr  - deregisters a memory region
49
50       rdma_post_recv  - post a buffer to receive a message
51
52       rdma_post_send  - post a buffer to send a message
53
54       rdma_post_read  - post an RDMA to read data into a buffer
55
56       rdma_post_write - post an RDMA to send data from a buffer
57
58       rdma_post_recvv  - post a vector of buffers to receive a message
59
60       rdma_post_sendv  - post a vector of buffers to send a message
61
62       rdma_post_readv  - post a vector of buffers to receive an RDMA read
63
64       rdma_post_writev - post a vector of buffers to send an RDMA write
65
66       rdma_post_ud_send - post a buffer to send a message on a UD QP
67
68       rdma_get_send_comp - get completion status for a send or RDMA operation
69
70       rdma_get_recv_comp - get information about a completed receive
71

CLIENT OPERATION

73       This section provides a general overview of the basic operation for the
74       active, or client, side of communication.  This flow  assume  asynchro‐
75       nous  operation  with  low  level  call details shown.  For synchronous
76       operation,  calls  to   rdma_create_event_channel,   rdma_get_cm_event,
77       rdma_ack_cm_event,  and rdma_destroy_event_channel would be eliminated.
78       Abstracted calls, such as rdma_create_ep encapsulate several  of  these
79       calls under a single API.  Users may also refer to the example applica‐
80       tions for code samples.  A general connection flow would be:
81
82       rdma_getaddrinfo
83              retrieve address information of the destination
84
85       rdma_create_event_channel
86              create channel to receive events
87
88       rdma_create_id
89              allocate an rdma_cm_id, this is conceptually similar to a socket
90
91       rdma_resolve_addr
92              obtain a local RDMA device to reach the remote address
93
94       rdma_get_cm_event
95              wait for RDMA_CM_EVENT_ADDR_RESOLVED event
96
97       rdma_ack_cm_event
98              ack event
99
100       rdma_create_qp
101              allocate a QP for the communication
102
103       rdma_resolve_route
104              determine the route to the remote address
105
106       rdma_get_cm_event
107              wait for RDMA_CM_EVENT_ROUTE_RESOLVED event
108
109       rdma_ack_cm_event
110              ack event
111
112       rdma_connect
113              connect to the remote server
114
115       rdma_get_cm_event
116              wait for RDMA_CM_EVENT_ESTABLISHED event
117
118       rdma_ack_cm_event
119              ack event
120
121       Perform data transfers over connection
122
123       rdma_disconnect
124              tear-down connection
125
126       rdma_get_cm_event
127              wait for RDMA_CM_EVENT_DISCONNECTED event
128
129       rdma_ack_cm_event
130              ack event
131
132       rdma_destroy_qp
133              destroy the QP
134
135       rdma_destroy_id
136              release the rdma_cm_id
137
138       rdma_destroy_event_channel
139              release the event channel
140
141       An almost identical process is used to setup unreliable  datagram  (UD)
142       communication  between  nodes.   No actual connection is formed between
143       QPs however, so disconnection is not needed.
144
145       Although this example  shows  the  client  initiating  the  disconnect,
146       either side of a connection may initiate the disconnect.
147

SERVER OPERATION

149       This section provides a general overview of the basic operation for the
150       passive, or server, side of communication.  A general  connection  flow
151       would be:
152
153       rdma_create_event_channel
154              create channel to receive events
155
156       rdma_create_id
157              allocate an rdma_cm_id, this is conceptually similar to a socket
158
159       rdma_bind_addr
160              set the local port number to listen on
161
162       rdma_listen
163              begin listening for connection requests
164
165       rdma_get_cm_event
166              wait   for   RDMA_CM_EVENT_CONNECT_REQUEST   event  with  a  new
167              rdma_cm_id
168
169       rdma_create_qp
170              allocate a QP for the communication on the new rdma_cm_id
171
172       rdma_accept
173              accept the connection request
174
175       rdma_ack_cm_event
176              ack event
177
178       rdma_get_cm_event
179              wait for RDMA_CM_EVENT_ESTABLISHED event
180
181       rdma_ack_cm_event
182              ack event
183
184       Perform data transfers over connection
185
186       rdma_get_cm_event
187              wait for RDMA_CM_EVENT_DISCONNECTED event
188
189       rdma_ack_cm_event
190              ack event
191
192       rdma_disconnect
193              tear-down connection
194
195       rdma_destroy_qp
196              destroy the QP
197
198       rdma_destroy_id
199              release the connected rdma_cm_id
200
201       rdma_destroy_id
202              release the listening rdma_cm_id
203
204       rdma_destroy_event_channel
205              release the event channel
206

RETURN CODES

208       =  0   success
209
210       = -1   error - see errno for more details
211
212       Most librdmacm functions return 0 to indicate success, and a -1  return
213       value  to  indicate  failure.  If a function operates asynchronously, a
214       return value of 0 means that the operation  was  successfully  started.
215       The  operation  could  still  complete in error; users should check the
216       status of the related event.  If the return value  is  -1,  then  errno
217       will  contain additional information regarding the reason for the fail‐
218       ure.
219
220       Prior versions of the library would return -errno and not set errno for
221       some  cases  related  to  ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNO‐
222       TAVAIL codes. Applications that want to check these codes and have com‐
223       patibility  with  prior library versions must manually set errno to the
224       negative of the return code if it is < -1.
225