1fi_eq(3) Libfabric v1.12.1 fi_eq(3)
2
6 fi_eq - Event queue operations
7 fi_eq_open / fi_close
9 Open/close an event queue
10 fi_control
12 Control operation of EQ
13 fi_eq_read / fi_eq_readerr
15 Read an event from an event queue
16 fi_eq_write
18 Writes an event to an event queue
19 fi_eq_sread
21 A synchronous (blocking) read of an event queue
22 fi_eq_strerror
24 Converts provider specific error information into a printable
25 string
26
28 #include <rdma/fi_domain.h>
29 int fi_eq_open(struct fid_fabric *fabric, struct fi_eq_attr *attr,
31 struct fid_eq **eq, void *context);
32 int fi_close(struct fid *eq);
34 int fi_control(struct fid *eq, int command, void *arg);
36 ssize_t fi_eq_read(struct fid_eq *eq, uint32_t *event,
38 void *buf, size_t len, uint64_t flags);
39 ssize_t fi_eq_readerr(struct fid_eq *eq, struct fi_eq_err_entry *buf,
41 uint64_t flags);
42 ssize_t fi_eq_write(struct fid_eq *eq, uint32_t event,
44 const void *buf, size_t len, uint64_t flags);
45 ssize_t fi_eq_sread(struct fid_eq *eq, uint32_t *event,
47 void *buf, size_t len, int timeout, uint64_t flags);
48 const char * fi_eq_strerror(struct fid_eq *eq, int prov_errno,
50 const void *err_data, char *buf, size_t len);
51
53 fabric Opened fabric descriptor
54 eq Event queue
56 attr Event queue attributes
58 context
60 User specified context associated with the event queue.
61 event Reported event
63 buf For read calls, the data buffer to write events into. For write
65 calls, an event to insert into the event queue. For fi_eq_str‐
66 error, an optional buffer that receives printable error informa‐
67 tion.
68 len Length of data buffer
70 flags Additional flags to apply to the operation
72 command
74 Command of control operation to perform on EQ.
75 arg Optional control argument
77 prov_errno
79 Provider specific error value
80 err_data
82 Provider specific error data related to a completion
83 timeout
85 Timeout specified in milliseconds
86
88 Event queues are used to report events associated with control opera‐
89 tions. They are associated with memory registration, address vectors,
90 connection management, and fabric and domain level events. Reported
91 events are either associated with a requested operation or affiliated
92 with a call that registers for specific types of events, such as lis‐
93 tening for connection requests.
94 fi_eq_open
96 fi_eq_open allocates a new event queue.
97 The properties and behavior of an event queue are defined by
99 struct fi_eq_attr.
100 struct fi_eq_attr {
102 size_t size; /* # entries for EQ */
103 uint64_t flags; /* operation flags */
104 enum fi_wait_obj wait_obj; /* requested wait object */
105 int signaling_vector; /* interrupt affinity */
106 struct fid_wait *wait_set; /* optional wait set */
107 };
108 size Specifies the minimum size of an event queue.
110 flags Flags that control the configuration of the EQ.
112 - FI_WRITE
114 Indicates that the application requires support for inserting
115 user events into the EQ. If this flag is set, then the
116 fi_eq_write operation must be supported by the provider. If the
117 FI_WRITE flag is not set, then the application may not invoke
118 fi_eq_write.
119 - FI_AFFINITY
121 Indicates that the signaling_vector field (see below) is valid.
122 wait_obj
124 EQ's may be associated with a specific wait object. Wait ob‐
125 jects allow applications to block until the wait object is sig‐
126 naled, indicating that an event is available to be read. Users
127 may use fi_control to retrieve the underlying wait object asso‐
128 ciated with an EQ, in order to use it in other system calls.
129 The following values may be used to specify the type of wait ob‐
130 ject associated with an EQ:
131 - FI_WAIT_NONE
133 Used to indicate that the user will not block (wait) for events
134 on the EQ. When FI_WAIT_NONE is specified, the application may
135 not call fi_eq_sread. This is the default is no wait object is
136 specified.
137 - FI_WAIT_UNSPEC
139 Specifies that the user will only wait on the EQ using fabric
140 interface calls, such as fi_eq_sread. In this case, the under‐
141 lying provider may select the most appropriate or highest per‐
142 forming wait object available, including custom wait mechanisms.
143 Applications that select FI_WAIT_UNSPEC are not guaranteed to
144 retrieve the underlying wait object.
145 - FI_WAIT_SET
147 Indicates that the event queue should use a wait set object to
148 wait for events. If specified, the wait_set field must refer‐
149 ence an existing wait set object.
150 - FI_WAIT_FD
152 Indicates that the EQ should use a file descriptor as its wait
153 mechanism. A file descriptor wait object must be usable in se‐
154 lect, poll, and epoll routines. However, a provider may signal
155 an FD wait object by marking it as readable or with an error.
156 - FI_WAIT_MUTEX_COND
158 Specifies that the EQ should use a pthread mutex and cond vari‐
159 able as a wait object.
160 - FI_WAIT_YIELD
162 Indicates that the EQ will wait without a wait object but in‐
163 stead yield on every wait. Allows usage of fi_eq_sread through
164 a spin.
165 signaling_vector
167 If the FI_AFFINITY flag is set, this indicates the logical cpu
168 number (0..max cpu - 1) that interrupts associated with the EQ
169 should target. This field should be treated as a hint to the
170 provider and may be ignored if the provider does not support in‐
171 terrupt affinity.
172 wait_set
174 If wait_obj is FI_WAIT_SET, this field references a wait object
175 to which the event queue should attach. When an event is in‐
176 serted into the event queue, the corresponding wait set will be
177 signaled if all necessary conditions are met. The use of a
178 wait_set enables an optimized method of waiting for events
179 across multiple event queues. This field is ignored if wait_obj
180 is not FI_WAIT_SET.
181 fi_close
183 The fi_close call releases all resources associated with an event
184 queue. Any events which remain on the EQ when it is closed are lost.
185 The EQ must not be bound to any other objects prior to being closed,
187 otherwise the call will return -FI_EBUSY.
188 fi_control
190 The fi_control call is used to access provider or implementation spe‐
191 cific details of the event queue. Access to the EQ should be serial‐
192 ized across all calls when fi_control is invoked, as it may redirect
193 the implementation of EQ operations. The following control commands
194 are usable with an EQ.
195 FI_GETWAIT (void **)
197 This command allows the user to retrieve the low-level wait ob‐
198 ject associated with the EQ. The format of the wait-object is
199 specified during EQ creation, through the EQ attributes. The
200 fi_control arg parameter should be an address where a pointer to
201 the returned wait object will be written. This should be an
202 'int *' for FI_WAIT_FD, or 'struct fi_mutex_cond' for
203 FI_WAIT_MUTEX_COND.
204 struct fi_mutex_cond {
206 pthread_mutex_t *mutex;
207 pthread_cond_t *cond;
208 };
209 fi_eq_read
211 The fi_eq_read operations performs a non-blocking read of event data
212 from the EQ. The format of the event data is based on the type of
213 event retrieved from the EQ, with all events starting with a struct
214 fi_eq_entry header. At most one event will be returned per EQ read op‐
215 eration. The number of bytes successfully read from the EQ is returned
216 from the read. The FI_PEEK flag may be used to indicate that event da‐
217 ta should be read from the EQ without being consumed. A subsequent
218 read without the FI_PEEK flag would then remove the event from the EQ.
219 The following types of events may be reported to an EQ, along with in‐
221 formation regarding the format associated with each event.
222 Asynchronous Control Operations
224 Asynchronous control operations are basic requests that simply
225 need to generate an event to indicate that they have completed.
226 These include the following types of events: memory registra‐
227 tion, address vector resolution, and multicast joins.
228 Control requests report their completion by inserting a
230 struct fi_eq_entry into the EQ. The format of this structure is:
231 struct fi_eq_entry {
233 fid_t fid; /* fid associated with request */
234 void *context; /* operation context */
235 uint64_t data; /* completion-specific data */
236 };
237 For the completion of basic asynchronous control operations, the re‐
239 turned event will indicate the operation that has completed, and the
240 fid will reference the fabric descriptor associated with the event.
241 For memory registration, this will be an FI_MR_COMPLETE event and the
242 fid_mr. Address resolution will reference an FI_AV_COMPLETE event and
243 fid_av. Multicast joins will report an FI_JOIN_COMPLETE and fid_mc.
244 The context field will be set to the context specified as part of the
245 operation, if available, otherwise the context will be associated with
246 the fabric descriptor. The data field will be set as described in the
247 man page for the corresponding object type (e.g., see fi_av(3) for a
248 description of how asynchronous address vector insertions are complet‐
249 ed).
250 Connection Notification
252 Connection notifications are connection management notifications
253 used to setup or tear down connections between endpoints. There
254 are three connection notification events: FI_CONNREQ, FI_CON‐
255 NECTED, and FI_SHUTDOWN. Connection notifications are reported
256 using struct fi_eq_cm_entry:
257 struct fi_eq_cm_entry {
259 fid_t fid; /* fid associated with request */
260 struct fi_info *info; /* endpoint information */
261 uint8_t data[]; /* app connection data */
262 };
263 A connection request (FI_CONNREQ) event indicates that a remote end‐
265 point wishes to establish a new connection to a listening, or passive,
266 endpoint. The fid is the passive endpoint. Information regarding the
267 requested, active endpoint's capabilities and attributes are available
268 from the info field. The application is responsible for freeing this
269 structure by calling fi_freeinfo when it is no longer needed. The
270 fi_info connreq field will reference the connection request associated
271 with this event. To accept a connection, an endpoint must first be
272 created by passing an fi_info structure referencing this connreq field
273 to fi_endpoint(). This endpoint is then passed to fi_accept() to com‐
274 plete the acceptance of the connection attempt. Creating the endpoint
275 is most easily accomplished by passing the fi_info returned as part of
276 the CM event into fi_endpoint(). If the connection is to be rejected,
277 the connreq is passed to fi_reject().
278 Any application data exchanged as part of the connection request is
280 placed beyond the fi_eq_cm_entry structure. The amount of data avail‐
281 able is application dependent and limited to the buffer space provided
282 by the application when fi_eq_read is called. The amount of returned
283 data may be calculated using the return value to fi_eq_read. Note that
284 the amount of returned data is limited by the underlying connection
285 protocol, and the length of any data returned may include protocol pad‐
286 ding. As a result, the returned length may be larger than that speci‐
287 fied by the connecting peer.
288 If a connection request has been accepted, an FI_CONNECTED event will
290 be generated on both sides of the connection. The active side -- one
291 that called fi_connect() -- may receive user data as part of the
292 FI_CONNECTED event. The user data is passed to the connection manager
293 on the passive side through the fi_accept call. User data is not pro‐
294 vided with an FI_CONNECTED event on the listening side of the connec‐
295 tion.
296 Notification that a remote peer has disconnected from an active end‐
298 point is done through the FI_SHUTDOWN event. Shutdown notification us‐
299 es struct fi_eq_cm_entry as declared above. The fid field for a shut‐
300 down notification refers to the active endpoint's fid_ep.
301 Asynchronous Error Notification
303 Asynchronous errors are used to report problems with fabric re‐
304 sources. Reported errors may be fatal or transient, based on
305 the error, and result in the resource becoming disabled. Dis‐
306 abled resources will fail operations submitted against them un‐
307 til they are explicitly re-enabled by the application.
308 Asynchronous errors may be reported for completion queues and endpoints
310 of all types. CQ errors can result when resource management has been
311 disabled, and the provider has detected a queue overrun. Endpoint er‐
312 rors may be result of numerous actions, but are often associated with a
313 failed operation. Operations may fail because of buffer overruns, in‐
314 valid permissions, incorrect memory access keys, network routing fail‐
315 ures, network reach-ability issues, etc.
316 Asynchronous errors are reported using struct fi_eq_err_entry, as de‐
318 fined below. The fabric descriptor (fid) associated with the error is
319 provided as part of the error data. An error code is also available to
320 determine the cause of the error.
321 fi_eq_sread
323 The fi_eq_sread call is the blocking (or synchronous) equivalent to
324 fi_eq_read. It behaves is similar to the non-blocking call, with the
325 exception that the calls will not return until either an event has been
326 read from the EQ or an error or timeout occurs. Specifying a negative
327 timeout means an infinite timeout.
328 Threads blocking in this function will return to the caller if they are
330 signaled by some external source. This is true even if the timeout has
331 not occurred or was specified as infinite.
332 It is invalid for applications to call this function if the EQ has been
334 configured with a wait object of FI_WAIT_NONE or FI_WAIT_SET.
335 fi_eq_readerr
337 The read error function, fi_eq_readerr, retrieves information regarding
338 any asynchronous operation which has completed with an unexpected er‐
339 ror. fi_eq_readerr is a non-blocking call, returning immediately
340 whether an error completion was found or not.
341 EQs are optimized to report operations which have completed successful‐
343 ly. Operations which fail are reported 'out of band'. Such operations
344 are retrieved using the fi_eq_readerr function. When an operation that
345 completes with an unexpected error is inserted into an EQ, it is placed
346 into a temporary error queue. Attempting to read from an EQ while an
347 item is in the error queue results in an FI_EAVAIL failure. Applica‐
348 tions may use this return code to determine when to call fi_eq_readerr.
349 Error information is reported to the user through struct fi_eq_err_en‐
351 try. The format of this structure is defined below.
352 struct fi_eq_err_entry {
354 fid_t fid; /* fid associated with error */
355 void *context; /* operation context */
356 uint64_t data; /* completion-specific data */
357 int err; /* positive error code */
358 int prov_errno; /* provider error code */
359 void *err_data; /* additional error data */
360 size_t err_data_size; /* size of err_data */
361 };
362 The fid will reference the fabric descriptor associated with the event.
364 For memory registration, this will be the fid_mr, address resolution
365 will reference a fid_av, and CM events will refer to a fid_ep. The
366 context field will be set to the context specified as part of the oper‐
367 ation.
368 The data field will be set as described in the man page for the corre‐
370 sponding object type (e.g., see fi_av(3) for a description of how asyn‐
371 chronous address vector insertions are completed).
372 The general reason for the error is provided through the err field.
374 Provider or operational specific error information may also be avail‐
375 able through the prov_errno and err_data fields. Users may call
376 fi_eq_strerror to convert provider specific error information into a
377 printable string for debugging purposes.
378 On input, err_data_size indicates the size of the err_data buffer in
380 bytes. On output, err_data_size will be set to the number of bytes
381 copied to the err_data buffer. The err_data information is typically
382 used with fi_eq_strerror to provide details about the type of error
383 that occurred.
384 For compatibility purposes, if err_data_size is 0 on input, or the fab‐
386 ric was opened with release < 1.5, err_data will be set to a data buf‐
387 fer owned by the provider. The contents of the buffer will remain
388 valid until a subsequent read call against the EQ. Applications must
389 serialize access to the EQ when processing errors to ensure that the
390 buffer referenced by err_data does not change.
391
393 The EQ entry data structures share many of the same fields. The mean‐
394 ings are the same or similar for all EQ structure formats, with specif‐
395 ic details described below.
396 fid This corresponds to the fabric descriptor associated with the
398 event. The type of fid depends on the event being reported.
399 For FI_CONNREQ this will be the fid of the passive endpoint.
400 FI_CONNECTED and FI_SHUTDOWN will reference the active endpoint.
401 FI_MR_COMPLETE and FI_AV_COMPLETE will refer to the MR or AV
402 fabric descriptor, respectively. FI_JOIN_COMPLETE will point to
403 the multicast descriptor returned as part of the join operation.
404 Applications can use fid->context value to retrieve the context
405 associated with the fabric descriptor.
406 context
408 The context value is set to the context parameter specified with
409 the operation that generated the event. If no context parameter
410 is associated with the operation, this field will be NULL.
411 data Data is an operation specific value or set of bytes. For con‐
413 nection events, data is application data exchanged as part of
414 the connection protocol.
415 err This err code is a positive fabric errno associated with an
417 event. The err value indicates the general reason for an error,
418 if one occurred. See fi_errno.3 for a list of possible error
419 codes.
420 prov_errno
422 On an error, prov_errno may contain a provider specific error
423 code. The use of this field and its meaning is provider specif‐
424 ic. It is intended to be used as a debugging aid. See
425 fi_eq_strerror for additional details on converting this error
426 value into a human readable string.
427 err_data
429 On an error, err_data may reference a provider specific amount
430 of data associated with an error. The use of this field and its
431 meaning is provider specific. It is intended to be used as a
432 debugging aid. See fi_eq_strerror for additional details on
433 converting this error data into a human readable string.
434 err_data_size
436 On input, err_data_size indicates the size of the err_data buf‐
437 fer in bytes. On output, err_data_size will be set to the num‐
438 ber of bytes copied to the err_data buffer. The err_data infor‐
439 mation is typically used with fi_eq_strerror to provide details
440 about the type of error that occurred.
441 For compatibility purposes, if err_data_size is 0 on input, or the fab‐
443 ric was opened with release < 1.5, err_data will be set to a data buf‐
444 fer owned by the provider. The contents of the buffer will remain
445 valid until a subsequent read call against the EQ. Applications must
446 serialize access to the EQ when processing errors to ensure that the
447 buffer referenced by err_data does no change.
448
450 If an event queue has been overrun, it will be placed into an 'overrun'
451 state. Write operations against an overrun EQ will fail with
452 -FI_EOVERRUN. Read operations will continue to return any valid,
453 non-corrupted events, if available. After all valid events have been
454 retrieved, any attempt to read the EQ will result in it returning an
455 FI_EOVERRUN error event. Overrun event queues are considered fatal and
456 may not be used to report additional events once the overrun occurs.
457
459 fi_eq_open
460 Returns 0 on success. On error, a negative value corresponding
461 to fabric errno is returned.
462 fi_eq_read / fi_eq_readerr
464 On success, returns the number of bytes read from the event
465 queue. On error, a negative value corresponding to fabric errno
466 is returned. If no data is available to be read from the event
467 queue, -FI_EAGAIN is returned.
468 fi_eq_sread
470 On success, returns the number of bytes read from the event
471 queue. On error, a negative value corresponding to fabric errno
472 is returned. If the timeout expires or the calling thread is
473 signaled and no data is available to be read from the event
474 queue, -FI_EAGAIN is returned.
475 fi_eq_write
477 On success, returns the number of bytes written to the event
478 queue. On error, a negative value corresponding to fabric errno
479 is returned.
480 fi_eq_strerror
482 Returns a character string interpretation of the provider spe‐
483 cific error returned with a completion.
484 Fabric errno values are defined in rdma/fi_errno.h.
486
488 fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_cntr(3), fi_poll(3)
489
491 OpenFabrics.
492Libfabric Programmer's Manual 2019-12-13 fi_eq(3)