1fi_poll(3) Libfabric v1.6.1 fi_poll(3)
2
6 fi_poll - Polling and wait set operations
7 fi_poll_open / fi_close : Open/close a polling set
9 fi_poll_add / fi_poll_del : Add/remove a completion queue or counter
11 to/from a poll set.
12 fi_poll : Poll for progress and events across multiple completion
14 queues and counters.
15 fi_wait_open / fi_close : Open/close a wait set
17 fi_wait : Waits for one or more wait objects in a set to be signaled.
19 fi_trywait : Indicate when it is safe to block on wait objects using
21 native OS calls.
22 fi_control : Control wait set operation or attributes.
24
26 #include <rdma/fi_domain.h>
27 int fi_poll_open(struct fid_domain *domain, struct fi_poll_attr *attr,
29 struct fid_poll **pollset);
30 int fi_close(struct fid *pollset);
32 int fi_poll_add(struct fid_poll *pollset, struct fid *event_fid,
34 uint64_t flags);
35 int fi_poll_del(struct fid_poll *pollset, struct fid *event_fid,
37 uint64_t flags);
38 int fi_poll(struct fid_poll *pollset, void **context, int count);
40 int fi_wait_open(struct fid_fabric *fabric, struct fi_wait_attr *attr,
42 struct fid_wait **waitset);
43 int fi_close(struct fid *waitset);
45 int fi_wait(struct fid_wait *waitset, int timeout);
47 int fi_trywait(struct fid_fabric *fabric, struct fid **fids, size_t count);
49 int fi_control(struct fid *waitset, int command, void *arg);
51
53 fabric : Fabric provider
54 domain : Resource domain
56 pollset : Event poll set
58 waitset : Wait object set
60 attr : Poll or wait set attributes
62 context : On success, an array of user context values associated with
64 completion queues or counters.
65 fids : An array of fabric descriptors, each one associated with a
67 native wait object.
68 count : Number of entries in context or fids array.
70 timeout : Time to wait for a signal, in milliseconds.
72 command : Command of control operation to perform on the wait set.
74 arg : Optional control argument.
76
78 fi_poll_open
79 fi_poll_open creates a new polling set. A poll set enables an opti‐
80 mized method for progressing asynchronous operations across multiple
81 completion queues and counters and checking for their completions.
82 A poll set is defined with the following attributes.
84 struct fi_poll_attr {
86 uint64_t flags; /* operation flags */
87 };
88 flags : Flags that set the default operation of the poll set. The use
90 of this field is reserved and must be set to 0 by the caller.
91 fi_close
93 The fi_close call releases all resources associated with a poll set.
94 The poll set must not be associated with any other resources prior to
95 being closed, otherwise the call will return -FI_EBUSY.
96 fi_poll_add
98 Associates a completion queue or counter with a poll set.
99 fi_poll_del
101 Removes a completion queue or counter from a poll set.
102 fi_poll
104 Progresses all completion queues and counters associated with a poll
105 set and checks for events. If events might have occurred, contexts
106 associated with the completion queues and/or counters are returned.
107 Completion queues will return their context if they are not empty. The
108 context associated with a counter will be returned if the counter's
109 success value or error value have changed since the last time fi_poll,
110 fi_cntr_set, or fi_cntr_add were called. The number of contexts is
111 limited to the size of the context array, indicated by the count param‐
112 eter.
113 Note that fi_poll only indicates that events might be available. In
115 some cases, providers may consume such events internally, to drive
116 progress, for example. This can result in fi_poll returning false pos‐
117 itives. Applications should drive their progress based on the results
118 of reading events from a completion queue or reading counter values.
119 The fi_poll function will always return all completion queues and coun‐
120 ters that do have new events.
121 fi_wait_open
123 fi_wait_open allocates a new wait set. A wait set enables an optimized
124 method of waiting for events across multiple completion queues and
125 counters. Where possible, a wait set uses a single underlying wait
126 object that is signaled when a specified condition occurs on an associ‐
127 ated completion queue or counter.
128 The properties and behavior of a wait set are defined by struct
130 fi_wait_attr.
131 struct fi_wait_attr {
133 enum fi_wait_obj wait_obj; /* requested wait object */
134 uint64_t flags; /* operation flags */
135 };
136 wait_obj : Wait sets are associated with specific wait object(s). Wait
138 objects allow applications to block until the wait object is signaled,
139 indicating that an event is available to be read. The following values
140 may be used to specify the type of wait object associated with a wait
141 set: FI_WAIT_UNSPEC, FI_WAIT_FD, and FI_WAIT_MUTEX_COND.
142 · FI_WAIT_UNSPEC : Specifies that the user will only wait on the wait
144 set using fabric interface calls, such as fi_wait. In this case, the
145 underlying provider may select the most appropriate or highest per‐
146 forming wait object available, including custom wait mechanisms.
147 Applications that select FI_WAIT_UNSPEC are not guaranteed to
148 retrieve the underlying wait object.
149 · FI_WAIT_FD : Indicates that the wait set should use file descrip‐
151 tor(s) as its wait mechanism. It may not always be possible for a
152 wait set to be implemented using a single underlying file descriptor,
153 but all wait objects will be file descriptors. File descriptor wait
154 objects must be usable in the POSIX select(2), poll(2), and epoll(7)
155 routines (if available). However, a provider may signal an FD wait
156 object by marking it as readable or with an error.
157 · FI_WAIT_MUTEX_COND : Specifies that the wait set should use a pthread
159 mutex and cond variable as a wait object.
160 · FI_WAIT_CRITSEC_COND : Windows specific. Specifies that the EQ
162 should use a critical section and condition variable as a wait
163 object.
164 flags : Flags that set the default operation of the wait set. The use
166 of this field is reserved and must be set to 0 by the caller.
167 fi_close
169 The fi_close call releases all resources associated with a wait set.
170 The wait set must not be bound to any other opened resources prior to
171 being closed, otherwise the call will return -FI_EBUSY.
172 fi_wait
174 Waits on a wait set until one or more of its underlying wait objects is
175 signaled.
176 fi_trywait
178 The fi_trywait call was introduced in libfabric version 1.3. The
179 behavior of using native wait objects without the use of fi_trywait is
180 provider specific and should be considered non-deterministic.
181 The fi_trywait() call is used in conjunction with native operating sys‐
183 tem calls to block on wait objects, such as file descriptors. The
184 application must call fi_trywait and obtain a return value of FI_SUC‐
185 CESS prior to blocking on a native wait object. Failure to do so may
186 result in the wait object not being signaled, and the application not
187 observing the desired events. The following pseudo-code demonstrates
188 the use of fi_trywait in conjunction with the OS select(2) call.
189 fi_control(&cq->fid, FI_GETWAIT, (void *) &fd);
191 FD_ZERO(&fds);
192 FD_SET(fd, &fds);
193 while (1) {
195 if (fi_trywait(&cq, 1) == FI_SUCCESS)
196 select(fd + 1, &fds, NULL, &fds, &timeout);
197 do {
199 ret = fi_cq_read(cq, &comp, 1);
200 } while (ret > 0);
201 }
202 fi_trywait() will return FI_SUCCESS if it is safe to block on the wait
204 object(s) corresponding to the fabric descriptor(s), or -FI_EAGAIN if
205 there are events queued on the fabric descriptor or if blocking could
206 hang the application.
207 The call takes an array of fabric descriptors. For each wait object
209 that will be passed to the native wait routine, the corresponding fab‐
210 ric descriptor should first be passed to fi_trywait. All fabric
211 descriptors passed into a single fi_trywait call must make use of the
212 same underlying wait object type.
213 The following types of fabric descriptors may be passed into fi_try‐
215 wait: event queues, completion queues, counters, and wait sets. Appli‐
216 cations that wish to use native wait calls should select specific wait
217 objects when allocating such resources. For example, by setting the
218 item's creation attribute wait_obj value to FI_WAIT_FD.
219 In the case the wait object to check belongs to a wait set, only the
221 wait set itself needs to be passed into fi_trywait. The fabric
222 resources associated with the wait set do not.
223 On receiving a return value of -FI_EAGAIN from fi_trywait, an applica‐
225 tion should read all queued completions and events, and call fi_trywait
226 again before attempting to block. Applications can make use of a fab‐
227 ric poll set to identify completion queues and counters that may
228 require processing.
229 fi_control
231 The fi_control call is used to access provider or implementation spe‐
232 cific details of the wait set. Access to the wait set should be seri‐
233 alized across all calls when fi_control is invoked, as it may redirect
234 the implementation of wait set operations. The following control com‐
235 mands are usable with a wait set.
236 FI_GETWAIT (void **) : This command allows the user to retrieve the
238 low-level wait object associated with the wait set. The format of the
239 wait set is specified during wait set creation, through the wait set
240 attributes. The fi_control arg parameter should be an address where a
241 pointer to the returned wait object will be written. This should be an
242 'int *' for FI_WAIT_FD, or 'struct fi_mutex_cond' for
243 FI_WAIT_MUTEX_COND. Support for FI_GETWAIT is provider specific and
244 may fail if not supported or if the wait set is implemented using more
245 than one wait object.
246
248 Returns FI_SUCCESS on success. On error, a negative value correspond‐
249 ing to fabric errno is returned.
250 Fabric errno values are defined in rdma/fi_errno.h.
252 fi_poll : On success, if events are available, returns the number of
254 entries written to the context array.
255
258 fi_getinfo(3), fi_domain(3), fi_cntr(3), fi_eq(3)
259
261 OpenFabrics.
262Libfabric Programmer's Manual 2016-12-07 fi_poll(3)