1fabtests(7) @VERSION@ fabtests(7)
2
6 Fabtests
7
9 Fabtests is a set of examples for fabric providers that demonstrates
10 various features of libfabric- high-performance fabric software li‐
11 brary.
12
14 Libfabric defines sets of interface that fabric providers can support.
15 The purpose of Fabtests examples is to demonstrate some of the major
16 features. The goal is to familiarize users with different functionali‐
17 ties libfabric offers and how to use them. Although most tests report
18 performance numbers, they are designed to test functionality and not
19 performance. The exception are the benchmarks and ubertest.
20 The tests are divided into the following categories. Except the unit
22 tests all of them are client-server tests. Not all providers will sup‐
23 port each test.
24 The test names try to indicate the type of functionality each test is
26 verifying. Although some tests work with any endpoint type, many are
27 restricted to verifying a single endpoint type. These tests typically
28 include the endpoint type as part of the test name, such as dgram, msg,
29 or rdm.
30
32 These tests are a mix of very basic functionality tests that show major
33 features of libfabric.
34 fi_av_xfer
36 Tests communication for unconnected endpoints, as addresses are
37 inserted and removed from the local address vector.
38 fi_cm_data
40 Verifies exchanging CM data as part of connecting endpoints.
41 fi_cq_data
43 Tranfers messages with CQ data.
44 fi_dgram
46 A basic datagram endpoint example.
47 fi_dgram_waitset
49 Transfers datagrams using waitsets for completion notifcation.
50 fi_inj_complete
52 Sends messages using the FI_INJECT_COMPLETE operation flag.
53 fi_mcast
55 A simple multicast test.
56 fi_msg A basic message endpoint example.
58 fi_msg_epoll
60 Transfers messages with completion queues configured to use file
61 descriptors as wait objetcts. The file descriptors are re‐
62 trieved by the program and used directly with the Linux epoll
63 API.
64 fi_msg_sockets
66 Verifies that the address assigned to a passive endpoint can be
67 transitioned to an active endpoint. This is required applica‐
68 tions that need socket API semantics over RDMA implementations
69 (e.g. rsockets).
70 fi_multi_ep
72 Performs data transfers over multiple endpoints in parallel.
73 fi_multi_mr
75 Issues RMA write operations to multiple memory regions, using
76 completion counters of inbound writes as the notification mecha‐
77 nism.
78 fi_poll
80 Exchanges data over RDM endpoints using poll sets to drive com‐
81 pletion notifications.
82 fi_rdm A basic RDM endpoint example.
84 fi_rdm_atomic
86 Test and verifies atomic operations over an RDM endpoint.
87 fi_rdm_deferred_wq
89 Test triggered operations and deferred work queue support.
90 fi_rdm_multi_domain
92 Performs data transfers over multiple endpoints, with each end‐
93 point belonging to a different opened domain.
94 fi_rdm_multi_recv
96 Transfers multiple messages over an RDM endpoint that are re‐
97 ceived into a single buffer, posted using the FI_MULTI_RECV
98 flag.
99 fi_rdm_rma_simple
101 A simple RMA write example over an RDM endpoint.
102 fi_rdm_rma_trigger
104 A basic example of queuing an RMA write operation that is initi‐
105 ated upon the firing of a triggering completion. Works with RDM
106 endpoints.
107 fi_rdm_shared_av
109 Spawns child processes to verify basic functionality of using a
110 shared address vector with RDM endpoints.
111 fi_rdm_tagged_peek
113 Basic test of using the FI_PEEK operation flag with tagged mes‐
114 sages. Works with RDM endpoints.
115 fi_recv_cancel
117 Tests canceling posted receives for tagged messages.
118 fi_resmgmt_test
120 Tests the resource management enabled feature. This verifies
121 that the provider prevents applications from overruning local
122 and remote command queues and completion queues. This corre‐
123 sponds to setting the domain attribute resource_mgmt to
124 FI_RM_ENABLED.
125 fi_scalable_ep
127 Performs data transfers over scalable endpoints, endpoints asso‐
128 ciated with multiple transmit and receive contexts.
129 fi_shared_ctx
131 Performs data transfers between multiple endpoints, where the
132 endpoints share transmit and/or receive contexts.
133 fi_unexpected_msg
135 Tests the send and receive handling of unexpected tagged mes‐
136 sages.
137 fi_unmap_mem
139 Tests data transfers where the transmit buffer is mmapped and
140 unmapped between each transfer, but the virtual address of the
141 transmit buffer tries to remain the same. This test is used to
142 validate the correct behavior of memory registration caches.
143
145 The client and the server exchange messages in either a ping-pong man‐
146 ner, for pingpong named tests, or transfer messages one-way, for bw
147 named tests. These tests can transfer various messages sizes, with
148 controls over which features are used by the test, and report perfor‐
149 mance numbers. The tests are structured based on the benchmarks pro‐
150 vided by OSU MPI. They are not guaranteed to provide the best latency
151 or bandwidth performance numbers a given provider or system may
152 achieve.
153 fi_dgram_pingpong
155 Latency test for datagram endpoints
156 fi_msg_bw
158 Message transfer bandwidth test for connected (MSG) endpoints.
159 fi_msg_pingpong
161 Message transfer latency test for connected (MSG) endpoints.
162 fi_rdm_cntr_pingpong
164 Message transfer latency test for reliable-datagram (RDM) end‐
165 points that uses counters as the completion mechanism.
166 fi_rdm_pingpong
168 Message transfer latency test for reliable-datagram (RDM) end‐
169 points.
170 fi_rdm_tagged_bw
172 Tagged message bandwidth test for reliable-datagram (RDM) end‐
173 points.
174 fi_rdm_tagged_pingpong
176 Tagged message latency test for reliable-datagram (RDM) end‐
177 points.
178 fi_rma_bw
180 An RMA read and write bandwidth test for reliable (MSG and RDM)
181 endpoints.
182
184 These are simple one-sided unit tests that validate basic behavior of
185 the API. Because these are single system tests that do not perform da‐
186 ta transfers their testing scope is limited.
187 fi_av_test
189 Verify address vector interfaces.
190 fi_cntr_test
192 Tests counter creation and destruction.
193 fi_cq_test
195 Tests completion queue creation and destruction.
196 fi_dom_test
198 Tests domain creation and destruction.
199 fi_eq_test
201 Tests event queue creation, destruction, and capabilities.
202 fi_getinfo_test
204 Tests provider response to fi_getinfo calls with varying hints.
205 fi_mr_test
207 Tests memory registration.
208 fi_resource_freeing
210 Allocates and closes fabric resources to check for proper
211 cleanup.
212
214 This is a comprehensive latency, bandwidth, and functionality test that
215 can handle a variety of test configurations. The test is able to run a
216 large number of tests by iterating over a large number of test vari‐
217 ables. As a result, a full ubertest run can take a significant amount
218 of time. Because ubertest iterates over input variables, it relies on
219 a test configuration file for control, rather than extensive command
220 line options that are used by other fabtests. A configuration file
221 must be constructured for each provider. Example test configurations
222 are at /test_configs.
223 fi_ubertest
225 This test takes a configure file as input. The file contains a
226 list of variables and their values to iterate over. The test
227 will run a set of latency, bandwidth, and functionality tests
228 over a given provider. It will perform one execution for every
229 possible combination of all variables. For example, if there
230 are 8 test variables, with 6 having 2 possible values and 2 hav‐
231 ing 3 possible values, ubertest will execute 576 total itera‐
232 tions of each test.
233 Config file options
235 TODO: add all supported config options
236 · threading Specify a list of threading levels. This is a hints only
238 config: ubertest doesn't spawn multiple threads to verify functional‐
239 ity.
240
242 (1) Fabtests requires that libfabric be installed on the system, and at
243 least one provider be usable.
244 (2) Install fabtests on the system. By default all the test executa‐
246 bles are installed in /usr/bin directory unless specified other‐
247 wise.
248 (3) All the client-server tests have the following usage model:
250 fi_ [OPTIONS] start server fi_ connect to server
252
254 Tests share command line options where appropriate. The following com‐
255 mand line options are available for one or more test. To see which op‐
256 tions apply for a given test, you can use the '-h' help option to see
257 the list available for that test.
258 -h Displays help output for the test.
260 -f Restrict test to the specified fabric name.
262 -d Restrict test to the specified domain name.
264 -p Restrict test to the specified provider name.
266 -e Use the specified endpoint type for the test. Valid options are
268 msg, dgram, and rdm. The default endpoint type is rdm.
269 *-a
270 · : The name of a shared address vector. This option only applies to
272 tests that support shared address vectors.
273 -B
275 Specifies the port number of the local endpoint, overriding the de‐
276 fault.
277 -P
279 Specifies the port number of the peer endpoint, overriding the de‐
280 fault.
281 *-s
282 · : Specifies the address of the local endpoint.
284 -b[=oob_port]
286 Enables out-of-band (via sockets) address exchange and test synchro‐
287 nization. A port for the out-of-band connection may be specified as
288 part of this option to override the default.
289 -E[=oob_port]
291 Enables out-of-band (via sockets) address exchange only. A port for
292 the out-of-band connection may be specified as part of this option to
293 override the default. Cannot be used together with the '-b' option.
294 -I
296 Number of data transfer iterations.
297 -w
299 Number of warm-up data transfer iterations.
300 -S
302 Data transfer size or 'all' for a full range of sizes. By default a
303 select number of sizes will be tested.
304 -l
306 If specified, the starting address of transmit and receive buffers
307 will be aligned along a page boundary.
308 -m
310 Use machine readable output. This is useful for post-processing the
311 test output with scripts.
312 -t
314 Specify the type of completion mechanism to use. Valid values are
315 queue and counter. The default is to use completion queues.
316 -c
318 Indicate the type of processing to use checking for completed opera‐
319 tions. Valid values are spin, sread, and fd. The default is to busy
320 wait (spin) until the desired operation has completed. The sread op‐
321 tion indicates that the application will invoke a blocking read call
322 in libfabric, such as fi_cq_sread. Fd indicates that the application
323 will retrieve the native operating system wait object (file descrip‐
324 tor) and use either poll() or select() to block until the fd has been
325 signaled, prior to checking for completions.
326 -o
328 For RMA based tests, specify the type of RMA operation to perform.
329 Valid values are read, write, and writedata. Write operations are
330 the default.
331 -M
333 For multicast tests, specifies the address of the multicast group to
334 join.
335
337 A simple example
338 run server: <test_name> -p <provider_name> -s <source_addr>
339 e.g. fi_msg_rma -p sockets -s 192.168.0.123
340 run client: <test_name> <server_addr> -p <provider_name>
341 e.g. fi_msg_rma 192.168.0.123 -p sockets
342 An example with various options
344 run server: fi_rdm_atomic -p psm -s 192.168.0.123 -I 1000 -S 1024
345 run client: fi_rdm_atomic 192.168.0.123 -p psm -I 1000 -S 1024
346 This will run "fi_rdm_atomic" for all atomic operations with
348 - PSM provider
350 - 1000 iterations
351 - 1024 bytes message size
352 - server node as 123.168.0.123
353 Run fi_ubertest
355 run server: fi_ubertest
356 run client: fi_ubertest -u /usr/share/fabtests/test_configs/sockets/quick.test 192.168.0.123
357 This will run "fi_ubertest" with
359 - sockets provider
361 - configurations defined in /usr/share/fabtests/test_configs/sockets/quick.test
362 - server node as 192.168.0.123
363 The config files are provided in /test_configs for sockets, verbs, udp,
365 and usnic providers and distributed with fabtests installation.
366 For more usage options: fi_ubertest -h
368 Run the whole fabtests suite
370 A runscript scripts/runfabtests.sh is provided that runs all the tests
371 in fabtests and reports the number of pass/fail/notrun.
372 Usage: runfabtests.sh [OPTIONS] [provider] [host] [client]
374 By default if none of the options are provided, it runs all the tests
376 using
377 - sockets provider
379 - 127.0.0.1 as both server and client address
380 - for small number of optiond and iterations
381 Various options can be used to choose provider, subset tests to run,
383 level of verbosity etc.
384 runfabtests.sh -vvv -t all psm 192.168.0.123 192.168.0.124
386 This will run all fabtests using
388 - psm provider
390 - for different options and larger iterations
391 - server node as 192.168.0.123 and client node as 192.168.0.124
392 - print test output for all the tests
393 For detailed usage options: runfabtests.sh -h
395
397 OpenFabrics.
398Libfabric Programmer's Manual 2019-05-14 fabtests(7)