1fabtests(7) @VERSION@ fabtests(7)
2
3
4
6 Fabtests
7
9 Fabtests is a set of examples for fabric providers that demonstrates
10 various features of libfabric- high-performance fabric software li‐
11 brary.
12
14 Libfabric defines sets of interface that fabric providers can support.
15 The purpose of Fabtests examples is to demonstrate some of the major
16 features. The goal is to familiarize users with different functionali‐
17 ties libfabric offers and how to use them. Although most tests report
18 performance numbers, they are designed to test functionality and not
19 performance. The exception are the benchmarks and ubertest.
20
21 The tests are divided into the following categories. Except the unit
22 tests all of them are client-server tests. Not all providers will sup‐
23 port each test.
24
25 The test names try to indicate the type of functionality each test is
26 verifying. Although some tests work with any endpoint type, many are
27 restricted to verifying a single endpoint type. These tests typically
28 include the endpoint type as part of the test name, such as dgram, msg,
29 or rdm.
30
32 These tests are a mix of very basic functionality tests that show major
33 features of libfabric.
34
35 fi_av_xfer
36 Tests communication for connectionless endpoints, as addresses
37 are inserted and removed from the local address vector.
38
39 fi_cm_data
40 Verifies exchanging CM data as part of connecting endpoints.
41
42 fi_cq_data
43 Tranfers messages with CQ data.
44
45 fi_dgram
46 A basic datagram endpoint example.
47
48 fi_dgram_waitset
49 Transfers datagrams using waitsets for completion notification.
50
51 fi_inj_complete
52 Sends messages using the FI_INJECT_COMPLETE operation flag.
53
54 fi_mcast
55 A simple multicast test.
56
57 fi_msg A basic message endpoint example.
58
59 fi_msg_epoll
60 Transfers messages with completion queues configured to use file
61 descriptors as wait objects. The file descriptors are retrieved
62 by the program and used directly with the Linux epoll API.
63
64 fi_msg_sockets
65 Verifies that the address assigned to a passive endpoint can be
66 transitioned to an active endpoint. This is required applica‐
67 tions that need socket API semantics over RDMA implementations
68 (e.g. rsockets).
69
70 fi_multi_ep
71 Performs data transfers over multiple endpoints in parallel.
72
73 fi_multi_mr
74 Issues RMA write operations to multiple memory regions, using
75 completion counters of inbound writes as the notification mecha‐
76 nism.
77
78 fi_poll
79 Exchanges data over RDM endpoints using poll sets to drive com‐
80 pletion notifications.
81
82 fi_rdm A basic RDM endpoint example.
83
84 fi_rdm_atomic
85 Test and verifies atomic operations over an RDM endpoint.
86
87 fi_rdm_deferred_wq
88 Test triggered operations and deferred work queue support.
89
90 fi_rdm_multi_domain
91 Performs data transfers over multiple endpoints, with each end‐
92 point belonging to a different opened domain.
93
94 fi_rdm_multi_recv
95 Transfers multiple messages over an RDM endpoint that are re‐
96 ceived into a single buffer, posted using the FI_MULTI_RECV
97 flag.
98
99 fi_rdm_rma_event
100 An RMA write example over an RDM endpoint that uses RMA events
101 to notify the peer that the RMA transfer has completed.
102
103 fi_rdm_rma_trigger
104 A basic example of queuing an RMA write operation that is initi‐
105 ated upon the firing of a triggering completion. Works with RDM
106 endpoints.
107
108 fi_rdm_shared_av
109 Spawns child processes to verify basic functionality of using a
110 shared address vector with RDM endpoints.
111
112 fi_rdm_tagged_peek
113 Basic test of using the FI_PEEK operation flag with tagged mes‐
114 sages. Works with RDM endpoints.
115
116 fi_recv_cancel
117 Tests canceling posted receives for tagged messages.
118
119 fi_resmgmt_test
120 Tests the resource management enabled feature. This verifies
121 that the provider prevents applications from overrunning local
122 and remote command queues and completion queues. This corre‐
123 sponds to setting the domain attribute resource_mgmt to
124 FI_RM_ENABLED.
125
126 fi_scalable_ep
127 Performs data transfers over scalable endpoints, endpoints asso‐
128 ciated with multiple transmit and receive contexts.
129
130 fi_shared_ctx
131 Performs data transfers between multiple endpoints, where the
132 endpoints share transmit and/or receive contexts.
133
134 fi_unexpected_msg
135 Tests the send and receive handling of unexpected tagged mes‐
136 sages.
137
138 fi_unmap_mem
139 Tests data transfers where the transmit buffer is mmapped and
140 unmapped between each transfer, but the virtual address of the
141 transmit buffer tries to remain the same. This test is used to
142 validate the correct behavior of memory registration caches.
143
144 fi_bw Performs a one-sided bandwidth test with an option for data ver‐
145 ification. A sleep time on the receiving side can be enabled in
146 order to allow the sender to get ahead of the receiver.
147
149 The client and the server exchange messages in either a ping-pong man‐
150 ner, for pingpong named tests, or transfer messages one-way, for bw
151 named tests. These tests can transfer various messages sizes, with
152 controls over which features are used by the test, and report perfor‐
153 mance numbers. The tests are structured based on the benchmarks pro‐
154 vided by OSU MPI. They are not guaranteed to provide the best latency
155 or bandwidth performance numbers a given provider or system may
156 achieve.
157
158 fi_dgram_pingpong
159 Latency test for datagram endpoints
160
161 fi_msg_bw
162 Message transfer bandwidth test for connected (MSG) endpoints.
163
164 fi_msg_pingpong
165 Message transfer latency test for connected (MSG) endpoints.
166
167 fi_rdm_cntr_pingpong
168 Message transfer latency test for reliable-datagram (RDM) end‐
169 points that uses counters as the completion mechanism.
170
171 fi_rdm_pingpong
172 Message transfer latency test for reliable-datagram (RDM) end‐
173 points.
174
175 fi_rdm_tagged_bw
176 Tagged message bandwidth test for reliable-datagram (RDM) end‐
177 points.
178
179 fi_rdm_tagged_pingpong
180 Tagged message latency test for reliable-datagram (RDM) end‐
181 points.
182
183 fi_rma_bw
184 An RMA read and write bandwidth test for reliable (MSG and RDM)
185 endpoints.
186
188 These are simple one-sided unit tests that validate basic behavior of
189 the API. Because these are single system tests that do not perform da‐
190 ta transfers their testing scope is limited.
191
192 fi_av_test
193 Verify address vector interfaces.
194
195 fi_cntr_test
196 Tests counter creation and destruction.
197
198 fi_cq_test
199 Tests completion queue creation and destruction.
200
201 fi_dom_test
202 Tests domain creation and destruction.
203
204 fi_eq_test
205 Tests event queue creation, destruction, and capabilities.
206
207 fi_getinfo_test
208 Tests provider response to fi_getinfo calls with varying hints.
209
210 fi_mr_test
211 Tests memory registration.
212
213 fi_mr_cache_evict
214 Tests provider MR cache eviction capabilities.
215
217 This test runs a series of tests over multiple formats and patterns to
218 help validate at scale. The patterns are an all to all, one to all,
219 all to one and a ring. The tests also run across multiple capabili‐
220 ties, such as messages, rma, atomics, and tagged messages. Currently,
221 there is no option to run these capabilities and patterns independent‐
222 ly, however the test is short enough to be all run at once.
223
225 This is a comprehensive latency, bandwidth, and functionality test that
226 can handle a variety of test configurations. The test is able to run a
227 large number of tests by iterating over a large number of test vari‐
228 ables. As a result, a full ubertest run can take a significant amount
229 of time. Because ubertest iterates over input variables, it relies on
230 a test configuration file for control, rather than extensive command
231 line options that are used by other fabtests. A configuration file
232 must be constructed for each provider. Example test configurations are
233 at test_configs.
234
235 fi_ubertest
236 This test takes a configure file as input. The file contains a
237 list of variables and their values to iterate over. The test
238 will run a set of latency, bandwidth, and functionality tests
239 over a given provider. It will perform one execution for every
240 possible combination of all variables. For example, if there
241 are 8 test variables, with 6 having 2 possible values and 2 hav‐
242 ing 3 possible values, ubertest will execute 576 total itera‐
243 tions of each test.
244
245 Config file options
246 The following keys and respective key values may be used in the config
247 file.
248
249 prov_name
250 Identify the provider(s) to test. E.g. udp, tcp, verbs,
251 ofi_rxm;verbs; ofi_rxd;udp.
252
253 test_type
254 FT_TEST_LATENCY, FT_TEST_BANDWIDTH, FT_TEST_UNIT
255
256 test_class
257 FT_CAP_MSG, FT_CAP_TAGGED, FT_CAP_RMA, FT_CAP_ATOMIC
258
259 class_function
260 For FT_CAP_MSG and FT_CAP_TAGGED: FT_FUNC_SEND, FT_FUNC_SENDV,
261 FT_FUNC_SENDMSG, FT_FUNC_INJECT, FT_FUNC_INJECTDATA,
262 FT_FUNC_SENDDATA
263
264 For FT_CAP_RMA: FT_FUNC_WRITE, FT_FUNC_WRITEV, FT_FUNC_WRITEMSG,
265 FT_FUNC_WRITEDATA, FT_FUNC_INJECT_WRITE, FT_FUNC_INJECT_WRITEDATA
266 FT_FUNC_READ, FT_FUNC_READV, FT_FUNC_READMSG
267
268 For FT_CAP_ATOMIC: FT_FUNC_ATOMIC, FT_FUNC_ATOMICV, FT_FUNC_ATOMICMSG,
269 FT_FUNC_INJECT_ATOMIC, FT_FUNC_FETCH_ATOMIC, FT_FUNC_FETCH_ATOMICV,
270 FT_FUNC_FETCH_ATOMICMSG, FT_FUNC_COMPARE_ATOMIC, FT_FUNC_COMPARE_ATOM‐
271 ICV, FT_FUNC_COMPARE_ATOMICMSG
272
273 constant_caps - values OR'ed together
274 FI_RMA, FI_MSG, FI_SEND, FI_RECV, FI_READ, FI_WRITE, FI_RE‐
275 MOTE_READ, FI_REMOTE_WRITE, FI_TAGGED, FI_DIRECTED_RECV
276
277 mode - values OR'ed together
278 FI_CONTEXT, FI_RX_CQ_DATA
279
280 ep_type
281 FI_EP_MSG, FI_EP_DGRAM, FI_EP_RDM
282
283 comp_type
284 FT_COMP_QUEUE, FT_COMP_CNTR, FT_COMP_ALL
285
286 av_type
287 FI_AV_MAP, FI_AV_TABLE
288
289 eq_wait_obj
290 FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
291
292 cq_wait_obj
293 FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
294
295 cntr_wait_obj
296 FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
297
298 threading
299 FI_THREAD_UNSPEC, FI_THREAD_SAFE, FI_THREAD_FID, FI_THREAD_DO‐
300 MAIN, FI_THREAD_COMPLETION, FI_THREAD_ENDPOINT
301
302 progress
303 FI_PROGRESS_MANUAL, FI_PROGRESS_AUTO, FI_PROGRESS_UNSPEC
304
305 mr_mode
306 (Values OR'ed together) FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_AL‐
307 LOCATED, FI_MR_PROV_KEY
308
309 op For FT_CAP_ATOMIC: FI_MIN, FI_MAX, FI_SUM, FI_PROD, FI_LOR,
310 FI_LAND, FI_BOR, FI_BAND, FI_LXOR, FI_BXOR, FI_ATOMIC_READ,
311 FI_ATOMIC_WRITE, FI_CSWAP, FI_CSWAP_NE, FI_CSWAP_LE,
312 FI_CSWAP_LT, FI_CSWAP_GE, FI_CSWAP_GT, FI_MSWAP
313
314 datatype
315 For FT_CAP_ATOMIC: FI_INT8, FI_UINT8, FI_INT16, FI_UINT16,
316 FI_INT32, FI_UINT32, FI_INT64, FI_UINT64, FI_FLOAT, FI_DOUBLE,
317 FI_FLOAT_COMPLEX, FI_DOUBLE_COMPLEX, FI_LONG_DOUBLE,
318 FI_LONG_DOUBLE_COMPLEX
319
320 msg_flags - values OR'ed together
321 For FT_FUNC_[SEND,WRITE,READ,ATOMIC]MSG: FI_REMOTE_CQ_DATA,
322 FI_COMPLETION
323
324 rx_cq_bind_flags - values OR'ed together
325 FI_SELECTIVE_COMPLETION
326
327 tx_cq_bind_flags - values OR'ed together
328 FI_SELECTIVE_COMPLETION
329
330 rx_op_flags - values OR'ed together
331 FI_COMPLETION
332
333 tx_op_flags - values OR'ed together
334 FI_COMPLETION
335
336 test_flags - values OR'ed together
337 FT_FLAG_QUICKTEST
338
340 (1) Fabtests requires that libfabric be installed on the system, and at
341 least one provider be usable.
342
343 (2) Install fabtests on the system. By default all the test executa‐
344 bles are installed in /usr/bin directory unless specified other‐
345 wise.
346
347 (3) All the client-server tests have the following usage model:
348
349 fi_ [OPTIONS] start server fi_ connect to server
350
352 Tests share command line options where appropriate. The following com‐
353 mand line options are available for one or more test. To see which op‐
354 tions apply for a given test, you can use the '-h' help option to see
355 the list available for that test.
356
357 -h Displays help output for the test.
358
359 -f Restrict test to the specified fabric name.
360
361 -d Restrict test to the specified domain name.
362
363 -p Restrict test to the specified provider name.
364
365 -e Use the specified endpoint type for the test. Valid options are
366 msg, dgram, and rdm. The default endpoint type is rdm.
367
368 -D Allocate data buffers on the specified device, rather than in
369 host memory. Valid options are ze and cuda.
370 *-a
371
372 · : The name of a shared address vector. This option only applies to
373 tests that support shared address vectors.
374
375 -B
376 Specifies the port number of the local endpoint, overriding the de‐
377 fault.
378
379 -P
380 Specifies the port number of the peer endpoint, overriding the de‐
381 fault.
382 *-s
383
384 · : Specifies the address of the local endpoint.
385
386 *-F
387 Specifies the address format.
388
389 -b[=oob_port]
390 Enables out-of-band (via sockets) address exchange and test synchro‐
391 nization. A port for the out-of-band connection may be specified as
392 part of this option to override the default.
393
394 -E[=oob_port]
395 Enables out-of-band (via sockets) address exchange only. A port for
396 the out-of-band connection may be specified as part of this option to
397 override the default. Cannot be used together with the '-b' option.
398
399 -U
400 Run fabtests with FI_DELIVERY_COMPLETE.
401
402 -I
403 Number of data transfer iterations.
404
405 -Q
406 Associated any EQ with the domain, rather than directly with the EP.
407
408 -w
409 Number of warm-up data transfer iterations.
410
411 -S
412 Data transfer size or 'all' for a full range of sizes. By default a
413 select number of sizes will be tested.
414
415 -l
416 If specified, the starting address of transmit and receive buffers
417 will be aligned along a page boundary.
418
419 -m
420 Use machine readable output. This is useful for post-processing the
421 test output with scripts.
422
423 -t
424 Specify the type of completion mechanism to use. Valid values are
425 queue and counter. The default is to use completion queues.
426
427 -c
428 Indicate the type of processing to use checking for completed opera‐
429 tions. Valid values are spin, sread, and fd. The default is to busy
430 wait (spin) until the desired operation has completed. The sread op‐
431 tion indicates that the application will invoke a blocking read call
432 in libfabric, such as fi_cq_sread. Fd indicates that the application
433 will retrieve the native operating system wait object (file descrip‐
434 tor) and use either poll() or select() to block until the fd has been
435 signaled, prior to checking for completions.
436
437 -o
438 For RMA based tests, specify the type of RMA operation to perform.
439 Valid values are read, write, and writedata. Write operations are
440 the default.
441
442 -M
443 For multicast tests, specifies the address of the multicast group to
444 join.
445
446 -v
447 Add data verification check to data transfers.
448
450 A simple example
451 run server: <test_name> -p <provider_name> -s <source_addr>
452 e.g. fi_msg_rma -p sockets -s 192.168.0.123
453 run client: <test_name> <server_addr> -p <provider_name>
454 e.g. fi_msg_rma 192.168.0.123 -p sockets
455
456 An example with various options
457 run server: fi_rdm_atomic -p psm -s 192.168.0.123 -I 1000 -S 1024
458 run client: fi_rdm_atomic 192.168.0.123 -p psm -I 1000 -S 1024
459
460 This will run "fi_rdm_atomic" for all atomic operations with
461
462 - PSM provider
463 - 1000 iterations
464 - 1024 bytes message size
465 - server node as 123.168.0.123
466
467 Run multinode tests
468 Server and clients are invoked with the same command:
469 fi_multinode -n <number of processes> -s <server_addr> -C <mode>
470
471 A process on the server must be started before any of the clients can be started
472 succesfully. -C lists the mode that the tests will run in. Currently the options are
473
474 for rma and msg. If not provided, the test will default to msg.
475
476 Run fi_ubertest
477 run server: fi_ubertest
478 run client: fi_ubertest -u /usr/share/fabtests/test_configs/sockets/quick.test 192.168.0.123
479
480 This will run "fi_ubertest" with
481
482 - sockets provider
483 - configurations defined in /usr/share/fabtests/test_configs/sockets/quick.test
484 - server node as 192.168.0.123
485
486 The config files are provided in /test_configs for sockets, verbs, udp,
487 and usnic providers and distributed with fabtests installation.
488
489 For more usage options: fi_ubertest -h
490
491 Run the whole fabtests suite
492 A runscript scripts/runfabtests.sh is provided that runs all the tests
493 in fabtests and reports the number of pass/fail/notrun.
494
495 Usage: runfabtests.sh [OPTIONS] [provider] [host] [client]
496
497 By default if none of the options are provided, it runs all the tests
498 using
499
500 - sockets provider
501 - 127.0.0.1 as both server and client address
502 - for small number of optiond and iterations
503
504 Various options can be used to choose provider, subset tests to run,
505 level of verbosity etc.
506
507 runfabtests.sh -vvv -t all psm 192.168.0.123 192.168.0.124
508
509 This will run all fabtests using
510
511 - psm provider
512 - for different options and larger iterations
513 - server node as 192.168.0.123 and client node as 192.168.0.124
514 - print test output for all the tests
515
516 For detailed usage options: runfabtests.sh -h
517
519 OpenFabrics.
520
521
522
523Libfabric Programmer's Manual 2020-12-01 fabtests(7)