1fabtests(7)                        @VERSION@                       fabtests(7)
2
3
4

NAME

6       Fabtests
7

SYNOPSIS

9       Fabtests  is  a  set of examples for fabric providers that demonstrates
10       various features of libfabric-  high-performance  fabric  software  li‐
11       brary.
12

OVERVIEW

14       Libfabric  defines sets of interface that fabric providers can support.
15       The purpose of Fabtests examples is to demonstrate some  of  the  major
16       features.  The goal is to familiarize users with different functionali‐
17       ties libfabric offers and how to use them.  Although most tests  report
18       performance  numbers,  they  are designed to test functionality and not
19       performance.  The exception are the benchmarks and ubertest.
20
21       The tests are divided into the following categories.  Except  the  unit
22       tests all of them are client-server tests.  Not all providers will sup‐
23       port each test.
24
25       The test names try to indicate the type of functionality each  test  is
26       verifying.   Although  some tests work with any endpoint type, many are
27       restricted to verifying a single endpoint type.  These tests  typically
28       include the endpoint type as part of the test name, such as dgram, msg,
29       or rdm.
30

Functional

32       These tests are a mix of very basic functionality tests that show major
33       features of libfabric.
34
35       fi_av_xfer
36              Tests  communication  for connectionless endpoints, as addresses
37              are inserted and removed from the local address vector.
38
39       fi_cm_data
40              Verifies exchanging CM data as part of connecting endpoints.
41
42       fi_cq_data
43              Tranfers messages with CQ data.
44
45       fi_dgram
46              A basic datagram endpoint example.
47
48       fi_dgram_waitset
49              Transfers datagrams using waitsets for completion notification.
50
51       fi_inj_complete
52              Sends messages using the FI_INJECT_COMPLETE operation flag.
53
54       fi_mcast
55              A simple multicast test.
56
57       fi_msg A basic message endpoint example.
58
59       fi_msg_epoll
60              Transfers messages with completion queues configured to use file
61              descriptors as wait objects.  The file descriptors are retrieved
62              by the program and used directly with the Linux epoll API.
63
64       fi_msg_sockets
65              Verifies that the address assigned to a passive endpoint can  be
66              transitioned  to  an active endpoint.  This is required applica‐
67              tions that need socket API semantics over  RDMA  implementations
68              (e.g.  rsockets).
69
70       fi_multi_ep
71              Performs data transfers over multiple endpoints in parallel.
72
73       fi_multi_mr
74              Issues  RMA  write  operations to multiple memory regions, using
75              completion counters of inbound writes as the notification mecha‐
76              nism.
77
78       fi_poll
79              Exchanges  data over RDM endpoints using poll sets to drive com‐
80              pletion notifications.
81
82       fi_rdm A basic RDM endpoint example.
83
84       fi_rdm_atomic
85              Test and verifies atomic operations over an RDM endpoint.
86
87       fi_rdm_deferred_wq
88              Test triggered operations and deferred work queue support.
89
90       fi_rdm_multi_domain
91              Performs data transfers over multiple endpoints, with each  end‐
92              point belonging to a different opened domain.
93
94       fi_rdm_multi_recv
95              Transfers  multiple  messages  over an RDM endpoint that are re‐
96              ceived into a single  buffer,  posted  using  the  FI_MULTI_RECV
97              flag.
98
99       fi_rdm_rma_event
100              An  RMA  write example over an RDM endpoint that uses RMA events
101              to notify the peer that the RMA transfer has completed.
102
103       fi_rdm_rma_trigger
104              A basic example of queuing an RMA write operation that is initi‐
105              ated upon the firing of a triggering completion.  Works with RDM
106              endpoints.
107
108       fi_rdm_shared_av
109              Spawns child processes to verify basic functionality of using  a
110              shared address vector with RDM endpoints.
111
112       fi_rdm_tagged_peek
113              Basic  test of using the FI_PEEK operation flag with tagged mes‐
114              sages.  Works with RDM endpoints.
115
116       fi_recv_cancel
117              Tests canceling posted receives for tagged messages.
118
119       fi_resmgmt_test
120              Tests the resource management enabled  feature.   This  verifies
121              that  the  provider prevents applications from overrunning local
122              and remote command queues and completion  queues.   This  corre‐
123              sponds   to   setting  the  domain  attribute  resource_mgmt  to
124              FI_RM_ENABLED.
125
126       fi_scalable_ep
127              Performs data transfers over scalable endpoints, endpoints asso‐
128              ciated with multiple transmit and receive contexts.
129
130       fi_shared_ctx
131              Performs  data  transfers  between multiple endpoints, where the
132              endpoints share transmit and/or receive contexts.
133
134       fi_unexpected_msg
135              Tests the send and receive handling of  unexpected  tagged  mes‐
136              sages.
137
138       fi_unmap_mem
139              Tests  data  transfers  where the transmit buffer is mmapped and
140              unmapped between each transfer, but the virtual address  of  the
141              transmit  buffer tries to remain the same.  This test is used to
142              validate the correct behavior of memory registration caches.
143
144       fi_bw  Performs a one-sided bandwidth test with an option for data ver‐
145              ification.  A sleep time on the receiving side can be enabled in
146              order to allow the sender to get ahead of the receiver.
147

Benchmarks

149       The client and the server exchange messages in either a ping-pong  man‐
150       ner,  for  pingpong  named  tests, or transfer messages one-way, for bw
151       named tests.  These tests can transfer  various  messages  sizes,  with
152       controls  over  which features are used by the test, and report perfor‐
153       mance numbers.  The tests are structured based on the  benchmarks  pro‐
154       vided  by OSU MPI.  They are not guaranteed to provide the best latency
155       or bandwidth  performance  numbers  a  given  provider  or  system  may
156       achieve.
157
158       fi_dgram_pingpong
159              Latency test for datagram endpoints
160
161       fi_msg_bw
162              Message transfer bandwidth test for connected (MSG) endpoints.
163
164       fi_msg_pingpong
165              Message transfer latency test for connected (MSG) endpoints.
166
167       fi_rdm_cntr_pingpong
168              Message  transfer  latency test for reliable-datagram (RDM) end‐
169              points that uses counters as the completion mechanism.
170
171       fi_rdm_pingpong
172              Message transfer latency test for reliable-datagram  (RDM)  end‐
173              points.
174
175       fi_rdm_tagged_bw
176              Tagged  message  bandwidth test for reliable-datagram (RDM) end‐
177              points.
178
179       fi_rdm_tagged_pingpong
180              Tagged message latency test  for  reliable-datagram  (RDM)  end‐
181              points.
182
183       fi_rma_bw
184              An  RMA read and write bandwidth test for reliable (MSG and RDM)
185              endpoints.
186

Unit

188       These are simple one-sided unit tests that validate basic  behavior  of
189       the API.  Because these are single system tests that do not perform da‐
190       ta transfers their testing scope is limited.
191
192       fi_av_test
193              Verify address vector interfaces.
194
195       fi_cntr_test
196              Tests counter creation and destruction.
197
198       fi_cq_test
199              Tests completion queue creation and destruction.
200
201       fi_dom_test
202              Tests domain creation and destruction.
203
204       fi_eq_test
205              Tests event queue creation, destruction, and capabilities.
206
207       fi_getinfo_test
208              Tests provider response to fi_getinfo calls with varying hints.
209
210       fi_mr_test
211              Tests memory registration.
212
213       fi_mr_cache_evict
214              Tests provider MR cache eviction capabilities.
215

Multinode

217       This test runs a series of tests over multiple formats and patterns  to
218       help  validate  at  scale.  The patterns are an all to all, one to all,
219       all to one and a ring.  The tests also run  across  multiple  capabili‐
220       ties,  such as messages, rma, atomics, and tagged messages.  Currently,
221       there is no option to run these capabilities and patterns  independent‐
222       ly, however the test is short enough to be all run at once.
223

Ubertest

225       This is a comprehensive latency, bandwidth, and functionality test that
226       can handle a variety of test configurations.  The test is able to run a
227       large  number  of  tests by iterating over a large number of test vari‐
228       ables.  As a result, a full ubertest run can take a significant  amount
229       of  time.  Because ubertest iterates over input variables, it relies on
230       a test configuration file for control, rather  than  extensive  command
231       line  options  that  are  used by other fabtests.  A configuration file
232       must be constructed for each provider.  Example test configurations are
233       at test_configs.
234
235       fi_ubertest
236              This  test takes a configure file as input.  The file contains a
237              list of variables and their values to iterate  over.   The  test
238              will  run  a  set of latency, bandwidth, and functionality tests
239              over a given provider.  It will perform one execution for  every
240              possible  combination  of  all variables.  For example, if there
241              are 8 test variables, with 6 having 2 possible values and 2 hav‐
242              ing  3  possible  values, ubertest will execute 576 total itera‐
243              tions of each test.
244
245   Config file options
246       The following keys and respective key values may be used in the  config
247       file.
248
249       prov_name
250              Identify  the  provider(s)  to  test.   E.g.   udp,  tcp, verbs,
251              ofi_rxm;verbs, ofi_rxd;udp.
252
253       test_type
254              FT_TEST_LATENCY, FT_TEST_BANDWIDTH, FT_TEST_UNIT
255
256       test_class
257              FT_CAP_MSG, FT_CAP_TAGGED, FT_CAP_RMA, FT_CAP_ATOMIC
258
259       class_function
260              For FT_CAP_MSG and FT_CAP_TAGGED:  FT_FUNC_SEND,  FT_FUNC_SENDV,
261              FT_FUNC_SENDMSG,       FT_FUNC_INJECT,       FT_FUNC_INJECTDATA,
262              FT_FUNC_SENDDATA
263
264       For  FT_CAP_RMA:   FT_FUNC_WRITE,   FT_FUNC_WRITEV,   FT_FUNC_WRITEMSG,
265       FT_FUNC_WRITEDATA,    FT_FUNC_INJECT_WRITE,    FT_FUNC_INJECT_WRITEDATA
266       FT_FUNC_READ, FT_FUNC_READV, FT_FUNC_READMSG
267
268       For FT_CAP_ATOMIC: FT_FUNC_ATOMIC, FT_FUNC_ATOMICV,  FT_FUNC_ATOMICMSG,
269       FT_FUNC_INJECT_ATOMIC,   FT_FUNC_FETCH_ATOMIC,   FT_FUNC_FETCH_ATOMICV,
270       FT_FUNC_FETCH_ATOMICMSG, FT_FUNC_COMPARE_ATOMIC,  FT_FUNC_COMPARE_ATOM‐
271       ICV, FT_FUNC_COMPARE_ATOMICMSG
272
273       constant_caps - values OR'ed together
274              FI_RMA,  FI_MSG,  FI_SEND,  FI_RECV,  FI_READ,  FI_WRITE, FI_RE‐
275              MOTE_READ, FI_REMOTE_WRITE, FI_TAGGED, FI_DIRECTED_RECV
276
277       mode - values OR'ed together
278              FI_CONTEXT, FI_RX_CQ_DATA
279
280       ep_type
281              FI_EP_MSG, FI_EP_DGRAM, FI_EP_RDM
282
283       comp_type
284              FT_COMP_QUEUE, FT_COMP_CNTR, FT_COMP_ALL
285
286       av_type
287              FI_AV_MAP, FI_AV_TABLE
288
289       eq_wait_obj
290              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
291
292       cq_wait_obj
293              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
294
295       cntr_wait_obj
296              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
297
298       threading
299              FI_THREAD_UNSPEC, FI_THREAD_SAFE,  FI_THREAD_FID,  FI_THREAD_DO‐
300              MAIN, FI_THREAD_COMPLETION, FI_THREAD_ENDPOINT
301
302       progress
303              FI_PROGRESS_MANUAL, FI_PROGRESS_AUTO, FI_PROGRESS_UNSPEC
304
305       mr_mode
306              (Values  OR'ed together) FI_MR_LOCAL, FI_MR_VIRT_ADDR, FI_MR_AL‐
307              LOCATED, FI_MR_PROV_KEY
308
309       op     For FT_CAP_ATOMIC:  FI_MIN,  FI_MAX,  FI_SUM,  FI_PROD,  FI_LOR,
310              FI_LAND,  FI_BOR,  FI_BAND,  FI_LXOR,  FI_BXOR,  FI_ATOMIC_READ,
311              FI_ATOMIC_WRITE,     FI_CSWAP,     FI_CSWAP_NE,     FI_CSWAP_LE,
312              FI_CSWAP_LT, FI_CSWAP_GE, FI_CSWAP_GT, FI_MSWAP
313
314       datatype
315              For   FT_CAP_ATOMIC:  FI_INT8,  FI_UINT8,  FI_INT16,  FI_UINT16,
316              FI_INT32, FI_UINT32, FI_INT64, FI_UINT64,  FI_FLOAT,  FI_DOUBLE,
317              FI_FLOAT_COMPLEX,       FI_DOUBLE_COMPLEX,       FI_LONG_DOUBLE,
318              FI_LONG_DOUBLE_COMPLEX
319
320       msg_flags - values OR'ed together
321              For   FT_FUNC_[SEND,WRITE,READ,ATOMIC]MSG:    FI_REMOTE_CQ_DATA,
322              FI_COMPLETION
323
324       rx_cq_bind_flags - values OR'ed together
325              FI_SELECTIVE_COMPLETION
326
327       tx_cq_bind_flags - values OR'ed together
328              FI_SELECTIVE_COMPLETION
329
330       rx_op_flags - values OR'ed together
331              FI_COMPLETION
332
333       tx_op_flags - values OR'ed together
334              FI_COMPLETION
335
336       test_flags - values OR'ed together
337              FT_FLAG_QUICKTEST
338

HOW TO RUN TESTS

340       (1) Fabtests requires that libfabric be installed on the system, and at
341           least one provider be usable.
342
343       (2) Install fabtests on the system.  By default all the  test  executa‐
344           bles  are  installed  in /usr/bin directory unless specified other‐
345           wise.
346
347       (3) All the client-server tests have the following usage model:
348
349           fi_ [OPTIONS] start server fi_ connect to server
350

COMMAND LINE OPTIONS

352       Tests share command line options where appropriate.  The following com‐
353       mand line options are available for one or more test.  To see which op‐
354       tions apply for a given test, you can use the '-h' help option  to  see
355       the list available for that test.
356
357       -h     Displays help output for the test.
358
359       -f     Restrict test to the specified fabric name.
360
361       -d     Restrict test to the specified domain name.
362
363       -p     Restrict test to the specified provider name.
364
365       -e     Use the specified endpoint type for the test.  Valid options are
366              msg, dgram, and rdm.  The default endpoint type is rdm.
367
368       -D     Allocate data buffers on the specified device,  rather  than  in
369              host memory.  Valid options are ze and cuda.
370       *-a
371
372       • :  The  name of a shared address vector.  This option only applies to
373         tests that support shared address vectors.
374
375       -B
376         Specifies the port number of the local endpoint, overriding  the  de‐
377         fault.
378
379       -P
380         Specifies  the  port  number of the peer endpoint, overriding the de‐
381         fault.
382       *-s
383
384       • : Specifies the address of the local endpoint.
385
386       *-F
387         Specifies the address format.
388
389       -b[=oob_port]
390         Enables out-of-band (via sockets) address exchange and test  synchro‐
391         nization.   A port for the out-of-band connection may be specified as
392         part of this option to override the default.
393
394       -E[=oob_port]
395         Enables out-of-band (via sockets) address exchange only.  A port  for
396         the out-of-band connection may be specified as part of this option to
397         override the default.  Cannot be used together with the '-b' option.
398
399       -U
400         Run fabtests with FI_DELIVERY_COMPLETE.
401
402       -I
403         Number of data transfer iterations.
404
405       -Q
406         Associated any EQ with the domain, rather than directly with the EP.
407
408       -w
409         Number of warm-up data transfer iterations.
410
411       -S
412         Data transfer size or 'all' for a full range of sizes.  By default  a
413         select number of sizes will be tested.
414
415       -l
416         If  specified,  the  starting address of transmit and receive buffers
417         will be aligned along a page boundary.
418
419       -m
420         Use machine readable output.  This is useful for post-processing  the
421         test output with scripts.
422
423       -t
424         Specify  the  type  of completion mechanism to use.  Valid values are
425         queue and counter.  The default is to use completion queues.
426
427       -c
428         Indicate the type of processing to use checking for completed  opera‐
429         tions.  Valid values are spin, sread, and fd.  The default is to busy
430         wait (spin) until the desired operation has completed.  The sread op‐
431         tion  indicates that the application will invoke a blocking read call
432         in libfabric, such as fi_cq_sread.  Fd indicates that the application
433         will  retrieve the native operating system wait object (file descrip‐
434         tor) and use either poll() or select() to block until the fd has been
435         signaled, prior to checking for completions.
436
437       -o
438         For  RMA  based  tests, specify the type of RMA operation to perform.
439         Valid values are read, write, and writedata.   Write  operations  are
440         the default.
441
442       -M
443         For  multicast tests, specifies the address of the multicast group to
444         join.
445
446       -v
447         Add data verification check to data transfers.
448

USAGE EXAMPLES

450   A simple example
451              run server: <test_name> -p <provider_name> -s <source_addr>
452                  e.g.    fi_msg_rma -p sockets -s 192.168.0.123
453              run client: <test_name> <server_addr> -p <provider_name>
454                  e.g.    fi_msg_rma 192.168.0.123 -p sockets
455
456   An example with various options
457              run server: fi_rdm_atomic -p psm -s 192.168.0.123 -I 1000 -S 1024
458              run client: fi_rdm_atomic 192.168.0.123 -p psm -I 1000 -S 1024
459
460       This will run "fi_rdm_atomic" for all atomic operations with
461
462              - PSM provider
463              - 1000 iterations
464              - 1024 bytes message size
465              - server node as 123.168.0.123
466
467   Run multinode tests
468              Server and clients are invoked with the same command:
469                  fi_multinode -n <number of processes> -s <server_addr> -C <mode>
470
471              A process on the server must be started before any of the clients can be started
472              succesfully. -C lists the mode that the tests will run in. Currently the options are
473
474       for rma and msg.  If not provided, the test will default to msg.
475
476   Run fi_ubertest
477              run server: fi_ubertest
478              run client: fi_ubertest -u /usr/share/fabtests/test_configs/sockets/quick.test 192.168.0.123
479
480       This will run "fi_ubertest" with
481
482              - sockets provider
483              - configurations defined in /usr/share/fabtests/test_configs/sockets/quick.test
484              - server node as 192.168.0.123
485
486       The config files are provided in /test_configs for sockets, verbs, udp,
487       and usnic providers and distributed with fabtests installation.
488
489       For more usage options: fi_ubertest -h
490
491   Run the whole fabtests suite
492       A  runscript scripts/runfabtests.sh is provided that runs all the tests
493       in fabtests and reports the number of pass/fail/notrun.
494
495              Usage: runfabtests.sh [OPTIONS] [provider] [host] [client]
496
497       By default if none of the options are provided, it runs all  the  tests
498       using
499
500              - sockets provider
501              - 127.0.0.1 as both server and client address
502              - for small number of optiond and iterations
503
504       Various  options  can  be used to choose provider, subset tests to run,
505       level of verbosity etc.
506
507              runfabtests.sh -vvv -t all psm 192.168.0.123 192.168.0.124
508
509       This will run all fabtests using
510
511              - psm provider
512              - for different options and larger iterations
513              - server node as 192.168.0.123 and client node as 192.168.0.124
514              - print test output for all the tests
515
516       For detailed usage options: runfabtests.sh -h
517

AUTHORS

519       OpenFabrics.
520
521
522
523Libfabric Programmer's Manual     2020-12-01                       fabtests(7)
Impressum