1fabtests(7)                        @VERSION@                       fabtests(7)
2
3
4

NAME

6       Fabtests
7

SYNOPSIS

9       Fabtests  is  a  set of examples for fabric providers that demonstrates
10       various features of libfabric-  high-performance  fabric  software  li‐
11       brary.
12

OVERVIEW

14       Libfabric  defines sets of interface that fabric providers can support.
15       The purpose of Fabtests examples is to demonstrate some  of  the  major
16       features.  The goal is to familiarize users with different functionali‐
17       ties libfabric offers and how to use them.  Although most tests  report
18       performance  numbers,  they  are designed to test functionality and not
19       performance.  The exception are the benchmarks and ubertest.
20
21       The tests are divided into the following categories.  Except  the  unit
22       tests all of them are client-server tests.  Not all providers will sup‐
23       port each test.
24
25       The test names try to indicate the type of functionality each  test  is
26       verifying.   Although  some tests work with any endpoint type, many are
27       restricted to verifying a single endpoint type.  These tests  typically
28       include the endpoint type as part of the test name, such as dgram, msg,
29       or rdm.
30

Functional

32       These tests are a mix of very basic functionality tests that show major
33       features of libfabric.
34
35       fi_av_xfer
36              Tests  communication for unconnected endpoints, as addresses are
37              inserted and removed from the local address vector.
38
39       fi_cm_data
40              Verifies exchanging CM data as part of connecting endpoints.
41
42       fi_cq_data
43              Tranfers messages with CQ data.
44
45       fi_dgram
46              A basic datagram endpoint example.
47
48       fi_dgram_waitset
49              Transfers datagrams using waitsets for completion notifcation.
50
51       fi_inj_complete
52              Sends messages using the FI_INJECT_COMPLETE operation flag.
53
54       fi_mcast
55              A simple multicast test.
56
57       fi_msg A basic message endpoint example.
58
59       fi_msg_epoll
60              Transfers messages with completion queues configured to use file
61              descriptors  as  wait  objetcts.   The  file descriptors are re‐
62              trieved by the program and used directly with  the  Linux  epoll
63              API.
64
65       fi_msg_sockets
66              Verifies  that the address assigned to a passive endpoint can be
67              transitioned to an active endpoint.  This is  required  applica‐
68              tions  that  need socket API semantics over RDMA implementations
69              (e.g.  rsockets).
70
71       fi_multi_ep
72              Performs data transfers over multiple endpoints in parallel.
73
74       fi_multi_mr
75              Issues RMA write operations to multiple  memory  regions,  using
76              completion counters of inbound writes as the notification mecha‐
77              nism.
78
79       fi_poll
80              Exchanges data over RDM endpoints using poll sets to drive  com‐
81              pletion notifications.
82
83       fi_rdm A basic RDM endpoint example.
84
85       fi_rdm_atomic
86              Test and verifies atomic operations over an RDM endpoint.
87
88       fi_rdm_deferred_wq
89              Test triggered operations and deferred work queue support.
90
91       fi_rdm_multi_domain
92              Performs  data transfers over multiple endpoints, with each end‐
93              point belonging to a different opened domain.
94
95       fi_rdm_multi_recv
96              Transfers multiple messages over an RDM endpoint  that  are  re‐
97              ceived  into  a  single  buffer,  posted using the FI_MULTI_RECV
98              flag.
99
100       fi_rdm_rma_simple
101              A simple RMA write example over an RDM endpoint.
102
103       fi_rdm_rma_trigger
104              A basic example of queuing an RMA write operation that is initi‐
105              ated upon the firing of a triggering completion.  Works with RDM
106              endpoints.
107
108       fi_rdm_shared_av
109              Spawns child processes to verify basic functionality of using  a
110              shared address vector with RDM endpoints.
111
112       fi_rdm_tagged_peek
113              Basic  test of using the FI_PEEK operation flag with tagged mes‐
114              sages.  Works with RDM endpoints.
115
116       fi_recv_cancel
117              Tests canceling posted receives for tagged messages.
118
119       fi_resmgmt_test
120              Tests the resource management enabled  feature.   This  verifies
121              that  the  provider  prevents applications from overruning local
122              and remote command queues and completion  queues.   This  corre‐
123              sponds   to   setting  the  domain  attribute  resource_mgmt  to
124              FI_RM_ENABLED.
125
126       fi_scalable_ep
127              Performs data transfers over scalable endpoints, endpoints asso‐
128              ciated with multiple transmit and receive contexts.
129
130       fi_shared_ctx
131              Performs  data  transfers  between multiple endpoints, where the
132              endpoints share transmit and/or receive contexts.
133
134       fi_unexpected_msg
135              Tests the send and receive handling of  unexpected  tagged  mes‐
136              sages.
137
138       fi_unmap_mem
139              Tests  data  transfers  where the transmit buffer is mmapped and
140              unmapped between each transfer, but the virtual address  of  the
141              transmit  buffer tries to remain the same.  This test is used to
142              validate the correct behavior of memory registration caches.
143
144       fi_bw  Performs a one-sided bandwidth test with an option for data ver‐
145              ification.  A sleep time on the receiving side can be enabled in
146              order to allow the sender to get ahead of the receiver.
147

Benchmarks

149       The client and the server exchange messages in either a ping-pong  man‐
150       ner,  for  pingpong  named  tests, or transfer messages one-way, for bw
151       named tests.  These tests can transfer  various  messages  sizes,  with
152       controls  over  which features are used by the test, and report perfor‐
153       mance numbers.  The tests are structured based on the  benchmarks  pro‐
154       vided  by OSU MPI.  They are not guaranteed to provide the best latency
155       or bandwidth  performance  numbers  a  given  provider  or  system  may
156       achieve.
157
158       fi_dgram_pingpong
159              Latency test for datagram endpoints
160
161       fi_msg_bw
162              Message transfer bandwidth test for connected (MSG) endpoints.
163
164       fi_msg_pingpong
165              Message transfer latency test for connected (MSG) endpoints.
166
167       fi_rdm_cntr_pingpong
168              Message  transfer  latency test for reliable-datagram (RDM) end‐
169              points that uses counters as the completion mechanism.
170
171       fi_rdm_pingpong
172              Message transfer latency test for reliable-datagram  (RDM)  end‐
173              points.
174
175       fi_rdm_tagged_bw
176              Tagged  message  bandwidth test for reliable-datagram (RDM) end‐
177              points.
178
179       fi_rdm_tagged_pingpong
180              Tagged message latency test  for  reliable-datagram  (RDM)  end‐
181              points.
182
183       fi_rma_bw
184              An  RMA read and write bandwidth test for reliable (MSG and RDM)
185              endpoints.
186

Unit

188       These are simple one-sided unit tests that validate basic  behavior  of
189       the API.  Because these are single system tests that do not perform da‐
190       ta transfers their testing scope is limited.
191
192       fi_av_test
193              Verify address vector interfaces.
194
195       fi_cntr_test
196              Tests counter creation and destruction.
197
198       fi_cq_test
199              Tests completion queue creation and destruction.
200
201       fi_dom_test
202              Tests domain creation and destruction.
203
204       fi_eq_test
205              Tests event queue creation, destruction, and capabilities.
206
207       fi_getinfo_test
208              Tests provider response to fi_getinfo calls with varying hints.
209
210       fi_mr_test
211              Tests memory registration.
212
213       fi_resource_freeing
214              Allocates and  closes  fabric  resources  to  check  for  proper
215              cleanup.
216

Multinode

218       This  test runs a series of tests over multiple formats and patterns to
219       help validate at scale.  The patterns are an all to all,  one  to  all,
220       all  to  one  and  a  ring.   The tests also run accross multiple capa‐
221       bilites, such as messages, rma, atomics, and tagged messages.  Current‐
222       ly,  there is no option to run these capabilities and patterns indepen‐
223       dently, however the test is short enough to be all run at once.
224

Ubertest

226       This is a comprehensive latency, bandwidth, and functionality test that
227       can handle a variety of test configurations.  The test is able to run a
228       large number of tests by iterating over a large number  of  test  vari‐
229       ables.   As a result, a full ubertest run can take a significant amount
230       of time.  Because ubertest iterates over input variables, it relies  on
231       a  test  configuration  file for control, rather than extensive command
232       line options that are used by other  fabtests.   A  configuration  file
233       must  be  constructured for each provider.  Example test configurations
234       are at test_configs.
235
236       fi_ubertest
237              This test takes a configure file as input.  The file contains  a
238              list  of  variables  and their values to iterate over.  The test
239              will run a set of latency, bandwidth,  and  functionality  tests
240              over  a given provider.  It will perform one execution for every
241              possible combination of all variables.  For  example,  if  there
242              are 8 test variables, with 6 having 2 possible values and 2 hav‐
243              ing 3 possible values, ubertest will execute  576  total  itera‐
244              tions of each test.
245
246   Config file options
247       The  following keys and respective key values may be used in the config
248       file.
249
250       prov_name
251              Identify the  provider(s)  to  test.   E.g.   udp,  tcp,  verbs,
252              ofi_rxm;verbs; ofi_rxd;udp.
253
254       test_type
255              FT_TEST_LATENCY, FT_TEST_BANDWIDTH, FT_TEST_UNIT
256
257       test_class
258              FT_CAP_MSG, FT_CAP_TAGGED, FT_CAP_RMA, FT_CAP_ATOMIC
259
260       class_function
261              For  FT_CAP_MSG  and FT_CAP_TAGGED: FT_FUNC_SEND, FT_FUNC_SENDV,
262              FT_FUNC_SENDMSG,       FT_FUNC_INJECT,       FT_FUNC_INJECTDATA,
263              FT_FUNC_SENDDATA
264
265       For   FT_CAP_RMA:   FT_FUNC_WRITE,   FT_FUNC_WRITEV,  FT_FUNC_WRITEMSG,
266       FT_FUNC_WRITEDATA,    FT_FUNC_INJECT_WRITE,    FT_FUNC_INJECT_WRITEDATA
267       FT_FUNC_READ, FT_FUNC_READV, FT_FUNC_READMSG
268
269       For  FT_CAP_ATOMIC: FT_FUNC_ATOMIC, FT_FUNC_ATOMICV, FT_FUNC_ATOMICMSG,
270       FT_FUNC_INJECT_ATOMIC,   FT_FUNC_FETCH_ATOMIC,   FT_FUNC_FETCH_ATOMICV,
271       FT_FUNC_FETCH_ATOMICMSG,  FT_FUNC_COMPARE_ATOMIC, FT_FUNC_COMPARE_ATOM‐
272       ICV, FT_FUNC_COMPARE_ATOMICMSG
273
274       constant_caps - values OR'ed together
275              FI_RMA, FI_MSG,  FI_SEND,  FI_RECV,  FI_READ,  FI_WRITE,  FI_RE‐
276              MOTE_READ, FI_REMOTE_WRITE, FI_TAGGED, FI_DIRECTED_RECV
277
278       mode - values OR'ed together
279              FI_CONTEXT, FI_RX_CQ_DATA
280
281       ep_type
282              FI_EP_MSG, FI_EP_DGRAM, FI_EP_RDM
283
284       comp_type
285              FT_COMP_QUEUE, FT_COMP_CNTR, FT_COMP_ALL
286
287       av_type
288              FI_AV_MAP, FI_AV_TABLE
289
290       eq_wait_obj
291              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
292
293       cq_wait_obj
294              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
295
296       cntr_wait_obj
297              FI_WAIT_NONE, FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND
298
299       threading
300              FI_THREAD_UNSPEC,  FI_THREAD_SAFE,  FI_THREAD_FID, FI_THREAD_DO‐
301              MAIN, FI_THREAD_COMPLETION, FI_THREAD_ENDPOINT
302
303       progress
304              FI_PROGRESS_MANUAL, FI_PROGRESS_AUTO, FI_PROGRESS_UNSPEC
305
306       mr_mode
307              (Values OR'ed together) FI_MR_LOCAL, FI_MR_VIRT_ADDR,  FI_MR_AL‐
308              LOCATED, FI_MR_PROV_KEY
309
310       op     For  FT_CAP_ATOMIC:  FI_MIN,  FI_MAX,  FI_SUM,  FI_PROD, FI_LOR,
311              FI_LAND,  FI_BOR,  FI_BAND,  FI_LXOR,  FI_BXOR,  FI_ATOMIC_READ,
312              FI_ATOMIC_WRITE,     FI_CSWAP,     FI_CSWAP_NE,     FI_CSWAP_LE,
313              FI_CSWAP_LT, FI_CSWAP_GE, FI_CSWAP_GT, FI_MSWAP
314
315       datatype
316              For  FT_CAP_ATOMIC:  FI_INT8,  FI_UINT8,  FI_INT16,   FI_UINT16,
317              FI_INT32,  FI_UINT32,  FI_INT64, FI_UINT64, FI_FLOAT, FI_DOUBLE,
318              FI_FLOAT_COMPLEX,       FI_DOUBLE_COMPLEX,       FI_LONG_DOUBLE,
319              FI_LONG_DOUBLE_COMPLE
320
321       msg_flags - values OR'ed together
322              For FT_FUNC_XXXMSG: FI_REMOTE_CQ_DATA, FI_COMPLETION
323
324       rx_cq_bind_flags - values OR'ed together
325              FI_SELECTIVE_COMPLETION
326
327       tx_cq_bind_flags - values OR'ed together
328              FI_SELECTIVE_COMPLETION
329
330       rx_op_flags - values OR'ed together
331              FI_COMPLETION
332
333       tx_op_flags - values OR'ed together
334              FI_COMPLETION
335
336       test_flags - values OR'ed together
337              FT_FLAG_QUICKTEST
338

HOW TO RUN TESTS

340       (1) Fabtests requires that libfabric be installed on the system, and at
341           least one provider be usable.
342
343       (2) Install fabtests on the system.  By default all the  test  executa‐
344           bles  are  installed  in /usr/bin directory unless specified other‐
345           wise.
346
347       (3) All the client-server tests have the following usage model:
348
349           fi_ [OPTIONS] start server fi_ connect to server
350

COMMAND LINE OPTIONS

352       Tests share command line options where appropriate.  The following com‐
353       mand line options are available for one or more test.  To see which op‐
354       tions apply for a given test, you can use the '-h' help option  to  see
355       the list available for that test.
356
357       -h     Displays help output for the test.
358
359       -f     Restrict test to the specified fabric name.
360
361       -d     Restrict test to the specified domain name.
362
363       -p     Restrict test to the specified provider name.
364
365       -e     Use the specified endpoint type for the test.  Valid options are
366              msg, dgram, and rdm.  The default endpoint type is rdm.
367       *-a
368
369       · : The name of a shared address vector.  This option only  applies  to
370         tests that support shared address vectors.
371
372       -B
373         Specifies  the  port number of the local endpoint, overriding the de‐
374         fault.
375
376       -P
377         Specifies the port number of the peer endpoint,  overriding  the  de‐
378         fault.
379       *-s
380
381       · : Specifies the address of the local endpoint.
382
383       -b[=oob_port]
384         Enables  out-of-band (via sockets) address exchange and test synchro‐
385         nization.  A port for the out-of-band connection may be specified  as
386         part of this option to override the default.
387
388       -E[=oob_port]
389         Enables  out-of-band (via sockets) address exchange only.  A port for
390         the out-of-band connection may be specified as part of this option to
391         override the default.  Cannot be used together with the '-b' option.
392
393       -I
394         Number of data transfer iterations.
395
396       -w
397         Number of warm-up data transfer iterations.
398
399       -S
400         Data  transfer size or 'all' for a full range of sizes.  By default a
401         select number of sizes will be tested.
402
403       -l
404         If specified, the starting address of transmit  and  receive  buffers
405         will be aligned along a page boundary.
406
407       -m
408         Use  machine readable output.  This is useful for post-processing the
409         test output with scripts.
410
411       -t
412         Specify the type of completion mechanism to use.   Valid  values  are
413         queue and counter.  The default is to use completion queues.
414
415       -c
416         Indicate  the type of processing to use checking for completed opera‐
417         tions.  Valid values are spin, sread, and fd.  The default is to busy
418         wait (spin) until the desired operation has completed.  The sread op‐
419         tion indicates that the application will invoke a blocking read  call
420         in libfabric, such as fi_cq_sread.  Fd indicates that the application
421         will retrieve the native operating system wait object (file  descrip‐
422         tor) and use either poll() or select() to block until the fd has been
423         signaled, prior to checking for completions.
424
425       -o
426         For RMA based tests, specify the type of RMA  operation  to  perform.
427         Valid  values  are  read, write, and writedata.  Write operations are
428         the default.
429
430       -M
431         For multicast tests, specifies the address of the multicast group  to
432         join.
433
434       -v
435         Add data verification check to data transfers.
436

USAGE EXAMPLES

438   A simple example
439              run server: <test_name> -p <provider_name> -s <source_addr>
440                  e.g.    fi_msg_rma -p sockets -s 192.168.0.123
441              run client: <test_name> <server_addr> -p <provider_name>
442                  e.g.    fi_msg_rma 192.168.0.123 -p sockets
443
444   An example with various options
445              run server: fi_rdm_atomic -p psm -s 192.168.0.123 -I 1000 -S 1024
446              run client: fi_rdm_atomic 192.168.0.123 -p psm -I 1000 -S 1024
447
448       This will run "fi_rdm_atomic" for all atomic operations with
449
450              - PSM provider
451              - 1000 iterations
452              - 1024 bytes message size
453              - server node as 123.168.0.123
454
455   Run multinode tests
456              Server and clients are invoked with the same command:
457                  fi_multinode -n <number of processes> -s <server_addr> -C <mode>
458
459              A process on the server must be started before any of the clients can be started
460              succesfully. -C lists the mode that the tests will run in. Currently the options are
461
462       for rma and msg.  If not provided, the test will default to msg.
463
464   Run fi_ubertest
465              run server: fi_ubertest
466              run client: fi_ubertest -u /usr/share/fabtests/test_configs/sockets/quick.test 192.168.0.123
467
468       This will run "fi_ubertest" with
469
470              - sockets provider
471              - configurations defined in /usr/share/fabtests/test_configs/sockets/quick.test
472              - server node as 192.168.0.123
473
474       The config files are provided in /test_configs for sockets, verbs, udp,
475       and usnic providers and distributed with fabtests installation.
476
477       For more usage options: fi_ubertest -h
478
479   Run the whole fabtests suite
480       A runscript scripts/runfabtests.sh is provided that runs all the  tests
481       in fabtests and reports the number of pass/fail/notrun.
482
483              Usage: runfabtests.sh [OPTIONS] [provider] [host] [client]
484
485       By  default  if none of the options are provided, it runs all the tests
486       using
487
488              - sockets provider
489              - 127.0.0.1 as both server and client address
490              - for small number of optiond and iterations
491
492       Various options can be used to choose provider, subset  tests  to  run,
493       level of verbosity etc.
494
495              runfabtests.sh -vvv -t all psm 192.168.0.123 192.168.0.124
496
497       This will run all fabtests using
498
499              - psm provider
500              - for different options and larger iterations
501              - server node as 192.168.0.123 and client node as 192.168.0.124
502              - print test output for all the tests
503
504       For detailed usage options: runfabtests.sh -h
505

AUTHORS

507       OpenFabrics.
508
509
510
511Libfabric Programmer's Manual     2020-03-02                       fabtests(7)
Impressum