1IB_ATOMIC_BW(1)             General Commands Manual            IB_ATOMIC_BW(1)
2
3
4

NAME

6       ib_atomic_bw,   ib_atomic_lat,   ib_read_bw,  ib_read_lat,  ib_send_bw,
7       ib_send_lat, ib_write_bw, ib_write_lat - Collection  of  tests  written
8       over uverbs intended for use as a performance micro-benchmark
9

SYNOPSIS

11       ib_atomic_bw [<host>] [options]
12
13       ib_atomic_lat [<host>] [options]
14
15       ib_read_bw [<host>] [options]
16
17       ib_read_lat [<host>] [options]
18
19       ib_write_bw [<host>] [options]
20
21       ib_write_lat [<host>] [options]
22

DESCRIPTION

24       This is a collection of tests written over uverbs intended for use as a
25       performance micro-benchmark. As an example, the tests can be  used  for
26       HW or SW tuning and/or functional testing.
27
28       The collection conatains a set of BW and latency benchmark such as :
29
30       * Read   - ib_read_bw and ib_read_lat.
31
32       * Write  - ib_write_bw and ib_wriet_lat.
33
34       * Send   - ib_send_bw and ib_send_lat.
35
36       * Atomic - ib_atomic_bw and ib_atomic_lat
37
38       * Raw Etherent (when working with MOFED2) - raw_ethernet_bw, raw_ether‐
39       net_lat
40
41       The benchmark used the CPU cycle counter to  get  time  stamps  without
42       context  switch.   Some CPU architectures (e.g., Intel's 80486 or older
43       PPC) do NOT have such capability.
44
45       The latency benchmarks measures round-trip time  but  reports  half  of
46       that  as  one-way  latency.  This means that it may not be sufficiently
47       accurate for asymmetrical configurations.
48
49       On Bw benchmarks, we calculate the BW on send side only, as  he  calcu‐
50       lates  the  Bw  after  collecting completion from the receive side.  In
51       case we use the bidirectional flag , BW is calculated  on  both  sides.
52       in ib_send_bw, server side also calculate the received throughput.
53
54       Min/Median/Max  result  is  reported  in latency tests.  The median (vs
55       average) is less sensitive to extreme  scores.   Typically,  the  "Max"
56       value is the first value measured.
57
58       Larger samples help marginally only. The default (1000) is pretty good.
59       Note that an array of cycles_t (typically unsigned long)  is  allocated
60       once to collect samples and again to store the difference between them.
61       Really big sample sizes (e.g., 1 million) might expose  other  problems
62       with  the  program.  In  this  case  you  can  use -N flag (No Peak) to
63       instruct the test sample only 2 times (begining and end).
64
65       All throughput tests now have duration feature as well (-D <seconds  to
66       run>)  to  instruct the test to run for <seconds to run>.  Another fea‐
67       ture added is --run_infinitely, which instruct the test to run  all  te
68       time and print throughput every 5 seconds.
69
70       The  "-H"  option (latency) will dump the histogram for additional sta‐
71       tistical  analysis.   See   xgraph,   ygraph,   r-base   (http://www.r-
72       project.org/), pspp, or other statistical math programs.
73
74
75       Architectures tested: i686, x86_64, ia64
76

OPTIONS

78       The SAME OPTIONS must be passed to both server and client.
79
80       If  <host> is not presented, command starts a server and waits for con‐
81       nection.  If it is, command connects to server at <host>.
82
83       Common Options:
84
85           -h, --help
86                  Display this help message screen.
87
88           -p, --port=<port>
89                  Listen on/connect  to  port  <port>  (default:  18515)  when
90                  exchaning data.
91
92           -R, --rdma_cm
93                  Connect QPs with rdma_cm and run test on those QPs.
94
95           -z, --com_rdma_cm
96                  Communicate with rdma_cm module to exchange data - use regu‐
97                  lar QPs.
98
99           -m, --mtu=<mtu>
100                   QP Mtu size (default: active_mtu from ibv_devinfo).
101
102           -c, --connection=<RC/UC/UD>
103                  Connection type RC/UC/UD (default RC)
104
105           -d, --ib-dev=<dev>
106                  Use IB device <dev> (default: first device found).
107
108           -i, --ib-port=<port>
109                  Use port <port> of IB device (default: 1).
110
111           -s, --size=<size>
112                  Size of message to exchange (default: 1).
113
114           -a, --all
115                  Run sizes from 2 till 2^23.
116
117           -n, --iters=<iters>
118                  Number of exchanges (at least 100, default: 1000).
119
120           -x, --gid-index=<index>
121                  Test uses GID with GID index taken from command
122
123           -V, --version
124                  Display version number.
125
126           -e, --events
127                  Sleep on CQ events (default poll).
128
129           -F, --CPU-freq
130                  Do not fail even if cpufreq_ondemand module.
131
132           -I, --inline_size=<size>
133                  Max size of message to be sent in inline mode.
134
135           -u, --qp-timeout=<timeout>
136                  QP timeout, timeout value is  4  usec*2  ^timeout  (default:
137                  14).
138
139           -S, --sl=<sl>
140                  SL - Service Level (default 0)
141
142           -r, --rx-depth=<dep>
143                  Make rx queue bigger than tx (default 600).
144
145       Latenct tests options:
146
147           -C, --report-cycles
148                  Report times in cpu cycle units.
149
150           -H, --report-histogram
151                  Print out all results (Default: summary only).
152
153           -U, --report-unsorted
154                  Print out unsorted results (default sorted).
155
156       BW tests options:
157
158           -b, --bidirectional
159                  Measure bidirectional bandwidth (default uni).
160
161           -N, --no
162                  peak-bw               Cancel  peak-bw  calculation  (default
163                  with peak-bw)
164
165           -Q, --cq-mod
166                  Generate Cqe only after <cq-mod> completion
167
168           -t, --tx-depth=<dep>
169                  Size of tx queue (default: 128).
170
171           -O, --dualport
172                  Run test in dual-port mode  (2  QPs).  both  ports  must  be
173                  active (default OFF).
174
175           -D, --duration=<sec>
176                  Run test for <sec> period of seconds.
177
178           -f, --margin=<sec>
179                  When  in  Duration, measure results within margins (default:
180                  2)
181
182           -l, --post_list=<list_size>
183                  Post list of WQEs of <list size>  size  (instead  of  single
184                  post).
185
186           -q, --qp=<num_of_qps>
187                  Num of QPs running in the process (default: 1).
188
189           --run_infinitely
190                  Run test forever, print results every 5 seconds.
191
192       SEND tests options:
193
194           -r, --rx-depth=<dep>
195                  Size of RX queue (default: 512 in BW test).
196
197           -g, --mcg=<num_of_qps>
198                  Send  messages  to  multicast  group  with  <num_of_qps> qps
199                  attached to it.
200
201           -M, --MGID=<multicast_gid>
202                  In multicast, uses <multicast_gid> as the group MGID.
203
204       Raw Ethernet BW test options:
205
206           -A, --atomic_type=<type>
207                  type of atomic operation from {CMP_AND_SWAP,FETCH_AND_ADD}.
208
209           -o, --outs=<num>
210                  Number of outstanding read/atomic requests -  also  on  READ
211                  tests.
212
213           -B, --source_mac
214                  source MAC address by this format XX:XX:XX:XX:XX:XX (default
215                  take the MAC address form GID).
216
217           -E, --dest_mac
218                  destination MAC address  by  this  format  XX:XX:XX:XX:XX:XX
219                  **MUST** be entered.
220
221           -J, --server_ip
222                  server  ip  address  by  this  format X.X.X.X (using to send
223                  packets with IP header).
224
225           -j, --client_ip
226                  client ip address by this  format  X.X.X.X  (using  to  send
227                  packets with IP header).
228
229           -K, --server_port
230                  server  udp  port  number  (using  to  send packets with UPD
231                  header).
232
233           -k, --client_port
234                  client udp port number  (using  to  send  packets  with  UDP
235                  header).
236
237           -Z, --server
238                  choose    server    side    for    the    current    machine
239                  (--server/--client must be selected ).
240
241           -P, --client
242                  choose    client    side    for    the    current    machine
243                  (--server/--client must be selected).
244

ENVIRONMENT

246       Prerequisites:
247              kernel 2.6
248              (kernel module) matches libibverbs
249              (kernel module) matches librdmacm
250              (kernel module) matches libibumad
251              (kernel module) matches libmath (lm).
252

NOTES

254       You  need to be running a Subnet Manager on the switch or on one of the
255       nodes in your fabric, in case you are in IB fabric.
256

BUGS

258       1. Multicast feauture in ib_send_lat and in ib_send_bw still have  many
259       problems!   Will  increase the support and bug fixes in this Q, but now
260       the tests may stuck and could produce undefine behaviours.
261
262       2. Bidirectional feature in ib_send_bw test, when running in UD  or  UC
263       mode.   The  algorithm  we  use  for  the  bidirectional measurement is
264       designed for RC connection type.  When running in UC or  UD  connection
265       types, there is a small probablity the test will be stuck.
266
267       3. RDMA_CM feature in read tests still doesn't work.
268
269       4. Dual-port support currently works only with ib_write_bw.
270
271       5.  Compabilty issues may occur between different versions of perftest.
272       Please make sure you work with the same version on both sides to ensure
273       consistency of the test.
274

AUTHORS

276       Please  post  results/observations  to the openib-general mailing list.
277       See "Contact Us"  at  http://openib.org/mailman/listinfo/openib-general
278       and http://www.openib.org.
279
280
281
282Open Fabrics Enterprise Distribution 2014                      IB_ATOMIC_BW(1)
Impressum