1IB_ATOMIC_BW(1) General Commands Manual IB_ATOMIC_BW(1)
2
6 ib_atomic_bw, ib_atomic_lat, ib_read_bw, ib_read_lat, ib_send_bw,
7 ib_send_lat, ib_write_bw, ib_write_lat - Collection of tests written
8 over uverbs intended for use as a performance micro-benchmark
9
11 ib_atomic_bw [<host>] [options]
12 ib_atomic_lat [<host>] [options]
14 ib_read_bw [<host>] [options]
16 ib_read_lat [<host>] [options]
18 ib_write_bw [<host>] [options]
20 ib_write_lat [<host>] [options]
22
24 This is a collection of tests written over uverbs intended for use as a
25 performance micro-benchmark. As an example, the tests can be used for
26 HW or SW tuning and/or functional testing.
27 The collection conatains a set of BW and latency benchmark such as :
29 * Read - ib_read_bw and ib_read_lat.
31 * Write - ib_write_bw and ib_wriet_lat.
33 * Send - ib_send_bw and ib_send_lat.
35 * Atomic - ib_atomic_bw and ib_atomic_lat
37 * Raw Etherent (when working with MOFED2) - raw_ethernet_bw, raw_ether‐
39 net_lat
40 The benchmark used the CPU cycle counter to get time stamps without
42 context switch. Some CPU architectures (e.g., Intel's 80486 or older
43 PPC) do NOT have such capability.
44 The latency benchmarks measures round-trip time but reports half of
46 that as one-way latency. This means that it may not be sufficiently
47 accurate for asymmetrical configurations.
48 On Bw benchmarks, we calculate the BW on send side only, as he calcu‐
50 lates the Bw after collecting completion from the receive side. In
51 case we use the bidirectional flag , BW is calculated on both sides.
52 in ib_send_bw, server side also calculate the received throughput.
53 Min/Median/Max result is reported in latency tests. The median (vs
55 average) is less sensitive to extreme scores. Typically, the "Max"
56 value is the first value measured.
57 Larger samples help marginally only. The default (1000) is pretty good.
59 Note that an array of cycles_t (typically unsigned long) is allocated
60 once to collect samples and again to store the difference between them.
61 Really big sample sizes (e.g., 1 million) might expose other problems
62 with the program. In this case you can use -N flag (No Peak) to
63 instruct the test sample only 2 times (begining and end).
64 All throughput tests now have duration feature as well (-D <seconds to
66 run>) to instruct the test to run for <seconds to run>. Another fea‐
67 ture added is --run_infinitely, which instruct the test to run all te
68 time and print throughput every 5 seconds.
69 The "-H" option (latency) will dump the histogram for additional sta‐
71 tistical analysis. See xgraph, ygraph, r-base (http://www.r-
72 project.org/), pspp, or other statistical math programs.
73 Architectures tested: i686, x86_64, ia64
76
78 The SAME OPTIONS must be passed to both server and client.
79 If <host> is not presented, command starts a server and waits for con‐
81 nection. If it is, command connects to server at <host>.
82 Common Options:
84 -h, --help
86 Display this help message screen.
87 -p, --port=<port>
89 Listen on/connect to port <port> (default: 18515) when
90 exchaning data.
91 -R, --rdma_cm
93 Connect QPs with rdma_cm and run test on those QPs.
94 -z, --com_rdma_cm
96 Communicate with rdma_cm module to exchange data - use regu‐
97 lar QPs.
98 -m, --mtu=<mtu>
100 QP Mtu size (default: active_mtu from ibv_devinfo).
101 -c, --connection=<RC/UC/UD>
103 Connection type RC/UC/UD (default RC)
104 -d, --ib-dev=<dev>
106 Use IB device <dev> (default: first device found).
107 -i, --ib-port=<port>
109 Use port <port> of IB device (default: 1).
110 -s, --size=<size>
112 Size of message to exchange (default: 1).
113 -a, --all
115 Run sizes from 2 till 2^23.
116 -n, --iters=<iters>
118 Number of exchanges (at least 100, default: 1000).
119 -x, --gid-index=<index>
121 Test uses GID with GID index taken from command
122 -V, --version
124 Display version number.
125 -e, --events
127 Sleep on CQ events (default poll).
128 -F, --CPU-freq
130 Do not fail even if cpufreq_ondemand module.
131 -I, --inline_size=<size>
133 Max size of message to be sent in inline mode.
134 -u, --qp-timeout=<timeout>
136 QP timeout, timeout value is 4 usec*2 ^timeout (default:
137 14).
138 -S, --sl=<sl>
140 SL - Service Level (default 0)
141 -r, --rx-depth=<dep>
143 Make rx queue bigger than tx (default 600).
144 Latenct tests options:
146 -C, --report-cycles
148 Report times in cpu cycle units.
149 -H, --report-histogram
151 Print out all results (Default: summary only).
152 -U, --report-unsorted
154 Print out unsorted results (default sorted).
155 BW tests options:
157 -b, --bidirectional
159 Measure bidirectional bandwidth (default uni).
160 -N, --no
162 peak-bw Cancel peak-bw calculation (default
163 with peak-bw)
164 -Q, --cq-mod
166 Generate Cqe only after <cq-mod> completion
167 -t, --tx-depth=<dep>
169 Size of tx queue (default: 128).
170 -O, --dualport
172 Run test in dual-port mode (2 QPs). both ports must be
173 active (default OFF).
174 -D, --duration=<sec>
176 Run test for <sec> period of seconds.
177 -f, --margin=<sec>
179 When in Duration, measure results within margins (default:
180 2)
181 -l, --post_list=<list_size>
183 Post list of WQEs of <list size> size (instead of single
184 post).
185 -q, --qp=<num_of_qps>
187 Num of QPs running in the process (default: 1).
188 --run_infinitely
190 Run test forever, print results every 5 seconds.
191 SEND tests options:
193 -r, --rx-depth=<dep>
195 Size of RX queue (default: 512 in BW test).
196 -g, --mcg=<num_of_qps>
198 Send messages to multicast group with <num_of_qps> qps
199 attached to it.
200 -M, --MGID=<multicast_gid>
202 In multicast, uses <multicast_gid> as the group MGID.
203 Raw Ethernet BW test options:
205 -A, --atomic_type=<type>
207 type of atomic operation from {CMP_AND_SWAP,FETCH_AND_ADD}.
208 -o, --outs=<num>
210 Number of outstanding read/atomic requests - also on READ
211 tests.
212 -B, --source_mac
214 source MAC address by this format XX:XX:XX:XX:XX:XX (default
215 take the MAC address form GID).
216 -E, --dest_mac
218 destination MAC address by this format XX:XX:XX:XX:XX:XX
219 **MUST** be entered.
220 -J, --server_ip
222 server ip address by this format X.X.X.X (using to send
223 packets with IP header).
224 -j, --client_ip
226 client ip address by this format X.X.X.X (using to send
227 packets with IP header).
228 -K, --server_port
230 server udp port number (using to send packets with UPD
231 header).
232 -k, --client_port
234 client udp port number (using to send packets with UDP
235 header).
236 -Z, --server
238 choose server side for the current machine
239 (--server/--client must be selected ).
240 -P, --client
242 choose client side for the current machine
243 (--server/--client must be selected).
244
246 Prerequisites:
247 kernel 2.6
248 (kernel module) matches libibverbs
249 (kernel module) matches librdmacm
250 (kernel module) matches libibumad
251 (kernel module) matches libmath (lm).
252
254 You need to be running a Subnet Manager on the switch or on one of the
255 nodes in your fabric, in case you are in IB fabric.
256
258 1. Multicast feauture in ib_send_lat and in ib_send_bw still have many
259 problems! Will increase the support and bug fixes in this Q, but now
260 the tests may stuck and could produce undefine behaviours.
261 2. Bidirectional feature in ib_send_bw test, when running in UD or UC
263 mode. The algorithm we use for the bidirectional measurement is
264 designed for RC connection type. When running in UC or UD connection
265 types, there is a small probablity the test will be stuck.
266 3. RDMA_CM feature in read tests still doesn't work.
268 4. Dual-port support currently works only with ib_write_bw.
270 5. Compabilty issues may occur between different versions of perftest.
272 Please make sure you work with the same version on both sides to ensure
273 consistency of the test.
274
276 Please post results/observations to the openib-general mailing list.
277 See "Contact Us" at http://openib.org/mailman/listinfo/openib-general
278 and http://www.openib.org.
279Open Fabrics Enterprise Distribution 2014 IB_ATOMIC_BW(1)