1netpipe(1) netpipe netpipe(1)
2
6 NetPIPE - Network Protocol Independent Performance Evaluator
7
10 NPtcp [-h receiver_hostname] [-b TCP_buffer_sizes] [options]
11 mpirun [-machinefile hostlist] -np 2 NPmpi [-a] [-S] [-z] [options]
14 mpirun [-machinefile hostlist] -np 2 NPmpi2 [-f] [-g] [options]
17 NPpvm [options]
21 See the TESTING sections below for a more complete description of how
23 to run NetPIPE in each environment. The OPTIONS section describes the
24 general options available for all modules. See the README file from
25 the tar-ball at http://www.bitspjoule.org/Projects/NetPIPE/ for docu‐
26 mentation on the InfiniBand, GM, SHMEM, LAPI, and memcpy modules.
27
30 NetPIPE uses a simple series of ping-pong tests over a range of message
31 sizes to provide a complete measure of the performance of a network.
32 It bounces messages of increasing size between two processes, whether
33 across a network or within an SMP system. Message sizes are chosen at
34 regular intervals, and with slight perturbations, to provide a complete
35 evaluation of the communication system. Each data point involves many
36 ping-pong tests to provide an accurate timing. Latencies are calcu‐
37 lated by dividing the round trip time in half for small messages ( less
38 than 64 Bytes ).
39 The communication time for small messages is dominated by the overhead
41 in the communication layers, meaning that the transmission is latency
42 bound. For larger messages, the communication rate becomes bandwidth
43 limited by some component in the communication subsystem (PCI bus, net‐
44 work card link, network switch).
45 These measurements can be done at the message-passing layer (MPI,
47 MPI-2, and PVM) or at the native communications layers that that run
48 upon (TCP/IP, GM for Myrinet cards, InfiniBand, SHMEM for the Cray T3E
49 systems, and LAPI for IBM SP systems). Recent work is being aimed at
50 measuring some internal system properties such as the memcpy module
51 that measures the internal memory copy rates, or a disk module under
52 development that measures the performance to various I/O devices.
53 Some uses for NetPIPE include:
55 Comparing the latency and maximum throughput of various network
57 cards.
58 Comparing the performance between different types of networks.
60 Looking for inefficiencies in the message-passing layer by com‐
62 paring it to the native communication layer.
63 Optimizing the message-passing layer and tune OS and driver
65 parameters for optimal performance of the communication subsys‐
66 tem.
67 NetPIPE is provided with many modules allowing it to interface with a
70 wide variety of communication layers. It is fairly easy to write new
71 interfaces for other reliable protocols by using the existing modules
72 as examples.
73
78 NPtcp can now be launched in two ways, by manually starting NPtcp on
79 both systems or by using a nplaunch script. To manually start NPtcp,
80 the NetPIPE receiver must be started first on the remote system using
81 the command:
82 NPtcp [options]
84 then the primary transmitter is started on the local system with the
86 command
87 NPtcp -h receiver_hostname [options]
89 Any options used must be the same on both sides. The -P parameter can
91 be used to override the default port number. This is helpful when run‐
92 ning several streams through a router to a single endpoint.
93 The nplaunch script uses ssh to launch the remote receiver before
95 starting the local transmitter. To use rsh, simply change the nplaunch
96 script.
97 nplaunch NPtcp -h receiver_hostname [options]
99 The -b TCP_buffer_sizes option sets the TCP socket buffer size, which
101 can greatly influence the maximum throughput on some systems. A
102 throughput graph that flattens out suddenly may be a sign of the per‐
103 formance being limited by the socket buffer sizes.
104 Several other protocols are testable in the same way as TCP. These
106 include TCP6 (TCP over IPv6), SCTP and IPX. They are started in the
107 same way but the program names are NPtcp6, NPsctp, and NPipx respec‐
108 tively.
109
113 Use of the MPI interface for NetPIPE depends on the MPI implementation
114 being used. All will require the number of processes to be specified,
115 usually with a -np 2 argument. Clusters environments may require a
116 list of the hosts being used, either during initialization of MPI (dur‐
117 ing lamboot for LAM-MPI) or when each job is run (using a -machinefile
118 argument for MPICH). For LAM-MPI, for example, put the list of hosts
119 in hostlist then boot LAM and run NetPIPE using:
120 lamboot -v -b hostlist
122 mpirun -np 2 NPmpi [NetPIPE options]
124 For MPICH use a command like:
126 mpirun -machinefile hostlist -np 2 NPmpi [NetPIPE options]
128 To test the 1-sided communications of the MPI-2 standard, compile
130 using:
131 make mpi2
133 Running as described above and MPI will use 1-sided MPI_Put() calls in
135 both directions, with each receiver blocking until the last byte has
136 been overwritten before bouncing the message back. Use the -f option
137 to force usage of a fence to block rather than an overwrite of the last
138 byte. The -g option will use MP_Get() functions to transfer the data
139 rather than MP_Put().
140
144 Start the pvm system using:
145 pvm
147 and adding a second machine with the PVM command
149 add receiver_hostname
151 Exit the PVM command line interface using quit, then run the PVM Net‐
153 PIPE receiver on one system with the command:
154 NPpvm [options]
156 and run the TCP NetPIPE transmitter on the other system with the com‐
158 mand:
159 NPpvm -h receiver hostname [options]
161 Any options used must be the same on both sides. The nplaunch script
163 may also be used with NPpvm as described above for NPtcp.
164
167 NetPIPE tests network performance by sending a number of messages at
168 each block size, starting from the lower bound on the message sizes.
169 The message size is incremented until the upper bound on the message
171 size is reached or the time to transmit a block exceeds one second,
172 which ever occurs first. Message sizes are chosen at regular inter‐
173 vals, and for slight perturbations from them to provide a more complete
174 evaluation of the communication subsystem.
175 The NetPIPE output file may be graphed using a program such as gnu‐
177 plot(1). The output file contains three columns: the number of bytes
178 in the block, the transfer rate in bits per second, and the time to
179 transfer the block (half the round-trip time). The first two columns
180 are normally used to graph the throughput vs block size, while the
181 third column provides the latency. For example, the throughput versus
182 block size graph can be created by graphing bytes versus bits per sec‐
183 ond. Sample gnuplot(1) commands for such a graph would be
184 set logscale x
186 plot "np.out"
188
191 -a asynchronous mode: prepost receives (MPI, IB modules)
192 -b TCP_buffer_sizes
194 Set the send and receive TCP buffer sizes (TCP module only).
195 -B Burst mode where all receives are preposted at once (MPI, IB
198 modules).
199 -f Use a fence to block for completion (MPI2 module only).
202 -g Use MPI_Get() instead of MPI_Put() (MPI2 module only).
205 -h hostname
208 Specify the name of the receiver host to connect to (TCP, PVM,
209 IB, GM).
210 -I Invalidate cache to measure performance without cache effects
213 (mostly affects IB and memcpy modules).
214 -i Do an integrity check instead of a performance evaluation.
217 -l starting_msg_size
220 Specify the lower bound for the size of messages to be tested.
221 -n nrepeats
225 Set the number of repeats for each test to a constant.
226 Otherwise, the number of repeats is chosen to provide an
227 accurate timing for each test. Be very careful if speci‐
228 fying a low number so that the time for the ping-pong
229 test exceeds the timer accuracy.
230 -O source_offset,dest_offset
233 Specify the source and destination offsets of the buffers
234 from perfect page alignment.
235 -o output_filename
238 Specify the output filename (default is np.out).
239 -p perturbation_size
242 NetPIPE chooses the message sizes at regular intervals,
243 increasing them exponentially from the lower boundary to
244 the upper boundary. At each point, it also tests pertur‐
245 bations of 3 bytes above and 3 bytes below each test
246 point to find idiosyncrasies in the system. This pertur‐
247 bation value can be changed using the -p option, or
248 turned off using -p 0 .
249 -r This option resets the TCP sockets after every test (TCP
252 module only). It is necessary for some streaming tests
253 to get good measurements since the socket window size may
254 otherwise collapse.
255 -s Set streaming mode where data is only transmitted in one
258 direction.
259 -S Use synchronous sends (MPI module only).
262 -u upper_bound
265 Specify the upper boundary to the size of message being
266 tested. By default, NetPIPE will stop when the time to
267 transmit a block exceeds one second.
268 -z Receive messages using MPI_ANY_SOURCE (MPI module only)
271 -2 Set bi-directional mode where both sides send and receive
274 at the same time (supported by most modules). You may
275 need to use -a to choose asynchronous communications for
276 MPI to avoid freeze-ups. For TCP, the maximum test size
277 will be limited by the TCP buffer sizes.
278
281 np.out Default output file for NetPIPE. Overridden by the -o
282 option.
283
286 The original NetPIPE core plus TCP and MPI modules were written
287 by Quinn Snell, Armin Mikler, Guy Helmer, and John Gustafson.
288 NetPIPE is currently being developed and maintained by Dave
289 Turner with contributions from many students (Bogdan Vasiliu,
290 Adam Oline, Xuehua Chen, and Brian Smith).
291 Send comments/bug-reports to: <netpipe@bitspjoule.org>.
294 Additional information about NetPIPE can be found on the World
296 Wide Web at http://www.bitspjoule.org/Projects/NetPIPE/
297
300 As of version 3.6.1, there is a bug that causes NetPIPE to seg‐
301 fault on RedHat Enterprise systems. I will debug this as soon as
302 I get access to a few such systems. -Dave Turner (turner@ames‐
303 lab.gov)
304NetPIPE June 1, 2004 netpipe(1)