1ethhostadmin(8) EFSFFCLIRG (Man Page) ethhostadmin(8)
3
7 ethhostadmin
8 Performs a number of multi-step host initialization and verification
12 operations, including upgrading software, rebooting hosts, and other
13 operations. In general, operations performed by ethhostadmin involve a
14 login to one or more host systems.
15
17 ethhostadmin [-c] [-f hostfile] [-h 'hosts']
18 [-r release] [-I install_options] [-U upgrade_options] [-d dir]
19 [-T product] [-P packages] [-S] operation ...
20
22 --help
23 Produces full help text.
25 -c
28 Overwrites the result files from any previous run before
30 starting this run.
31 -f hostfile
34 Specifies the file with the names of hosts in a cluster. De‐
36 fault is /etc/eth-tools/hosts file.
37 -h hosts
40 Specifies the list of hosts to execute the operation against.
42 -r release
45 Specifies the software version to load/upgrade to. Default is
47 the version of Intel(R) Ethernet Fabric Suite Software pres‐
48 ently being run on the server.
49 -d dir
52 Specifies the directory to retrieve product. release.tgz for
54 load or upgrade.
55 -I install_options
58 Specifies the software install options.
60 -U upgrade_options
63 Specifies the software upgrade options.
65 -T product
68 Specifies the product type to install. Options include:
70 • IntelEth-Basic. <distro> (default)
74 • IntelEth-FS. <distro>
76 where <distro> is the distribution and CPU, such as
80 RHEL81-x86_64.
81 -P packages
83 Specifies the packages to install. Default is eth_tools
85 eth_rdma mpi. Refer to INSTALL -C for compete list of pack‐
86 ages.
87 -S
90 Securely prompts for user password on remote system.
92 operation
95 Performs the specified operation, which can be one or more of
97 the following:
98 load Starts initial installation of all hosts.
103 upgrade Upgrades installation of all hosts.
109 reboot Reboots hosts, ensures they go down and come back.
115 rping Verifies this host can ping each host through RDMA.
121 mpiperf Verifies latency and bandwidth for each host.
127 mpiperfdeviation
133 Verifies latency and bandwidth for each host
134 against a defined threshold (or relative to average
135 host performance).
136
140 ethhostadmin -c reboot
141 ethhostadmin upgrade
143 ethhostadmin -h 'elrond arwen' reboot
145 HOSTS='elrond arwen' ethhostadmin reboot
147
149 ethhostadmin provides detailed logging of its results. During each run,
150 the following files are produced:
151 • test.res : Appended with summary results of run.
153 • test.log : Appended with detailed results of run.
155 • save_tmp/ : Contains a directory per failed test with detailed
157 logs.
158 • test_tmp*/ : Intermediate result files while test is running.
160 The -c option removes all log files.
162 Results from ethhostadmin are grouped into test suites, test cases, and
164 test items. A given run of ethhostadmin represents a single test suite.
165 Within a test suite, multiple test cases occur; typically one test case
166 per host being operated on. Some of the more complex operations may
167 have multiple test items per test case. Each test item represents a ma‐
168 jor step in the overall test case.
169 Each ethhostadmin run appends to test.res and test.log, and creates
171 temporary files in test_tmp$PID in the current directory. test.res pro‐
172 vides an overall summary of operations performed and their results. The
173 same information is also displayed while ethhostadmin is executing.
174 test.log contains detailed information about what was performed, in‐
175 cluding the specific commands executed and the resulting output. The
176 test_tmp directories contain temporary files which reflect tests in
177 progress (or killed). The logs for any failures are logged in the
178 save_temp directory with a directory per failed test case. If the same
179 test case fails more than once, save_temp retains the information from
180 the first failure. Subsequent runs of ethhostadmin are appended to
181 test.log. Intel recommends reviewing failures and using the -c option
182 to remove old logs before subsequent runs of ethhostadmin.
183 ethhostadmin implicitly performs its operations in parallel. However,
185 as for the other tools, FF_MAX_PARALLEL can be exported to change the
186 degree of parallelism. 1000 parallel operations is the default.
187
189 The following environment variables are also used by this command:
190 HOSTS
192 List of hosts, used if -h option not supplied.
194 HOSTS_FILE
197 File containing list of hosts, used in absence of -f and -h.
199 FF_MAX_PARALLEL
202 Maximum concurrent operations are performed.
204 FF_SERIALIZE_OUTPUT
207 Serialize output of parallel operations (yes or no).
209 FF_TIMEOUT_MULT
212 Multiplier for all timeouts associated with this command.
214 Used if the systems are slow for some reason.
215
219 (Host) Intel recommends that you set up password SSH or SCP for use
220 during this operation. Alternatively, the -S option can be used to se‐
221 curely prompt for a password, in which case the same password is used
222 for all hosts. Alternately, the password may be put in the environment
223 or the ethfastfabric.conf file using FF_PASSWORD and FF_ROOTPASS.
224 load
226 Performs an initial installation of Intel(R) Ethernet Fabric
228 Suite Software on a group of hosts. Any existing installation
229 is uninstalled and existing configuration files are removed.
230 Subsequently, the hosts are installed with a default Intel(R)
231 Ethernet Fabric Suite Software configuration. The -I option
232 can be used to select different install packages. Default is
233 eth_tools eth_rdma mpi The -r option can be used to specify a
234 release to install other than the one that this host is pres‐
235 ently running. The FF_PRODUCT. FF_PRODUCT_VERSION.tgz file
236 (for example, IntelEth-Basic. version.tgz) is expected to ex‐
237 ist in the directory specified by -d. Default is the current
238 working directory. The specified software is copied to all
239 the selected hosts and installed.
240 upgrade
243 Upgrades all selected hosts without modifying existing con‐
245 figurations. This operation is comparable to the -U option
246 when running ./INSTALL manually. The -r option can be used to
247 upgrade to a release different from this host. The default is
248 to upgrade to the same release as this host. The FF_PRODUCT.
249 FF_PRODUCT_VERSION.tgz file (for example, IntelEth-Basic.
250 version.tgz) is expected to exist in the directory specified
251 by -d. The default is the current working directory. The
252 specified software is copied to all the end nodes and in‐
253 stalled.
254 NOTE: Only components that are currently installed are upgraded. This
259 operation fails for hosts that do not have Intel(R) Ethernet Fabric
260 Suite Software installed.
261 reboot
265 Reboots the given hosts and ensures they go down and come
267 back up by pinging them during the reboot process. The ping
268 rate is slow (5 seconds), so if the servers boot faster than
269 this, false failures may be seen.
270 rping
273 Verifies RDMA basic operation by ensuring that the nodes can
275 ping each other through RDMA. To run this command, Intel(R)
276 Ethernet Fabric software must be installed, RDMA must be con‐
277 figured and running on the host, and the given hosts, and
278 switches must be up.
279 mpiperf
282 Verifies that MPI is operational and checks MPI end-to-end
284 latency and bandwidth between pairs of nodes (for example,
285 1-2, 3-4, 5-6). Use this to verify switch latency/hops, PCI
286 bandwidth, and overall MPI performance. The test.res file
287 contains the results of each pair of nodes tested.
288 NOTE: This option is available for the Intel(R) Ethernet Host Software
294 OFA Delta packaging, but is not presently available for other packag‐
295 ings of OFED.
296 To obtain accurate results, this test should be run at a time
300 when no other stressful applications (for example, MPI jobs or
301 high stress file system operations) are running on the given
302 hosts.
303 Bandwidth issues typically indicate server configuration issues
305 (for example, incorrect slot used, incorrect BIOS settings, or
306 incorrect NIC model), or fabric issues (for example, symbol er‐
307 rors, incorrect link width, or speed). Assuming ethreport has
308 previously been used to check for link errors and link speed is‐
309 sues, the server configuration should be verified.
310 Note that BIOS settings and differences between server models
312 can account for 10-20% differences in bandwidth. For more de‐
313 tails about BIOS settings, consult the documentation from the
314 server supplier and/or the server PCI chipset manufacturer.
315 mpiperfdeviation
317 Specifies the enhanced version of mpiperf that verifies MPI
319 performance. Can be used to verify switch latency/hops, PCI
320 bandwidth, and overall MPI performance. It performs assorted
321 pair-wise bandwidth and latency tests, and reports pairs out‐
322 side an acceptable tolerance range. The tool identifies spe‐
323 cific nodes that have problems and provides a concise summary
324 of results. The test.res file contains the results of each
325 pair of nodes tested.
326 By default, concurrent mode is used to quickly analyze the
329 fabric and host performance. Pairs that have 20% less band‐
330 width or 50% more latency than the average pair are reported
331 as failures.
332 The tool can be run in a sequential or a concurrent mode. Se‐
334 quential mode runs each host against a reference host. By de‐
335 fault, the reference host is selected based on the best per‐
336 formance from a quick test of the first 40 hosts. In concur‐
337 rent mode, hosts are paired up and all pairs are run concur‐
338 rently. Since there may be fabric contention during such a
339 run, any poor performing pairs are then rerun sequentially
340 against the reference host.
341 Concurrent mode runs the tests in the shortest amount of
343 time, however, the results could be slightly less accurate
344 due to switch contention. In heavily oversubscribed fabric
345 designs, if concurrent mode is producing unexpectedly low
346 performance, try sequential mode.
347 NOTE: This option is available for the Intel(R) Ethernet Host Software
352 OFA Delta packaging, but is not presently available for other packag‐
353 ings of OFED.
354 To obtain accurate results, this test should be run at a time
358 when no other stressful applications (for example, MPI jobs,
359 high stress file system operations) are running on the given
360 hosts.
361 Bandwidth issues typically indicate server configuration issues
363 (for example, incorrect slot used, incorrect BIOS settings, or
364 incorrect NIC model), or fabric issues (for example, symbol er‐
365 rors, incorrect link width, or speed). Assuming ethreport has
366 previously been used to check for link errors and link speed is‐
367 sues, the server configuration should be verified.
368 Note that BIOS settings and differences between server models
370 can account for 10-20% differences in bandwidth. A result 5-10%
371 below the average is typically not cause for serious alarm, but
372 may reflect limitations in the server design or the chosen BIOS
373 settings.
374 For more details about BIOS settings, consult the documentation
376 from the server supplier and/or the server PCI chipset manufac‐
377 turer.
378 The deviation application supports a number of parameters which
380 allow for more precise control over the mode, benchmark and
381 pass/fail criteria. The parameters to use can be selected using
382 the FF_DEVIATION_ARGS configuration parameter in ethfastfab‐
383 ric.conf
384 Available parameters for deviation application:
386 [-bwtol bwtol] [-bwdelta MBs] [-bwthres MBs]
388 [-bwloop count] [-bwsize size] [-lattol latol]
390 [-latdelta usec] [-latthres usec] [-latloop count]
392 [-latsize size][-c] [-b] [-v] [-vv]
394 [-h reference_host]
396 -bwtol Specifies the percent of bandwidth degradation allowed
402 below average value.
403 -bwbidir Performs a bidirectional bandwidth test.
409 -bwunidir Performs a unidirectional bandwidth test (Default).
415 -bwdelta Specifies the limit in MB/s of bandwidth degradation
421 allowed below average value.
422 -bwthres Specifies the lower limit in MB/s of bandwidth al‐
428 lowed.
429 -bwloop Specifies the number of loops to execute each band‐
435 width test.
436 -bwsize Specifies the size of message to use for bandwidth
442 test.
443 -lattol Specifies the percent of latency degradation allowed
449 above average value.
450 -latdelta Specifies the imit in µsec of latency degradation
456 allowed above average value.
457 -latthres Specifies the lower limit in µsec of latency al‐
463 lowed.
464 -latloop Specifies the number of loops to execute each latency
470 test.
471 -latsize Specifies the size of message to use for latency test.
477 -c Runs test pairs concurrently instead of the default of
483 sequential.
484 -b When comparing results against tolerance and delta,
490 uses best instead of average.
491 -v Specifies the verbose output.
497 -vv Specifies the very verbose output.
503 -h Specifies the reference host to use for sequential
509 pairing.
510 Both bwtol and bwdelta must be exceeded to fail bandwidth test.
514 When bwthres is supplied, bwtol and bwdelta are ignored.
516 Both lattol and latdelta must be exceeded to fail latency test.
518 When latthres is supplied, lattol and latdelta are ignored.
520 For consistency with OSU benchmarks, MB/s is defined as 1000000
522 bytes/s.
523Copyright(C) 2020 Intel Corporation ethhostadmin(8)