1ethhostadmin(8) EFSFFCLIRG (Man Page) ethhostadmin(8)
3
7 ethhostadmin
8 Performs a number of multi-step host initialization and verification
12 operations, including upgrading software, rebooting hosts, and other
13 operations. In general, operations performed by ethhostadmin involve a
14 login to one or more host systems.
15
17 ethhostadmin [-c] [-f hostfile] [-h 'hosts']
18 [-r release] [-I install_options] [-U upgrade_options] [-d dir]
19 [-T product] [-P packages] [-S] operation ...
20
22 --help
23 Produces full help text.
25 -c
28 Overwrites the result files from any previous run before
30 starting this run.
31 -f hostfile
34 Specifies the file with the names of hosts in a cluster. De‐
36 fault is /etc/eth-tools/hosts file.
37 -h hosts
40 Specifies the list of hosts to execute the operation against.
42 -r release
45 Specifies the software version to load/upgrade to. Default is
47 the version of Intel(R) Ethernet Fabric Suite Software pres‐
48 ently being run on the server.
49 -d dir
52 Specifies the directory to retrieve product. release.tgz for
54 load or upgrade.
55 -I install_options
58 Specifies the software install options.
60 -U upgrade_options
63 Specifies the software upgrade options.
65 -T product
68 Specifies the product type to install. Options include:
70 • IntelEth-Basic. <distro> (default)
74 • IntelEth-FS. <distro>
76 where <distro> is the distribution and CPU, such as
80 RHEL81-x86_64.
81 -P packages
83 Specifies the packages to install. Default is eth eth_rdma.
85 Refer to INSTALL -C for compete list of packages.
86 -S
89 Securely prompts for user password on remote system.
91 operation
94 Performs the specified operation, which can be one or more of
96 the following:
97 load Starts initial installation of all hosts.
102 upgrade Upgrades installation of all hosts.
108 reboot Reboots hosts, ensures they go down and come back.
114 rping Verifies this host can ping each host through RDMA.
120 pfctest Verifies PFC works on all hosts.
126 mpiperf Verifies latency and bandwidth for each host.
132 mpiperfdeviation
138 Verifies latency and bandwidth for each host
139 against a defined threshold (or relative to average
140 host performance).
141
145 ethhostadmin -c reboot
146 ethhostadmin upgrade
148 ethhostadmin -h 'elrond arwen' reboot
150 HOSTS='elrond arwen' ethhostadmin reboot
152
154 ethhostadmin provides detailed logging of its results. During each run,
155 the following files are produced:
156 • test.res : Appended with summary results of run.
158 • test.log : Appended with detailed results of run.
160 • save_tmp/ : Contains a directory per failed test with detailed
162 logs.
163 • test_tmp*/ : Intermediate result files while test is running.
165 The -c option removes all log files.
167 Results from ethhostadmin are grouped into test suites, test cases, and
169 test items. A given run of ethhostadmin represents a single test suite.
170 Within a test suite, multiple test cases occur; typically one test case
171 per host being operated on. Some of the more complex operations may
172 have multiple test items per test case. Each test item represents a ma‐
173 jor step in the overall test case.
174 Each ethhostadmin run appends to test.res and test.log, and creates
176 temporary files in test_tmp$PID in the current directory. test.res pro‐
177 vides an overall summary of operations performed and their results. The
178 same information is also displayed while ethhostadmin is executing.
179 test.log contains detailed information about what was performed, in‐
180 cluding the specific commands executed and the resulting output. The
181 test_tmp directories contain temporary files which reflect tests in
182 progress (or killed). The logs for any failures are logged in the
183 save_temp directory with a directory per failed test case. If the same
184 test case fails more than once, save_temp retains the information from
185 the first failure. Subsequent runs of ethhostadmin are appended to
186 test.log. Intel recommends reviewing failures and using the -c option
187 to remove old logs before subsequent runs of ethhostadmin.
188 ethhostadmin implicitly performs its operations in parallel. However,
190 as for the other tools, FF_MAX_PARALLEL can be exported to change the
191 degree of parallelism. 1000 parallel operations is the default.
192
194 The following environment variables are also used by this command:
195 HOSTS
197 List of hosts, used if -h option not supplied.
199 HOSTS_FILE
202 File containing list of hosts, used in absence of -f and -h.
204 FF_MAX_PARALLEL
207 Maximum concurrent operations are performed.
209 FF_SERIALIZE_OUTPUT
212 Serialize output of parallel operations (yes or no).
214 FF_TIMEOUT_MULT
217 Multiplier for all timeouts associated with this command.
219 Used if the systems are slow for some reason.
220
224 (Host) Intel recommends that you set up password SSH or SCP for use
225 during this operation. Alternatively, the -S option can be used to se‐
226 curely prompt for a password, in which case the same password is used
227 for all hosts. Alternately, the password may be put in the environment
228 or the ethfastfabric.conf file using FF_PASSWORD and FF_ROOTPASS.
229 load
231 Performs an initial installation of Intel(R) Ethernet Fabric
233 Suite Software on a group of hosts. Any existing installation
234 is uninstalled and existing configuration files are removed.
235 Subsequently, the hosts are installed with a default Intel(R)
236 Ethernet Fabric Suite Software configuration. The -I option
237 can be used to select different install packages. Default is
238 eth_tools eth_rdma mpi The -r option can be used to specify a
239 release to install other than the one that this host is pres‐
240 ently running. The FF_PRODUCT. FF_PRODUCT_VERSION.tgz file
241 (for example, IntelEth-Basic. version.tgz) is expected to ex‐
242 ist in the directory specified by -d. Default is the current
243 working directory. The specified software is copied to all
244 the selected hosts and installed.
245 upgrade
248 Upgrades all selected hosts without modifying existing con‐
250 figurations. This operation is comparable to the -U option
251 when running ./INSTALL manually. The -r option can be used to
252 upgrade to a release different from this host. The default is
253 to upgrade to the same release as this host. The FF_PRODUCT.
254 FF_PRODUCT_VERSION.tgz file (for example, IntelEth-Basic.
255 version.tgz) is expected to exist in the directory specified
256 by -d. The default is the current working directory. The
257 specified software is copied to all the end nodes and in‐
258 stalled.
259 NOTE: Only components that are currently installed are upgraded. This
264 operation fails for hosts that do not have Intel(R) Ethernet Fabric
265 Suite Software installed.
266 reboot
270 Reboots the given hosts and ensures they go down and come
272 back up by pinging them during the reboot process. The ping
273 rate is slow (5 seconds), so if the servers boot faster than
274 this, false failures may be seen.
275 rping
278 Verifies RDMA basic operation by ensuring that the nodes can
280 ping each other through RDMA. To run this command, Intel(R)
281 Ethernet Fabric software must be installed, RDMA must be con‐
282 figured and running on the host, and the given hosts, and
283 switches must be up.
284 pfctest
287 Empirical test which verifies PFC is working right. To run
289 this command, Intel(R) Ethernet Fabric software must be in‐
290 stalled, PFC must be configured on both hosts and switches,
291 and the given hosts and switches must be up.
292 mpiperf
295 Verifies that MPI is operational and checks MPI end-to-end
297 latency and bandwidth between pairs of nodes (for example,
298 1-2, 3-4, 5-6). Use this to verify switch latency/hops, PCI
299 bandwidth, and overall MPI performance. The test.res file
300 contains the results of each pair of nodes tested.
301 NOTE: This option is available for the Intel(R) Ethernet Host Software
307 OFA Delta packaging, but is not presently available for other packag‐
308 ings of OFED.
309 To obtain accurate results, this test should be run at a time
313 when no other stressful applications (for example, MPI jobs or
314 high stress file system operations) are running on the given
315 hosts.
316 Bandwidth issues typically indicate server configuration issues
318 (for example, incorrect slot used, incorrect BIOS settings, or
319 incorrect NIC model), or fabric issues (for example, symbol er‐
320 rors, incorrect link width, or speed). Assuming ethreport has
321 previously been used to check for link errors and link speed is‐
322 sues, the server configuration should be verified.
323 Note that BIOS settings and differences between server models
325 can account for 10-20% differences in bandwidth. For more de‐
326 tails about BIOS settings, consult the documentation from the
327 server supplier and/or the server PCI chipset manufacturer.
328 mpiperfdeviation
330 Specifies the enhanced version of mpiperf that verifies MPI
332 performance. Can be used to verify switch latency/hops, PCI
333 bandwidth, and overall MPI performance. It performs assorted
334 pair-wise bandwidth and latency tests, and reports pairs out‐
335 side an acceptable tolerance range. The tool identifies spe‐
336 cific nodes that have problems and provides a concise summary
337 of results. The test.res file contains the results of each
338 pair of nodes tested.
339 By default, concurrent mode is used to quickly analyze the
342 fabric and host performance. Pairs that have 20% less band‐
343 width or 50% more latency than the average pair are reported
344 as failures.
345 The tool can be run in a sequential or a concurrent mode. Se‐
347 quential mode runs each host against a reference host. By de‐
348 fault, the reference host is selected based on the best per‐
349 formance from a quick test of the first 40 hosts. In concur‐
350 rent mode, hosts are paired up and all pairs are run concur‐
351 rently. Since there may be fabric contention during such a
352 run, any poor performing pairs are then rerun sequentially
353 against the reference host.
354 Concurrent mode runs the tests in the shortest amount of
356 time, however, the results could be slightly less accurate
357 due to switch contention. In heavily oversubscribed fabric
358 designs, if concurrent mode is producing unexpectedly low
359 performance, try sequential mode.
360 NOTE: This option is available for the Intel(R) Ethernet Host Software
365 OFA Delta packaging, but is not presently available for other packag‐
366 ings of OFED.
367 To obtain accurate results, this test should be run at a time
371 when no other stressful applications (for example, MPI jobs,
372 high stress file system operations) are running on the given
373 hosts.
374 Bandwidth issues typically indicate server configuration issues
376 (for example, incorrect slot used, incorrect BIOS settings, or
377 incorrect NIC model), or fabric issues (for example, symbol er‐
378 rors, incorrect link width, or speed). Assuming ethreport has
379 previously been used to check for link errors and link speed is‐
380 sues, the server configuration should be verified.
381 Note that BIOS settings and differences between server models
383 can account for 10-20% differences in bandwidth. A result 5-10%
384 below the average is typically not cause for serious alarm, but
385 may reflect limitations in the server design or the chosen BIOS
386 settings.
387 For more details about BIOS settings, consult the documentation
389 from the server supplier and/or the server PCI chipset manufac‐
390 turer.
391 The deviation application supports a number of parameters which
393 allow for more precise control over the mode, benchmark and
394 pass/fail criteria. The parameters to use can be selected using
395 the FF_DEVIATION_ARGS configuration parameter in ethfastfab‐
396 ric.conf
397 Available parameters for deviation application:
399 [-bwtol bwtol] [-bwdelta MBs] [-bwthres MBs]
401 [-bwloop count] [-bwsize size] [-lattol latol]
403 [-latdelta usec] [-latthres usec] [-latloop count]
405 [-latsize size][-c] [-b] [-v] [-vv]
407 [-h reference_host]
409 -bwtol Specifies the percent of bandwidth degradation allowed
415 below average value.
416 -bwbidir Performs a bidirectional bandwidth test.
422 -bwunidir Performs a unidirectional bandwidth test (Default).
428 -bwdelta Specifies the limit in MB/s of bandwidth degradation
434 allowed below average value.
435 -bwthres Specifies the lower limit in MB/s of bandwidth al‐
441 lowed.
442 -bwloop Specifies the number of loops to execute each band‐
448 width test.
449 -bwsize Specifies the size of message to use for bandwidth
455 test.
456 -lattol Specifies the percent of latency degradation allowed
462 above average value.
463 -latdelta Specifies the imit in µsec of latency degradation
469 allowed above average value.
470 -latthres Specifies the lower limit in µsec of latency al‐
476 lowed.
477 -latloop Specifies the number of loops to execute each latency
483 test.
484 -latsize Specifies the size of message to use for latency test.
490 -c Runs test pairs concurrently instead of the default of
496 sequential.
497 -b When comparing results against tolerance and delta,
503 uses best instead of average.
504 -v Specifies the verbose output.
510 -vv Specifies the very verbose output.
516 -h Specifies the reference host to use for sequential
522 pairing.
523 Both bwtol and bwdelta must be exceeded to fail bandwidth test.
527 When bwthres is supplied, bwtol and bwdelta are ignored.
529 Both lattol and latdelta must be exceeded to fail latency test.
531 When latthres is supplied, lattol and latdelta are ignored.
533 For consistency with OSU benchmarks, MB/s is defined as 1000000
535 bytes/s.
536Copyright(C) 2020-2021 Intel Corporation ethhostadmin(8)