1ethhostadmin(8) EFSFFCLIRG (Man Page) ethhostadmin(8)
3
7 ethhostadmin
8 Performs a number of multi-step host initialization and verification
12 operations, including upgrading software, rebooting hosts, and other
13 operations. In general, operations performed by ethhostadmin involve a
14 login to one or more host systems.
15
17 ethhostadmin [-c] [-e] [-f hostfile] [-h 'hosts']
18 [-r release] [-I install_options] [-U upgrade_options] [-d dir]
19 [-T product] [-P packages] [-S] operation ...
20
22 --help
23 Produces full help text.
25 -c
28 Overwrites the result files from any previous run before
30 starting this run.
31 -e
34 exit after 1st operation which fails.
36 -f hostfile
39 Specifies the file with the names of hosts in a cluster. De‐
41 fault is /etc/eth-tools/hosts file.
42 -h hosts
45 Specifies the list of hosts to execute the operation against.
47 -r release
50 Specifies the software version to load/upgrade to. Default is
52 the version of Intel(R) Ethernet Fabric Suite Software pres‐
53 ently being run on the server.
54 -d dir
57 Specifies the directory to retrieve product. release.tgz for
59 load or upgrade.
60 -I install_options
63 Specifies the software install options.
65 -U upgrade_options
68 Specifies the software upgrade options.
70 -T product
73 Specifies the product type to install. Options include:
75 • IntelEth-Basic. <distro> (default)
79 • IntelEth-FS. <distro>
81 where <distro> is the distribution and CPU, such as
85 RHEL81-x86_64.
86 -P packages
88 Specifies the packages to install. Default is eth eth_rdma.
90 Refer to INSTALL -C for compete list of packages.
91 -S
94 Securely prompts for user password on remote system.
96 operation
99 Performs the specified operation, which can be one or more of
101 the following:
102 load Starts initial installation of all hosts.
107 upgrade Upgrades installation of all hosts.
113 reboot Reboots hosts, ensures they go down and come back.
119 rping Verifies this host can ping each host through RDMA.
125 pfctest Verifies PFC works on all hosts.
131 mpiperf Verifies latency and bandwidth for each host.
137 mpiperfdeviation
143 Verifies latency and bandwidth for each host
144 against a defined threshold (or relative to average
145 host performance).
146
150 ethhostadmin -c reboot
151 ethhostadmin upgrade
153 ethhostadmin -h 'elrond arwen' reboot
155 HOSTS='elrond arwen' ethhostadmin reboot
157
159 ethhostadmin provides detailed logging of its results. During each run,
160 the following files are produced:
161 • test.res : Appended with summary results of run.
163 • test.log : Appended with detailed results of run.
165 • save_tmp/ : Contains a directory per failed test with detailed
167 logs.
168 • test_tmp*/ : Intermediate result files while test is running.
170 The -c option removes all log files.
172 Results from ethhostadmin are grouped into test suites, test cases, and
174 test items. A given run of ethhostadmin represents a single test suite.
175 Within a test suite, multiple test cases occur; typically one test case
176 per host being operated on. Some of the more complex operations may
177 have multiple test items per test case. Each test item represents a ma‐
178 jor step in the overall test case.
179 Each ethhostadmin run appends to test.res and test.log, and creates
181 temporary files in test_tmp$PID in the current directory. test.res pro‐
182 vides an overall summary of operations performed and their results. The
183 same information is also displayed while ethhostadmin is executing.
184 test.log contains detailed information about what was performed, in‐
185 cluding the specific commands executed and the resulting output. The
186 test_tmp directories contain temporary files which reflect tests in
187 progress (or killed). The logs for any failures are logged in the
188 save_temp directory with a directory per failed test case. If the same
189 test case fails more than once, save_temp retains the information from
190 the first failure. Subsequent runs of ethhostadmin are appended to
191 test.log. Intel recommends reviewing failures and using the -c option
192 to remove old logs before subsequent runs of ethhostadmin.
193 ethhostadmin implicitly performs its operations in parallel. However,
195 as for the other tools, FF_MAX_PARALLEL can be exported to change the
196 degree of parallelism. 1000 parallel operations is the default.
197
199 The following environment variables are also used by this command:
200 HOSTS
202 List of hosts, used if -h option not supplied.
204 HOSTS_FILE
207 File containing list of hosts, used in absence of -f and -h.
209 FF_MAX_PARALLEL
212 Maximum concurrent operations are performed.
214 FF_SERIALIZE_OUTPUT
217 Serialize output of parallel operations (yes or no).
219 FF_TIMEOUT_MULT
222 Multiplier for all timeouts associated with this command.
224 Used if the systems are slow for some reason.
225
229 (Host) Intel recommends that you set up password SSH or SCP for use
230 during this operation. Alternatively, the -S option can be used to se‐
231 curely prompt for a password, in which case the same password is used
232 for all hosts. Alternately, the password may be put in the environment
233 or the ethfastfabric.conf file using FF_PASSWORD and FF_ROOTPASS.
234 load
236 Performs an initial installation of Intel(R) Ethernet Fabric
238 Suite Software on a group of hosts. Any existing installation
239 is uninstalled and existing configuration files are removed.
240 Subsequently, the hosts are installed with a default Intel(R)
241 Ethernet Fabric Suite Software configuration. The -I option
242 can be used to select different install packages. Default is
243 eth_tools eth_rdma mpi The -r option can be used to specify a
244 release to install other than the one that this host is pres‐
245 ently running. The FF_PRODUCT. FF_PRODUCT_VERSION.tgz file
246 (for example, IntelEth-Basic. version.tgz) is expected to ex‐
247 ist in the directory specified by -d. Default is the current
248 working directory. The specified software is copied to all
249 the selected hosts and installed.
250 upgrade
253 Upgrades all selected hosts without modifying existing con‐
255 figurations. This operation is comparable to the -U option
256 when running ./INSTALL manually. The -r option can be used to
257 upgrade to a release different from this host. The default is
258 to upgrade to the same release as this host. The FF_PRODUCT.
259 FF_PRODUCT_VERSION.tgz file (for example, IntelEth-Basic.
260 version.tgz) is expected to exist in the directory specified
261 by -d. The default is the current working directory. The
262 specified software is copied to all the end nodes and in‐
263 stalled.
264 NOTE: Only components that are currently installed are upgraded. This
269 operation fails for hosts that do not have Intel(R) Ethernet Fabric
270 Suite Software installed.
271 reboot
275 Reboots the given hosts and ensures they go down and come
277 back up by pinging them during the reboot process. The ping
278 rate is slow (5 seconds), so if the servers boot faster than
279 this, false failures may be seen.
280 rping
283 Verifies RDMA basic operation by ensuring that the nodes can
285 ping each other through RDMA. To run this command, Intel(R)
286 Ethernet Fabric software must be installed, RDMA must be con‐
287 figured and running on the host, and the given hosts, and
288 switches must be up.
289 pfctest
292 Empirical test which verifies PFC is working right. To run
294 this command, Intel(R) Ethernet Fabric software must be in‐
295 stalled, PFC must be configured on both hosts and switches,
296 and the given hosts and switches must be up.
297 mpiperf
300 Verifies that MPI is operational and checks MPI end-to-end
302 latency and bandwidth between pairs of nodes (for example,
303 1-2, 3-4, 5-6). Use this to verify switch latency/hops, PCI
304 bandwidth, and overall MPI performance. The test.res file
305 contains the results of each pair of nodes tested.
306 NOTE: This option is available for the Intel(R) Ethernet Host Software
312 OFA Delta packaging, but is not presently available for other packag‐
313 ings of OFED.
314 To obtain accurate results, this test should be run at a time
318 when no other stressful applications (for example, MPI jobs or
319 high stress file system operations) are running on the given
320 hosts.
321 Bandwidth issues typically indicate server configuration issues
323 (for example, incorrect slot used, incorrect BIOS settings, or
324 incorrect NIC model), or fabric issues (for example, symbol er‐
325 rors, incorrect link width, or speed). Assuming ethreport has
326 previously been used to check for link errors and link speed is‐
327 sues, the server configuration should be verified.
328 Note that BIOS settings and differences between server models
330 can account for 10-20% differences in bandwidth. For more de‐
331 tails about BIOS settings, consult the documentation from the
332 server supplier and/or the server PCI chipset manufacturer.
333 mpiperfdeviation
335 Specifies the enhanced version of mpiperf that verifies MPI
337 performance. Can be used to verify switch latency/hops, PCI
338 bandwidth, and overall MPI performance. It performs assorted
339 pair-wise bandwidth and latency tests, and reports pairs out‐
340 side an acceptable tolerance range. The tool identifies spe‐
341 cific nodes that have problems and provides a concise summary
342 of results. The test.res file contains the results of each
343 pair of nodes tested.
344 By default, concurrent mode is used to quickly analyze the
347 fabric and host performance. Pairs that have 20% less band‐
348 width or 50% more latency than the average pair are reported
349 as failures.
350 The tool can be run in a sequential or a concurrent mode. Se‐
352 quential mode runs each host against a reference host. By de‐
353 fault, the reference host is selected based on the best per‐
354 formance from a quick test of the first 40 hosts. In concur‐
355 rent mode, hosts are paired up and all pairs are run concur‐
356 rently. Since there may be fabric contention during such a
357 run, any poor performing pairs are then rerun sequentially
358 against the reference host.
359 Concurrent mode runs the tests in the shortest amount of
361 time, however, the results could be slightly less accurate
362 due to switch contention. In heavily oversubscribed fabric
363 designs, if concurrent mode is producing unexpectedly low
364 performance, try sequential mode.
365 NOTE: This option is available for the Intel(R) Ethernet Host Software
370 OFA Delta packaging, but is not presently available for other packag‐
371 ings of OFED.
372 To obtain accurate results, this test should be run at a time
376 when no other stressful applications (for example, MPI jobs,
377 high stress file system operations) are running on the given
378 hosts.
379 Bandwidth issues typically indicate server configuration issues
381 (for example, incorrect slot used, incorrect BIOS settings, or
382 incorrect NIC model), or fabric issues (for example, symbol er‐
383 rors, incorrect link width, or speed). Assuming ethreport has
384 previously been used to check for link errors and link speed is‐
385 sues, the server configuration should be verified.
386 Note that BIOS settings and differences between server models
388 can account for 10-20% differences in bandwidth. A result 5-10%
389 below the average is typically not cause for serious alarm, but
390 may reflect limitations in the server design or the chosen BIOS
391 settings.
392 For more details about BIOS settings, consult the documentation
394 from the server supplier and/or the server PCI chipset manufac‐
395 turer.
396 The deviation application supports a number of parameters which
398 allow for more precise control over the mode, benchmark and
399 pass/fail criteria. The parameters to use can be selected using
400 the FF_DEVIATION_ARGS configuration parameter in ethfastfab‐
401 ric.conf
402 Available parameters for deviation application:
404 [-bwtol bwtol] [-bwdelta MBs] [-bwthres MBs]
406 [-bwloop count] [-bwsize size] [-lattol latol]
408 [-latdelta usec] [-latthres usec] [-latloop count]
410 [-latsize size][-c] [-b] [-v] [-vv]
412 [-h reference_host]
414 -bwtol Specifies the percent of bandwidth degradation allowed
420 below average value.
421 -bwbidir Performs a bidirectional bandwidth test.
427 -bwunidir Performs a unidirectional bandwidth test (Default).
433 -bwdelta Specifies the limit in MB/s of bandwidth degradation
439 allowed below average value.
440 -bwthres Specifies the lower limit in MB/s of bandwidth al‐
446 lowed.
447 -bwloop Specifies the number of loops to execute each band‐
453 width test.
454 -bwsize Specifies the size of message to use for bandwidth
460 test.
461 -lattol Specifies the percent of latency degradation allowed
467 above average value.
468 -latdelta Specifies the imit in µsec of latency degradation
474 allowed above average value.
475 -latthres Specifies the lower limit in µsec of latency al‐
481 lowed.
482 -latloop Specifies the number of loops to execute each latency
488 test.
489 -latsize Specifies the size of message to use for latency test.
495 -c Runs test pairs concurrently instead of the default of
501 sequential.
502 -b When comparing results against tolerance and delta,
508 uses best instead of average.
509 -v Specifies the verbose output.
515 -vv Specifies the very verbose output.
521 -h Specifies the reference host to use for sequential
527 pairing.
528 Both bwtol and bwdelta must be exceeded to fail bandwidth test.
532 When bwthres is supplied, bwtol and bwdelta are ignored.
534 Both lattol and latdelta must be exceeded to fail latency test.
536 When latthres is supplied, lattol and latdelta are ignored.
538 For consistency with OSU benchmarks, MB/s is defined as 1000000
540 bytes/s.
541Copyright(C) 2020-2022 Intel Corporation ethhostadmin(8)