1scout(1) Scalasca Trace Tools scout(1)
2
6 scout - Scalasca parallel trace analyzer
7
9 scout.ser [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
10 scout.omp [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
11 scout.mpi [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
12 scout.hyb [OPTION] (ANCHOR_FILE | EXPERIMENT_DIR)
13
15 scout is the scalable automatic trace-analysis component of the
16 Scalasca Trace Tools. In particular, it provides the ability to
17 • identify wait states in communication and synchronization operations
19 that occur, for example, as a result of unevenly distributed
20 workloads,
21 • pinpoint the root causes of those wait states (i.e., delays), and
23 • identify the activities on the critical path of the target
25 application, highlighting those routines which determine the length
26 of the program execution and therefore constitute the best candidates
27 for optimization.
28 Usually, scout is launched automatically by the Scalasca measurement
30 collection and analysis nexus scan(1) after a successful measurement if
31 event tracing is configured. However, it can also be run manually on an
32 existing event trace measurement.
33 scout currently supports trace experiments in two different event trace
35 formats: OTF2 traces generated by the Score-P instrumentation and
36 measurement system, and legacy traces in EPILOG format generated by the
37 measurement system of the Scalasca 1.x release series. For OTF2 event
38 traces, scout has to be provided with the corresponding ANCHOR_FILE
39 (e.g., 'traces.otf2'), for EPILOG traces with the experiment directory
40 name EXPERIMENT_DIR.
41 Depending on the build configuration and the capabilities of the target
43 platform, the scout analyzer may be available in up to four forms:
44 scout.ser
46 is always built. It is used to analyze event traces generated by
47 serial applications. It can also be used to analyze event traces
48 from multi-threaded applications, however, it will then only
49 provide information about the master thread.
50 scout.omp
52 is built whenever the Scalasca Trace Tools are configured with
53 OpenMP support. It is used to analyze event traces generated by
54 pure multi-threaded applications (e.g., using OpenMP or POSIX
55 threads). It can also be used to analyze event traces from serial
56 applications, though analysis incurs a higher overhead than using
57 scout.ser.
58 scout.mpi
60 is built whenever the Scalasca Trace Tools are configured with MPI
61 support. It is used to analyze event traces generated by pure MPI
62 applications. It can also be used on traces from multi-threaded MPI
63 applications, however, it will then only provide information about
64 the master thread of each process and its MPI activities.
65 scout.hyb
67 is built if the Scalasca Trace Tools are configured with both MPI
68 and OpenMP support. It is used to analyze event traces generated by
69 multi-threaded MPI applications (e.g., MPI+OpenMP or MPI+Pthreads),
70 providing information about all OpenMP/POSIX threads of each MPI
71 process. It can also be used on traces from pure MPI applications,
72 though analysis incurs a slightly higher overhead than using
73 scout.mpi.
74 Note that scout.mpi and scout.hyb are implemented as MPI programs, and
76 therefore have to be executed using appropriate MPI launch commands and
77 flags. Also, the number of MPI processes for scout must be identical to
78 the number of MPI processes used for the target application execution.
79 If successful, scout produces the following output files in the
81 measurement archive directory:
82 • scout.cubex: the trace analysis result
84 • trace.stat: trace analysis pattern statistics
86
88 scout accepts a number of command-line options to enable/disable
89 particular analysis features. When scout is launched automatically from
90 the Scalasca measurement collection and analysis nexus scan(1), these
91 options can be passed to the analyzer via the SCAN_ANALYZE_OPTS
92 environment variable.
93 Common options
95 --statistics
96 Enables most-severe instance tracking and wait-state statistics.
97 This is the default.
98 --no-statistics
100 Disables most-severe instance tracking and wait-state statistics.
101 --critical-path
103 Enables critical-path analysis. This is the default.
104 --no-critical-path
106 Disables critical-path analysis.
107 --rootcause
109 Enables root-cause analysis. This is the default.
110 --no-rootcause
112 Disables root-cause analysis.
113 --single-pass
115 Use single-pass forward analysis only. Disables both critical-path
116 and root-cause analysis, as well as the detection of Late Receiver
117 wait states.
118 -v, --verbose
120 Increase verbosity.
121 --help
123 Print a brief usage summary, then exit.
124 MPI options (scout.mpi/scout.hyb only)
126 --time-correct
127 Enables enhanced timestamp correction. Event traces collected on
128 clusters without a synchronized clock may contain logical clock
129 condition violations (such as a receive completing before the
130 corresponding send is initiated). When scout detects such
131 situations, it issues a warning that the analysis may be
132 inconsistent and recommends (re-)running trace analysis with its
133 integrated timestamp correction algorithm activated.
134 --no-time-correct
136 Disables enhanced timestamp correction. This is the default.
137
139 scout exits with status 0 if automatic trace analysis was successful,
140 and greater than 0 if errors occur.
141
143 scout poses a number of requirements on the input event trace data,
144 which are documented in the OPEN_ISSUES file installed as part of the
145 Scalasca Trace Tools documentation. It is also available online at
146 https://www.scalasca.org. If those requirements are not met, scout may
147 abort, deadlock, or crash.
148 If scout crashes or deadlocks even though the documented requirements
150 are met (which usually indicates a bug), restricting the scope of the
151 analysis by disabling certain features (e.g., critical-path and/or
152 root-cause analysis) may help as a workaround. In any case, please
153 report such issues for further investigation (see BUGS below).
154
156 Please report bugs to scalasca@fz-juelich.de. Make sure to include at
157 least the following information in your bug report:
158 • The Scalasca Trace Tools version reported by 'scalasca -V'.
160 • The Scalasca Trace Tools configuration reported by 'scalasca -c'.
162 • The Score-P version reported by 'scorep --version'.
164 • The Score-P configuration reported by 'scorep-info config-summary'.
166 • The exact command line of the failing command.
168 • The exact failure/error message.
170 Also, please archive a copy of the entire experiment archive directory
172 including the event trace data, as this may be required to aid in
173 debugging. However, ONLY PROVIDE TRACE DATA IF EXPLICITLY REQUESTED, as
174 the data volume may be excessive.
175
177 scout.omp scorep_foo_Ox4_trace/traces.otf2
178 Perform the Scalasca OpenMP event trace analysis on the OTF2 event
179 trace with anchor file scorep_foo_Ox4_trace/traces.otf2.
180 mpiexec -n 16 scout.mpi --time-correct scorep_bar_16_trace/traces.otf2
182 Apply the enhanced timestamp correction and perform the Scalasca MPI
183 event trace analysis on the OTF2 event trace with anchor file
184 scorep_bar_16_trace/traces.otf.
185 mpiexec -n 4 scout.hyb epik_foobar_4x4_trace
187 Perform the hybrid Scalasca MPI+OpenMP event trace analysis on the
188 EPILOG event trace in the experiment archive epik_foobar_4x4_trace
189 generated by the Scalasca 1.x release series.
190
192 scalasca(1), scan(1), square(1)
193 The full Scalasca Trace Tools documentation is available online at
195 https://www.scalasca.org.
196Version 2.6 Mon Apr 19 2021 scout(1)