1PMLOGEXTRACT(1) General Commands Manual PMLOGEXTRACT(1)
2
6 pmlogextract - reduce, extract, concatenate and merge Performance Co-
7 Pilot archives
8
10 pmlogextract [-dfwz] [-c configfile] [-S starttime] [-s samples] [-T
11 endtime] [-v volsamples] [-Z timezone] input [...] output
12
14 pmlogextract reads one or more Performance Co-Pilot (PCP) archive logs
15 identified by input and creates a temporally merged and/or reduced PCP
16 archive log in output. The nature of merging is controlled by the num‐
17 ber of input archive logs, while the nature of data reduction is con‐
18 trolled by the command line arguments. The input(s) must be PCP ar‐
19 chive logs created by pmlogger(1) with performance data collected from
20 the same host, but usually over different time periods and possibly
21 (although not usually) with different performance metrics being logged.
22 If only one input is specified, then the default behavior simply copies
24 the input PCP archive log, into the output PCP archive log. When two
25 or more PCP archive logs are specified as input, the logs are merged
26 (or concatenated) and written to output.
27 In the output archive log a ``mark'' record will be inserted at a time
29 just past the end of each of the input archive logs to indicate a pos‐
30 sible temporal discontinuity between the end of one input archive log
31 and the start of the next input archive log. See the MARK RECORDS sec‐
32 tion below for more information. There is no ``mark'' record after the
33 end of the last (in temporal order) of the input archive logs.
34
36 The command line options for pmlogextract are as follows:
37 -c configfile
39 Extract only the metrics specified in configfile from the input
40 PCP archive log(s). The configfile syntax accepted by pmlogex‐
41 tract is explained in more detail in the Configuration File Syn‐
42 tax section.
43 -d Desperate mode. Normally if a fatal error occurs, all trace of
45 the partially written PCP archive output is removed. With the
46 -d option, the output archive log is not removed.
47 -f For most common uses, all of the input archive logs will have
49 been collected in the same timezone. But if this is not the
50 case, then pmlogextract must choose one of the timezones from
51 the input archive logs to be used as the timezone for the output
52 archive log. The default is to use the timezone from the last
53 input archive log. The -f option forces the timezone from the
54 first input archive log to be used.
55 -S starttime
57 Define the start of a time window to restrict the samples
58 retrieved or specify a ``natural'' alignment of the output sam‐
59 ple times; refer to PCPIntro(1). See also the -w option.
60 -s samples
62 The argument samples defines the number of samples to be written
63 to output. If samples is 0 or -s is not specified, pmlogextract
64 will sample until the end of the PCP archive log, or the end of
65 the time window as specified by -T, whichever comes first. The
66 -s option will override the -T option if it occurs sooner.
67 -T endtime
69 Define the termination of a time window to restrict the samples
70 retrieved or specify a ``natural'' alignment of the output sam‐
71 ple times; refer to PCPIntro(1). See also the -w option.
72 -v volsamples
74 The output archive log is potentially a multi-volume data set,
75 and the -v option causes pmlogextract to start a new volume
76 after volsamples log records have been written to the archive
77 log.
78 Independent of any -v option, each volume of an archive is lim‐
80 ited to no more than 2^31 bytes, so pmlogextract will automati‐
81 cally create a new volume for the archive before this limit is
82 reached.
83 -w Where -S and -T specify a time window within the same day, the
85 -w flag will cause the data within the time window to be
86 extracted, for every day in the archive log. For example, the
87 options -w -S @11:00 -T @15:00 specify that pmlogextract should
88 include archive log records only for the periods from 11am to
89 3pm on each day. When -w is used, the output archive log will
90 contain ``mark'' records to indicate the temporal discontinuity
91 between the end of one time window and the start of the next.
92 -Z timezone
94 Use timezone when displaying the date and time. Timezone is in
95 the format of the environment variable TZ as described in envi‐
96 ron(7).
97 -z Use the local timezone of the host from the input archive logs.
99 The default is to initially use the timezone of the local host.
100
102 The configfile contains metrics of interest - only those metrics (or
103 instances) mentioned explicitly or implicitly in the configuration file
104 will be included in the output archive. Each specifications must begin
105 on a new line, and may span multiple lines in the configuration file.
106 Instances may also be specified, but they are optional. The format for
107 each specification is
108 metric [[instance[,instance...]]]
110 where metric may be a leaf or a non-leaf name in the Performance Met‐
112 rics Name Space (PMNS, see pmns(5)). If a metric refers to a non-leaf
113 node in the PMNS, pmlogextract will recursively descend the PMNS and
114 include all metrics corresponding to descendent leaf nodes.
115 Instances are optional, and may be specified as a list of one or more
117 space (or comma) separated names, numbers or strings (enclosed in sin‐
118 gle or double quotes). Elements in the list that are numbers are
119 assumed to be internal instance identifiers - see pmGetInDom(3) for
120 more information. If no instances are given, then all instances of the
121 associated metric(s) will be extracted.
122 Any additional white space is ignored and comments may be added with a
124 `#' prefix.
125
127 This is an example of a valid configfile:
128 #
130 # config file for pmlogextract
131 #
132 kernel.all.cpu
134 kernel.percpu.cpu.sys ["cpu0","cpu1"]
135 disk.dev ["dks0d1"]
136
138 When more than one input archive log contributes performance data to
139 the output archive log, then ``mark'' records are inserted to indicate
140 a possible discontinuity in the performance data.
141 A ``mark'' record contains a timestamp and no performance data and is
143 used to indicate that there is a time period in the PCP archive log
144 where we do not know the values of any performance metrics, because
145 there was no pmlogger(1) collecting performance data during this
146 period. Since these periods are often associated with the restart of a
147 service or pmcd(1) or a system, there may be considerable doubt as to
148 the continuity of performance data across this time period.
149 The rationale behind ``mark'' records may be demonstrated with an exam‐
151 ple. Consider one input archive log that starts at 00:10 and ends at
152 09:15 on the same day, and another input archive log that starts at
153 09:20 on the same day and ends at 00:10 the following morning. The
154 would be a very common case for archives managed and rotated by pmlog‐
155 ger_check(1) and pmlogger_daily(1).
156 The output archive log would contain:
158 00:10.000 first record from first input archive log
159 ...
160 09:15.000 last record from first input archive log
161 09:15.001 <mark record>
162 09:20.000 first record from second input archive log
163 ...
164 01:10.000 last record from second input archive log
165 The time period where the performance data is missing starts just after
167 09:15 and ends just before 09:20. When the output archive log is pro‐
168 cessed with any of the PCP reporting tools, the ``mark'' record is used
169 to indicate a period of missing data. For example in the archive
170 above, if one was reporting the average I/O rate at 30 minute inter‐
171 vals, aligned on the hour, then there would be data for the intervals
172 ending at 09:00 and 10:00 but no data reported for the interval ending
173 at 09:30 as this spans a ``mark'' record.
174 The presence of ``mark'' records in a PCP archive log can be estab‐
176 lished using pmdumplog(1) where a timestamp and the annotation <mark>
177 is used to indicate a ``mark'' record.
178
180 When more than one input archive is specified, pmlogextract performs a
181 number of checks to ensure the metadata is consistent for metrics
182 appearing in more than one of the input archives. These checks
183 include:
184 * metric data type is the same
186 * metric semantics are the same
187 * metric units are the same
188 * metric is either always singular or always has the same instance
189 domain
190 * metrics with the same name have the same PMID
191 * metrics with the same PMID have the same name
192 If any of these checks fail, pmlogextract reports the details and
194 aborts without creating the output archive.
195 To address these semantic issues, use pmlogrewrite(1) to translate the
197 input archives into equivalent archives with consistent metdadata
198 before using pmlogextract.
199
201 For each of the input and output archive logs, several physical files
202 are used.
203 archive.meta
204 metadata (metric descriptions, instance domains, etc.) for
205 the archive log
206 archive.0 initial volume of metrics values (subsequent volumes have
207 suffixes 1, 2, ...) - for input these files may have been
208 previously compressed with bzip2(1) or gzip(1) and thus may
209 have an additional .bz2 or .gz suffix.
210 archive.index
211 temporal index to support rapid random access to the other
212 files in the archive log.
213
215 Environment variables with the prefix PCP_ are used to parameterize the
216 file and directory names used by PCP. On each installation, the file
217 /etc/pcp.conf contains the local values for these variables. The
218 $PCP_CONF variable may be used to specify an alternative configuration
219 file, as described in pcp.conf(5).
220
222 PCPIntro(1), pmdumplog(1), pmlc(1), pmlogger(1), pmlogreduce(1), pmlo‐
223 grewrite(1), pcp.conf(5) and pcp.env(5).
224
226 All error conditions detected by pmlogextract are reported on stderr
227 with textual (if sometimes terse) explanation.
228 Should one of the input archive logs be corrupted (this can happen if
230 the pmlogger instance writing the log suddenly dies), then pmlogextract
231 will detect and report the position of the corruption in the file, and
232 any subsequent information from that archive log will not be processed.
233 If any error is detected, pmlogextract will exit with a non-zero sta‐
235 tus.
236
238 The preamble metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host, and
239 pmcd.pmlogger.port), which are automatically recorded by pmlogger at
240 the start of the archive, may not be present in the archive output by
241 pmlogextract. These metrics are only relevant while the archive is
242 being created, and have no significance once recording has finished.
243Performance Co-Pilot PCP PMLOGEXTRACT(1)