1PMLOGEXTRACT(1) General Commands Manual PMLOGEXTRACT(1)
2
6 pmlogextract - reduce, extract, concatenate and merge Performance Co-
7 Pilot archives
8
10 pmlogextract [-dfmwxz?] [-c configfile] [-S starttime] [-s samples]
11 [-T endtime] [-V version] [-v volsamples] [-Z timezone] input [...]
12 output
13
15 pmlogextract reads one or more Performance Co-Pilot (PCP) archives
16 identified by input and creates a merged and/or reduced PCP archive in
17 output. Each input argument is either a name or a comma-separated list
18 of names, and each name is the name of one file from an archive or the
19 base name of an archive or the name of a directory containing one or
20 more archives. The nature of merging is controlled by the number of
21 input archives, while the nature of data reduction is controlled by the
22 command line arguments. The input arguments must be archives created
23 by pmlogger(1) with performance data collected from the same host, but
24 usually over different time periods and possibly (although not usually)
25 with different performance metrics being logged.
26 If only one input is specified, then the default behavior simply copies
28 the input PCP archive (with possible conversion to a newer version of
29 the archive format, see -V below), into the output PCP archive. When
30 two or more PCP archives are specified as input, the archives are
31 merged (or concatenated) and written to output.
32 In the output archive a <mark> record may be inserted at a time just
34 past the end of each of the input archive to indicate a possible tempo‐
35 ral discontinuity between the end of one input archive and the start of
36 the next input archive. See the MARK RECORDS section below for more
37 information. There is no <mark> record after the end of the last (in
38 temporal order) of the records from the input archive(s).
39
41 The available command line options are:
42 -c config, --config=config
44 Extract only the metrics specified in config from the input PCP
45 archive(s). The config syntax accepted by pmlogextract is ex‐
46 plained in more detail in the CONFIGURATION FILE SYNTAX section.
47 -d, --desperate
49 Desperate mode. Normally if a fatal error occurs, all trace of
50 the partially written PCP archive output is removed. With the -d
51 option, the output archive is not removed.
52 -f, --first
54 For most common uses, all of the input archives will have been
55 collected in the same timezone. But if this is not the case, then
56 pmlogextract must choose one of the timezones from the input ar‐
57 chives to be used as the timezone for the output archive. The de‐
58 fault is to use the timezone from the last input archive. The -f
59 option forces the timezone from the first input archive to be
60 used.
61 -m, --mark
63 As described in the MARK RECORDS section below, sometimes it is
64 possible to safely omit <mark> records from the output archive.
65 If the -m option is specified, then the epilogue and prologue test
66 is skipped and a <mark> record will always be inserted at the end
67 of each input archive (except the last). This is the original be‐
68 haviour for pmlogextract.
69 -S starttime, --start=starttime
71 Define the start of a time window to restrict the records pro‐
72 cessed; refer to PCPIntro(1). See also the -w option.
73 -s samples, --samples=samples
75 The argument samples defines the number of samples (or records) to
76 be written to output. If samples is 0 or -s is not specified, pm‐
77 logextract will continue until the end of all the input archives
78 or until the end of the time window as specified by -T, whichever
79 comes first. The -s option will override the -T option if it oc‐
80 curs sooner.
81 -T endtime, --finish=endtime
83 Define the end of a time window to restrict the records processed;
84 refer to PCPIntro(1). See also the -w option.
85 -V version, --outputversion=version
87 Each PCP archive has a version for the physical record format,
88 currently 2 or 3. By default, the output archive is created with
89 a version equal to the maximum of the version of the input ar‐
90 chives. The -V option may be used to explicitly force the version
91 for output, provided version is no smaller than the archive ver‐
92 sion that would have been chosen by the default rule.
93 For example, specifying -V 3 may be used to produce a version 3
95 output archive from input archives that could be a mixture of ver‐
96 sion 2 and/or version 3.
97 -v volsamples
99 The output archive is potentially a multi-volume data set, and the
100 -v option causes pmlogextract to start a new volume after volsam‐
101 ples records have been written to the archive.
102 Independent of any -v option, each volume of an archive is limited
104 to no more than 2^31 bytes, so pmlogextract will automatically
105 create a new volume for the archive before this limit is reached.
106 -w Where -S and -T specify a time window within the same day, the -w
108 flag will cause the data within the time window to be extracted,
109 for every day in the archive. For example, the options -w -S
110 @11:00 -T @15:00 specify that pmlogextract should include archive
111 records only for the periods from 11am to 3pm on each day. When
112 -w is used, the output archive will contain <mark> records to in‐
113 dicate the temporal discontinuity between the end of one time win‐
114 dow and the start of the next.
115 -x It is expected that the metadata (name, PMID, type, semantics and
117 units) for each metric will be consistent across all of the input
118 PCP archive(s) in which that metric appears. In rare cases, e.g.
119 in development, in QA and when a PMDA is upgraded, this may not be
120 the case and pmlogextract will report the issue and abort without
121 creating the output archive. This is done so the problem can be
122 fixed with pmlogrewrite(1) before retrying the merge. In unat‐
123 tended or QA environments it may be preferable to force the merge
124 and omit the metrics with the mismatched metadata. The -x option
125 does this.
126 -Z timezone, --timezone=timezone
128 Use timezone when displaying the date and time in diagnostics.
129 Timezone is in the format of the environment variable TZ as de‐
130 scribed in environ(7). The default is to initially use the time‐
131 zone of the local host.
132 -z, --hostzone
134 Use the local timezone of the host from the input archive(s) when
135 displaying the date and time in diagnostics. The default is to
136 initially use the timezone of the local host.
137 -?, --help
139 Display usage message and exit.
140
142 The configfile contains metrics of interest - only those metrics (or
143 instances) mentioned explicitly or implicitly in the configuration file
144 will be included in the output archive. Each specification must begin
145 on a new line, and may span multiple lines in the configuration file.
146 Instances may also be specified, but they are optional. The format for
147 each specification is
148 metric
150 or
151 metric [ instance ... ]
152 where metric may be a leaf or a non-leaf name of a metric in the Per‐
154 formance Metrics Name Space (PMNS, see PMNS(5)). If a metric refers to
155 a non-leaf node in the PMNS, pmlogextract will recursively descend the
156 PMNS and include all metrics corresponding to descendent leaf nodes.
157 Instances are optional and are specified as a list space (or comma)
159 separated of instance identifiers, with the list enclosed by square
160 brackets. Each instance identifier may be a number or a string (en‐
161 closed in single or double quotes). instance identifiers that are num‐
162 bers are assumed to be internal instance identifiers, else the string
163 values are assumed to be external instance identifiers; see
164 pmGetInDom(3) for more information. If no instances are given, then
165 all instances of the associated metric(s) will be extracted.
166 Any additional white space is ignored and comments may be added with a
168 `#' prefix.
169
171 This is an example of a valid configfile:
172 #
174 # config file for pmlogextract
175 #
176 kernel.all.cpu
178 kernel.percpu.cpu.sys ["cpu0","cpu1"]
179 disk.dev ["dks0d1"]
180
182 When more than one input archive contributes performance data to the
183 output archive, then <mark> records may be inserted to indicate a pos‐
184 sible temporal discontinuity in the performance data.
185 A <mark> record contains a timestamp and no performance data and is
187 used to indicate that there is a time period in the PCP archive where
188 we do not know the values of any performance metrics, because there was
189 no pmlogger(1) collecting performance data during this period. Since
190 these periods are often associated with the restart of a service or
191 pmcd(1) or a system reboot, there may be considerable doubt as to the
192 continuity of performance data across this time period.
193 Most current archives are created with a prologue record at the begin‐
195 ning and an epilogue record at the end. These records identify the
196 state of pmcd(1) at the time, and may be used by pmlogextract to deter‐
197 mine that there is no discontinuity between the end of one archive and
198 the next output record, and as a consequence the <mark> record can
199 safely be omitted from the output archive.
200 The rationale behind <mark> records may be demonstrated with an exam‐
202 ple. Consider one input archive that starts at 00:10 and ends at 09:15
203 on the same day, and another input archive that starts at 09:20 on the
204 same day and ends at 00:10 the following morning. This would be a very
205 common case for archives managed and rotated by pmlogger_check(1) and
206 pmlogger_daily(1).
207 The output archive created by pmlogextract would contain:
209 00:10.000 first record from first input archive
210 ...
211 09:15.000 last record from first input archive
212 09:15.001 <mark> record
213 09:20.000 first record from second input archive
214 ...
215 01:10.000 last record from second input archive
216 The time period where the performance data is missing starts just after
218 09:15 and ends just before 09:20. When the output archive is processed
219 with any of the PCP reporting tools, the <mark> record is used to indi‐
220 cate a period of missing data. For example using the output archive
221 above, imagine one was reporting the average I/O rate at 30 minute in‐
222 tervals aligned on the hour and half-hour. The I/O count metric is a
223 counter, so the average I/O rate requires two valid values from consec‐
224 utive sample times. There would be values for all the intervals ending
225 at 09:00, then no values at 09:30 because of the <mark> record, then no
226 values at 10:00 because the ``prior'' value at 09:30 is not available,
227 then the rate would be reported again at 10:30 and continue every 30
228 minutes until the last reported value at 01:00.
229 The presence of <mark> records in a PCP archive can be established us‐
231 ing pmdumplog(1) where a timestamp and the annotation <mark> is used to
232 indicate a <mark> record.
233
235 When more than one input archive is specified, pmlogextract performs a
236 number of checks to ensure the metadata is consistent for metrics ap‐
237 pearing in more than one of the input archives. These checks include:
238 * metric data type is the same
240 * metric semantics are the same
241 * metric units are the same
242 * metric is either always singular or always has the same instance do‐
243 main
244 * metrics with the same name have the same PMID
245 * metrics with the same PMID have the same name
246 If any of these checks fail, pmlogextract reports the details and
248 aborts without creating the output archive.
249 To address these semantic issues, use pmlogrewrite(1) to translate the
251 input archives into equivalent archives with consistent metadata before
252 using pmlogextract.
253 Refer to the -x and -d command line options above for alternatives to
255 the default handling of errors during metadata checks.
256
258 The prologue metrics (pmcd.pmlogger.archive, pmcd.pmlogger.host, and
259 pmcd.pmlogger.port), which are automatically recorded by pmlogger at
260 the start of the archive, may not be present in the archive output by
261 pmlogextract. These metrics are only relevant while the archive is be‐
262 ing created, and have no significance once recording has finished.
263
265 All error conditions detected by pmlogextract are reported on stderr
266 with textual (if sometimes terse) explanation.
267 If one of the input archives contains no archive records then an
269 ``empty archive'' warning is issued and that archive is skipped.
270 Should one of the input archive(s) be corrupted (this can happen if the
272 pmlogger instance writing the archive suddenly dies), then pmlogextract
273 will detect and report the position of the corruption in the file, and
274 any subsequent information from that archive will not be processed.
275 If any error is detected, pmlogextract will exit with a non-zero sta‐
277 tus.
278
280 For each of the input and output archive, several physical files are
281 used.
282 archive.meta
284 metadata (metric descriptions, instance domains, etc.) for the ar‐
285 chive
286 archive.0
288 initial volume of metrics values (subsequent volumes have suffixes
289 1, 2, ...) - for input these files may have been previously com‐
290 pressed with bzip2(1) or gzip(1) and thus may have an additional
291 .bz2 or .gz suffix.
292 archive.index
294 temporal index to support rapid random access to the other files
295 in the archive.
296
298 Environment variables with the prefix PCP_ are used to parameterize the
299 file and directory names used by PCP. On each installation, the file
300 /etc/pcp.conf contains the local values for these variables. The
301 $PCP_CONF variable may be used to specify an alternative configuration
302 file, as described in pcp.conf(5).
303 For environment variables affecting PCP tools, see pmGetOptions(3).
305
307 PCPIntro(1), pmdumplog(1), pmlc(1), pmlogger(1), pmlogreduce(1), pmlo‐
308 grewrite(1), pcp.conf(5), pcp.env(5) and PMNS(5).
309Performance Co-Pilot PCP PMLOGEXTRACT(1)