1PMLOGREWRITE(1) General Commands Manual PMLOGREWRITE(1)
2
6 pmlogrewrite - rewrite Performance Co-Pilot archives
7
9 $PCP_BINADM_DIR/pmlogrewrite [-Cdiqsvw ] [-c config] inlog [outlog]
10
12 pmlogrewrite reads a Performance Co-Pilot (PCP) archive log identified
13 by inlog and creates a PCP archive log in outlog. Under normal usage,
14 the -c option will be used to nominate a configuration file or files
15 that contains specifications (see the REWRITING RULES SYNTAX section
16 below) that describe how the data and metadata from inlog should be
17 transformed to produce outlog.
18 The typical uses for pmlogrewrite would be to accommodate the evolution
20 of Performance Metric Domain Agents (PMDAs) where the names, metadata
21 and semantics of metrics and their associated instance domains may
22 change over time, e.g. promoting the type of a metric from a 32-bit to
23 a 64-bit integer, or renaming a group of metrics. Refer to the EXAM‐
24 PLES section for some additional use cases.
25 pmlogrewrite is most useful where PMDA changes, or errors in the pro‐
27 duction environment, result in archives that cannot be combined with
28 pmlogextract(1). By pre-processing the archives with pmlogrewrite the
29 resulting archives may be able to be merged with pmlogextract(1).
30 The input inlog must be a PCP archive log created by pmlogger(1), or
32 possibly one of the tools that read and create PCP archives, e.g.
33 pmlogextract(1) and pmlogreduce(1).
34 If no -c option is specified, then the default behavior simply creates
36 outlog as a copy of inlog. This is a little more complicated than
37 cat(1), as each PCP archive is made up of several physical files.
38 While pmlogrewrite may be used to repair some data consistency issues
40 in PCP archives, there is also a class of repair tasks that cannot be
41 handled by pmlogrewrite and pmloglabel(1) may be a useful tool in these
42 cases.
43
45 The command line options for pmlogrewrite are as follows:
46 -C Parse the rewriting rules and quit. outlog is not created.
48 When -C is specified, this also sets -v and -w so that all warn‐
49 ings and verbose messages are displayed as config is parsed.
50 -c config
52 If config is a file or symbolic link, read and parse rewriting
53 rules from there. If config is a directory, then all of the
54 files or symbolic links in that directory (excluding those
55 beginning with a period ``.'') will be used to provide the
56 rewriting rules. Multiple -c options are allowed.
57 -d Desperate mode. Normally if a fatal error occurs, all trace of
59 the partially written PCP archive outlog is removed. With the
60 -d option, the partially created outlog archive log is not
61 removed.
62 -i Rather than creating outlog, inlog is rewritten in place when
64 the -i option is used. A new archive is created using temporary
65 file names and then renamed to inlog in such a way that if any
66 errors (not warnings) are encountered, inlog remains unaltered.
67 -q Quick mode, where if there are no rewriting actions to be per‐
69 formed (none of the global data, instance domains or metrics
70 from inlog will be changed), then pmlogrewrite will exit (with
71 status 0, so success) immediately after parsing the configura‐
72 tion file(s) and outlog is not created.
73 -s When the ``units'' of a metric are changed, if the dimension in
75 terms of space, time and count is unaltered, then the scaling
76 factor is being changed, e.g. BYTE to KBYTE, or MSEC-1 to
77 USEC-1, or the composite MBYTE.SEC-1 to KBYTE.USEC-1. The moti‐
78 vation may be (a) that the original metadata was wrong but the
79 values in inlog are correct, or (b) the metadata is changing so
80 the values need to change as well. The default pmlogrewrite be‐
81 haviour matches case (a). If case (b) applies, then use the -s
82 option and the values of all the metrics with a scale factor
83 change in each result will be rescaled. For finer control over
84 value rescaling refer to the RESCALE option for the UNITS clause
85 of the metric rewriting rule described below.
86 -v Increase verbosity of diagnostic output.
88 -w Emit warnings. Normally pmlogrewrite remains silent for any
90 warning that is not fatal and it is expected that for a particu‐
91 lar archive, some (or indeed, all) of the rewriting specifica‐
92 tions may not apply. For example, changes to a PMDA may be cap‐
93 tured in a set of rewriting rules, but a single archive may not
94 contain all of the modified metrics nor all of the modified
95 instance domains and/or instances. Because these cases are
96 expected, they do not prevent pmlogrewrite executing, and rules
97 that do not apply to inlog are silently ignored by default.
98 Similarly, some rewriting rules may involve no change because
99 the metadata in inlog already matches the intent of the rewrit‐
100 ing rule to correct data from a previous version of a PMDA. The
101 -w flag forces warnings to be emitted for all of these cases.
102 The argument outlog is required in all cases, except when -i is speci‐
104 fied.
105
107 A configuration file contains zero or more rewriting rules as defined
108 below.
109 Keywords and special punctuation characters are shown below in
111 bolditalic font and are case-insensitive, so METRIC, metric and Metric
112 are all equivalent in rewriting rules.
113 The character ``#'' introduces a comment and the remainder of the line
115 is ignored. Otherwise the input is relatively free format with
116 optional white space (spaces, tabs or newlines) between lexical items
117 in the rules.
118 A global rewriting rule has the form:
120 GLOBAL { globalspec ... }
122 where globalspec is zero or more of the following clauses:
124 HOSTNAME -> hostname
126 Modifies the label records in the outlog PCP archive, so that
128 the metrics will appear to have been collected from the host
129 hostname.
130 TIME -> delta
132 Both metric values and the instance domain metadata in a PCP
134 archive carry timestamps. This clause forces all the time‐
135 stamps to be adjusted by delta, where delta is an optional sign
136 ``+'' (the default) or ``-'', an optional number of hours fol‐
137 lowed by a colon ``:'', an optional number of minutes followed
138 by a colon ``:'', a number of seconds, an optional fraction of
139 seconds following a period ``.''. The simplest example would
140 be ``30'' to increase the timestamps by 30 seconds. A more
141 complex example would be ``-23:59:59.999'' to move the time‐
142 stamps backwards by one millisecond less than one day.
143 TZ -> "timezone"
145 Modifies the label records in the outlog PCP archive, so that
147 the metrics will appear to have been collected from a host with
148 a local timezone of timezone. timezone must be enclosed in
149 quotes, and should conform to the valid timezone syntax rules
150 for the local platform.
151 An indom rewriting rule modifies an instance domain and has the form:
153 INDOM domain.serial { indomspec ... }
155 where domain and serial identify one or more existing instance domains
157 from inlog - typically domain would be an integer in the range 1 to 510
158 and serial would be an integer in the range 0 to 4194304.
159 As a special case serial could be an asterisk ``*'' which means the
161 rule applies to every instance domain with a domain number of domain.
162 If a designated instance domain is not in inlog the rule has no effect.
164 The indomspec is zero or more of the following clauses:
166 INAME "oldname" -> "newname"
168 The instance identified by the external instance name oldname
170 is renamed to newname. Both oldname and newname must be
171 enclosed in quotes.
172 As a special case, the new name may be the keyword DELETE (with
174 no quotes), and then the instance oldname will be expunged from
175 outlog which removes it from the instance domain metadata and
176 removes all values of this instance for all the associated met‐
177 rics.
178 If the instance names contain any embedded spaces then special
180 care needs to be taken in respect of the PCP instance naming
181 rule that treats the leading non-space part of the instance
182 name as the unique portion of the name for the purposes of
183 matching and ensuring uniqueness within an instance domain,
184 refer to pmdaInstance(3) for a discussion of this issue.
185 As an illustration, consider the hypothetical instance domain
187 for a metric which contains 2 instances with the following
188 names:
189 red
190 eek urk
191 Then some possible INAME clauses might be:
193 "eek" -> "yellow like a flower"
195 Acceptable, oldname "eek" matches the "eek urk"
196 instance.
197 "red" -> "eek"
199 Error, newname "eek" matches the existing "eek urk"
200 instance.
201 "eek urk" -> "red of another hue"
203 Error, newname "red of another hue" matches the
204 existing "red" instance.
205 INDOM -> newdomain.newserial
207 Modifies the metadata for the instance domain and every metric
209 associated with the instance domain. As a special case, newse‐
210 rial could be an asterisk ``*'' which means use serial from the
211 indom rewriting rule, although this is most useful when serial
212 is also an asterisk. So for example:
213 indom 29.* { indom -> 109.* }
214 will move all instance domains from domain 29 to domain 109.
215 INDOM -> DUPLICATE newdomain.newserial
217 A special case of the previous INDOM clause where the instance
219 domain is a duplicate copy of the domain.serial instance domain
220 from the indom rewriting rule, and then any mapping rules are
221 applied to the copied newdomain.newserial instance domain.
222 This is useful when a PMDA is split and the same instance
223 domain needs to be replicated for domain domain and domain new‐
224 domain. So for example if the metrics foo.one and foo.two are
225 both defined over instance domain 12.34, and foo.two is moved
226 to another PMDA using domain 27, then the following rewriting
227 rules could be used:
228 indom 12.34 { indom -> duplicate 27.34 }
229 metric foo.two { indom -> 27.34 pmid -> 27.*.* }
230 INST oldid -> newid
232 The instance identified by the internal instance identifier
234 oldid is renumbered to newid. Both oldid and newid are inte‐
235 gers in the range 0 to 231-1.
236 As a special case, newid may be the keyword DELETE and then the
238 instance oldid will be expunged from outlog which removes it
239 from the instance domain metadata and removes all values of
240 this instance for all the associated metrics.
241 A metric rewriting rule has the form:
243 METRIC metricid { metricspec ... }
245 where metricid identifies one or more existing metrics from inlog using
247 either a metric name, or the internal encoding for a metric's PMID as
248 domain.cluster.item. In the latter case, typically domain would be an
249 integer in the range 1 to 510, cluster would be an integer in the range
250 0 to 4095, and item would be an integer in the range 0 to 1023.
251 As special cases item could be an asterisk ``*'' which means the rule
253 applies to every metric with a domain number of domain and a cluster
254 number of cluster, or cluster could be an asterisk which means the rule
255 applies to every metric with a domain number of domain and an item num‐
256 ber of item, or both cluster and item could be asterisks, and rule
257 applies to every metric with a domain number of domain.
258 If a designated metric is not in inlog the rule has no effect.
260 The metricspec is zero or more of the following clauses:
262 DELETE
265 The metric is completely removed from outlog, both the metadata
267 and all values in results are expunged.
268 INDOM -> newdomain.newserial [ pick ]
271 Modifies the metadata to change the instance domain for this
273 metric. The new instance domain must exist in outlog.
274 The optional pick clause may be used to select one input value,
276 or compute an aggregate value from the instances in an input
277 result, or assign an internal instance identifier to a single
278 output value. If no pick clause is specified, the default be‐
279 haviour is to copy all input values from each input result to
280 an output result, however if the input instance domain is sin‐
281 gular (indom PM_INDOM_NULL) then the one output value must be
282 assigned an internal instance identifier, which is 0 by
283 default, unless over-ridden by a INST or INAME clause as
284 defined below.
285 The choices for pick are as follows:
287 OUTPUT FIRST
289 choose the value of the first instance from each
290 input result
291 OUTPUT LAST choose the value of the last instance from each
293 input result
294 OUTPUT INST instid
296 choose the value of the instance with internal
297 instance identifier instid from each result; the
298 sequence of rewriting rules ensures the OUTPUT pro‐
299 cessing happens before instance identifier renum‐
300 bering from any associated indom rule, so instid
301 should be one of the internal instance identifiers
302 that appears in inlog
303 OUTPUT INAME "name"
305 choose the value of the instance with name for its
306 external instance name from each result; the
307 sequence of rewriting rules ensures the OUTPUT pro‐
308 cessing happens before instance renaming from any
309 associated indom rule, so name should be one of the
310 external instance names that appears in inlog
311 OUTPUT MIN choose the smallest value in each result (metric
313 type must be numeric and output instance will be 0
314 for a non-singular instance domain)
315 OUTPUT MAX choose the largest value in each result (metric
317 type must be numeric and output instance will be 0
318 for a non-singular instance domain)
319 OUTPUT SUM choose the sum of all values in each result (metric
321 type must be numeric and output instance will be 0
322 for a non-singular instance domain)
323 OUTPUT AVG choose the average of all values in each result
325 (metric type must be numeric and output instance
326 will be 0 for a non-singular instance domain)
327 If the input instance domain is singular (indom PM_INDOM_NULL)
329 then independent of any pick specifications, there is at most
330 one value in each input result and so FIRST, LAST, MIN, MAX,
331 SUM and AVG are all equivalent and the output instance identi‐
332 fier will be 0.
333 In general it is an error to specify a rewriting action for the
335 same metadata or result values more than once, e.g. more than
336 one INDOM clause for the same instance domain. The one excep‐
337 tion is the possible interaction between the INDOM clauses in
338 the indom and metric rules. For example the metric sample.bin
339 is defined over the instance domain 29.2 in inlog and the fol‐
340 lowing is acceptable (albeit redundant):
341 indom 29.* { indom -> 109.* }
342 metric sample.bin { indom -> 109.2 }
343 However the following is an error, because the instance domain
344 for sample.bin has two conflicting definitions:
345 indom 29.* { indom -> 109.* }
346 metric sample.bin { indom -> 123.2 }
347 INDOM -> NULL[ pick ]
350 The metric (which must have been previously defined over an
352 instance domain) is being modified to be a singular metric.
353 This involves a metadata change and collapsing all results for
354 this metric so that multiple values become one value.
355 The optional pick part of the clause defines how the one value
357 for each result should be calculated and follows the same rules
358 as described for the non-NULL INDOM case above.
359 In the absence of pick, the default is OUTPUT FIRST.
361 NAME -> newname
364 Renames the metric in the PCP archive's metadata that supports
366 the Performance Metrics Name Space (PMNS). newname should not
367 match any existing name in the archive's PMNS and must follow
368 the syntactic rules for valid metric names as outlined in
369 pmns(5).
370 PMID -> newdomain.newcluster.newitem
373 Modifies the metadata and results to renumber the metric's
375 PMID. As special cases, newcluster could be an asterisk ``*''
376 which means use cluster from the metric rewriting rule and/or
377 item could be an asterisk which means use item from the metric
378 rewriting rule. This is most useful when cluster and/or item
379 is also an asterisk. So for example:
380 metric 30.*.* { pmid -> 123.*.* }
381 will move all metrics from domain 30 to domain 123.
382 SEM -> newsem
385 Change the semantics of the metric. newsem should be the XXX
387 part of the name of one of the PM_SEM_XXX macros defined in
388 <pcp/pmapi.h> or pmLookupDesc(3), e.g. COUNTER for
389 PM_TYPE_COUNTER.
390 No data value rewriting is performed as a result of the SEM
392 clause, so the usefulness is limited to cases where a version
393 of the associated PMDA was exporting incorrect semantics for
394 the metric. pmlogreduce(1) may provide an alternative in cases
395 where re-computation of result values is desired.
396 TYPE -> newtype
399 Change the type of the metric which alters the metadata and may
401 change the encoding of values in results. newtype should be
402 the XXX part of the name of one of the PM_TYPE_XXX macros
403 defined in <pcp/pmapi.h> or pmLookupDesc(3), e.g. FLOAT for
404 PM_TYPE_FLOAT.
405 Type conversion is only supported for cases where the old and
407 new metric type is numeric, so PM_TYPE_STRING, PM_TYPE_AGGRE‐
408 GATE and PM_TYPE_EVENT are not allowed. Even for the numeric
409 cases, some conversions may produce run-time errors, e.g. inte‐
410 ger overflow, or attempting to rewrite a negative value into an
411 unsigned type.
412 UNITS -> newunits [ RESCALE ]
415 newunits is six values separated by commas. The first 3 values
417 describe the dimension of the metric along the dimensions of
418 space, time and count; these are integer values, usually 0, 1
419 or -1. The remaining 3 values describe the scale of the met‐
420 ric's values in the dimensions of space, time and count. Space
421 scale values should be 0 (if the space dimension is 0), else
422 the XXX part of the name of one of the PM_SPACE_XXX macros,
423 e.g. KBYTE for PM_TYPE_KBYTE. Time scale values should be 0
424 (if the time dimension is 0), else the XXX part of the name of
425 one of the PM_TIME_XXX macros, e.g. SEC for PM_TIME_SEC.
426 Count scale values should be 0 (if the time dimension is 0),
427 else ONE for PM_COUNT_ONE.
428 The PM_SPACE_XXX, PM_TIME_XXX and PM_COUNT_XXX macros are
430 defined in <pcp/pmapi.h> or pmLookupDesc(3).
431 When the scale is changed (but the dimension is unaltered) the
433 optional keyword RESCALE may be used to chose value rescaling
434 as per the -s command line option, but applied to just this
435 metric.
436 When changing the domain number for a metric or instance domain,
439 the new domain number will usually match an existing PMDA's domain
440 number. If this is not the case, then the new domain number should
441 not be randomly chosen; consult $PCP_VAR_DIR/pmns/stdpmid for
442 domain numbers that are already assigned to PMDAs.
443
445 To promote the values of the per-disk IOPS metrics to 64-bit to allow
446 aggregation over a long time period for capacity planning, or because
447 the PMDA has changed to export 64-bit counters and we want to convert
448 old archives so they can be processed alongside new archives.
449 metric disk.dev.read { type -> U64 }
450 metric disk.dev.write { type -> U64 }
451 metric disk.dev.total { type -> U64 }
452 The instances associated with the load average metric kernel.all.load
454 could be renamed and renumbered by the rules below.
455 # for the Linux PMDA, the kernel.all.load metric is defined
456 # over instance domain 60.2
457 indom 60.2 {
458 inst 1 -> 60 iname "1 minute" -> "60 second"
459 inst 5 -> 300 iname "5 minute" -> "300 second"
460 inst 15 -> 900 iname "15 minute" -> "900 second"
461 }
462 If we decide to split the ``proc'' metrics out of the Linux PMDA, this
464 will involve changing the domain number for the PMID of these metrics
465 and the associated instance domains. The rules below would rewrite an
466 old archive to match the changes after the PMDA split.
467 # all Linux proc metrics are in 7 clusters
468 metric 60.8.* { pmid -> 123.*.* }
469 metric 60.9.* { pmid -> 123.*.* }
470 metric 60.13.* { pmid -> 123.*.* }
471 metric 60.24.* { pmid -> 123.*.* }
472 metric 60.31.* { pmid -> 123.*.* }
473 metric 60.32.* { pmid -> 123.*.* }
474 metric 60.51.* { pmid -> 123.*.* }
475 # only one instance domain for Linux proc metrics
476 indom 60.9 { indom -> 123.0 }
477
479 For each of the inlog and outlog archive logs, several physical files
480 are used.
481 archive.meta
482 metadata (metric descriptions, instance domains, etc.) for
483 the archive log
484 archive.0 initial volume of metrics values (subsequent volumes have
485 suffixes 1, 2, ...).
486 archive.index
487 temporal index to support rapid random access to the other
488 files in the archive log.
489
491 Environment variables with the prefix PCP_ are used to parameterize the
492 file and directory names used by PCP. On each installation, the file
493 /etc/pcp.conf contains the local values for these variables. The
494 $PCP_CONF variable may be used to specify an alternative configuration
495 file, as described in pcp.conf(5).
496
498 PCPIntro(1), pmdaInstance(3), pmdumplog(1), pmlogger(1), pmlogex‐
499 tract(1), pmloglabel(1), pmlogreduce(1), pmLookupDesc(3), pmns(5),
500 pcp.conf(5) and pcp.env(5).
501
503 All error conditions detected by pmlogrewrite are reported on stderr
504 with textual (if sometimes terse) explanation.
505 Should the input archive log be corrupted (this can happen if the
507 pmlogger instance writing the log suddenly dies), then pmlogrewrite
508 will detect and report the position of the corruption in the file, and
509 any subsequent information from that archive log will not be processed.
510 If any error is detected, pmlogrewrite will exit with a non-zero sta‐
512 tus.
513Performance Co-Pilot PMLOGREWRITE(1)