1PMSERIES(1) General Commands Manual PMSERIES(1)
2
6 pmseries - display information about performance metric timeseries
7
9 pmseries [-adFiIlLmMnqsStvV?] [-c config] [-g pattern] [-h host] [-p
10 port] [-Z timezone] [query | labels ... | series ... | source ... ]
11
13 pmseries displays various types of information about performance met‐
14 rics available through the scalable timeseries facilities of the Per‐
15 formance Co-Pilot (PCP) and the Redis distributed data store.
16 By default pmseries communicates with a local redis-server(1), however
18 the -h and -p options can be used to specify an alternate Redis in‐
19 stance. If this instance is a node of a Redis cluster, all other in‐
20 stances in the cluster will be discovered and used automatically.
21 pmseries runs in several different modes - either querying timeseries
23 identifiers, metadata or values (already stored in Redis), or manually
24 loading timeseries into Redis. The latter mode is generally only used
25 for loading previously collected (inactive) archives, since pmproxy(1)
26 will automatically perform this function for "live" (actively growing)
27 local pmlogger(1) instances, when running in its default time series
28 mode. See the TIMESERIES LOADING section below and the -L option for
29 further details.
30 Without command line options specifying otherwise, pmseries will issue
32 a timeseries query to find matching timeseries and values. All time‐
33 series are identified using a unique SHA-1 hash which is always dis‐
34 played in a 40-hexdigit human readable form. These hashes are formed
35 using the metadata associated with every metric.
36 Importantly, this includes all metric metadata (labels, names, descrip‐
38 tors). Metric labels in particular are (as far as possible) unique for
39 every machine - on Linux for example the labels associated with every
40 metric include the unique /etc/machine-id, the hostname, domainname,
41 and other automatically generated machine labels, as well as any admin‐
42 istrator-defined labels from /etc/pcp/labels. These labels can be re‐
43 ported with pminfo(1) and the pmcd.labels metric.
44 See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and pm‐
46 LookupDesc(3) for detailed information about metric labels and other
47 metric metadata used in each timeseries identifier hash calculation.
48 The timeseries identifiers provide a higher level (and machine indepen‐
50 dent) identifier than the traditional PCP performance metric identi‐
51 fiers (pmID), instance domain identifiers (pmInDom) and metric names.
52 See PCPIntro(1) for more details about these traditional identifiers.
53 However, pmseries uses timeseries identifiers in much the same way that
54 pminfo(1) uses the lower level indom, metric identifiers and metric
55 names.
56 The default mode of pmseries operation (i.e. with no command line op‐
58 tions) depends on the arguments it is presented. If all non-option ar‐
59 guments appear to be timeseries identifiers (in 40 hex digit form) pm‐
60 series will report metadata for these timeseries - refer to the -a op‐
61 tion for details. Otherwise, the parameters will be treated as a time‐
62 series query.
63
65 Query expressions are formed using the pmseries query language de‐
66 scribed below, but can be as simple as a metric name.
67 The following is an example of querying timeseries from all hosts that
69 match a metric name pattern (globbed):
70 $ pmseries kernel.all.cpu*
72 1d7b0bb3f6aec0f49c54f5210885464a53629b60
73 379db729afd63fb9eff436625bd6c55a7adc5cfd
74 3dd3b45bb05f96636043e5d58b52b441ce542285
75 [...]
76 ed2bf325ff6dc7589ec966698e5404b67252306a
77 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
78 To identify timeseries expression operands, the query language uses the
80 general syntax:
81 [metric.name] '{metadata qualifiers}' '[time specification]'
83 The metric.name component restricts the timeseries query to any match‐
85 ing PCP metric name (the list of metric names for a PCP archive or live
86 host is reported by pminfo(1) with no arguments beyond --host or --ar‐
87 chive). The pmseries syntax extends on that of pminfo and allows for
88 glob(7) based pattern matching within the metric name. The above de‐
89 scribes operands available as the leaves of pmseries expressions, which
90 may include functions, arithmetic operators and other features. See
91 the EXPRESSIONS section below for further details.
92
94 Metadata qualifiers are enclosed by ``curly'' braces ({}), and further
95 restrict the query results to timeseries operands with various metadata
96 properties. These qualifiers are based on metric or instance names,
97 and metric label values, and take the general form metadata.name OPERA‐
98 TOR value, such as:
99 instance.name == "cpu0"
101 metric.name != "kernel.all.pswitch"
102 When using label names, the metadata qualifier is optional and can be
104 dropped, such as:
105 label.hostname == "www.acme.com"
107 hostname == "www.acme.com"
108 For metric and instance names only the string operators apply, but for
110 metric label values all operators are available. The set of available
111 operators is:
112 Boolean operators
114 All string (label, metrics and instances) and numeric (label) values
115 can be tested for equality ("==") or inequality ("!=").
116 String operators
118 Strings can be subject to pattern matching in the form of glob matching
119 ("~~"), regular expression matching ("=~"), and regular expression non-
120 matching ("!~"). The ":" operator is equivalent to "~~" - i.e., glob
121 matching.
122 Relational operators (numeric label values only)
124 Numeric label values can be subject to the less than ("<"), greater
125 than (">"), less than or equal ("<="), greater than or equal (">="),
126 equal ("==") and not equal ("!=") operators.
127 Logical operators
129 Multiple metadata qualifiers can be combined with the logical operators
130 for AND ("&&") and OR ("||") as in many programming languages. The
131 comma (",") character is equivalent to logical AND ("&&").
132
134 The final (optional) component of a query allows the user to specify a
135 specific time window of interest. Any time specification will result
136 in values being returned for all matching timeseries only for the time
137 window specified.
138 The specification is ``square'' bracket ([]) enclosed, and consists of
140 one or more comma-separated components. Each component specifies some
141 aspect related to time, taking the general form: keyword: value, such
142 as:
143 samples:10
145 Sample count
147 The number of samples to return, specified via either the samples or
148 (equivalent) count keyword. The value provided must be a positive in‐
149 teger. If no end time is explicitly set (see ``Time window'' later)
150 then the most recent samples will be returned.
151 Sample interval
153 An interval between successive samples can be requested using the in‐
154 terval or (equivalent) delta keyword. The value provided should be ei‐
155 ther a numeric or string value that will be parsed by pmParseInter‐
156 val(3), such as 5 (seconds) or 2min (minutes).
157 Time window
159 Start and end times, and alignments, affecting the returned values.
160 The keywords match the parameters to the pmParseTimeWindow(3) function
161 which will be used to parse them, and are: start or (equivalent) begin,
162 finish or (equivalent) end, align and offset.
163 Time zones
165 The resulting timestamps can be returned having been evaluated for a
166 specific timezone, using the timezone or hostzone keywords. The value
167 associated with timezone will be interpreted by pmNewZone(3). A true
168 or false value should be associated with hostzone, and when set to true
169 this has the same effect as described by pmNewContextZone(3).
170
172 As described above, operands are the leaves of a query expression tree.
173 [metric.name] '{metadata qualifiers}' '[time specification]'
175 Note in most of the query expression examples below, the metadata qual‐
176 ifiers have been omitted for brevity. In all cases, multiple time se‐
177 ries may qualify, particularly for the hostname label.
178 In the simple case, a query expression consists of a single operand and
180 may just be a metric name. In the more general case, a query expres‐
181 sion is either an operand or the argument to a function, or two oper‐
182 ands in a binary arithmetic or logical expression. Most functions take
183 a single argument (an expression), though some require additional argu‐
184 ments, e.g. rescale.
185 operand | expr operator expr | func(expr[, arg])
187 This grammar shows expressions may be nested, e.g. using the addition
189 (+) operator as an example,
190 func1(func2(expr))
192 func1(expr) + func2(expr)
193 expr + func(expr)
194 func(expr) + expr
195 expr + expr
196 Rules governing compatibility of operands in an expression generally
198 depend on the function and/or operators and are described below indi‐
199 vidually. An important rule is that if any time windows are specified,
200 then all operands must cover the same number of samples, though the
201 time windows may differ individually. If no time windows or sample
202 counts are given, then pmseries will return a series identifier (SID)
203 instead of a series of timestamps and values. This SID may be used in
204 subsequent /series/values?series=SID RESTAPI calls, along with a spe‐
205 cific time window.
206 Arithmetic Operators
208 pmseries support addition, subtraction, division and multiplication on
209 each value in the time series of a binary pair of operands. No unary
210 or ternary operators are supported (yet). In all cases, the instance
211 domain and the number of samples of time series operands must be the
212 same. The metadata (units and dimensions) must also be compatible.
213 Depending on the function, the result will usually have the same in‐
214 stance domain and (unless noted otherwise), the same units as the oper‐
215 ands. The metadata dimensions (space, time, count) of the result may
216 differ (see below).
217 Expression operands may have different qualifiers, e.g. you can perform
219 binary arithmetic on metrics qualified by different labels (such as
220 hostname), or metric names. For example, to add the two most recent
221 samples of the process context switch (pswitch) counter metric for
222 hosts node88 and node89, and then perform rate conversion:
223 $ pmseries 'rate(kernel.all.pswitch{hostname:node88}[count:2] +
225 kernel.all.pswitch{hostname:node89}[count:2])'
226 1cf1a85d5978640ef94c68264d3ae8866cc11f7c
227 [Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924
228 Note the resulting time series of values has one less sample than the
230 expression operand passed to the rate function.
231 Other rules for arithmetic expressions:
233 1. if both operands have the semantics of a counter, then only addition
235 and subtraction are allowed
236 2. if the left operand is a counter and the right operand is not, then
238 only multiplication or division are allowed
239 3. if the left operand is not a counter and the right operand is a
241 counter, then only multiplication is allowed.
242 4. addition and subtraction - the dimensions of the result are the same
244 as the dimensions of the operands.
245 5. multiplication - the dimensions of the result are the sum of the di‐
247 mensions of the operands.
248 6. division - the dimensions of the result are the difference of the
250 dimensions of the operands.
251 Functions
253 Expression functions operate on vectors of time series values, and may
254 be nested with other functions or expressions as described above. When
255 an operand has multiple instances, there will generally be one result
256 for each series of instances. For example, the result for
257 $ pmseries 'min(kernel.all.load[count:100])'
259 will be the smallest value of the 100 most recent samples, treating
261 each of the three load average instances as a separate time series. As
262 an example, for the two most recent samples for each of the three in‐
263 stances of the load average metric:
264 $ pmseries 'kernel.all.load[count:2]'
266 726a325c4c1ba4339ecffcdebd240f441ea77848
267 [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
268 [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
269 [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
270 [Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
271 [Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
272 [Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
273 Using the min function :
275 $ pmseries 'min(kernel.all.load[count:2])'
277 11b965bc5f9598034ed9139fb3a78c6c0b7065ba
278 [Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
279 [Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
280 [Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
281 For singular metrics (with no instance domain), a single value will re‐
283 sult, e.g. for the five most recent samples of the context switching
284 metric:
285 $ pmseries 'kernel.all.pswitch[count:5]'
287 d7832c4fba33bcc980b1a1b614e0508043288480
288 [Tue Nov 10 12:44:59.380666000 2020] 460774294
289 [Tue Nov 10 12:44:49.382070000 2020] 460747232
290 [Tue Nov 10 12:44:39.378545000 2020] 460722370
291 [Tue Nov 10 12:44:29.379029000 2020] 460697388
292 [Tue Nov 10 12:44:19.379096000 2020] 460657412
293 $ pmseries 'min(kernel.all.pswitch[count:5])'
295 1b6e92fb5bc012372f54452734dd03f0f131fa06
296 [Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480
297 Future versions of pmseries may provide functions that perform aggrega‐
300 tion, interpolation, filtering or transforms in other ways, e.g. across
301 instances instead of time.
302 Function Reference
304 max(expr) the maximum value in the time series for each instance of ex‐
305 pr
306 min(expr) the minimum value in the time series for each instance of ex‐
308 pr
309 rate(expr) the rate with respect to time of each sample. The given ex‐
311 pr must have counter semantics and the result will have instant seman‐
312 tics (the time dimension reduced by one). In addition, the result will
313 have one less sample than the operand - this is because the first sam‐
314 ple cannot be rate converted (two samples are required).
315 rescale(expr,scale) rescale the values in the time series for each in‐
317 stance of expr to scale (units). Note that expr should have instant or
318 discrete semantics (not counter - rate conversion should be done first
319 if needed). The time, space and count dimensions between expr and
320 scale must be compatible. Example: rate convert the read throughput
321 counter for each disk instance and then rescale to mbytes per second.
322 Note the native units of disk.dev.read_bytes is a counter of kbytes
323 read from each device instance since boot.
324 $ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'
326 abs(expr) the absolute value of each value in the time series for each
328 instance of expr. This has no effect if the type of expr is unsigned.
329 floor(expr) rounded down to the nearest integer value of the time se‐
331 ries for each instance of expr.
332 round(expr) rounded up or down to the nearest integer for each value in
334 the time series for each instance of expr.
335 log(expr) logarithm of the values in the time series for each instance
337 of expr
338 sqrt(expr) square root of the values in the time series for each in‐
340 stance of expr
341 Compatibility
343 All operands in an expression must have the same number of samples, but
344 not necessarily the same time window. e.g. you could subtract some met‐
345 ric time series from today from that of yesterday by giving different
346 time windows and different metrics or qualifiers, ensuring the same
347 number of samples are given as the operands.
348 Operands in an expression must either all have a time window, or none.
350 If no operands have a time window, then instead of a series of time
351 stamps and values, the result will be a time series identifier (SID)
352 that may be passed to the /series/values?series=SID REST API function,
353 along with a time window. For further details, see PMWEBAPI(3).
354 If the semantics of both operands in an arithmetic expression are not
356 counter (i.e. PM_SEM_INSTANT or PM_SEM_DISCRETE) then the result will
357 have semantics PM_SEM_INSTANT unless both operands are PM_SEM_DISCRETE
358 in which case the result is also PM_SEM_DISCRETE.
359
361 Using command line options, pmseries can be requested to provide meta‐
362 data (metric names, instance names, labels, descriptors) associated
363 with either individual timeseries or a group of timeseries, for exam‐
364 ple:
365 $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
367 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
369 PMID: 60.0.21
370 Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
371 Semantics: counter Units: millisec
372 Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
373 Metric: kernel.all.cpu.nice
374 labels {"agent":"linux","domainname":"localdomain",\
375 "groupid":1000,"hostname":"shard",\
376 "latitude":-25.28496,"longitude":152.87886,\
377 "machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
378 "userid":1000}
379 The complete set of pmseries metadata reporting options are:
381 -a, --all
383 Convenience option to report all metadata for the given time‐
384 series, equivalent to -deilms.
385 -d, --desc
387 Metric descriptions detailing the PMID, data type, data semantics,
388 units, scale and associated instance domain. This option has a
389 direct pminfo(1) equivalent.
390 -F, --fast
392 Query or load series metadata only, not values.
393 -g pattern, --glob=pattern
395 Provide a glob(7) pattern to restrict the report provided by the
396 -i, -l, -m, and -S.
397 -i, --instances
399 Metric descriptions detailing the PMID, data type, data semantics,
400 units, scale and associated instance domain.
401 -I, --fullindom
403 Print the InDom in verbose mode. This option has a direct pmin‐
404 fo(1) equivalent.
405 -l, --labels
407 Print label sets associated with metrics and instances. Labels
408 are optional metric metadata described in detail in pmLookupLa‐
409 bels(3). This option has a direct pminfo(1) equivalent.
410 -m, --metrics
412 Print metric names.
413 -M, --fullpmid
415 Print the PMID in verbose mode. This option has a direct pmin‐
416 fo(1) equivalent.
417 -n, --names
419 Print comma-separated label names only (not values) for the labels
420 associated with metrics and instances.
421 -s, --series
423 Print timeseries identifiers associated with metrics, instances
424 and sources. These unique identifiers are calculated from intrin‐
425 sic (non-optional) labels and other metric metadata associated
426 with each PMAPI context (sources), metrics and instances. Ar‐
427 chive, local context or pmcd(1) connections for the same host all
428 produce the same source identifier. This option has a direct
429 pminfo(1) equivalent. See also pmLookupLabels(3) and the -l/--la‐
430 bels option.
431
433 A source is a unique identifier (represented externally as a 40-byte
434 hexadecimal SHA-1 hash) that represents both the live host and/or ar‐
435 chives from which each timeseries originated. The context for a source
436 identifier (obtained with -s) can be reported with:
437 -S, --sources
439 Print names for timeseries sources. These names are either host‐
440 names or fully qualified archive paths.
441 It is important to note that live and archived sources can and will
443 generate the same SHA-1 source identifier hash, provided that the con‐
444 text labels remain the same for that host (labels are stored in PCP ar‐
445 chives and can also be fetched live from pmcd(1)).
446
448 Timeseries metadata and data are loaded either automatically by a local
449 pmproxy(1), or manually using a specially crafted pmseries query and
450 the -L/--load option:
451 $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
453 pmseries: [Info] processed 2275 archive records from [...]
454 This query must specify a source archive path, but can also restrict
456 the import to specific timeseries (using metric names, labels, etc) and
457 to a specific time window using the time specification component of the
458 query language.
459 As a convenience, if the argument to load is a valid file path as de‐
461 termined by access(2), then a short-hand form can be used:
462 $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
464
466 The available command line options, in addition to timeseries metadata
467 and sources options described above, are:
468 -c config, --config=config
470 Specify the config file to use.
471 -h host, --host=host
473 Connect Redis server at host, rather than the one the localhost.
474 -L, --load
476 Load timeseries metadata and data into the Redis cluster.
477 -p port, --port=port
479 Connect Redis server at port, rather than the default 6379.
480 -q, --query
482 Perform a timeseries query. This is the default action.
483 -t, --times
485 Report time stamps numerically (in milliseconds) instead of the
486 default human readable form.
487 -v, --values
489 Report all of the known values for given label name(s).
490 -V, --version
492 Display version number and exit.
493 -Z timezone, --timezone=timezone
495 Use timezone for the date and time. Timezone is in the format of
496 the environment variable TZ as described in environ(7).
497 -?, --help
499 Display usage message and exit.
500
502 The following sample query shows several fundamental aspects of the pm‐
503 series query language:
504 $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'
506 eb713a9cf472f775aa59ae90c43cd7f960f7870f
508 [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
509 [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
510 [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
511 [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
512 [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
513 [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
514 This query returns the two most recent values for all instances of the
516 kernel.all.load metric with a label.hostname matching the regular ex‐
517 pression "toium". This is a set-valued metric (i.e., a metric with an
518 ``instance domain'' which in this case consists of three instances: 1,
519 5 and 15 minute averages). The first column returned is a timestamp,
520 then a floating point value, and finally an instance identifier time‐
521 series hash (two values returned for three instances, so six rows are
522 returned). The metadata for these timeseries can then be further exam‐
523 ined:
524 $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
526 eb713a9cf472f775aa59ae90c43cd7f960f7870f
528 PMID: 60.2.0
529 Data Type: float InDom: 60.2 0xf000002
530 Semantics: instant Units: none
531 Source: 0e89c1192db79326900d82131c31399524f0b3ee
532 Metric: kernel.all.load
533 inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
534 inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
535 inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
536 inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
537 inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
538 inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
539
541 Environment variables with the prefix PCP_ are used to parameterize the
542 file and directory names used by PCP. On each installation, the file
543 /etc/pcp.conf contains the local values for these variables. The
544 $PCP_CONF variable may be used to specify an alternative configuration
545 file, as described in pcp.conf(5).
546 For environment variables affecting PCP tools, see pmGetOptions(3).
548
550 PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), redis-server(1), ac‐
551 cess(2), PMAPI(3), PMWEBAPI(3), pmLookupDesc(3), pmLookupInDom(3), pm‐
552 LookupLabels(3), pmLookupName(3), pmNewContextZone(3), pmNewZone(3),
553 pmParseInterval(3), pmParseTimeWindow(3), pcp.conf(5), environ(7),
554 glob(7) and regex(7).
555Performance Co-Pilot PCP PMSERIES(1)