1PMSERIES(1) General Commands Manual PMSERIES(1)
2
6 pmseries - display information about performance metric timeseries
7
9 pmseries [-adFiIlLmMnqsStvV?] [-c config] [-g pattern] [-h host] [-p
10 port] [-Z timezone] [query | labels ... | series ... | source ... ]
11
13 pmseries displays various types of information about performance met‐
14 rics available through the scalable timeseries facilities of the Per‐
15 formance Co-Pilot (PCP) and the Redis distributed data store.
16 By default pmseries communicates with a local redis-server(1), however
18 the -h and -p options can be used to specify an alternate Redis
19 instance. If this instance is a node of a Redis cluster, all other
20 instances in the cluster will be discovered and used automatically.
21 pmseries runs in several different modes - either querying timeseries
23 identifiers, metadata or values (already stored in Redis), or manually
24 loading timeseries into Redis. The latter mode is seldom used, how‐
25 ever, since pmproxy(1) will automatically perform this function for
26 local pmlogger(1) instances, when running in its default time series
27 mode.
28 Without command line options specifying otherwise, pmseries will issue
30 a timeseries query to find matching timeseries and values. All time‐
31 series are identified using a unique SHA-1 hash which is always dis‐
32 played in a 40-hexdigit human readable form. These hashes are formed
33 using the metadata associated with every metric.
34 Importantly, this includes all metric metadata (labels, names, descrip‐
36 tors). Metric labels in particular are (as far as possible) unique for
37 every machine - on Linux for example the labels associated with every
38 metric include the unique /etc/machine-id, the hostname, domainname,
39 and other automatically generated machine labels, as well as any admin‐
40 istrator-defined labels from /etc/pcp/labels. These labels can be
41 reported with pminfo(1) and the pmcd.labels metric.
42 See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
44 pmLookupDesc(3) for detailed information about metric labels and other
45 metric metadata used in each timeseries identifier hash calculation.
46 The timeseries identifiers provide a higher level (and machine indepen‐
48 dent) identifier than the traditional PCP performance metric identi‐
49 fiers (pmID), instance domain identifiers (pmInDom) and metric names.
50 See PCPIntro(1) for more details about these traditional identifiers.
51 However, pmseries uses timeseries identifiers in much the same way that
52 pminfo(1) uses the lower level indom, metric identifiers and metric
53 names.
54 The default mode of pmseries operation (i.e. with no command line
56 options) depends on the arguments it is presented. If all non-option
57 arguments appear to be timeseries identifiers (in 40 hex digit form)
58 pmseries will report metadata for these timeseries - refer to the -a
59 option for details. Otherwise, the parameters will be treated as a
60 timeseries query.
61
63 Query expressions are formed using the pmseries query language
64 described below, but can be as simple as a metric name.
65 The following is an example of querying timeseries from all hosts that
67 match a metric name pattern (globbed):
68 $ pmseries kernel.all.cpu*
70 1d7b0bb3f6aec0f49c54f5210885464a53629b60
71 379db729afd63fb9eff436625bd6c55a7adc5cfd
72 3dd3b45bb05f96636043e5d58b52b441ce542285
73 [...]
74 ed2bf325ff6dc7589ec966698e5404b67252306a
75 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
76 To identify timeseries, the query language uses the general syntax:
78 [metric.name] '{metadata qualifiers}' '[time specification]'
80 The metric.name component restricts the timeseries query to any match‐
82 ing PCP metric name (the list of metric names for a PCP archive or live
83 host is reported by pminfo(1) with no arguments beyond --host or --ar‐
84 chive). The pmseries syntax extends on that of pminfo and allows for
85 glob(7) based pattern matching within the metric name.
86
88 Metadata qualifiers are enclosed by ``curly'' braces ({}), and further
89 restrict the query results to timeseries with various metadata proper‐
90 ties. These qualifiers are based on metric or instance names, and met‐
91 ric label values, and take the general form metadata.name OPERATOR val‐
92 ue, such as:
93 instance.name == "cpu0"
95 metric.name != "kernel.all.pswitch"
96 When using label names, the metadata qualifier is optional and can be
98 dropped, such as:
99 label.hostname == "www.acme.com"
101 hostname == "www.acme.com"
102 For metric and instance names only the string operators apply, but for
104 metric label values all operators are available. The set of available
105 operators is:
106 Boolean operators
108 All string (label, metrics and instances) and numeric (label) values
109 can be tested for equality ("==") or inequality ("!=").
110 String operators
112 Strings can be subject to pattern matching in the form of glob matching
113 ("~~"), regular expression matching ("=~"), and regular expression non-
114 matching ("!~"). The ":" operator is equivalent to "~~" - i.e., regu‐
115 lar expression matching.
116 Relational operators (numeric label values only)
118 Numeric label values can be subject to the less than ("<"), greater
119 than (">"), less than or equal ("<="), greater than or equal (">="),
120 equal ("==") and not equal ("!=") operators.
121 Logical operators
123 Multiple metadata qualifiers can be combined with the logical operators
124 for AND ("&&") and OR ("||") as in many programming languages. The
125 comma (",") character is equivalent to logical AND ("&&").
126
128 The final (optional) component of a query allows the user to specify a
129 specific time window of interest. Any time specification will result
130 in values being returned for all matching timeseries only for the time
131 window specified.
132 The specification is ``square'' bracket ([]) enclosed, and consists of
134 one or more comma-separated components. Each component specifies some
135 aspect related to time, taking the general form: keyword: value, such
136 as:
137 samples: 10
139 Sample count
141 The number of samples to return, specified via either the samples or
142 (equivalent) count keyword. The value provided must be a positive in‐
143 teger. If no end time is explicitly set (see ``Time window'' later)
144 then the most recent samples will be returned.
145 Sample interval
147 An interval between successive samples can be requested using the in‐
148 terval or (equivalent) delta keyword. The value provided should be ei‐
149 ther a numeric or string value that will be parsed by pmParseInter‐
150 val(3), such as 5 (seconds) or 2min (minutes).
151 Time window
153 Start and end times, and alignments, affecting the returned values.
154 The keywords match the parameters to the pmParseTimeWindow(3) function
155 which will be used to parse them, and are: start or (equivalent) begin,
156 finish or (equivalent) end, align and offset.
157 Time zones
159 The resulting timestamps can be returned having been evaluated for a
160 specific timezone, using the timezone or hostzone keywords. The value
161 associated with timezone will be interpreted by pmNewZone(3). A true
162 or false value should be associated with hostzone, and when set to true
163 this has the same effect as described by pmNewContextZone(3).
164
166 Using command line options, pmseries can be requested to provide meta‐
167 data (metric names, instance names, labels, descriptors) associated
168 with either individual timeseries or a group of timeseries, for exam‐
169 ple:
170 $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
172 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
174 PMID: 60.0.21
175 Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
176 Semantics: counter Units: millisec
177 Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
178 Metric: kernel.all.cpu.nice
179 labels {"agent":"linux","domainname":"localdomain",\
180 "groupid":1000,"hostname":"shard",\
181 "latitude":-25.28496,"longitude":152.87886,\
182 "machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
183 "userid":1000}
184 The complete set of pmseries metadata reporting options are:
186 -a, --all
188 Convenience option to report all metadata for the given time‐
189 series, equivalent to -dilms.
190 -d, --desc
192 Metric descriptions detailing the PMID, data type, data semantics,
193 units, scale and associated instance domain. This option has a
194 direct pminfo(1) equivalent.
195 -g pattern, --glob=pattern
197 Provide a glob(7) pattern to restrict the report provided by the
198 -i, -l, -m, and -S.
199 -i, --instances
201 Metric descriptions detailing the PMID, data type, data semantics,
202 units, scale and associated instance domain.
203 -I, --fullindom
205 Print the InDom in verbose mode. This option has a direct pmin‐
206 fo(1) equivalent.
207 -l, --labels
209 Print label sets associated with metrics and instances. Labels
210 are optional metric metadata described in detail in pmLookupLa‐
211 bels(3). This option has a direct pminfo(1) equivalent.
212 -m, --metrics
214 Print metric names.
215 -M, --fullpmid
217 Print the PMID in verbose mode. This option has a direct pmin‐
218 fo(1) equivalent.
219 -n, --names
221 Print comma-separated label names only (not values) for the labels
222 associated with metrics and instances.
223 -s, --series
225 Print timeseries identifiers associated with metrics, instances
226 and sources. These unique identifiers are calculated from intrin‐
227 sic (non-optional) labels and other metric metadata associated
228 with each PMAPI context (sources), metrics and instances. Ar‐
229 chive, local context or pmcd(1) connections for the same host all
230 produce the same source identifier. This option has a direct
231 pminfo(1) equivalent. See also pmLookupLabels(3) and the -l/--la‐
232 bels option.
233
235 A source is a unique identifier (represented externally as a 40-byte
236 hexadecimal SHA-1 hash) that represents both the live host and/or ar‐
237 chives from which each timeseries originated. The context for a source
238 identifier (obtained with -s) can be reported with:
239 -S, --sources
241 Print names for timeseries sources. These names are either host‐
242 names or fully qualified archive paths.
243 It is important to note that live and archived sources can and will
245 generate the same SHA-1 source identifier hash, provided that the con‐
246 text labels remain the same for that host (labels are stored in PCP ar‐
247 chives and can also be fetched live from pmcd(1)).
248
250 Timeseries metadata and data are loaded either automatically by a local
251 pmproxy(1), or manually using a specially crafted pmseries query and
252 the -L/--load option:
253 $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
255 pmseries: [Info] processed 2275 archive records from [...]
256 This query must specify a source archive path, but can also restrict
258 the import to specific timeseries (using metric names, labels, etc) and
259 to a specific time window using the time specification component of the
260 query language.
261 As a convenience, if the argument to load is a valid file path as de‐
263 termined by access(2), then a short-hand form can be used:
264 $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0
266
268 The available command line options, in addition to timeseries metadata
269 and sources options described above, are:
270 -c config, --config=config
272 Specify the config file to use.
273 -h host, --host=host
275 Connect Redis server at host, rather than the one the localhost.
276 -L, --load
278 Load timeseries metadata and data into the Redis cluster.
279 -p port, --port=port
281 Connect Redis server at port, rather than the default 6379.
282 -q, --query
284 Perform a timeseries query. This is the default action.
285 -t, --times
287 Report time stamps numerically (in milliseconds) instead of the
288 default human readable form.
289 -v, --values
291 Report all of the known values for given label name(s).
292 -V, --version
294 Display version number and exit.
295 -Z timezone, --timezone=timezone
297 Use timezone for the date and time. Timezone is in the format of
298 the environment variable TZ as described in environ(7).
299 -?, --help
301 Display usage message and exit.
302
304 The following sample query shows several fundamental aspects of the pm‐
305 series query language:
306 $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'
308 eb713a9cf472f775aa59ae90c43cd7f960f7870f
310 [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
311 [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
312 [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
313 [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
314 [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
315 [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
316 This query returns the two most recent values for all instances of the
318 kernel.all.load metric with a label.hostname matching the regular ex‐
319 pression "toium". This is a set-valued metric (i.e., a metric with an
320 ``instance domain'' which in this case consists of three instances: 1,
321 5 and 15 minute averages). The first column returned is a timestamp,
322 then a floating point value, and finally an instance identifier time‐
323 series hash (two values returned for three instances, so six rows are
324 returned). The metadata for these timeseries can then be further exam‐
325 ined:
326 $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
328 eb713a9cf472f775aa59ae90c43cd7f960f7870f
330 PMID: 60.2.0
331 Data Type: float InDom: 60.2 0xf000002
332 Semantics: instant Units: none
333 Source: 0e89c1192db79326900d82131c31399524f0b3ee
334 Metric: kernel.all.load
335 inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
336 inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
337 inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
338 inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
339 inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
340 inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
341
343 Environment variables with the prefix PCP_ are used to parameterize the
344 file and directory names used by PCP. On each installation, the file
345 /etc/pcp.conf contains the local values for these variables. The
346 $PCP_CONF variable may be used to specify an alternative configuration
347 file, as described in pcp.conf(5).
348 For environment variables affecting PCP tools, see pmGetOptions(3).
350
352 PCPIntro(1), pmcd(1), pminfo(1), pmproxy(1), redis-server(1), ac‐
353 cess(2), PMAPI(3), PMWEBAPI(3), pmLookupDesc(3), pmLookupInDom(3), pm‐
354 LookupLabels(3), pmLookupName(3), pmNewContextZone(3), pmNewZone(3),
355 pmParseInterval(3), pmParseTimeWindow(3), pcp.conf(5), environ(7),
356 glob(7) and regex(7).
357Performance Co-Pilot PCP PMSERIES(1)