1PMSERIES(1) General Commands Manual PMSERIES(1)
2
6 pmseries - display information about performance metric timeseries
7
9 pmseries [-adFilLmMnqsS] [-p port] [-h host] [query | series ... |
10 source ... ]
11
13 pmseries displays various types of information about performance met‐
14 rics available through the scalable timeseries facilities of the Per‐
15 formance Co-Pilot (PCP) and the Redis distributed data store.
16 By default pmseries communicates with a local redis-server(1), however
18 the -p/--port option and -h/--host options can be used to specify an
19 alternate Redis instance. If this instance is a node of a Redis clus‐
20 ter, all other instances in the cluster will be discovered and used
21 automatically.
22 pmseries runs in several different modes - either querying timeseries
24 identifiers, metadata or values (already stored in Redis), or manually
25 loading timeseries into Redis. The latter mode is seldom used, how‐
26 ever, since pmproxy(1) will automatically perform this function when it
27 is started with the -t/--timeseries option. In time, this option will
28 become enabled by default.
29 Without command line options specifying otherwise, pmseries will issue
31 a timeseries query to find matching timeseries and values. All time‐
32 series are identified using a unique SHA-1 hash which is always dis‐
33 played in a 40-hexdigit human readable form. These hashes are formed
34 using the metadata associated with every metric.
35 Importantly, this includes all metric metadata (labels, names, descrip‐
37 tors). Metric labels in particular are (as far as possible) unique for
38 every machine - on Linux for example the labels associated with every
39 metric include the unique /etc/machine-id, the hostname, domainname,
40 and other automatically generated machine labels, as well as any admin‐
41 istrator-defined labels from /etc/pcp/labels. These labels can be
42 reported with pminfo(1) and the pmcd.labels metric.
43 See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and
45 pmLookupDesc(3) for detailed information about metric labels and other
46 metric metadata used in each timeseries identifier hash calculation.
47 The timeseries identifiers provide a higher level (and machine indepen‐
49 dent) identifier than the traditional PCP performance metric identi‐
50 fiers (pmID), instance domain identifiers (pmInDom) and metric names.
51 However, pmseries uses timeseries identifiers in much the same way that
52 pminfo uses the lower level indom, metric identifiers and metric names.
53
55 The default mode of pmseries operation is to issue a query that will be
56 evaluated over all known timeseries. Query expressions are formed
57 using the pmseries query language described below, but can be as simple
58 as a metric name.
59 The following is an example of querying timeseries from all hosts that
61 match a metric name pattern (globbed):
62 $ pmseries kernel.all.cpu*
64 1d7b0bb3f6aec0f49c54f5210885464a53629b60
65 379db729afd63fb9eff436625bd6c55a7adc5cfd
66 3dd3b45bb05f96636043e5d58b52b441ce542285
67 [...]
68 ed2bf325ff6dc7589ec966698e5404b67252306a
69 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
70 To identify timeseries, the query language uses the general syntax:
72 [metric.name] '{' metadata qualifiers '}' '[' time specification ']'
74 The metric.name component restricts the timeseries query to any match‐
76 ing PCP metric name (the list of metric names for a PCP archive or live
77 host is the reported by pminfo with no arguments beyond --host or --ar‐
78 chive). The pmseries syntax extends on that of pminfo and allows for
79 glob(7) based pattern matching within the metric name.
80 Metadata qualifiers are enclosed by ('curly') braces, and further re‐
82 strict the query results to timeseries with various metadata proper‐
83 ties. These qualifiers are based on metric or instance names, and met‐
84 ric label values, and take the general form metadata.name OPERATOR val‐
85 ue, such as:
86 instance.name == "cpu0"
88 metric.name != "kernel.all.pswitch"
89 When using label names, the metadata qualifier is optional and can be
91 dropped, such as:
92 label.hostname == "www.acme.com"
94 hostname == "www.acme.com"
95 For metric and instance names only the string operators apply, but for
97 metric label values all operators are available. The set of available
98 operators is:
99 Boolean operators
101 All string (label, metrics and instances) and numeric (label) values
102 can be tested for equality ("==") or inequality ("!=").
103 String operators
105 Strings can be subject to pattern matching in the form of glob matching
106 ("~~"), regular expression matching ("=~"), and regular expression non-
107 matching ("!~"). The ":" operator is equivalent to "~~" - i.e. regular
108 expression matching.
109 Relational operators (numeric label values only)
111 Numeric label values can be subject to the less than ("<"), greater
112 than (">"), less than or equal ("<="), greater than or equal (">="),
113 equal ("==") and not equal ("!=") operators.
114 Logical operators
116 Multiple metadata qualifiers can be combined with the logical operators
117 for AND ("&&") and OR ("||") as in many programming languages. The
118 comma (",") character is equivalent to logical AND ("&&").
119
121 The final (optional) component of a query allows the user to specify a
122 specific time window of interest. Any time specification will result
123 in values being returned for all matching timeseries, for the time win‐
124 dow specified.
125 The specification is ('square') bracket enclosed, and consists of one
127 or more comma-separated components. Each component specifies some as‐
128 pect related to time, taking the general form: keyword: value, such as:
129 samples: 10
131 Sample count
133 The number of samples to return, specified via either the samples or
134 (equivalent) count keyword. The value provided must be a positive in‐
135 teger. If no end time is explicitly set (see ``Time window'' later)
136 then the most recent samples will be returned.
137 Sample interval
139 An interval between successive samples can be requested using the in‐
140 terval or (equivalent) delta keyword. The value provided should be ei‐
141 ther a numeric or string value that will be parsed by pmParseTimeInter‐
142 val(3).
143 Time window
145 Start and end times, and alignments, affecting the returned values.
146 The keywords match the parameters to the pmParseTimeWindow(3) function
147 which will be used to parse them, and are: start or (equivalent) begin,
148 finish or (equivalent) end, align and offset.
149 Time zones
151 The resulting timestamps can be returned having been evaluated for a
152 specific timezone, using the timezone or hostzone keywords. The value
153 associated with timezone will be interpreted by pmNewZone(3). A true
154 or false value should be associated with hostzone, and when set to true
155 this has the same effect as described by pmNewContextZone(3).
156
158 Using command line options, pmseries can be requested to provide meta‐
159 data (metric names, instance names, labels, descriptors) associated
160 with either individual timeseries or a group of timeseries, for exam‐
161 ple:
162 $ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
164 dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
166 PMID: 60.0.21
167 Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
168 Semantics: counter Units: millisec
169 Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
170 Metric: kernel.all.cpu.nice
171 labels {"agent":"linux","domainname":"localdomain",\
172 "groupid":1000,"hostname":"shard",\
173 "latitude":-25.28496,"longitude":152.87886,\
174 "machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
175 "userid":1000}
176 The complete set of pmseries metadata reporting options are:
178 -a, --all
180 Convenience option to report all metadata for the given time‐
181 series, equivalent to -dilms.
182 -d, --desc
184 Metric descriptions detailing the PMID, data type, data semantics,
185 units, scale and associated instance domain. This option has a
186 direct pminfo(1) equivalent.
187 -i, --instances
189 Metric descriptions detailing the PMID, data type, data semantics,
190 units, scale and associated instance domain.
191 -I, --fullindom
193 Print the InDom in verbose mode. This option has a direct pmin‐
194 fo(1) equivalent.
195 -l, --labels
197 Print label sets associated with metrics and instances. Labels
198 are optional metric metadata described in detail in pmLookupLa‐
199 bels(3). This option has a direct pminfo(1) equivalent.
200 -m, --metrics
202 Print metric names.
203 -M, --fullpmid
205 Print the PMID in verbose mode. This option has a direct pmin‐
206 fo(1) equivalent.
207 -s, --series
209 Print time series identifiers associated with metrics, instances
210 and sources. These unique identifiers are calculated from intrin‐
211 sic (non-optional) labels and other metric metadata associated
212 with each PMAPI context (sources), metrics and instances. Ar‐
213 chive, local context or pmcd(1) connections for the same host all
214 produce the same source identifier. This option has a direct
215 pminfo(1) equivalent. See also pmLookupLabels(3) and the -l/--la‐
216 bels option.
217
219 A source is a unique identifier (represented externally as a 40-byte
220 hexadecimal SHA-1 hash) that represents both the live host and/or ar‐
221 chives from which each timeseries originated.
222 The -S/--sources option reports names for timeseries sources. These
224 names are either hostnames or fully qualified archive paths.
225 It is important to note that live and archived sources can and will
227 generate the same SHA-1 source identifier hash - that is, provided the
228 context labels remain the same for that host (labels are stored in PCP
229 archives and can also be fetched live from pmcd).
230
232 Timeseries metadata and data are loaded either automatically by a local
233 pmproxy, or manually using a specially crafted pmseries query and the
234 -l/--load option.
235 $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
237 pmseries: [Info] processed 2275 archive records from [...]
238 This query must specify a source archive path, but can also restrict
240 the import to specific timeseries (using metric names, labels, etc) and
241 to specific a time window using the time specification component of the
242 query language.
243
245 The following sample query shows several fundamental aspects of the pm‐
246 series query language:
247 $ pmseries 'kernel.all.load{hostname:toium}[count:2]'
249 eb713a9cf472f775aa59ae90c43cd7f960f7870f
251 [1545346764128.128] 1.000000e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
252 [1545346764128.128] 6.800000e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
253 [1545346764128.128] 6.400000e-01 e1974a062375e6e62370ffadf5b0650dad739480
254 [1545346774112.112] 1.600000e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
255 [1545346774112.112] 6.700000e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
256 [1545346774112.112] 6.400000e-01 e1974a062375e6e62370ffadf5b0650dad739480
257 This query returns the two most recent values for all instances of the
259 kernel.all.load metric with a label.hostname matching the regular ex‐
260 pression "toium". This is a set-valued metric (i.e. with an instance
261 domain or 1, 5 and 15 minute averages). The first column returned is a
262 timestamp, then a floating point value, and finally an instance identi‐
263 fier timeseries hash (two values returned for three instances, so six
264 rows are returned). The metadata for these timeseries can then be fur‐
265 ther examined:
266 $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f
268 eb713a9cf472f775aa59ae90c43cd7f960f7870f
270 PMID: 60.2.0
271 Data Type: float InDom: 60.2 0xf000002
272 Semantics: instant Units: none
273 Source: 0e89c1192db79326900d82131c31399524f0b3ee
274 Metric: kernel.all.load
275 inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
276 inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
277 inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
278 inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
279 inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
280 inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}
281
283 Environment variables with the prefix PCP_ are used to parameterize the
284 file and directory names used by PCP. On each installation, the file
285 /etc/pcp.conf contains the local values for these variables. The
286 $PCP_CONF variable may be used to specify an alternative configuration
287 file, as described in pcp.conf(5).
288
290 pminfo(1), pmproxy(1), redis-server(1), PMAPI(3), PMWEBAPI(3), pmLooku‐
291 pLabels(3), pmLookupInDom(3), pmLookupName(3), pmLookupDesc(3), pmNew‐
292 Zone(3). pmNewContextZone(3). pmParseTimeInterval(3), pmParseTimeWin‐
293 dow(3), pcp.conf(5), pcp.env(5), glob(7) and regex(7).
294Performance Co-Pilot PCP PMSERIES(1)