1PMDAOPENMETRICS(1) General Commands Manual PMDAOPENMETRICS(1)
2
6 pmdaopenmetrics - OpenMetrics PMDA
7
9 $PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics [-D] [-n] [-c config] [-d
10 domain] [-l logfile] [-r root] [-t timeout] [-u user]
11
13 pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which dy‐
14 namically creates PCP metrics from configured OpenMetrics endpoints,
15 which provide HTTP based access to application metrics. The PMDA es‐
16 sentially implements a bridge between Prometheus and PCP, allowing PCP
17 to easily ingest performance data from more than 650 registered end-
18 points and many other application specific end-points.
19 The default config directory is $PCP_PMDAS_DIR/openmetrics/config.d/,
21 see ``CONFIGURATION SOURCES'' below. The default URL fetch timeout is
22 2 seconds. The default user, if not specified with the -u option, is
23 the current user. If the -n option is given, the list of configuration
24 files will not be sorted prior to processing. This list is sorted by
25 default but that can be expensive if there are a large number of con‐
26 figuration files (URLs and/or scripts).
27 If the -D option is given, additional diagnostic messages will be writ‐
29 ten to the PMDA log file, which is $PCP_LOG_DIR/pmcd/openmetrics.log by
30 default (see also -lbelow). In addition, the metric openmetrics.con‐
31 trol.debug controls the same debug flag and can be set with the follow‐
32 ing command:
33 pmstore openmetrics.control.debug value
34 where value is either 1 (to enable verbose log messages) or 0 (to dis‐
35 able verbose log messages). This is particularly useful for examining
36 the http headers passed to each fetch request, filter settings and
37 other processing details that are logged when the debugging flag is en‐
38 abled.
39 The -d option may be used to override the default performance metrics
41 domain number, which defaults to 144. It is strongly recommended not
42 to change this. The domain number should be different for every PMDA
43 on the one host, and the same domain number should be used for pm‐
44 daopenmetrics PMDA on all hosts. See also the -r option, which allows
45 the root of the dynamic namespace to be changed from the default open‐
46 metrics.
47 The -l option may be used to specify logfile as the destination for
49 PMDA messages instead of the default, $PCP_LOG_DIR/pmcd/open‐
50 metrics.log. As a special case, logfile may be "-" to send messages to
51 the stderr stream instead, e.g. -l-. This would normally be the
52 stderr stream for the parent process, pmcd(1), which may itself have
53 redirected stderr. This redirection is normally most useful in a con‐
54 tainerized environment, or when using dbpmda(1).
55 The -r option allows the root of the dynamic namespace to be changed to
57 root from the default, openmetrics. In conjunction with other command
58 line options, this allows pmdaopenmetrics to be deployed as a different
59 PMDA with distinct metrics namespace and metrics domain on the same
60 host system. Note that all PMDAs require a unique domain number so the
61 -d option must also be specified. Use of the -r option may also change
62 the defaults for some other command line options, e.g. the default log
63 file name and the default configuration directory.
64
66 As it runs, pmdaopenmetrics periodically recursively scans the $PCP_PM‐
67 DAS_DIR/openmetrics/config.d directory (or the directory specified with
68 the -c option), looking for source URL files (*.url) and executable
69 scripts or binaries. Any files that do not have the .url suffix or are
70 not executable, are ignored - this allows documentation files such as
71 "README" and non-executable "common" script function definitions to be
72 present without being considered as config files.
73 A remote server does not have to be up or stay running - the PMDA tol‐
75 erates remote URLs that may come and go over time. The PMDA will relay
76 data and metadata when/if they are available, and will return errors
77 when/if they are down. PCP metric IDs, internal and external instance
78 domain identifiers are persisted and will be restored when individual
79 metric sources become available and/or when the PMDA is restarted. In
80 addition, the PMDA checks directory modification times and will rescan
81 for new or changed configuration files dynamically. It is not neces‐
82 sary to restart the PMDA when adding, removing or changing configura‐
83 tion files.
84
86 Each file with the .url suffix found in the config directory or sub-di‐
87 rectory contains one complete HTTP or HTTPS URL at which pmdaopen‐
88 metrics can reach a OpenMetrics endpoint. Local file access is also
89 supported with a conventional file:///somepath/somefile URL, in which
90 case somepath/somefile should contain openmetrics formatted metric
91 data.
92 The first line of a .url config file should be the URL, as described
94 above. Subsequent lines, if any, are prefixed with a keyword that can
95 be used to alter the http GET request. A keyword must end with ':'
96 (colon) and the text extends to the end of the line. Comment lines
97 that start with # and blank lines are ignored. The only currently sup‐
98 ported keywords are HEADER: and FILTER:.
99 HEADER: headername: value ... to end of line
101 Adds headername and its value to the headers passed in the http GET re‐
102 quest for the configured URL. An example configuration file that pro‐
103 vides 3 commonly used headers and an authentication token might be :
104 http://somehost/path/endpoint.html
106 # this is a comment
107 HEADER: Accept: text/html
108 HEADER: Keep-Alive: 300
109 HEADER: Connection: keep-alive
110 HEADER: Authorization: token ABCDEF1234567890
111 As mentioned above, header values extend to the end of the line. They
113 may contain any valid characters, including colons. Multiple spaces
114 will be collapsed to a single space, and leading and trailing spaces
115 are trimmed. A common use for headers is to configure a proxy agent
116 and the assorted parameters it may require.
117
119 Metric filtering is a configuration file feature that allows ingested
120 metrics to be included or excluded, i.e. filtered. This is useful be‐
121 cause most end-points return multiple metrics, and usually only some
122 are interesting for monitoring purposes. The syntax is:
123 FILTER: INCLUDE METRIC regex
124 or
125 FILTER: EXCLUDE METRIC regex
126 Dynamically created metric names that match regex will be either in‐
127 cluded or excluded in the name space, as specified. Note that only the
128 PMNS leaf component of the metric name (as ingested from the URL
129 source) is compared with the regex pattern. The simple rule is that
130 the first matching filter regex for a particular metric leaf name is
131 the rule that prevails. If no filter regex matches (or there are no
132 filters), then the metric is included by default, i.e. the default fil‐
133 ter if none are specified is FILTER: INCLUDE METRIC .* This is back‐
134 ward compatible with older versions of the configuration file that did
135 not support filters. Multiple FILTER: lines would normally be used,
136 e.g. to include some metrics but exclude all others, use FILTER: EX‐
137 CLUDE METRIC .* as the last of several filters that include the de‐
138 sired metrics. Conversely, to exclude some metrics but include all
139 others, use FILTER: EXCLUDE METRIC regex. In this case it's not neces‐
140 sary (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC
141 .* because, as stated above, any metric that does not match any filter
142 regex will be included by default.
143
145 Label filtering uses similar FILTER: syntax and semantics as metric
146 filtering. FILTER: EXCLUDE LABEL regex will delete all labels with la‐
147 bel name matching regex from all metrics defined by the configuration
148 file. The same rules as for metric filters apply for label filters too
149 - an implicit rule: FILTER: INCLUDE LABEL .* applies to all labels
150 that do not match any earlier label filter rule. FILTER: OPTIONAL LA‐
151 BEL regex specifies that matching label names are to be included in the
152 returned metric labelsets (i.e. included), but are not to be used as
153 part of the the external instance names. All included labels that are
154 not optional (i.e. the intrinsic labels) will be concatenated together
155 and used for external instance naming. In addition, non-intrinsic la‐
156 bels (i.e. labels tagged as OPTIONAL) will have the PM_LABEL_OPTIONAL
157 flag set in the labelsets returned by notes callbacks. This flag af‐
158 fects how the labels are used in certain clients. For further details,
159 see pmLookupLabels(3) and related man pages for further details. Note
160 that external instance names begin with the unique numeric internal in‐
161 stance identifier followed by a space, so external instance names are
162 always unique.
163 Caution is needed with label filtering because by default, all labels
165 are used to construct the PCP instance name. By excluding some labels
166 (or changing them to optional), the instance names will change. In ad‐
167 dition, excluding all labels for a particular metric changes that met‐
168 ric to be singular, i.e. have no instance domain. By excluding some
169 labels, different instances returned by the URL or scripted configura‐
170 tion entry for the same metric may become duplicates. When such dupli‐
171 cates occur, the last duplicate instance returned by the end-point URL
172 or script prevails over any earlier instances. For these reasons, it
173 is recommended that label filtering rules be configured when the con‐
174 figuration file is first defined, and not changed thereafter. If a la‐
175 bel filtering change is required, the configuration file should be re‐
176 named, which effectively defines a new metric (or set of peer metrics
177 as returned by the URL or script), with the new (or changed) instance
178 naming.
179 Unrecognized keywords in configuration files are reported in the PMDA
181 log file but otherwise ignored.
182
184 Executable scripts present in the $PCP_PMDAS_DIR/openmetrics/config.d
185 directory or sub-directories will be executed and the stdout stream
186 containing openmetrics formatted metric data will be parsed as though
187 it had come from a URL or file. The stderr stream from a script will
188 be sent to the PMDA log file, which by default can be found in
189 $(PCP_LOG_DIR)/pmcd/openmetrics.log.
190 Note that scripted sources do not support label or metric filtering (as
192 described above for URL sources) - they can simply do their own filter‐
193 ing in the script itself with sed(1), awk(1), or whatever tool is de‐
194 sired.
195 A simple example of a scripted config entry follows:
197 #! /bin/sh
199 awk '{
200 print("# HELP loadavg local load average")
201 print("# TYPE loadavg gauge")
202 printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
203 printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
204 printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
205 }' /proc/loadavg
206 This script produces the following OpenMetrics-formatted metric data
208 when run:
209 # HELP loadavg local load average
211 # TYPE loadavg gauge
212 loadavg {interval="1-minute"} 0.12
213 loadavg {interval="5-minute"} 0.27
214 loadavg {interval="15-minute"} 0.54
215 If the above script was saved and made executable in a file named
217 $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh then this would re‐
218 sult in a new PCP metric named openmetrics.local.system.loadavg which
219 would have three instances for the current load average values:
220 1-minute, 5-minute and 15-minute.
221 Scripted config entries may produce more than one PCP leaf metric name.
223 For example, the above "system.sh" script could also export other met‐
224 rics such as CPU statistics, by reading /proc/stat on the local system.
225 Such additional metrics would appear as peer metrics in the same PCP
226 metric subtree. In the case of CPU counters, the metric type defini‐
227 tion should be counter, not gauge. For full details of the openmetrics
228 exposition formats, see https://github.com/OpenObservability/Open‐
229 Metrics/blob/master/specification/OpenMetrics.md.
230
232 Scripted config files are executed by the pmdaopenmetrics PMDA with the
233 same SELinux context and policy as the local pmcd(1). For simple
234 scripts, such as the load average example described above, this is nor‐
235 mally fine. However AVC errors may result for scripts that make li‐
236 brary or system calls that are restricted by the prevailing SELinux
237 context and policies. In these cases it is not feasible to unilater‐
238 ally grant pmcd or it's PMDAs an unconfined execution policy. In these
239 site specific cases it will be necessary to create a local SELinux pol‐
240 icy module. This can be done by capturing the AVC record(s) from the
241 local audit log, generate a local policy module using audit2allow, and
242 then load the new module using semodule, e.g. as follows :
243 $ sudo grep '^type=AVC.*pcp' /var/log/audit/audit.log \
245 | audit2allow -M mypolicy
246 $ sudo semodule -i mypolicy.pp
247 If these local policies need to be persistent across reboots, then a
249 scriptlet similar to the above example may be added to the local pmcd
250 RC file (typically /etc/pcp/pmcd/rc.local). For further details, see
251 audit2allow(1) and semodule(1).
252
254 All metrics from a file named JOB.* will be exported as PCP metrics
255 with the openmetrics.JOB metric name prefix. Therefore, the JOB name
256 must be a valid non-leaf name for PCP PMNS metric names. If the JOB
257 name has multiple dot-separated components, the resulting PMNS names
258 will include those components and care is needed to ensure there are no
259 overlapping definitions, e.g. metrics returned by JOB.response may
260 overlap or conflict with metrics returned by JOB.response.time.
261 Config file entries (URLs or scripts) found in subdirectories of the
263 config directory will also result in hierarchical metric names. For
264 example, a config file named $PCP_PMDAS_DIR/openmetrics/con‐
265 fig.d/mysource/latency/get.url will result in metrics being created (by
266 fetching that source URL) below openmetrics.mysource.latency.get in the
267 PCP namespace. Scripts found in subdirectories of the config directory
268 similarly result in hierarchical PCP metric names.
269
271 As described above, changes and new additions can be made to files in
272 the configuration directory without having to restart the PMDA. These
273 changes are detected automatically and the PCP metric names below open‐
274 metrics in the PMNS will be updated accordingly, i.e. new metrics will
275 be dynamically added and/or existing metrics removed. In addition, pm‐
276 daopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3) protocol that was
277 introduced in PCP version 4.0. In particular, if openmetrics metrics
278 are being logged by a PCP version 4.0 or later pmlogger(1), new metrics
279 that appear as a result of changes in the PMDA configuration directory
280 will automatically start to be logged, provided the root of the open‐
281 metrics PMDA namespace is configured for logging in the pmlogger con‐
282 figuration file. See pmlogger(1) for details. An example of such a
283 pmlogger configuration file is :
284 log mandatory on 2 second {
286 # log all metrics below the root of the openmetrics namespace
287 openmetrics
288 }
289
291 Metric data returned by URL or scripted configuration files may contain
292 metadata that can be used by the openmetrics PMDA to specify the seman‐
293 tics, data type, scaling and units of dynamically created metrics.
294 This metadata is prefixed with # PCP5 or # PCP in the ingested metric
295 data. For additional information about PCP metadata, see pm‐
296 LookupDesc(3) and pmParseUnitsStr(3) and examples in shipped configura‐
297 tion files.
298 In-line "PCP5" metadata must be supplied by the metrics source end-pont
300 (URL or script). An alternative is to specify this in the URL configu‐
301 ration file directly, which has the advantage of not having to modify
302 the source/end-point if the metadata is incorrect or missing. Metadata
303 specified in the URL configuration file over-rides any in-line meta‐
304 data.
305 The configuration file syntax for specifying metadata is:
307 METADATA: regex type indom semantics units ... to EOL
308 Where:
309 METADATA: is literal
310 regex is an extended regular expression to match one or more metric
311 names returned by the URL,
312 type is one of the PCP numeric types (32, u32, 64, u64, float or dou‐
313 ble),
314 indom is an arbitrary instance domain name, or PM_INDOM_NULL,
315 semantics is either counter, instant or discrete and
316 units is either none or a string extending to end of line that is
317 parseable by pmParseUnitsStr(3), i.e. the units, dimensions and scaling
318 to be used for matching metrics.
319 An example configuration file that ingests metrics from the Grafana
321 /metrics end-point on localhost, filters out all metrics returned by
322 that URL except for grafana_api_response_status_total and then speci‐
323 fies the metric type is an unsigned 32 bit integer with a non-singular
324 instance domain named response and counter semantics with units of
325 count.
326 http://localhost:3000/metrics
328 FILTER: INCLUDE METRIC grafana_api_response_status_total
329 FILTER: EXCLUDE METRIC .*
330 METADATA: grafana_api_response_status_total u32 response counter count
331 Note that the name in the indom field is presently ignored unless it is
333 PM_INDOM_NULL, in which case the metric has no instance domain (i.e.
334 singular), even if it has labels which would otherwise be used for in‐
335 stance naming.
336
338 The PMDA maintains special control metrics, as described below. Apart
339 from openmetrics.control.debug, each of these metrics has one instance
340 for each configured metric source. All of these metrics have integer
341 values with counter semantics, except openmetrics.control.status, which
342 has a string value. It is important to note that fetching any of the
343 openmetrics.control metrics will only update the counters and status
344 values if the corresponding URL is actually fetched. If the source URL
345 is not fetched, the control metric values do not trigger a refresh and
346 the control values reported represent the most recent fetch of each
347 corresponding source.
348 The instance domain for the openmetrics.control metrics is adjusted dy‐
350 namically as new sources are discovered. If there are no sources con‐
351 figured, the metric names are still defined but the instance domain
352 will be empty and a fetch will return no values.
353 openmetrics.control.status
355 A string representing the status of the last fetch of the corre‐
356 sponding source. This will generally be success for an http re‐
357 sponse code of 200. This metric can be used for service avail‐
358 ability monitoring - provided, as stated above, the correspond‐
359 ing source URL is fetched too.
360 openmetrics.control.status_code
362 This metric is similar to openmetrics.control.status except that
363 it is the integer response code of the last fetch. A value of
364 200 usually signifies success and any other value failure. This
365 metric can also be used for service availability monitoring,
366 with the same caveats as openmetrics.control.status.
367 openmetrics.control.calls
369 total number of times each configured metric source has been
370 fetched (if it's a URL) or executed (if it's a script), since
371 the PMDA started. This metric has counter semantics and would
372 normally be converted to a rate/second by client tools.
373 openmetrics.control.fetch_time
375 Total time in milliseconds that each configured metric source
376 has taken to return a document, excluding the time to parse the
377 document. This metric has counter semantics and would normally
378 be rate converted by client tools but is also useful in raw form
379 as the accumulated parse time since the PMDA was started.
380 openmetrics.control.parse_time
382 Total time in milliseconds that each configured metric source
383 has taken to parse each document, excluding the time to fetch
384 the document. This metric has counter semantics and would nor‐
385 mally be rate converted by client tools but is also useful in
386 raw form as the accumulated parse time since the PMDA was
387 started.
388 When converted to a rate, the calls metric represents the average fetch
390 rate of each source over the sampling interval (time delta between sam‐
391 ples). The fetch_time and parse_time counters, when converted to a
392 rate, represent the average fetch and parsing latency (respectfully),
393 during the sampling interval.
394 The openmetrics.control.debug metric has a singular value, defaulting
396 to 0. If a non-zero value is stored into this metric using pmstore(1),
397 additional debug messages will be written to the PMDA log file.
398
400 pmdaopenmetrics and libpcp internals impose some numerical constraints
401 about the number of sources (4095), metrics (1024) within each source,
402 and instances for each metric (4194304).
403
405 Install the OpenMetrics PMDA by using the Install script as root:
406 # cd $PCP_PMDAS_DIR/openmetrics
408 # ./Install
409 To uninstall, the following must be done as root:
411 # cd $PCP_PMDAS_DIR/openmetrics
413 # ./Remove
414 pmdaopenmetrics is launched by pmcd(1) and should never be executed di‐
416 rectly. The Install and Remove scripts notify pmcd when the agent is
417 installed or removed.
418 When scripts and .url files are added, removed or changed in the con‐
420 figuration directory, it is usually not necessary to restart the PMDA -
421 the changes will be detected and managed on subsequent requests to the
422 PMDA.
423
425 $PCP_PMDAS_DIR/openmetrics/Install
426 installation script for the pmdaopenmetrics agent
427 $PCP_PMDAS_DIR/openmetrics/Remove
429 undo installation script for the pmdaopenmetrics agent
430 $PCP_PMDAS_DIR/openmetrics/config.d/
432 contains URLs and scripts used by the pmdaopenmetrics agent as
433 sources of openmetrics metric data.
434 $PCP_LOG_DIR/pmcd/openmetrics.log
436 default log file for error messages from pmdaopenmetrics
437 $PCP_VAR_DIR/config/144.*
439 files containing internal tables for metric and instance ID number
440 persistence (domain 144).
441
443 Environment variables with the prefix PCP_ are used to parameterize the
444 file and directory names used by PCP. On each installation, the file
445 /etc/pcp.conf contains the local values for these variables. The
446 $PCP_CONF variable may be used to specify an alternative configuration
447 file, as described in pcp.conf(5).
448
450 PCPIntro(1), audit2allow(1), pmcd(1), pminfo(1), pmlogger(1), pm‐
451 store(1), PMWEBAPI(3), pmFetch(3), pmLookupLabels(3), semodule(1), and
452 https://prometheus.io/docs/instrumenting/exposition_formats.
453Performance Co-Pilot PCP PMDAOPENMETRICS(1)