1PMDAPROMETHEUS(1) General Commands Manual PMDAPROMETHEUS(1)
2
6 pmdaprometheus - Prometheus PMDA
7
9 $PCP_PMDAS_DIR/prometheus/pmdaprometheus [-D] [-n] [-c config] [-d
10 domain] [-l logfile] [-r root] [-t timeout] [-u user]
11
13 pmdaprometheus is a Performance Metrics Domain Agent (PMDA) which cre‐
14 ates PCP metrics from Prometheus endpoints, which provide HTTP based
15 access to application metrics. The default config directory is
16 $PCP_PMDAS_DIR/prometheus/config.d/, see ``CONFIGURATION SOURCES''
17 below. The default URL fetch timeout is 2 seconds. The default user,
18 if not specified with the -u option, is the current user. If the -n
19 option is given, the list of configuration files will not be sorted
20 prior to processing. This list is sorted by default but that can be
21 expensive if there are a large number of configuration files (URLs
22 and/or scripts).
23 If the -D option is given, additional diagnostic messages will be writ‐
25 ten to the PMDA log file, which is $PCP_LOG_DIR/pmcd/prometheus.log by
26 default (see also -lbelow). In addition, the metric prometheus.con‐
27 trol.debug controls the same debug flag and can be set with the follow‐
28 ing command:
29 pmstore prometheus.control.debug value
30 where value is either 1 (to enable verbose log messages) or 0 (to dis‐
31 able verbose log messages). This is particularly useful for examining
32 the http headers passed to each fetch request, filter settings and
33 other processing details that are logged when the debugging flag is
34 enabled.
35 The -d option may be used to override the default performance metrics
37 domain number, which defaults to 144. It is strongly recommended not
38 to change this. The domain number should be different for every PMDA
39 on the one host, and the same domain number should be used for
40 pmdaprometheus PMDA on all hosts. See also the -r option, which allows
41 the root of the dynamic namespace to be changed from the default
42 prometheus.
43 The -l option may be used to specify logfile as the destination for
45 PMDA messages instead of the default, $PCP_LOG_DIR/pmcd/prometheus.log.
46 As a special case, logfile may be "-" to send messages to the stderr
47 stream instead, e.g. -l-. This would normally be the stderr stream
48 for the parent process, pmcd(1), which may itself have redirected
49 stderr. This redirection is normally most useful in a containerized
50 environment, or when using dbpmda(1).
51 The -r option allows the root of the dynamic namespace to be changed to
53 root from the default, prometheus. In conjunction with other command
54 line options, this allows pmdaprometheus to be deployed as a different
55 PMDA with distinct metrics namespace and metrics domain on the same
56 host system. Note that all PMDAs require a unique domain number so the
57 -d option must also be specified. Use of the -r option may also change
58 the defaults for some other command line options, e.g. the default log
59 file name and the default configuration directory.
60
62 As it runs, pmdaprometheus periodically recursively scans the
63 $PCP_PMDAS_DIR/prometheus/config.d directory (or the directory speci‐
64 fied with the -c option), looking for source URL files (*.url) and exe‐
65 cutable scripts or binaries. Any files that do not have the .url suf‐
66 fix or are not executable, are ignored - this allows documentation
67 files such as "README" and non-executable "common" script function def‐
68 initions to be present without being considered as config files.
69 A remote server does not have to be up or stay running - the PMDA tol‐
71 erates remote URLs that may come and go over time. The PMDA will relay
72 data and metadata when/if they are available, and will return errors
73 when/if they are down. PCP metric IDs, internal and external instance
74 domain identifiers are persisted and will be restored when individual
75 metric sources become available and/or when the PMDA is restarted. In
76 addition, the PMDA checks directory modification times and will rescan
77 for new or changed configuration files dynamically. It is not neces‐
78 sary to restart the PMDA when adding, removing or changing configura‐
79 tion files.
80
82 Each file with the .url suffix found in the config directory or sub-
83 directory contains one complete HTTP or HTTPS URL at which
84 pmdaprometheus can reach a Prometheus endpoint. Local file access is
85 also supported with a conventional file://somepath/somefile URL, in
86 which case somepath/somefile should contain prometheus formatted metric
87 data.
88 The first line of a .url config file should be the URL, as described
90 above. Subsequent lines, if any, are prefixed with a keyword that can
91 be used to alter the http GET request. A keyword must end with ':'
92 (colon) and the text extends to the end of the line. Comment lines
93 that start with # and blank lines are ignored. The only currently sup‐
94 ported keywords are HEADER: and FILTER:.
95 HEADER: headername: value ... to end of line
97 Adds headername and its value to the headers passed in the http GET
98 request for the configured URL. An example configuration file that
99 provides 3 commonly used headers and an authentication token might be :
100 http://somehost/path/endpoint.html
102 # this is a comment
103 HEADER: Accept: text/html
104 HEADER: Keep-Alive: 300
105 HEADER: Connection: keep-alive
106 HEADER: Authorization: token ABCDEF1234567890
107 As mentioned above, header values extend to the end of the line. They
109 may contain any valid characters, including colons. Multiple spaces
110 will be collapsed to a single space, and leading and trailing spaces
111 are trimmed. A common use for headers is to configure a proxy agent
112 and the assorted parameters it may require.
113 FILTER: INCLUDE METRIC regex
115 or
116 FILTER: EXCLUDE METRIC regex
117 Dynamically created metric names that match regex will be either
118 included or excluded in the name space, as specified. The simple rule
119 is that the first matching filter regex for a particular metric name is
120 the rule that prevails. If no filter regex matches (or there are no
121 filters), then the metric is included by default, i.e. the default fil‐
122 ter if none are specified is FILTER: INCLUDE METRIC .* This is back‐
123 ward compatible with older versions of the configuration file that did
124 not support filters. Multiple FILTER: lines would normally be used,
125 e.g. to include some metrics but exclude all others, use FILTER:
126 EXCLUDE METRIC .* as the last of several filters that include the
127 desired metrics. Conversely, to exclude some metrics but include all
128 others, use FILTER: EXCLUDE METRIC regex. In this case it's not neces‐
129 sary (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC
130 .* because, as stated above, any metric that does not match any filter
131 regex will be included by default.
132 Label filtering uses similar FILTER: syntax and semantics. FILTER:
134 EXCLUDE LABEL regex will delete all labels matching regex from all met‐
135 rics defined in the configuration file. The same rules as for metrics
136 apply for labels too - an implicit rule: FILTER: INCLUDE LABEL .*
137 applies to all labels that do not match any earlier filter rule.
138 Caution is needed with label filtering because by default, all labels
140 are used to construct the PCP instance name. By excluding some labels,
141 the instance names will change. Excluding all labels for a particular
142 metric changes that metric to be singular, i.e. have no instance
143 domain. In addition, by excluding some labels, different instances of
144 the same metric may become duplicates. When such duplicates occur, the
145 last duplicate instance returned by the end-point URL prevails over any
146 earlier instances. For these reasons, it is recommended that label
147 filtering rules be configured when the configuration file is first
148 defined, and not changed thereafter. If a label filtering change is
149 required, the configuration file should be renamed, which effectively
150 defines a new metric, with the new (or changed) instance naming.
151 Unrecognized keywords in configuration files are reported in the PMDA
153 log file but otherwise ignored.
154
156 Executable scripts present in the $PCP_PMDAS_DIR/prometheus/config.d
157 directory or sub-directories will be executed and the stdout stream
158 containing prometheus formatted metric data will be parsed as though it
159 had come from a URL or file. The stderr stream from a script will be
160 sent to the PMDA log file, which by default can be found in
161 $(PCP_LOG_DIR)/pmcd/prometheus.log.
162 Note that scripted sources do not support label or metric filtering (as
164 described above for URL sources) - they can simply do their own filter‐
165 ing in the script itself with sed(1), awk(1), or whatever tool is
166 desired.
167 A simple example of a scripted config entry follows:
169 #! /bin/sh
171 awk '{
172 print("# HELP loadavg local load average")
173 print("# Type loadavg gauge")
174 printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
175 printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
176 printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
177 }' /proc/loadavg
178 This script produces the following Prometheus-formatted metric data
180 when run:
181 # HELP loadavg local load average
183 # Type loadavg gauge
184 loadavg {interval="1-minute"} 0.12
185 loadavg {interval="5-minute"} 0.27
186 loadavg {interval="15-minute"} 0.54
187 If the above script was saved and made executable in a file named
189 $PCP_PMDAS_DIR/prometheus/config.d/local/system.sh then this would
190 result in a new PCP metric named prometheus.local.system.loadavg which
191 would have three instances for the current load average values:
192 1-minute, 5-minute and 15-minute.
193 Scripted config entries may produce more than one PCP leaf metric name.
195 For example, the above "system.sh" script could also export other met‐
196 rics such as CPU statistics, by reading /proc/stat on the local system.
197 Such additional metrics would appear as peer metrics in the same PCP
198 metric subtree. In the case of CPU counters, the metric type defini‐
199 tion should be counter, not gauge. For full details of the prometheus
200 exposition formats, see https://prometheus.io/docs/instrumenting/expo‐
201 sition_formats.
202
204 All metrics from a file named JOB.* will be exported as PCP metrics
205 with the prometheus.JOB metric name prefix. Therefore, the JOB name
206 must be a valid non-leaf name for PCP PMNS metric names. If the JOB
207 name has multiple dot-separated components, the resulting PMNS names
208 will include those components and care is needed to ensure there are no
209 overlapping definitions, e.g. metrics returned by JOB.response may
210 overlap or conflict with metrics returned by JOB.response.time.
211 Config file entries (URLs or scripts) found in subdirectories of the
213 config directory will also result in hierarchical metric names. For
214 example, a config file named $PCP_PMDAS_DIR/prometheus/con‐
215 fig.d/mysource/latency/get.url will result in metrics being created (by
216 fetching that source URL) below prometheus.mysource.latency.get in the
217 PCP namespace. Scripts found in subdirectories of the config directory
218 similarly result in hierarchical PCP metric names.
219
221 As described above, changes and new additions can be made to files in
222 the configuration directory without having to restart the PMDA. These
223 changes are detected automatically and the PCP metric names below
224 prometheus in the PMNS will be updated accordingly, i.e. new metrics
225 will be dynamically added and/or existing metrics removed. In addi‐
226 tion, pmdaprometheus honors the PMCD_NAMES_CHANGE pmFetch(3) protocol
227 that was introduced in PCP version 4.0. In particular, if prometheus
228 metrics are being logged by a PCP version 4.0 or later pmlogger(1), new
229 metrics that appear as a result of changes in the PMDA configuration
230 directory will automatically start to be logged, provided the root of
231 the prometheus PMDA namespace is configured for logging in the pmlogger
232 configuration file. See pmlogger(1) for details. An example of such a
233 pmlogger configuration file is :
234 log mandatory on 2 second {
236 # log all metrics below the root of the prometheus namespace
237 prometheus
238 }
239
241 The PMDA maintains special control metrics, as described below. Apart
242 from prometheus.control.debug, each of these metrics is a counter and
243 has one instance for each configured metric source. The instance
244 domain is adjusted dynamically as new sources are discovered. If there
245 are no sources configured, the metric names are still defined but the
246 instance domain will be empty and a fetch will return no values.
247 prometheus.control.calls
249 total number of times each configured metric source has been
250 fetched (if it's a URL) or executed (if it's a script), since
251 the PMDA started.
252 prometheus.control.fetch_time
254 Total time in milliseconds that each configured metric source
255 has taken to return a document, excluding the time to parse the
256 document.
257 prometheus.control.parse_time
259 Total time in milliseconds that each configured metric source
260 has taken to parse each document, excluding the time to fetch
261 the document.
262 When converted to a rate, the calls metric represents the average fetch
264 rate of each source over the sampling interval (time delta between sam‐
265 ples). The fetch_time and parse_time counters, when converted to a
266 rate, represent the average fetch and parsing latency (respectfully),
267 during the sampling interval.
268 The prometheus.control.debug metric has a singular value, defaulting to
270 0. If a non-zero value is stored into this metric using pmstore(1),
271 additional debug messages will be written to the PMDA log file.
272
274 pmdaprometheus and libpcp internals impose some numerical constraints
275 about the number of sources (4095), metrics (1024) within each source,
276 and instances for each metric (4194304).
277
279 Install the Prometheus PMDA by using the Install script as root:
280 # cd $PCP_PMDAS_DIR/prometheus
282 # ./Install
283 To uninstall, do the following as root:
285 # cd $PCP_PMDAS_DIR/prometheus
287 # ./Remove
288 pmdaprometheus is launched by pmcd(1) and should never be executed
290 directly. The Install and Remove scripts notify pmcd when the agent is
291 installed or removed.
292 When scripts and .url files are added, removed or changed in the con‐
294 figuration directory, it is usually not necessary to restart the PMDA -
295 the changes will be detected and managed on subsequent requests to the
296 PMDA.
297
299 $PCP_PMDAS_DIR/prometheus/Install
300 installation script for the pmdaprometheus agent
301 $PCP_PMDAS_DIR/prometheus/Remove
303 undo installation script for the pmdaprometheus agent
304 $PCP_PMDAS_DIR/prometheus/config.d/
306 contains URLs and scripts used by the pmdaprometheus agent as
307 sources of prometheus metric data.
308 $PCP_LOG_DIR/pmcd/prometheus.log
310 default log file for error messages from pmdaprometheus
311 $PCP_VAR_DIR/config/144.*
313 files containing internal tables for metric and instance ID number
314 persistence (domain 144).
315
317 Environment variables with the prefix PCP_ are used to parameterize the
318 file and directory names used by PCP. On each installation, the file
319 /etc/pcp.conf contains the local values for these variables. The
320 $PCP_CONF variable may be used to specify an alternative configuration
321 file, as described in pcp.conf(5).
322
324 pmcd(1), pminfo(1), pmlogger(1), pmstore(1), PMWEBAPI(3), pmFetch(3)
325 and https://prometheus.io/docs/instrumenting/exposition_formats.
326Performance Co-Pilot PCP PMDAPROMETHEUS(1)