1PMDAPROMETHEUS(1)           General Commands Manual          PMDAPROMETHEUS(1)
2
3
4

NAME

6       pmdaprometheus - Prometheus PMDA
7

SYNOPSIS

9       $PCP_PMDAS_DIR/prometheus/pmdaprometheus  [-D]  [-n]  [-c  config]  [-d
10       domain] [-l logfile] [-r root] [-t timeout] [-u user]
11

DESCRIPTION

13       pmdaprometheus is a Performance Metrics Domain Agent (PMDA) which  cre‐
14       ates  PCP  metrics  from Prometheus endpoints, which provide HTTP based
15       access  to  application  metrics.   The  default  config  directory  is
16       $PCP_PMDAS_DIR/prometheus/config.d/,   see   ``CONFIGURATION  SOURCES''
17       below.  The default URL fetch timeout is 2 seconds.  The default  user,
18       if  not  specified  with the -u option, is the current user.  If the -n
19       option is given, the list of configuration files  will  not  be  sorted
20       prior  to  processing.   This list is sorted by default but that can be
21       expensive if there are a large  number  of  configuration  files  (URLs
22       and/or scripts).
23
24       If the -D option is given, additional diagnostic messages will be writ‐
25       ten to the PMDA log file, which is $PCP_LOG_DIR/pmcd/prometheus.log  by
26       default  (see  also  -lbelow).  In addition, the metric prometheus.con‐
27       trol.debug controls the same debug flag and can be set with the follow‐
28       ing command:
29            pmstore prometheus.control.debug value
30       where  value is either 1 (to enable verbose log messages) or 0 (to dis‐
31       able verbose log messages).  This is particularly useful for  examining
32       the  http  headers  passed  to  each fetch request, filter settings and
33       other processing details that are logged when  the  debugging  flag  is
34       enabled.
35
36       The  -d  option may be used to override the default performance metrics
37       domain number, which defaults to 144.  It is strongly  recommended  not
38       to  change  this.  The domain number should be different for every PMDA
39       on the one host,  and  the  same  domain  number  should  be  used  for
40       pmdaprometheus PMDA on all hosts.  See also the -r option, which allows
41       the root of the dynamic  namespace  to  be  changed  from  the  default
42       prometheus.
43
44       The  -l  option  may  be used to specify logfile as the destination for
45       PMDA messages instead of the default, $PCP_LOG_DIR/pmcd/prometheus.log.
46       As  a  special  case, logfile may be "-" to send messages to the stderr
47       stream instead, e.g.  -l-.  This would normally be  the  stderr  stream
48       for  the  parent  process,  pmcd(1),  which  may itself have redirected
49       stderr.  This redirection is normally most useful  in  a  containerized
50       environment, or when using dbpmda(1).
51
52       The -r option allows the root of the dynamic namespace to be changed to
53       root from the default, prometheus.  In conjunction with  other  command
54       line  options, this allows pmdaprometheus to be deployed as a different
55       PMDA with distinct metrics namespace and metrics  domain  on  the  same
56       host system.  Note that all PMDAs require a unique domain number so the
57       -d option must also be specified.  Use of the -r option may also change
58       the  defaults for some other command line options, e.g. the default log
59       file name and the default configuration directory.
60

CONFIGURATION SOURCES

62       As  it  runs,  pmdaprometheus  periodically   recursively   scans   the
63       $PCP_PMDAS_DIR/prometheus/config.d  directory  (or the directory speci‐
64       fied with the -c option), looking for source URL files (*.url) and exe‐
65       cutable  scripts or binaries.  Any files that do not have the .url suf‐
66       fix or are not executable, are  ignored  -  this  allows  documentation
67       files such as "README" and non-executable "common" script function def‐
68       initions to be present without being considered as config files.
69
70       A remote server does not have to be up or stay running - the PMDA  tol‐
71       erates remote URLs that may come and go over time.  The PMDA will relay
72       data and metadata when/if they are available, and  will  return  errors
73       when/if  they are down.  PCP metric IDs, internal and external instance
74       domain identifiers are persisted and will be restored  when  individual
75       metric  sources become available and/or when the PMDA is restarted.  In
76       addition, the PMDA checks directory modification times and will  rescan
77       for  new  or changed configuration files dynamically.  It is not neces‐
78       sary to restart the PMDA when adding, removing or  changing  configura‐
79       tion files.
80

URL SOURCES

82       Each  file  with  the .url suffix found in the config directory or sub-
83       directory  contains  one  complete  HTTP  or   HTTPS   URL   at   which
84       pmdaprometheus  can  reach a Prometheus endpoint.  Local file access is
85       also supported with a  conventional  file://somepath/somefile  URL,  in
86       which case somepath/somefile should contain prometheus formatted metric
87       data.
88
89       The first line of a .url config file should be the  URL,  as  described
90       above.   Subsequent lines, if any, are prefixed with a keyword that can
91       be used to alter the http GET request.  A keyword  must  end  with  ':'
92       (colon)  and  the  text  extends to the end of the line.  Comment lines
93       that start with # and blank lines are ignored.  The only currently sup‐
94       ported keywords are HEADER: and FILTER:.
95
96       HEADER: headername: value ... to end of line
97       Adds  headername  and  its  value to the headers passed in the http GET
98       request for the configured URL.  An  example  configuration  file  that
99       provides 3 commonly used headers and an authentication token might be :
100
101          http://somehost/path/endpoint.html
102          # this is a comment
103          HEADER: Accept: text/html
104          HEADER: Keep-Alive: 300
105          HEADER: Connection: keep-alive
106          HEADER: Authorization: token ABCDEF1234567890
107
108       As  mentioned above, header values extend to the end of the line.  They
109       may contain any valid characters, including  colons.   Multiple  spaces
110       will  be  collapsed  to a single space, and leading and trailing spaces
111       are trimmed.  A common use for headers is to configure  a  proxy  agent
112       and the assorted parameters it may require.
113
114       FILTER: INCLUDE METRIC regex
115       or
116       FILTER: EXCLUDE METRIC regex
117       Dynamically  created  metric  names  that  match  regex  will be either
118       included or excluded in the name space, as specified.  The simple  rule
119       is that the first matching filter regex for a particular metric name is
120       the rule that prevails.  If no filter regex matches (or  there  are  no
121       filters), then the metric is included by default, i.e. the default fil‐
122       ter if none are specified is FILTER: INCLUDE METRIC .*  This  is  back‐
123       ward  compatible with older versions of the configuration file that did
124       not support filters.  Multiple FILTER: lines would  normally  be  used,
125       e.g.  to  include  some  metrics  but  exclude  all others, use FILTER:
126       EXCLUDE METRIC .*  as the last of  several  filters  that  include  the
127       desired  metrics.   Conversely, to exclude some metrics but include all
128       others, use FILTER: EXCLUDE METRIC regex.  In this case it's not neces‐
129       sary  (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC
130       .*  because, as stated above, any metric that does not match any filter
131       regex will be included by default.
132
133       Label  filtering  uses  similar  FILTER: syntax and semantics.  FILTER:
134       EXCLUDE LABEL regex will delete all labels matching regex from all met‐
135       rics  defined in the configuration file.  The same rules as for metrics
136       apply for labels too - an  implicit  rule:  FILTER:  INCLUDE  LABEL  .*
137       applies to all labels that do not match any earlier filter rule.
138
139       Caution  is  needed with label filtering because by default, all labels
140       are used to construct the PCP instance name.  By excluding some labels,
141       the  instance names will change.  Excluding all labels for a particular
142       metric changes that metric  to  be  singular,  i.e.  have  no  instance
143       domain.   In addition, by excluding some labels, different instances of
144       the same metric may become duplicates.  When such duplicates occur, the
145       last duplicate instance returned by the end-point URL prevails over any
146       earlier instances.  For these reasons, it  is  recommended  that  label
147       filtering  rules  be  configured  when  the configuration file is first
148       defined, and not changed thereafter.  If a label  filtering  change  is
149       required,  the  configuration file should be renamed, which effectively
150       defines a new metric, with the new (or changed) instance naming.
151
152       Unrecognized keywords in configuration files are reported in  the  PMDA
153       log file but otherwise ignored.
154

SCRIPTED SOURCES

156       Executable  scripts  present  in the $PCP_PMDAS_DIR/prometheus/config.d
157       directory or sub-directories will be executed  and  the  stdout  stream
158       containing prometheus formatted metric data will be parsed as though it
159       had come from a URL or file.  The stderr stream from a script  will  be
160       sent  to  the  PMDA  log  file,  which  by  default  can  be  found  in
161       $(PCP_LOG_DIR)/pmcd/prometheus.log.
162
163       Note that scripted sources do not support label or metric filtering (as
164       described above for URL sources) - they can simply do their own filter‐
165       ing in the script itself with  sed(1),  awk(1),  or  whatever  tool  is
166       desired.
167
168       A simple example of a scripted config entry follows:
169
170          #! /bin/sh
171          awk '{
172              print("# HELP loadavg local load average")
173              print("# Type loadavg gauge")
174              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
175              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
176              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
177          }' /proc/loadavg
178
179       This  script  produces  the  following Prometheus-formatted metric data
180       when run:
181
182          # HELP loadavg local load average
183          # Type loadavg gauge
184          loadavg {interval="1-minute"} 0.12
185          loadavg {interval="5-minute"} 0.27
186          loadavg {interval="15-minute"} 0.54
187
188       If the above script was saved and  made  executable  in  a  file  named
189       $PCP_PMDAS_DIR/prometheus/config.d/local/system.sh   then   this  would
190       result in a new PCP metric named prometheus.local.system.loadavg  which
191       would  have  three  instances  for  the  current  load  average values:
192       1-minute, 5-minute and 15-minute.
193
194       Scripted config entries may produce more than one PCP leaf metric name.
195       For  example, the above "system.sh" script could also export other met‐
196       rics such as CPU statistics, by reading /proc/stat on the local system.
197       Such  additional  metrics  would appear as peer metrics in the same PCP
198       metric subtree.  In the case of CPU counters, the metric  type  defini‐
199       tion  should be counter, not gauge.  For full details of the prometheus
200       exposition formats, see  https://prometheus.io/docs/instrumenting/expo
201       sition_formats.
202

METRIC NAMING

204       All  metrics  from  a file named JOB.*  will be exported as PCP metrics
205       with the prometheus.JOB metric name prefix.  Therefore,  the  JOB  name
206       must  be  a  valid non-leaf name for PCP PMNS metric names.  If the JOB
207       name has multiple dot-separated components, the  resulting  PMNS  names
208       will include those components and care is needed to ensure there are no
209       overlapping definitions, e.g.  metrics  returned  by  JOB.response  may
210       overlap or conflict with metrics returned by JOB.response.time.
211
212       Config  file  entries  (URLs or scripts) found in subdirectories of the
213       config directory will also result in hierarchical  metric  names.   For
214       example,    a    config   file   named   $PCP_PMDAS_DIR/prometheus/con‐
215       fig.d/mysource/latency/get.url will result in metrics being created (by
216       fetching  that source URL) below prometheus.mysource.latency.get in the
217       PCP namespace.  Scripts found in subdirectories of the config directory
218       similarly result in hierarchical PCP metric names.
219

DYNAMIC METRIC NAMES

221       As  described  above, changes and new additions can be made to files in
222       the configuration directory without having to restart the PMDA.   These
223       changes  are  detected  automatically  and  the  PCP metric names below
224       prometheus in the PMNS will be updated accordingly,  i.e.  new  metrics
225       will  be  dynamically  added and/or existing metrics removed.  In addi‐
226       tion, pmdaprometheus honors the PMCD_NAMES_CHANGE  pmFetch(3)  protocol
227       that  was  introduced in PCP version 4.0.  In particular, if prometheus
228       metrics are being logged by a PCP version 4.0 or later pmlogger(1), new
229       metrics  that  appear  as a result of changes in the PMDA configuration
230       directory will automatically start to be logged, provided the  root  of
231       the prometheus PMDA namespace is configured for logging in the pmlogger
232       configuration file.  See pmlogger(1) for details.  An example of such a
233       pmlogger configuration file is :
234
235          log mandatory on 2 second {
236               # log all metrics below the root of the prometheus namespace
237               prometheus
238          }
239

CONTROL METRICS

241       The  PMDA maintains special control metrics, as described below.  Apart
242       from prometheus.control.debug, each of these metrics is a  counter  and
243       has  one  instance  for  each  configured  metric source.  The instance
244       domain is adjusted dynamically as new sources are discovered.  If there
245       are  no  sources configured, the metric names are still defined but the
246       instance domain will be empty and a fetch will return no values.
247
248       prometheus.control.calls
249              total number of times each configured  metric  source  has  been
250              fetched  (if  it's  a URL) or executed (if it's a script), since
251              the PMDA started.
252
253       prometheus.control.fetch_time
254              Total time in milliseconds that each  configured  metric  source
255              has  taken to return a document, excluding the time to parse the
256              document.
257
258       prometheus.control.parse_time
259              Total time in milliseconds that each  configured  metric  source
260              has  taken  to  parse each document, excluding the time to fetch
261              the document.
262
263       When converted to a rate, the calls metric represents the average fetch
264       rate of each source over the sampling interval (time delta between sam‐
265       ples).  The fetch_time and parse_time counters,  when  converted  to  a
266       rate,  represent  the average fetch and parsing latency (respectfully),
267       during the sampling interval.
268
269       The prometheus.control.debug metric has a singular value, defaulting to
270       0.   If  a  non-zero value is stored into this metric using pmstore(1),
271       additional debug messages will be written to the PMDA log file.
272

LIMITATIONS

274       pmdaprometheus and libpcp internals impose some  numerical  constraints
275       about  the number of sources (4095), metrics (1024) within each source,
276       and instances for each metric (4194304).
277

INSTALLATION

279       Install the Prometheus PMDA by using the Install script as root:
280
281             # cd $PCP_PMDAS_DIR/prometheus
282             # ./Install
283
284       To uninstall, do the following as root:
285
286             # cd $PCP_PMDAS_DIR/prometheus
287             # ./Remove
288
289       pmdaprometheus is launched by pmcd(1)  and  should  never  be  executed
290       directly.  The Install and Remove scripts notify pmcd when the agent is
291       installed or removed.
292
293       When scripts and .url files are added, removed or changed in  the  con‐
294       figuration directory, it is usually not necessary to restart the PMDA -
295       the changes will be detected and managed on subsequent requests to  the
296       PMDA.
297

FILES

299       $PCP_PMDAS_DIR/prometheus/Install
300           installation script for the pmdaprometheus agent
301
302       $PCP_PMDAS_DIR/prometheus/Remove
303           undo installation script for the pmdaprometheus agent
304
305       $PCP_PMDAS_DIR/prometheus/config.d/
306           contains  URLs  and  scripts  used  by  the pmdaprometheus agent as
307           sources of prometheus metric data.
308
309       $PCP_LOG_DIR/pmcd/prometheus.log
310           default log file for error messages from pmdaprometheus
311
312       $PCP_VAR_DIR/config/144.*
313           files containing internal tables for metric and instance ID  number
314           persistence (domain 144).
315

PCP ENVIRONMENT

317       Environment variables with the prefix PCP_ are used to parameterize the
318       file and directory names used by PCP.  On each installation,  the  file
319       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
320       $PCP_CONF variable may be used to specify an alternative  configuration
321       file, as described in pcp.conf(5).
322

SEE ALSO

324       pmcd(1),  pminfo(1),  pmlogger(1),  pmstore(1), PMWEBAPI(3), pmFetch(3)
325       and https://prometheus.io/docs/instrumenting/exposition_formats.
326
327
328
329Performance Co-Pilot                  PCP                    PMDAPROMETHEUS(1)
Impressum