1PMDAOPENMETRICS(1)          General Commands Manual         PMDAOPENMETRICS(1)
2
3
4

NAME

6       pmdaopenmetrics - OpenMetrics PMDA
7

SYNOPSIS

9       $PCP_PMDAS_DIR/openmetrics/pmdaopenmetrics  [-D]  [-n]  [-c config] [-d
10       domain] [-l logfile] [-r root] [-t timeout] [-u user]
11

DESCRIPTION

13       pmdaopenmetrics is a Performance Metrics Domain Agent (PMDA) which cre‐
14       ates  PCP  metrics from OpenMetrics endpoints, which provide HTTP based
15       access  to  application  metrics.   The  default  config  directory  is
16       $PCP_PMDAS_DIR/openmetrics/config.d/,   see  ``CONFIGURATION  SOURCES''
17       below.  The default URL fetch timeout is 2 seconds.  The default  user,
18       if  not  specified  with the -u option, is the current user.  If the -n
19       option is given, the list of configuration files  will  not  be  sorted
20       prior  to  processing.   This list is sorted by default but that can be
21       expensive if there are a large  number  of  configuration  files  (URLs
22       and/or scripts).
23
24       If the -D option is given, additional diagnostic messages will be writ‐
25       ten to the PMDA log file, which is $PCP_LOG_DIR/pmcd/openmetrics.log by
26       default  (see  also -lbelow).  In addition, the metric openmetrics.con‐
27       trol.debug controls the same debug flag and can be set with the follow‐
28       ing command:
29            pmstore openmetrics.control.debug value
30       where  value is either 1 (to enable verbose log messages) or 0 (to dis‐
31       able verbose log messages).  This is particularly useful for  examining
32       the  http  headers  passed  to  each fetch request, filter settings and
33       other processing details that are logged when  the  debugging  flag  is
34       enabled.
35
36       The  -d  option may be used to override the default performance metrics
37       domain number, which defaults to 144.  It is strongly  recommended  not
38       to  change  this.  The domain number should be different for every PMDA
39       on the one host,  and  the  same  domain  number  should  be  used  for
40       pmdaopenmetrics  PMDA  on  all  hosts.   See  also the -r option, which
41       allows the root of the dynamic namespace to be changed from the default
42       openmetrics.
43
44       The  -l  option  may  be used to specify logfile as the destination for
45       PMDA  messages  instead   of   the   default,   $PCP_LOG_DIR/pmcd/open‐
46       metrics.log.  As a special case, logfile may be "-" to send messages to
47       the stderr stream instead, e.g.   -l-.   This  would  normally  be  the
48       stderr  stream  for  the parent process, pmcd(1), which may itself have
49       redirected stderr.  This redirection is normally most useful in a  con‐
50       tainerized environment, or when using dbpmda(1).
51
52       The -r option allows the root of the dynamic namespace to be changed to
53       root from the default, openmetrics.  In conjunction with other  command
54       line options, this allows pmdaopenmetrics to be deployed as a different
55       PMDA with distinct metrics namespace and metrics  domain  on  the  same
56       host system.  Note that all PMDAs require a unique domain number so the
57       -d option must also be specified.  Use of the -r option may also change
58       the  defaults for some other command line options, e.g. the default log
59       file name and the default configuration directory.
60

CONFIGURATION SOURCES

62       As  it  runs,  pmdaopenmetrics  periodically  recursively   scans   the
63       $PCP_PMDAS_DIR/openmetrics/config.d  directory (or the directory speci‐
64       fied with the -c option), looking for source URL files (*.url) and exe‐
65       cutable  scripts or binaries.  Any files that do not have the .url suf‐
66       fix or are not executable, are  ignored  -  this  allows  documentation
67       files such as "README" and non-executable "common" script function def‐
68       initions to be present without being considered as config files.
69
70       A remote server does not have to be up or stay running - the PMDA  tol‐
71       erates remote URLs that may come and go over time.  The PMDA will relay
72       data and metadata when/if they are available, and  will  return  errors
73       when/if  they are down.  PCP metric IDs, internal and external instance
74       domain identifiers are persisted and will be restored  when  individual
75       metric  sources become available and/or when the PMDA is restarted.  In
76       addition, the PMDA checks directory modification times and will  rescan
77       for  new  or changed configuration files dynamically.  It is not neces‐
78       sary to restart the PMDA when adding, removing or  changing  configura‐
79       tion files.
80

URL SOURCES

82       Each  file  with  the .url suffix found in the config directory or sub-
83       directory contains one complete HTTP or HTTPS URL  at  which  pmdaopen‐
84       metrics  can  reach  a OpenMetrics endpoint.  Local file access is also
85       supported with a conventional file:///somepath/somefile URL,  in  which
86       case  somepath/somefile  should  contain  openmetrics  formatted metric
87       data.
88
89       The first line of a .url config file should be the  URL,  as  described
90       above.   Subsequent lines, if any, are prefixed with a keyword that can
91       be used to alter the http GET request.  A keyword  must  end  with  ':'
92       (colon)  and  the  text  extends to the end of the line.  Comment lines
93       that start with # and blank lines are ignored.  The only currently sup‐
94       ported keywords are HEADER: and FILTER:.
95
96       HEADER: headername: value ... to end of line
97       Adds  headername  and  its  value to the headers passed in the http GET
98       request for the configured URL.  An  example  configuration  file  that
99       provides 3 commonly used headers and an authentication token might be :
100
101          http://somehost/path/endpoint.html
102          # this is a comment
103          HEADER: Accept: text/html
104          HEADER: Keep-Alive: 300
105          HEADER: Connection: keep-alive
106          HEADER: Authorization: token ABCDEF1234567890
107
108       As  mentioned above, header values extend to the end of the line.  They
109       may contain any valid characters, including  colons.   Multiple  spaces
110       will  be  collapsed  to a single space, and leading and trailing spaces
111       are trimmed.  A common use for headers is to configure  a  proxy  agent
112       and the assorted parameters it may require.
113
114       FILTER: INCLUDE METRIC regex
115       or
116       FILTER: EXCLUDE METRIC regex
117       Dynamically  created  metric  names  that  match  regex  will be either
118       included or excluded in the name space, as specified.  The simple  rule
119       is that the first matching filter regex for a particular metric name is
120       the rule that prevails.  If no filter regex matches (or  there  are  no
121       filters), then the metric is included by default, i.e. the default fil‐
122       ter if none are specified is FILTER: INCLUDE METRIC .*  This  is  back‐
123       ward  compatible with older versions of the configuration file that did
124       not support filters.  Multiple FILTER: lines would  normally  be  used,
125       e.g.  to  include  some  metrics  but  exclude  all others, use FILTER:
126       EXCLUDE METRIC .*  as the last of  several  filters  that  include  the
127       desired  metrics.   Conversely, to exclude some metrics but include all
128       others, use FILTER: EXCLUDE METRIC regex.  In this case it's not neces‐
129       sary  (though doesn't hurt) to specify the final FILTER: INCLUDE METRIC
130       .*  because, as stated above, any metric that does not match any filter
131       regex will be included by default.
132
133       Label  filtering  uses  similar  FILTER: syntax and semantics.  FILTER:
134       EXCLUDE LABEL regex will delete all labels matching regex from all met‐
135       rics  defined in the configuration file.  The same rules as for metrics
136       apply for labels too - an  implicit  rule:  FILTER:  INCLUDE  LABEL  .*
137       applies to all labels that do not match any earlier filter rule.
138
139       Caution  is  needed with label filtering because by default, all labels
140       are used to construct the PCP instance name.  By excluding some labels,
141       the  instance names will change.  Excluding all labels for a particular
142       metric changes that metric  to  be  singular,  i.e.  have  no  instance
143       domain.   In addition, by excluding some labels, different instances of
144       the same metric may become duplicates.  When such duplicates occur, the
145       last duplicate instance returned by the end-point URL prevails over any
146       earlier instances.  For these reasons, it  is  recommended  that  label
147       filtering  rules  be  configured  when  the configuration file is first
148       defined, and not changed thereafter.  If a label  filtering  change  is
149       required,  the  configuration file should be renamed, which effectively
150       defines a new metric, with the new (or changed) instance naming.
151
152       Unrecognized keywords in configuration files are reported in  the  PMDA
153       log file but otherwise ignored.
154

SCRIPTED SOURCES

156       Executable  scripts  present in the $PCP_PMDAS_DIR/openmetrics/config.d
157       directory or sub-directories will be executed  and  the  stdout  stream
158       containing  openmetrics  formatted metric data will be parsed as though
159       it had come from a URL or file.  The stderr stream from a  script  will
160       be  sent  to  the  PMDA  log  file,  which  by  default can be found in
161       $(PCP_LOG_DIR)/pmcd/openmetrics.log.
162
163       Note that scripted sources do not support label or metric filtering (as
164       described above for URL sources) - they can simply do their own filter‐
165       ing in the script itself with  sed(1),  awk(1),  or  whatever  tool  is
166       desired.
167
168       A simple example of a scripted config entry follows:
169
170          #! /bin/sh
171          awk '{
172              print("# HELP loadavg local load average")
173              print("# Type loadavg gauge")
174              printf("loadavg {interval=\"1-minute\"} %.2f\n", $1)
175              printf("loadavg {interval=\"5-minute\"} %.2f\n", $2)
176              printf("loadavg {interval=\"15-minute\"} %.2f\n", $3)
177          }' /proc/loadavg
178
179       This  script  produces  the following OpenMetrics-formatted metric data
180       when run:
181
182          # HELP loadavg local load average
183          # Type loadavg gauge
184          loadavg {interval="1-minute"} 0.12
185          loadavg {interval="5-minute"} 0.27
186          loadavg {interval="15-minute"} 0.54
187
188       If the above script was saved and  made  executable  in  a  file  named
189       $PCP_PMDAS_DIR/openmetrics/config.d/local/system.sh   then  this  would
190       result in a new PCP metric named openmetrics.local.system.loadavg which
191       would  have  three  instances  for  the  current  load  average values:
192       1-minute, 5-minute and 15-minute.
193
194       Scripted config entries may produce more than one PCP leaf metric name.
195       For  example, the above "system.sh" script could also export other met‐
196       rics such as CPU statistics, by reading /proc/stat on the local system.
197       Such  additional  metrics  would appear as peer metrics in the same PCP
198       metric subtree.  In the case of CPU counters, the metric  type  defini‐
199       tion should be counter, not gauge.  For full details of the openmetrics
200       exposition formats, see https://openmetrics.io/docs/instrumenting/expo
201       sition_formats.
202

METRIC NAMING

204       All  metrics  from  a file named JOB.*  will be exported as PCP metrics
205       with the openmetrics.JOB metric name prefix.  Therefore, the  JOB  name
206       must  be  a  valid non-leaf name for PCP PMNS metric names.  If the JOB
207       name has multiple dot-separated components, the  resulting  PMNS  names
208       will include those components and care is needed to ensure there are no
209       overlapping definitions, e.g.  metrics  returned  by  JOB.response  may
210       overlap or conflict with metrics returned by JOB.response.time.
211
212       Config  file  entries  (URLs or scripts) found in subdirectories of the
213       config directory will also result in hierarchical  metric  names.   For
214       example,    a   config   file   named   $PCP_PMDAS_DIR/openmetrics/con‐
215       fig.d/mysource/latency/get.url will result in metrics being created (by
216       fetching that source URL) below openmetrics.mysource.latency.get in the
217       PCP namespace.  Scripts found in subdirectories of the config directory
218       similarly result in hierarchical PCP metric names.
219

DYNAMIC METRIC NAMES

221       As  described  above, changes and new additions can be made to files in
222       the configuration directory without having to restart the PMDA.   These
223       changes are detected automatically and the PCP metric names below open‐
224       metrics in the PMNS will be updated accordingly, i.e. new metrics  will
225       be  dynamically  added  and/or  existing metrics removed.  In addition,
226       pmdaopenmetrics honors the PMCD_NAMES_CHANGE pmFetch(3)  protocol  that
227       was  introduced in PCP version 4.0.  In particular, if openmetrics met‐
228       rics are being logged by a PCP version 4.0 or  later  pmlogger(1),  new
229       metrics  that  appear  as a result of changes in the PMDA configuration
230       directory will automatically start to be logged, provided the  root  of
231       the  openmetrics PMDA namespace is configured for logging in the pmlog‐
232       ger configuration file.  See pmlogger(1) for details.   An  example  of
233       such a pmlogger configuration file is :
234
235          log mandatory on 2 second {
236               # log all metrics below the root of the openmetrics namespace
237               openmetrics
238          }
239

CONTROL METRICS

241       The  PMDA maintains special control metrics, as described below.  Apart
242       from openmetrics.control.debug, each of these metrics is a counter  and
243       has  one  instance  for  each  configured  metric source.  The instance
244       domain is adjusted dynamically as new sources are discovered.  If there
245       are  no  sources configured, the metric names are still defined but the
246       instance domain will be empty and a fetch will return no values.
247
248       openmetrics.control.calls
249              total number of times each configured  metric  source  has  been
250              fetched  (if  it's  a URL) or executed (if it's a script), since
251              the PMDA started.
252
253       openmetrics.control.fetch_time
254              Total time in milliseconds that each  configured  metric  source
255              has  taken to return a document, excluding the time to parse the
256              document.
257
258       openmetrics.control.parse_time
259              Total time in milliseconds that each  configured  metric  source
260              has  taken  to  parse each document, excluding the time to fetch
261              the document.
262
263       When converted to a rate, the calls metric represents the average fetch
264       rate of each source over the sampling interval (time delta between sam‐
265       ples).  The fetch_time and parse_time counters,  when  converted  to  a
266       rate,  represent  the average fetch and parsing latency (respectfully),
267       during the sampling interval.
268
269       The openmetrics.control.debug metric has a singular  value,  defaulting
270       to 0.  If a non-zero value is stored into this metric using pmstore(1),
271       additional debug messages will be written to the PMDA log file.
272

LIMITATIONS

274       pmdaopenmetrics and libpcp internals impose some numerical  constraints
275       about  the number of sources (4095), metrics (1024) within each source,
276       and instances for each metric (4194304).
277

INSTALLATION

279       Install the OpenMetrics PMDA by using the Install script as root:
280
281             # cd $PCP_PMDAS_DIR/openmetrics
282             # ./Install
283
284       To uninstall, do the following as root:
285
286             # cd $PCP_PMDAS_DIR/openmetrics
287             # ./Remove
288
289       pmdaopenmetrics is launched by pmcd(1) and  should  never  be  executed
290       directly.  The Install and Remove scripts notify pmcd when the agent is
291       installed or removed.
292
293       When scripts and .url files are added, removed or changed in  the  con‐
294       figuration directory, it is usually not necessary to restart the PMDA -
295       the changes will be detected and managed on subsequent requests to  the
296       PMDA.
297

FILES

299       $PCP_PMDAS_DIR/openmetrics/Install
300           installation script for the pmdaopenmetrics agent
301
302       $PCP_PMDAS_DIR/openmetrics/Remove
303           undo installation script for the pmdaopenmetrics agent
304
305       $PCP_PMDAS_DIR/openmetrics/config.d/
306           contains  URLs  and  scripts  used  by the pmdaopenmetrics agent as
307           sources of openmetrics metric data.
308
309       $PCP_LOG_DIR/pmcd/openmetrics.log
310           default log file for error messages from pmdaopenmetrics
311
312       $PCP_VAR_DIR/config/144.*
313           files containing internal tables for metric and instance ID  number
314           persistence (domain 144).
315

PCP ENVIRONMENT

317       Environment variables with the prefix PCP_ are used to parameterize the
318       file and directory names used by PCP.  On each installation,  the  file
319       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
320       $PCP_CONF variable may be used to specify an alternative  configuration
321       file, as described in pcp.conf(5).
322

SEE ALSO

324       PCPIntro(1),  pmcd(1), pminfo(1), pmlogger(1), pmstore(1), PMWEBAPI(3),
325       pmFetch(3)    and     https://openmetrics.io/docs/instrumenting/exposi
326       tion_formats.
327
328
329
330Performance Co-Pilot                  PCP                   PMDAOPENMETRICS(1)
Impressum