1PCP2ELASTICSEARCH(1)        General Commands Manual       PCP2ELASTICSEARCH(1)
2
3
4

NAME

6       pcp2elasticsearch - pcp-to-elasticsearch metrics exporter
7

SYNOPSIS

9       pcp2elasticsearch  [-5CGHIjLmnrRvV?]  [-4 action] [-8|-9 limit] [-a ar‐
10       chive] [-A align] [--archive-folio folio] [-b|-B space-scale] [-c  con‐
11       fig]  [--container  container]  [--daemonize] [-e derived] [-g URL] [-h
12       host] [-i instances] [-J rank] [-K spec] [-N predicate] [-O origin] [-p
13       type] [-P|-0 precision] [-q|-Q count-scale] [-s samples] [-S starttime]
14       [-t interval] [-T endtime] [-x index] [-X  hostid]  [-y|-Y  time-scale]
15       metricspec [...]
16

DESCRIPTION

18       pcp2elasticsearch  is  a customizable performance metrics exporter tool
19       from PCP to Elasticsearch.  Any available performance metric,  live  or
20       archived,  system  and/or  application,  can  be selected for exporting
21       using either command line arguments or a configuration file.
22
23       pcp2elasticsearch is a close relative of  pmrep(1).   Please  refer  to
24       pmrep(1)  for  the metricspec description accepted on pcp2elasticsearch
25       command line and pmrep.conf(5)  for  description  of  the  pcp2elastic‐
26       search.conf  configuration  file  overall  syntax,  this page describes
27       pcp2elasticsearch specific options and configuration  file  differences
28       with  pmrep.conf(5).   pmrep(1) also lists some usage examples of which
29       most are applicable with pcp2elasticsearch as well.
30
31       Only the command line options listed on this page are supported,  other
32       options recognized by pmrep(1) are not supported.
33
34       Options  via environment values (see pmGetOptions(3)) override the cor‐
35       responding  built-in  default  values  (if  any).   Configuration  file
36       options  override  the  corresponding  environment  variables (if any).
37       Command line options  override  the  corresponding  configuration  file
38       options (if any).
39

CONFIGURATION FILE

41       pcp2elasticsearch   uses  a  configuration  file  with  overall  syntax
42       described in pmrep.conf(5).  The  following  options  are  common  with
43       pmrep.conf:  version, source, speclocal, derived, header, globals, sam‐
44       ples,  interval,  type,  type_prefer,  ignore_incompat,   names_change,
45       instances,   live_filter,   rank,   limit_filter,   limit_filter_force,
46       invert_filter, predicate, omit_flat, include_labels, precision,  preci‐
47       sion_force,      count_scale,      count_scale_force,      space_scale,
48       space_scale_force, time_scale, time_scale_force.  The output option  is
49       recognized but ignored for pmrep.conf compatibility.
50
51   pcp2elasticsearch specific options
52       es_server (string)
53           Specify  the  URL  of  Elasticsearch server to send the metrics to.
54           Corresponding command line option is -g.  Default is  http://local
55           host:9200/.
56
57       es_index (string)
58           Specify  the  Elasticsearch  index for metric names.  Corresponding
59           command line option is -x.  Default is pcp.
60
61       es_hostid (string)
62           Specify the Elasticsearch host-id for measurements.   Corresponding
63           command line option is -X.  Defaults to the metrics source host.
64
65       es_search_type (string)
66           Specify  the  Elasticsearch  search  type for measurements.  Corre‐
67           sponding command line option is -p.  Defaults to pcp-metric.
68

OPTIONS

70       The available command line options are:
71
72       -0 precision, --precision-force=precision
73            Like -P but this option will override per-metric specifications.
74
75       -4 action, --names-change=action
76            Specify which action to take on receiving a  metric  names  change
77            event  during  sampling.  These events occur when a PMDA discovers
78            new metrics sometime after starting up, and informs running client
79            tools  like pcp2elasticsearch.  Valid values for action are update
80            (refresh metrics being sampled), ignore (do nothing - the  default
81            behaviour) and abort (exit the program if such an event happens).
82
83       -5, --ignore-unknown
84            Silently ignore any metric name that cannot be resolved.  At least
85            one metric must be found for the tool to start.
86
87       -8 limit, --limit-filter=limit
88            Limit results to instances with values above/below limit.  A posi‐
89            tive  integer  will  include instances with values at or above the
90            limit in reporting.  A negative  integer  will  include  instances
91            with  values  at or below the limit in reporting.  A value of zero
92            performs no limit filtering.  This option will not override possi‐
93            ble per-metric specifications.  See also -J and -N.
94
95       -9 limit, --limit-filter-force=limit
96            Like -8 but this option will override per-metric specifications.
97
98       -a archive, --archive=archive
99            Performance  metric  values  are retrieved from the set of Perfor‐
100            mance Co-Pilot (PCP) archive log files identified by  the  archive
101            argument,  which is a comma-separated list of names, each of which
102            may be the base name of an archive or the name of a directory con‐
103            taining one or more archives.
104
105       -A align, --align=align
106            Force  the initial sample to be aligned on the boundary of a natu‐
107            ral time unit align.  Refer to PCPIntro(1) for a complete descrip‐
108            tion of the syntax for align.
109
110       --archive-folio=folio
111            Read  metric source archives from the PCP archive folio created by
112            tools like pmchart(1) or, less often, manually with mkaf(1).
113
114       -b scale, --space-scale=scale
115            Unit/scale for  space  (byte)  metrics,  possible  values  include
116            bytes, Kbytes, KB, Mbytes, MB, and so forth.  This option will not
117            override possible per-metric specifications.  See  also  pmParseU‐
118            nitsStr(3).
119
120       -B scale, --space-scale-force=scale
121            Like -b but this option will override per-metric specifications.
122
123       -c config, --config=config
124            Specify  the  config  file to use.  The default is the first found
125            of:    ./pcp2elasticsearch.conf,    $HOME/.pcp2elasticsearch.conf,
126            $HOME/pcp/pcp2elasticsearch.conf,  and  $PCP_SYSCONF_DIR/pcp2elas‐
127            ticsearch.conf.   For  details,  see   the   above   section   and
128            pmrep.conf(5).
129
130       --container=container
131            Fetch  performance  metrics  from  the specified container, either
132            local or remote (see -h).
133
134       -C, --check
135            Exit before reporting any values, but after parsing the configura‐
136            tion and metrics and printing possible headers.
137
138       --daemonize
139            Daemonize on startup.
140
141       -e derived, --derived=derived
142            Specify  derived  performance  metrics.   If derived starts with a
143            slash (``/'') or with a dot (``.'') it will be  interpreted  as  a
144            derived  metrics  configuration  file, otherwise it will be inter‐
145            preted as comma- or  semicolon-separated  derived  metric  expres‐
146            sions.   For  details  see  pmLoadDerivedConfig(3) and pmRegister‐
147            Derived(3).
148
149       -g URL, --es-host=URL
150            Elasticsearch server URL to send the metrics to.
151
152       -G, --no-globals
153            Do not include global metrics in reporting (see pmrep.conf(5)).
154
155       -h host, --host=host
156            Fetch performance metrics from pmcd(1) on host, rather  than  from
157            the default localhost.
158
159       -H, --no-header
160            Do not print any headers.
161
162       -i instances, --instances=instances
163            Report  only  the  listed  instances  from  current  instances (if
164            present, see also -j).  By  default  all  instances,  present  and
165            future,  are  reported.   This is a global option that is used for
166            all metrics unless a metric-specific instance definition  is  pro‐
167            vided  as part of a metricspec.  By default single-valued ``flat''
168            metrics without multiple instances are still  reported  as  usual,
169            use  -v to change this.  Please refer to pmrep(1) for more details
170            on this option.
171
172       -I, --ignore-incompat
173            Ignore incompatible  metrics.   By  default  incompatible  metrics
174            (that  is,  their  type is unsupported or they cannot be scaled as
175            requested) will cause pcp2elasticsearch to terminate with an error
176            message.   With  this option all incompatible metrics are silently
177            omitted from  reporting.   This  may  be  especially  useful  when
178            requesting non-leaf nodes of the PMNS tree for reporting.
179
180       -j, --live-filter
181            Perform  instance  live filtering.  This allows capturing all fil‐
182            tered instances even if processes  are  restarted  at  some  point
183            (unlike without live filtering).  Performing live filtering over a
184            huge amount of instances will add some internal overhead so a  bit
185            of user caution is advised.  See also -n.
186
187       -J rank, --rank=rank
188            Limit  results  to  highest/lowest  ranked instances of set-valued
189            metrics.  A positive integer will include highest valued instances
190            in  reporting.   A  negative  integer  will  include lowest valued
191            instances in reporting.  A value  of  zero  performs  no  ranking.
192            Ranking does not imply sorting, see -6.  See also -8.
193
194       -K spec, --spec-local=spec
195            When fetching metrics from a local context (see -L), the -K option
196            may be used to control the DSO PMDAs that should be made  accessi‐
197            ble.   The  spec  argument  conforms  to  the  syntax described in
198            pmSpecLocalPMDA(3).  More than one -K option may be used.
199
200       -L, --local-PMDA
201            Use a local context to collect metrics from DSO PMDAs on the local
202            host without PMCD.  See also -K.
203
204       -m, --include-labels
205            Include metric labels in the output.
206
207       -n, --invert-filter
208            Perform  ranking  before live filtering.  By default instance live
209            filtering (when requested, see -j) happens before instance ranking
210            (when  requested, see -J).  With this option the logic is inverted
211            and ranking happens before live filtering.
212
213       -N predicate, --predicate=predicate
214            Specify a comma-separated list of predicate filter reference  met‐
215            rics.   By  default ranking (see -J) happens for each metric indi‐
216            vidually.  With predicates, ranking is done only for the specified
217            predicate  metrics.   When  reporting, rest of the metrics sharing
218            the same instance domain (see PCPIntro(1)) as the  predicate  will
219            include  only  the  highest/lowest ranking instances of the corre‐
220            sponding predicate.  Ranking does not imply sorting, see -6.
221
222            So for example, using proc.memory.rss  (resident  memory  size  of
223            process) as the predicate metric together with proc.io.total_bytes
224            and mem.util.used as metrics to be reported,  only  the  processes
225            using  most/least (as per -J) memory will be included when report‐
226            ing total bytes written by processes.  Since  mem.util.used  is  a
227            single-valued metric (thus not sharing the same instance domain as
228            the process-related metrics), it will be reported as usual.
229
230       -O origin, --origin=origin
231            When reporting archived metrics, start reporting at origin  within
232            the  time window (see -S and -T).  Refer to PCPIntro(1) for a com‐
233            plete description of the syntax for origin.
234
235       -p type, --es-search-type=type
236            Elasticsearch search type for measurements.
237
238       -P precision, --precision=precision
239            Use precision for numeric non-integer output values.  The  default
240            is  to  use  3 decimal places (when applicable).  This option will
241            not override possible per-metric specifications.
242
243       -q scale, --count-scale=scale
244            Unit/scale for count metrics,  possible  values  include  count  x
245            10^-1, count, count x 10, count x 10^2, and so forth from 10^-8 to
246            10^7.  (These values are currently space-sensitive.)  This  option
247            will  not  override  possible per-metric specifications.  See also
248            pmParseUnitsStr(3).
249
250       -Q scale, --count-scale-force=scale
251            Like -q but this option will override per-metric specifications.
252
253       -r, --raw
254            Output raw metric values, do not convert  cumulative  counters  to
255            rates.   This  option will override possible per-metric specifica‐
256            tions.
257
258       -R, --raw-prefer
259            Like -r but this option will not  override  per-metric  specifica‐
260            tions.
261
262       -s samples, --samples=samples
263            The samples argument defines the number of samples to be retrieved
264            and reported.  If samples is 0 or -s is not  specified,  pcp2elas‐
265            ticsearch  will sample and report continuously (in real time mode)
266            or until the end of the set of PCP  archives  (in  archive  mode).
267            See also -T.
268
269       -S starttime, --start=starttime
270            When  reporting archived metrics, the report will be restricted to
271            those records logged at or after starttime.  Refer to  PCPIntro(1)
272            for a complete description of the syntax for starttime.
273
274       -t interval, --interval=interval
275            Set  the  reporting interval to something other than the default 1
276            second.  The interval argument follows  the  syntax  described  in
277            PCPIntro(1),  and  in the simplest form may be an unsigned integer
278            (the implied units in this case are seconds).   See  also  the  -T
279            option.
280
281       -T endtime, --finish=endtime
282            When  reporting archived metrics, the report will be restricted to
283            those records logged before or at endtime.  Refer  to  PCPIntro(1)
284            for a complete description of the syntax for endtime.
285
286            When  used  to  define  the  runtime before pcp2elasticsearch will
287            exit, if no samples is given (see -s) then the number of  reported
288            samples  depends  on  interval (see -t).  If samples is given then
289            interval will be adjusted to allow  reporting  of  samples  during
290            runtime.   In case all of -T, -s, and -t are given, endtime deter‐
291            mines the actual time pcp2elasticsearch will run.
292
293       -v, --omit-flat
294            Omit single-valued ``flat'' metrics from reporting, only  consider
295            set-valued  metrics  (i.e.,  metrics  with  multiple  values)  for
296            reporting.  See -i and -I.
297
298       -V, --version
299            Display version number and exit.
300
301       -x index, --es-index=index
302            Elasticsearch index for metric names.
303
304       -X hostid, --es-hostid=hostid
305            Elasticsearch hostid for measurements.
306
307       -y scale, --time-scale=scale
308            Unit/scale for time metrics, possible values include nanosec,  ns,
309            microsec,  us,  millisec,  ms,  and so forth up to hour, hr.  This
310            option will not override possible per-metric specifications.   See
311            also pmParseUnitsStr(3).
312
313       -Y scale, --time-scale-force=scale
314            Like -y but this option will override per-metric specifications.
315
316       -?, --help
317            Display usage message and exit.
318

OUTPUT FORMAT

320       First, pcp2elasticsearch sends this type definition to Elasticsearch:
321
322               {
323                   "mappings": {
324                       "pcp-metric": {
325                           "properties": {
326                               "@timestamp": {
327                                   "type": "date"
328                               },
329                               "host-id": {
330                                   "type": "string"
331                               }
332                           }
333                       }
334                   }
335               }
336
337       The actual metrics data document is similar to this example:
338
339               {
340                   "@host-id": "bozo",
341                   "mem": {
342                       "util": {
343                           "used": 4525820,
344                           "free": 3344000
345                       }
346                   },
347                   "@timestamp": 1507637520,
348                   "disk": {
349                       "partitions": {
350                           "@instances": [
351                               {
352                                   "read": 1121,
353                                   "@id": "sda1"
354                               },
355                               {
356                                   "read": 77306,
357                                   "@id": "sda2"
358                               },
359                               {
360                                   "read": 131,
361                                   "@id": "sda3"
362                               }
363                           ]
364                       }
365                   }
366               }
367

FILES

369       pcp2elasticsearch.conf
370            pcp2elasticsearch configuration file (see -c)
371

PCP ENVIRONMENT

373       Environment variables with the prefix PCP_ are used to parameterize the
374       file and directory names used by PCP.  On each installation,  the  file
375       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
376       $PCP_CONF variable may be used to specify an alternative  configuration
377       file, as described in pcp.conf(5).
378
379       For environment variables affecting PCP tools, see pmGetOptions(3).
380

SEE ALSO

382       mkaf(1),   PCPIntro(1),   pcp(1),   pcp2graphite(1),   pcp2influxdb(1),
383       pcp2json(1),  pcp2spark(1),  pcp2xlsx(1),  pcp2xml(1),   pcp2zabbix(1),
384       pmcd(1),   pminfo(1),  pmrep(1),  pmGetOptions(3),  pmSpecLocalPMDA(3),
385       pmLoadDerivedConfig(3), pmParseUnitsStr(3), pmRegisterDerived(3), LOGA‐
386       RCHIVE(5), pcp.conf(5), PMNS(5) and pmrep.conf(5).
387
388
389
390Performance Co-Pilot                  PCP                 PCP2ELASTICSEARCH(1)
Impressum