1PCP2ELASTICSEARCH(1)        General Commands Manual       PCP2ELASTICSEARCH(1)
2
3
4

NAME

6       pcp2elasticsearch - pcp-to-elasticsearch metrics exporter
7

SYNOPSIS

9       pcp2elasticsearch  [-5CGHIjLmnrRvV?]  [-4 action] [-8|-9 limit] [-a ar‐
10       chive] [-A align] [--archive-folio folio] [-b|-B space-scale] [-c  con‐
11       fig]  [--container  container]  [--daemonize] [-e derived] [-g URL] [-h
12       host] [-i instances] [-J rank] [-K spec] [-N predicate] [-O origin] [-p
13       type] [-P|-0 precision] [-q|-Q count-scale] [-s samples] [-S starttime]
14       [-t interval] [-T endtime] [-x index] [-X  hostid]  [-y|-Y  time-scale]
15       metricspec [...]
16

DESCRIPTION

18       pcp2elasticsearch  is  a customizable performance metrics exporter tool
19       from PCP to Elasticsearch.  Any available performance metric,  live  or
20       archived,  system and/or application, can be selected for exporting us‐
21       ing either command line arguments or a configuration file.
22
23       pcp2elasticsearch is a close relative of pmrep(1).  Refer  to  pmrep(1)
24       for  the  metricspec  description accepted on pcp2elasticsearch command
25       line.  See pmrep.conf(5) for description of the  pcp2elasticsearch.conf
26       configuration  file syntax.  This page describes pcp2elasticsearch spe‐
27       cific options and configuration file  differences  with  pmrep.conf(5).
28       pmrep(1)  also  lists  some usage examples of which most are applicable
29       with pcp2elasticsearch as well.
30
31       Only the command line options listed on this page are supported,  other
32       options available for pmrep(1) are not supported.
33
34       Options  via environment values (see pmGetOptions(3)) override the cor‐
35       responding built-in default values (if any).   Configuration  file  op‐
36       tions  override the corresponding environment variables (if any).  Com‐
37       mand line options override the corresponding configuration file options
38       (if any).
39

CONFIGURATION FILE

41       pcp2elasticsearch  uses  a  configuration file with syntax described in
42       pmrep.conf(5).  The following options are common with pmrep.conf:  ver‐
43       sion,  source,  speclocal, derived, header, globals, samples, interval,
44       type, type_prefer, ignore_incompat, names_change, instances,  live_fil‐
45       ter,  rank, limit_filter, limit_filter_force, invert_filter, predicate,
46       omit_flat,  include_labels,  precision,  precision_force,  count_scale,
47       count_scale_force,    space_scale,    space_scale_force,    time_scale,
48       time_scale_force.  The rest of the pmrep.conf  options  are  recognized
49       but ignored for compatibility.
50
51   pcp2elasticsearch specific options
52       es_server (string)
53           Specify  the  destination  Elasticsearch server metric URL.  Corre‐
54           sponding command line option  is  -g.   Defaults  to  http://local
55           host:9200/.
56
57       es_auth (string)
58           Specify the user name to use with Elasticsearch requests.  Defaults
59           to none.
60
61       es_password (string)
62           Specify the pass word to use with Elasticsearch requests.  Defaults
63           to none.
64
65       es_index (string)
66           Specify  the  Elasticsearch  index for metric names.  Corresponding
67           command line option is -x.  Defaults to pcp.
68
69       es_hostid (string)
70           Specify the Elasticsearch host-id for measurements.   Corresponding
71           command line option is -X.  Defaults to the metrics source host.
72
73       es_search_type (string)
74           Specify  the  Elasticsearch  search  type for measurements.  Corre‐
75           sponding command line option is -p.  Defaults to pcp-metric.
76

OPTIONS

78       The available command line options are:
79
80       -0 precision, --precision-force=precision
81            Like -P but this option will override per-metric specifications.
82
83       -4 action, --names-change=action
84            Specify which action to take on receiving a  metric  names  change
85            event  during  sampling.  These events occur when a PMDA discovers
86            new metrics sometime after starting up, and informs running client
87            tools  like pcp2elasticsearch.  Valid values for action are update
88            (refresh metrics being sampled), ignore (do nothing - the  default
89            behaviour) and abort (exit the program if such an event occurs).
90
91       -5, --ignore-unknown
92            Silently ignore any metric name that cannot be resolved.  At least
93            one metric must be found for the tool to start.
94
95       -8 limit, --limit-filter=limit
96            Limit results to instances with values above/below limit.  A posi‐
97            tive  integer  will  include instances with values at or above the
98            limit in reporting.  A negative  integer  will  include  instances
99            with  values  at or below the limit in reporting.  A value of zero
100            performs no limit filtering.  This option will not override possi‐
101            ble per-metric specifications.  See also -J and -N.
102
103       -9 limit, --limit-filter-force=limit
104            Like -8 but this option will override per-metric specifications.
105
106       -a archive, --archive=archive
107            Performance  metric  values  are retrieved from the set of Perfor‐
108            mance Co-Pilot (PCP) archive files identified by the archive argu‐
109            ment,  which is a comma-separated list of names, each of which may
110            be the base name of an archive or the name of a directory contain‐
111            ing one or more archives.
112
113       -A align, --align=align
114            Force  the initial sample to be aligned on the boundary of a natu‐
115            ral time unit align.  Refer to PCPIntro(1) for a complete descrip‐
116            tion of the syntax for align.
117
118       --archive-folio=folio
119            Read  metric source archives from the PCP archive folio created by
120            tools like pmchart(1) or, less often, manually with mkaf(1).
121
122       -b scale, --space-scale=scale
123            Unit/scale for  space  (byte)  metrics,  possible  values  include
124            bytes, Kbytes, KB, Mbytes, MB, and so forth.  This option will not
125            override possible per-metric specifications.  See  also  pmParseU‐
126            nitsStr(3).
127
128       -B scale, --space-scale-force=scale
129            Like -b but this option will override per-metric specifications.
130
131       -c config, --config=config
132            Specify  the config file or directory to use.  In case config is a
133            directory all files in it ending .conf will be included.  The  de‐
134            fault   is   the   first   found   of:   ./pcp2elasticsearch.conf,
135            $HOME/.pcp2elasticsearch.conf,   $HOME/pcp/pcp2elasticsearch.conf,
136            and $PCP_SYSCONF_DIR/pcp2elasticsearch.conf.  For details, see the
137            above section and pmrep.conf(5).
138
139       --container=container
140            Fetch performance metrics from the specified container, either lo‐
141            cal or remote (see -h).
142
143       -C, --check
144            Exit before reporting any values, but after parsing the configura‐
145            tion and metrics and printing possible headers.
146
147       --daemonize
148            Daemonize on startup.
149
150       -e derived, --derived=derived
151            Specify derived performance metrics.  If  derived  starts  with  a
152            slash  (``/'')  or  with a dot (``.'') it will be interpreted as a
153            PCP derived metrics configuration file, otherwise it will  be  in‐
154            terpreted  as comma- or semicolon-separated derived metric expres‐
155            sions.  For complete description of derived metrics  and  PCP  de‐
156            rived  metrics  configuration files see pmLoadDerivedConfig(3) and
157            pmRegisterDerived(3).  Alternatively, using pmrep.conf(5) configu‐
158            ration  syntax  allows defining derived metrics as part of metric‐
159            sets.
160
161       -g URL, --es-host=URL
162            Destination Elasticsearch server metric URL.
163
164       -G, --no-globals
165            Do not include global metrics in reporting (see pmrep.conf(5)).
166
167       -h host, --host=host
168            Fetch performance metrics from pmcd(1) on host, rather  than  from
169            the default localhost.
170
171       -H, --no-header
172            Do not print any headers.
173
174       -i instances, --instances=instances
175            Retrieve  and  report only the specified metric instances.  By de‐
176            fault all instances, present and future, are reported.
177
178            Refer to pmrep(1) for complete description of this option.
179
180       -I, --ignore-incompat
181            Ignore incompatible  metrics.   By  default  incompatible  metrics
182            (that  is,  their  type is unsupported or they cannot be scaled as
183            requested) will cause pcp2elasticsearch to terminate with an error
184            message.   With  this option all incompatible metrics are silently
185            omitted from reporting.  This may be especially  useful  when  re‐
186            questing non-leaf nodes of the PMNS tree for reporting.
187
188       -j, --live-filter
189            Perform  instance live filtering.  This allows capturing all named
190            instances even if processes are restarted at  some  point  (unlike
191            without  live  filtering).   Performing live filtering over a huge
192            number of instances will add some internal overhead so  a  bit  of
193            user caution is advised.  See also -n.
194
195       -J rank, --rank=rank
196            Limit  results  to  highest/lowest  ranked instances of set-valued
197            metrics.  A positive integer will include highest valued instances
198            in  reporting.   A negative integer will include lowest valued in‐
199            stances in reporting.  A value of zero performs no ranking.  Rank‐
200            ing does not imply sorting, see -6.  See also -8.
201
202       -K spec, --spec-local=spec
203            When fetching metrics from a local context (see -L), the -K option
204            may be used to control the DSO PMDAs that should be made  accessi‐
205            ble.   The  spec  argument conforms to the syntax described in pm‐
206            SpecLocalPMDA(3).  More than one -K option may be used.
207
208       -L, --local-PMDA
209            Use a local context to collect metrics from DSO PMDAs on the local
210            host without PMCD.  See also -K.
211
212       -m, --include-labels
213            Include PCP metric labels in the output.
214
215       -n, --invert-filter
216            Perform  ranking  before live filtering.  By default instance live
217            filtering (when requested, see -j) happens before instance ranking
218            (when  requested, see -J).  With this option the logic is inverted
219            and ranking happens before live filtering.
220
221       -N predicate, --predicate=predicate
222            Specify a comma-separated list of predicate filter reference  met‐
223            rics.   By  default ranking (see -J) happens for each metric indi‐
224            vidually.  With predicates, ranking is done only for the specified
225            predicate  metrics.   When  reporting, rest of the metrics sharing
226            the same instance domain (see PCPIntro(1)) as the  predicate  will
227            include  only  the  highest/lowest ranking instances of the corre‐
228            sponding predicate.  Ranking does not imply sorting, see -6.
229
230            So for example, using proc.memory.rss  (resident  memory  size  of
231            process) as the predicate metric together with proc.io.total_bytes
232            and mem.util.used as metrics to be reported,  only  the  processes
233            using  most/least (as per -J) memory will be included when report‐
234            ing total bytes written by processes.  Since  mem.util.used  is  a
235            single-valued metric (thus not sharing the same instance domain as
236            the process related metrics), it will be reported as usual.
237
238       -O origin, --origin=origin
239            When reporting archived metrics, start reporting at origin  within
240            the  time window (see -S and -T).  Refer to PCPIntro(1) for a com‐
241            plete description of the syntax for origin.
242
243       -p type, --es-search-type=type
244            Elasticsearch search type for measurements.
245
246       -P precision, --precision=precision
247            Use precision for numeric non-integer output values.  The  default
248            is  to  use  3 decimal places (when applicable).  This option will
249            not override possible per-metric specifications.
250
251       -q scale, --count-scale=scale
252            Unit/scale for count metrics,  possible  values  include  count  x
253            10^-1, count, count x 10, count x 10^2, and so forth from 10^-8 to
254            10^7.  (These values are currently space-sensitive.)  This  option
255            will  not  override  possible per-metric specifications.  See also
256            pmParseUnitsStr(3).
257
258       -Q scale, --count-scale-force=scale
259            Like -q but this option will override per-metric specifications.
260
261       -r, --raw
262            Output raw metric values, do not convert  cumulative  counters  to
263            rates.   This  option will override possible per-metric specifica‐
264            tions.
265
266       -R, --raw-prefer
267            Like -r but this option will not  override  per-metric  specifica‐
268            tions.
269
270       -s samples, --samples=samples
271            The samples argument defines the number of samples to be retrieved
272            and reported.  If samples is 0 or -s is not  specified,  pcp2elas‐
273            ticsearch  will sample and report continuously (in real time mode)
274            or until the end of the set of PCP  archives  (in  archive  mode).
275            See also -T.
276
277       -S starttime, --start=starttime
278            When  reporting archived metrics, the report will be restricted to
279            those records logged at or after starttime.  Refer to  PCPIntro(1)
280            for a complete description of the syntax for starttime.
281
282       -t interval, --interval=interval
283            Set  the  reporting interval to something other than the default 1
284            second.  The interval argument follows  the  syntax  described  in
285            PCPIntro(1),  and  in the simplest form may be an unsigned integer
286            (the implied units in this case are seconds).  See also the -T op‐
287            tion.
288
289       -T endtime, --finish=endtime
290            When  reporting archived metrics, the report will be restricted to
291            those records logged before or at endtime.  Refer  to  PCPIntro(1)
292            for a complete description of the syntax for endtime.
293
294            When  used  to  define  the  runtime before pcp2elasticsearch will
295            exit, if no samples is given (see -s) then the number of  reported
296            samples  depends  on  interval (see -t).  If samples is given then
297            interval will be adjusted to allow  reporting  of  samples  during
298            runtime.   In case all of -T, -s, and -t are given, endtime deter‐
299            mines the actual time pcp2elasticsearch will run.
300
301       -v, --omit-flat
302            Report only set-valued metrics with instances (e.g. disk.dev.read)
303            and  omit  single-valued  ``flat'' metrics without instances (e.g.
304            kernel.all.sysfork).  See -i and -I.
305
306       -V, --version
307            Display version number and exit.
308
309       -x index, --es-index=index
310            Elasticsearch index for metric names.
311
312       -X hostid, --es-hostid=hostid
313            Elasticsearch hostid for measurements.
314
315       -y scale, --time-scale=scale
316            Unit/scale for time metrics, possible values include nanosec,  ns,
317            microsec, us, millisec, ms, and so forth up to hour, hr.  This op‐
318            tion will not override possible  per-metric  specifications.   See
319            also pmParseUnitsStr(3).
320
321       -Y scale, --time-scale-force=scale
322            Like -y but this option will override per-metric specifications.
323
324       -?, --help
325            Display usage message and exit.
326

OUTPUT FORMAT

328       First, pcp2elasticsearch sends this type definition to Elasticsearch:
329
330               {
331                   "mappings": {
332                       "pcp-metric": {
333                           "properties": {
334                               "@timestamp": {
335                                   "type": "date"
336                               },
337                               "host-id": {
338                                   "type": "string"
339                               }
340                           }
341                       }
342                   }
343               }
344
345       The actual metrics data document is similar to this example:
346
347               {
348                   "@host-id": "bozo",
349                   "mem": {
350                       "util": {
351                           "used": 4525820,
352                           "free": 3344000
353                       }
354                   },
355                   "@timestamp": 1507637520,
356                   "disk": {
357                       "partitions": {
358                           "@instances": [
359                               {
360                                   "read": 1121,
361                                   "@id": "sda1"
362                               },
363                               {
364                                   "read": 77306,
365                                   "@id": "sda2"
366                               },
367                               {
368                                   "read": 131,
369                                   "@id": "sda3"
370                               }
371                           ]
372                       }
373                   }
374               }
375

FILES

377       pcp2elasticsearch.conf
378            pcp2elasticsearch configuration file (see -c)
379
380       $PCP_SYSCONF_DIR/pmrep/*.conf
381            system provided default pmrep configuration files
382

PCP ENVIRONMENT

384       Environment variables with the prefix PCP_ are used to parameterize the
385       file and directory names used by PCP.  On each installation,  the  file
386       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
387       $PCP_CONF variable may be used to specify an alternative  configuration
388       file, as described in pcp.conf(5).
389
390       For environment variables affecting PCP tools, see pmGetOptions(3).
391

SEE ALSO

393       PCPIntro(1),   mkaf(1),   pcp(1),   pcp2graphite(1),   pcp2influxdb(1),
394       pcp2json(1),  pcp2spark(1),  pcp2xlsx(1),  pcp2xml(1),   pcp2zabbix(1),
395       pmcd(1),  pminfo(1), pmrep(1), pmGetOptions(3), pmLoadDerivedConfig(3),
396       pmParseUnitsStr(3),  pmRegisterDerived(3),  pmSpecLocalPMDA(3),   LOGA‐
397       RCHIVE(5), pcp.conf(5), pmrep.conf(5) and PMNS(5).
398
399
400
401Performance Co-Pilot                  PCP                 PCP2ELASTICSEARCH(1)
Impressum