1PCP2ELASTICSEARCH(1)        General Commands Manual       PCP2ELASTICSEARCH(1)
2
3
4

NAME

6       pcp2elasticsearch - pcp-to-elasticsearch metrics exporter
7

SYNOPSIS

9       pcp2elasticsearch  [-5CGHIjLmnrRvV?]  [-4 action] [-8|-9 limit] [-a ar‐
10       chive] [-A align] [--archive-folio folio] [-b|-B space-scale] [-c  con‐
11       fig]  [--container  container]  [--daemonize] [-e derived] [-g URL] [-h
12       host] [-i instances] [-J rank] [-K spec] [-N predicate] [-O origin] [-p
13       type] [-P|-0 precision] [-q|-Q count-scale] [-s samples] [-S starttime]
14       [-t interval] [-T endtime] [-x index] [-X  hostid]  [-y|-Y  time-scale]
15       metricspec [...]
16

DESCRIPTION

18       pcp2elasticsearch  is  a customizable performance metrics exporter tool
19       from PCP to Elasticsearch.  Any available performance metric,  live  or
20       archived,  system and/or application, can be selected for exporting us‐
21       ing either command line arguments or a configuration file.
22
23       pcp2elasticsearch is a close relative of pmrep(1).  Refer  to  pmrep(1)
24       for  the  metricspec  description accepted on pcp2elasticsearch command
25       line.  See pmrep.conf(5) for description of the  pcp2elasticsearch.conf
26       configuration  file  overall  syntax.  This page describes pcp2elastic‐
27       search specific options and configuration  file  differences  with  pm‐
28       rep.conf(5).  pmrep(1) also lists some usage examples of which most are
29       applicable with pcp2elasticsearch as well.
30
31       Only the command line options listed on this page are supported,  other
32       options recognized by pmrep(1) are not supported.
33
34       Options  via environment values (see pmGetOptions(3)) override the cor‐
35       responding built-in default values (if any).   Configuration  file  op‐
36       tions  override the corresponding environment variables (if any).  Com‐
37       mand line options override the corresponding configuration file options
38       (if any).
39

CONFIGURATION FILE

41       pcp2elasticsearch  uses  a  configuration  file with overall syntax de‐
42       scribed in pmrep.conf(5).  The following options are  common  with  pm‐
43       rep.conf:  version,  source,  speclocal, derived, header, globals, sam‐
44       ples, interval, type, type_prefer, ignore_incompat,  names_change,  in‐
45       stances,   live_filter,  rank,  limit_filter,  limit_filter_force,  in‐
46       vert_filter, predicate, omit_flat,  include_labels,  precision,  preci‐
47       sion_force,      count_scale,      count_scale_force,      space_scale,
48       space_scale_force, time_scale, time_scale_force.  The output option  is
49       recognized but ignored for pmrep.conf compatibility.
50
51   pcp2elasticsearch specific options
52       es_server (string)
53           Specify  the  URL  of  Elasticsearch server to send the metrics to.
54           Corresponding command line option is -g.  Default is  http://local
55           host:9200/.
56
57       es_auth (string)
58           Specify the user name to use with Elasticsearch requests.  Defaults
59           to none.
60
61       es_password (string)
62           Specify the pass word to use with Elasticsearch requests.  Defaults
63           to none.
64
65       es_index (string)
66           Specify  the  Elasticsearch  index for metric names.  Corresponding
67           command line option is -x.  Default is pcp.
68
69       es_hostid (string)
70           Specify the Elasticsearch host-id for measurements.   Corresponding
71           command line option is -X.  Defaults to the metrics source host.
72
73       es_search_type (string)
74           Specify  the  Elasticsearch  search  type for measurements.  Corre‐
75           sponding command line option is -p.  Defaults to pcp-metric.
76

OPTIONS

78       The available command line options are:
79
80       -0 precision, --precision-force=precision
81            Like -P but this option will override per-metric specifications.
82
83       -4 action, --names-change=action
84            Specify which action to take on receiving a  metric  names  change
85            event  during  sampling.  These events occur when a PMDA discovers
86            new metrics sometime after starting up, and informs running client
87            tools  like pcp2elasticsearch.  Valid values for action are update
88            (refresh metrics being sampled), ignore (do nothing - the  default
89            behaviour) and abort (exit the program if such an event happens).
90
91       -5, --ignore-unknown
92            Silently ignore any metric name that cannot be resolved.  At least
93            one metric must be found for the tool to start.
94
95       -8 limit, --limit-filter=limit
96            Limit results to instances with values above/below limit.  A posi‐
97            tive  integer  will  include instances with values at or above the
98            limit in reporting.  A negative  integer  will  include  instances
99            with  values  at or below the limit in reporting.  A value of zero
100            performs no limit filtering.  This option will not override possi‐
101            ble per-metric specifications.  See also -J and -N.
102
103       -9 limit, --limit-filter-force=limit
104            Like -8 but this option will override per-metric specifications.
105
106       -a archive, --archive=archive
107            Performance  metric  values  are retrieved from the set of Perfor‐
108            mance Co-Pilot (PCP) archive log files identified by  the  archive
109            argument,  which is a comma-separated list of names, each of which
110            may be the base name of an archive or the name of a directory con‐
111            taining one or more archives.
112
113       -A align, --align=align
114            Force  the initial sample to be aligned on the boundary of a natu‐
115            ral time unit align.  Refer to PCPIntro(1) for a complete descrip‐
116            tion of the syntax for align.
117
118       --archive-folio=folio
119            Read  metric source archives from the PCP archive folio created by
120            tools like pmchart(1) or, less often, manually with mkaf(1).
121
122       -b scale, --space-scale=scale
123            Unit/scale for  space  (byte)  metrics,  possible  values  include
124            bytes, Kbytes, KB, Mbytes, MB, and so forth.  This option will not
125            override possible per-metric specifications.  See  also  pmParseU‐
126            nitsStr(3).
127
128       -B scale, --space-scale-force=scale
129            Like -b but this option will override per-metric specifications.
130
131       -c config, --config=config
132            Specify  the config file or directory to use.  In case config is a
133            directory all files under it ending .conf will be  included.   The
134            default   is   the   first   found  of:  ./pcp2elasticsearch.conf,
135            $HOME/.pcp2elasticsearch.conf,   $HOME/pcp/pcp2elasticsearch.conf,
136            and $PCP_SYSCONF_DIR/pcp2elasticsearch.conf.  For details, see the
137            above section and pmrep.conf(5).
138
139       --container=container
140            Fetch performance metrics from the specified container, either lo‐
141            cal or remote (see -h).
142
143       -C, --check
144            Exit before reporting any values, but after parsing the configura‐
145            tion and metrics and printing possible headers.
146
147       --daemonize
148            Daemonize on startup.
149
150       -e derived, --derived=derived
151            Specify derived performance metrics.  If  derived  starts  with  a
152            slash  (``/'')  or  with a dot (``.'') it will be interpreted as a
153            derived metrics configuration file, otherwise it  will  be  inter‐
154            preted  as  comma-  or  semicolon-separated derived metric expres‐
155            sions.  For details  see  pmLoadDerivedConfig(3)  and  pmRegister‐
156            Derived(3).
157
158       -g URL, --es-host=URL
159            Elasticsearch server URL to send the metrics to.
160
161       -G, --no-globals
162            Do not include global metrics in reporting (see pmrep.conf(5)).
163
164       -h host, --host=host
165            Fetch  performance  metrics from pmcd(1) on host, rather than from
166            the default localhost.
167
168       -H, --no-header
169            Do not print any headers.
170
171       -i instances, --instances=instances
172            Retrieve and report only the specified metric instances.   By  de‐
173            fault all instances, present and future, are reported.
174
175            Refer to pmrep(1) for complete description of this option.
176
177       -I, --ignore-incompat
178            Ignore  incompatible  metrics.   By  default  incompatible metrics
179            (that is, their type is unsupported or they cannot  be  scaled  as
180            requested) will cause pcp2elasticsearch to terminate with an error
181            message.  With this option all incompatible metrics  are  silently
182            omitted  from  reporting.   This may be especially useful when re‐
183            questing non-leaf nodes of the PMNS tree for reporting.
184
185       -j, --live-filter
186            Perform instance live filtering.  This allows capturing all  named
187            instances  even  if  processes are restarted at some point (unlike
188            without live filtering).  Performing live filtering  over  a  huge
189            number  of  instances  will add some internal overhead so a bit of
190            user caution is advised.  See also -n.
191
192       -J rank, --rank=rank
193            Limit results to highest/lowest  ranked  instances  of  set-valued
194            metrics.  A positive integer will include highest valued instances
195            in reporting.  A negative integer will include lowest  valued  in‐
196            stances in reporting.  A value of zero performs no ranking.  Rank‐
197            ing does not imply sorting, see -6.  See also -8.
198
199       -K spec, --spec-local=spec
200            When fetching metrics from a local context (see -L), the -K option
201            may  be used to control the DSO PMDAs that should be made accessi‐
202            ble.  The spec argument conforms to the syntax  described  in  pm‐
203            SpecLocalPMDA(3).  More than one -K option may be used.
204
205       -L, --local-PMDA
206            Use a local context to collect metrics from DSO PMDAs on the local
207            host without PMCD.  See also -K.
208
209       -m, --include-labels
210            Include metric labels in the output.
211
212       -n, --invert-filter
213            Perform ranking before live filtering.  By default  instance  live
214            filtering (when requested, see -j) happens before instance ranking
215            (when requested, see -J).  With this option the logic is  inverted
216            and ranking happens before live filtering.
217
218       -N predicate, --predicate=predicate
219            Specify  a comma-separated list of predicate filter reference met‐
220            rics.  By default ranking (see -J) happens for each  metric  indi‐
221            vidually.  With predicates, ranking is done only for the specified
222            predicate metrics.  When reporting, rest of  the  metrics  sharing
223            the  same  instance domain (see PCPIntro(1)) as the predicate will
224            include only the highest/lowest ranking instances  of  the  corre‐
225            sponding predicate.  Ranking does not imply sorting, see -6.
226
227            So  for  example,  using  proc.memory.rss (resident memory size of
228            process) as the predicate metric together with proc.io.total_bytes
229            and  mem.util.used  as  metrics to be reported, only the processes
230            using most/least (as per -J) memory will be included when  report‐
231            ing  total  bytes  written by processes.  Since mem.util.used is a
232            single-valued metric (thus not sharing the same instance domain as
233            the process related metrics), it will be reported as usual.
234
235       -O origin, --origin=origin
236            When  reporting archived metrics, start reporting at origin within
237            the time window (see -S and -T).  Refer to PCPIntro(1) for a  com‐
238            plete description of the syntax for origin.
239
240       -p type, --es-search-type=type
241            Elasticsearch search type for measurements.
242
243       -P precision, --precision=precision
244            Use  precision for numeric non-integer output values.  The default
245            is to use 3 decimal places (when applicable).   This  option  will
246            not override possible per-metric specifications.
247
248       -q scale, --count-scale=scale
249            Unit/scale  for  count  metrics,  possible  values include count x
250            10^-1, count, count x 10, count x 10^2, and so forth from 10^-8 to
251            10^7.   (These values are currently space-sensitive.)  This option
252            will not override possible per-metric  specifications.   See  also
253            pmParseUnitsStr(3).
254
255       -Q scale, --count-scale-force=scale
256            Like -q but this option will override per-metric specifications.
257
258       -r, --raw
259            Output  raw  metric  values, do not convert cumulative counters to
260            rates.  This option will override possible  per-metric  specifica‐
261            tions.
262
263       -R, --raw-prefer
264            Like  -r  but  this option will not override per-metric specifica‐
265            tions.
266
267       -s samples, --samples=samples
268            The samples argument defines the number of samples to be retrieved
269            and  reported.   If samples is 0 or -s is not specified, pcp2elas‐
270            ticsearch will sample and report continuously (in real time  mode)
271            or  until  the  end  of the set of PCP archives (in archive mode).
272            See also -T.
273
274       -S starttime, --start=starttime
275            When reporting archived metrics, the report will be restricted  to
276            those  records logged at or after starttime.  Refer to PCPIntro(1)
277            for a complete description of the syntax for starttime.
278
279       -t interval, --interval=interval
280            Set the reporting interval to something other than the  default  1
281            second.   The  interval  argument  follows the syntax described in
282            PCPIntro(1), and in the simplest form may be an  unsigned  integer
283            (the implied units in this case are seconds).  See also the -T op‐
284            tion.
285
286       -T endtime, --finish=endtime
287            When reporting archived metrics, the report will be restricted  to
288            those  records  logged before or at endtime.  Refer to PCPIntro(1)
289            for a complete description of the syntax for endtime.
290
291            When used to define  the  runtime  before  pcp2elasticsearch  will
292            exit,  if no samples is given (see -s) then the number of reported
293            samples depends on interval (see -t).  If samples  is  given  then
294            interval  will  be  adjusted  to allow reporting of samples during
295            runtime.  In case all of -T, -s, and -t are given, endtime  deter‐
296            mines the actual time pcp2elasticsearch will run.
297
298       -v, --omit-flat
299            Report only set-valued metrics with instances (e.g. disk.dev.read)
300            and omit single-valued ``flat'' metrics  without  instances  (e.g.
301            kernel.all.sysfork).  See -i and -I.
302
303       -V, --version
304            Display version number and exit.
305
306       -x index, --es-index=index
307            Elasticsearch index for metric names.
308
309       -X hostid, --es-hostid=hostid
310            Elasticsearch hostid for measurements.
311
312       -y scale, --time-scale=scale
313            Unit/scale  for time metrics, possible values include nanosec, ns,
314            microsec, us, millisec, ms, and so forth up to hour, hr.  This op‐
315            tion  will  not  override possible per-metric specifications.  See
316            also pmParseUnitsStr(3).
317
318       -Y scale, --time-scale-force=scale
319            Like -y but this option will override per-metric specifications.
320
321       -?, --help
322            Display usage message and exit.
323

OUTPUT FORMAT

325       First, pcp2elasticsearch sends this type definition to Elasticsearch:
326
327               {
328                   "mappings": {
329                       "pcp-metric": {
330                           "properties": {
331                               "@timestamp": {
332                                   "type": "date"
333                               },
334                               "host-id": {
335                                   "type": "string"
336                               }
337                           }
338                       }
339                   }
340               }
341
342       The actual metrics data document is similar to this example:
343
344               {
345                   "@host-id": "bozo",
346                   "mem": {
347                       "util": {
348                           "used": 4525820,
349                           "free": 3344000
350                       }
351                   },
352                   "@timestamp": 1507637520,
353                   "disk": {
354                       "partitions": {
355                           "@instances": [
356                               {
357                                   "read": 1121,
358                                   "@id": "sda1"
359                               },
360                               {
361                                   "read": 77306,
362                                   "@id": "sda2"
363                               },
364                               {
365                                   "read": 131,
366                                   "@id": "sda3"
367                               }
368                           ]
369                       }
370                   }
371               }
372

FILES

374       pcp2elasticsearch.conf
375            pcp2elasticsearch configuration file (see -c)
376

PCP ENVIRONMENT

378       Environment variables with the prefix PCP_ are used to parameterize the
379       file  and  directory names used by PCP.  On each installation, the file
380       /etc/pcp.conf contains the  local  values  for  these  variables.   The
381       $PCP_CONF  variable may be used to specify an alternative configuration
382       file, as described in pcp.conf(5).
383
384       For environment variables affecting PCP tools, see pmGetOptions(3).
385

SEE ALSO

387       mkaf(1),   PCPIntro(1),   pcp(1),   pcp2graphite(1),   pcp2influxdb(1),
388       pcp2json(1),   pcp2spark(1),  pcp2xlsx(1),  pcp2xml(1),  pcp2zabbix(1),
389       pmcd(1), pminfo(1), pmrep(1), pmGetOptions(3), pmSpecLocalPMDA(3),  pm‐
390       LoadDerivedConfig(3),  pmParseUnitsStr(3),  pmRegisterDerived(3), LOGA‐
391       RCHIVE(5), pcp.conf(5), PMNS(5) and pmrep.conf(5).
392
393
394
395Performance Co-Pilot                  PCP                 PCP2ELASTICSEARCH(1)
Impressum