1PCP2ELASTICSEARCH(1)        General Commands Manual       PCP2ELASTICSEARCH(1)
2
3
4

NAME

6       pcp2elasticsearch - pcp-to-elasticsearch metrics exporter
7

SYNOPSIS

9       pcp2elasticsearch  [-5CGHIjLnrRvV?]   [-4 action] [-8|-9 limit] [-a ar‐
10       chive] [-A align] [--archive-folio folio] [-b|-B space-scale] [-c  con‐
11       fig]  [--container  container]  [--daemonize] [-e derived] [-g URL] [-h
12       host] [-i instances] [-J rank] [-K spec] [-N predicate] [-O origin] [-p
13       type] [-P|-0 precision] [-q|-Q count-scale] [-s samples] [-S starttime]
14       [-t interval] [-T endtime] [-x index] [-X  hostid]  [-y|-Y  time-scale]
15       metricspec [...]
16

DESCRIPTION

18       pcp2elasticsearch  is  a customizable performance metrics exporter tool
19       from PCP to Elasticsearch.  Any available performance metric,  live  or
20       archived,  system  and/or  application,  can  be selected for exporting
21       using either command line arguments or a configuration file.
22
23       pcp2elasticsearch is a close relative of  pmrep(1).   Please  refer  to
24       pmrep(1)  for  the metricspec description accepted on pcp2elasticsearch
25       command line and pmrep.conf(5)  for  description  of  the  pcp2elastic‐
26       search.conf  configuration  file  overall  syntax,  this page describes
27       pcp2elasticsearch specific options and configuration  file  differences
28       with  pmrep.conf(5).   pmrep(1) also lists some usage examples of which
29       most are applicable with pcp2elasticsearch as well.
30
31       Only the command line options listed on this page are supported,  other
32       options recognized by pmrep(1) are not supported.
33
34       Options  via environment values (see pmGetOptions(3)) override the cor‐
35       responding  built-in  default  values  (if  any).   Configuration  file
36       options  override  the  corresponding  environment  variables (if any).
37       Command line options  override  the  corresponding  configuration  file
38       options (if any).
39

CONFIGURATION FILE

41       pcp2elasticsearch   uses  a  configuration  file  with  overall  syntax
42       described in pmrep.conf(5).  The  following  options  are  common  with
43       pmrep.conf:  version, source, speclocal, derived, header, globals, sam‐
44       ples,  interval,  type,  type_prefer,  ignore_incompat,   names_change,
45       instances,   live_filter,   rank,   limit_filter,   limit_filter_force,
46       invert_filter,  predicate,   omit_flat,   precision,   precision_force,
47       count_scale,    count_scale_force,    space_scale,   space_scale_force,
48       time_scale, time_scale_force.  The  output  option  is  recognized  but
49       ignored for pmrep.conf compatibility.
50
51   pcp2elasticsearch specific options
52       es_server (string)
53           Specify  the  URL  of  Elasticsearch server to send the metrics to.
54           Corresponding command line option is -g.  Default is  http://local
55           host:9200/.
56
57       es_index (string)
58           Specify  the  Elasticsearch  index for metric names.  Corresponding
59           command line option is -x.  Default is pcp.
60
61       es_hostid (string)
62           Specify the Elasticsearch host-id for measurements.   Corresponding
63           command line option is -X.  Defaults to the metrics source host.
64
65       es_search_type (string)
66           Specify  the  Elasticsearch  search  type for measurements.  Corre‐
67           sponding command line option is -p.  Defaults to pcp-metric.
68

OPTIONS

70       The available command line options are:
71
72       -0 precision, --precision-force=precision
73            Like -P but this option will override per-metric specifications.
74
75       -4 action, --names-change=action
76            Specify which action to take on receiving a  metric  names  change
77            event  during  sampling.  These events occur when a PMDA discovers
78            new metrics sometime after starting up, and informs running client
79            tools  like pcp2elasticsearch.  Valid values for action are update
80            (refresh metrics being sampled), ignore (do nothing - the  default
81            behaviour) and abort (exit the program if such an event happens).
82
83       -5, --ignore-unknown
84            Silently ignore any metric name that cannot be resolved.  At least
85            one metric must be found for the tool to start.
86
87       -8 limit, --limit-filter=limit
88            Limit results to instances with values above/below limit.  A posi‐
89            tive  integer  will  include instances with values at or above the
90            limit in reporting.  A negative  integer  will  include  instances
91            with  values  at or below the limit in reporting.  A value of zero
92            performs no limit filtering.  This option will not override possi‐
93            ble per-metric specifications.  See also -J and -N.
94
95       -9 limit, --limit-filter-force=limit
96            Like -8 but this option will override per-metric specifications.
97
98       -a archive, --archive=archive
99            Performance  metric  values  are retrieved from the set of Perfor‐
100            mance Co-Pilot (PCP) archive log files identified by  the  archive
101            argument,  which is a comma-separated list of names, each of which
102            may be the base name of an archive or the name of a directory con‐
103            taining one or more archives.
104
105       -A align, --align=align
106            Force  the initial sample to be aligned on the boundary of a natu‐
107            ral time unit align.  Refer to PCPIntro(1) for a complete descrip‐
108            tion of the syntax for align.
109
110       --archive-folio=folio
111            Read  metric source archives from the PCP archive folio created by
112            tools like pmchart(1) or, less often, manually with mkaf(1).
113
114       -b scale, --space-scale=scale
115            Unit/scale for  space  (byte)  metrics,  possible  values  include
116            bytes, Kbytes, KB, Mbytes, MB, and so forth.  This option will not
117            override possible per-metric specifications.  See  also  pmParseU‐
118            nitsStr(3).
119
120       -B scale, --space-scale-force=scale
121            Like -b but this option will override per-metric specifications.
122
123       -c config, --config=config
124            Specify  the  config  file to use.  The default is the first found
125            of:    ./pcp2elasticsearch.conf,    $HOME/.pcp2elasticsearch.conf,
126            $HOME/pcp/pcp2elasticsearch.conf,  and  $PCP_SYSCONF_DIR/pcp2elas‐
127            ticsearch.conf.   For  details,  see   the   above   section   and
128            pmrep.conf(5).
129
130       --container=container
131            Fetch  performance  metrics  from  the specified container, either
132            local or remote (see -h).
133
134       -C, --check
135            Exit before reporting any values, but after parsing the configura‐
136            tion and metrics and printing possible headers.
137
138       --daemonize
139            Daemonize on startup.
140
141       -e derived, --derived=derived
142            Specify  derived  performance  metrics.   If derived starts with a
143            slash (``/'') or with a dot (``.'') it will be  interpreted  as  a
144            derived  metrics  configuration  file, otherwise it will be inter‐
145            preted as comma- or  semicolon-separated  derived  metric  expres‐
146            sions.   For  details  see  pmLoadDerivedConfig(3) and pmRegister‐
147            Derived(3).
148
149       -g URL, --es-host=URL
150            Elasticsearch server URL to send the metrics to.
151
152       -G, --no-globals
153            Do not include global metrics in reporting (see pmrep.conf(5)).
154
155       -h host, --host=host
156            Fetch performance metrics from pmcd(1) on host, rather  than  from
157            the default localhost.
158
159       -H, --no-header
160            Do not print any headers.
161
162       -i instances, --instances=instances
163            Report  only  the  listed  instances  from  current  instances (if
164            present, see also -j).  By  default  all  instances,  present  and
165            future,  are  reported.   This is a global option that is used for
166            all metrics unless a metric-specific instance definition  is  pro‐
167            vided  as part of a metricspec.  By default single-valued ``flat''
168            metrics without multiple instances are still  reported  as  usual,
169            use  -v to change this.  Please refer to pmrep(1) for more details
170            on this option.
171
172       -I, --ignore-incompat
173            Ignore incompatible  metrics.   By  default  incompatible  metrics
174            (that  is,  their  type is unsupported or they cannot be scaled as
175            requested) will cause pcp2elasticsearch to terminate with an error
176            message.   With  this option all incompatible metrics are silently
177            omitted from  reporting.   This  may  be  especially  useful  when
178            requesting non-leaf nodes of the PMNS tree for reporting.
179
180       -j, --live-filter
181            Perform  instance  live filtering.  This allows capturing all fil‐
182            tered instances even if processes  are  restarted  at  some  point
183            (unlike without live filtering).  Performing live filtering over a
184            huge amount of instances will add some internal overhead so a  bit
185            of user caution is advised.  See also -n.
186
187       -J rank, --rank=rank
188            Limit  results  to  highest/lowest  ranked instances of set-valued
189            metrics.  A positive integer will include highest valued instances
190            in  reporting.   A  negative  integer  will  include lowest valued
191            instances in reporting.  A value  of  zero  performs  no  ranking.
192            Ranking does not imply sorting, see -6.  See also -8.
193
194       -K spec, --spec-local=spec
195            When fetching metrics from a local context (see -L), the -K option
196            may be used to control the DSO PMDAs that should be made  accessi‐
197            ble.   The  spec  argument  conforms  to  the  syntax described in
198            pmSpecLocalPMDA(3).  More than one -K option may be used.
199
200       -L, --local-PMDA
201            Use a local context to collect metrics from DSO PMDAs on the local
202            host without PMCD.  See also -K.
203
204       -n, --invert-filter
205            Perform  ranking  before live filtering.  By default instance live
206            filtering (when requested, see -j) happens before instance ranking
207            (when  requested, see -J).  With this option the logic is inverted
208            and ranking happens before live filtering.
209
210       -N predicate, --predicate=predicate
211            Specify a comma-separated list of predicate filter reference  met‐
212            rics.   By  default ranking (see -J) happens for each metric indi‐
213            vidually.  With predicates, ranking is done only for the specified
214            predicate  metrics.   When  reporting, rest of the metrics sharing
215            the same instance domain (see PCPIntro(1)) as the  predicate  will
216            include  only  the  highest/lowest ranking instances of the corre‐
217            sponding predicate.  Ranking does not imply sorting, see -6.
218
219            So for example, using proc.memory.rss  (resident  memory  size  of
220            process) as the predicate metric together with proc.io.total_bytes
221            and mem.util.used as metrics to be reported,  only  the  processes
222            using  most/least (as per -J) memory will be included when report‐
223            ing total bytes written by processes.  Since  mem.util.used  is  a
224            single-valued metric (thus not sharing the same instance domain as
225            the process-related metrics), it will be reported as usual.
226
227       -O origin, --origin=origin
228            When reporting archived metrics, start reporting at origin  within
229            the  time window (see -S and -T).  Refer to PCPIntro(1) for a com‐
230            plete description of the syntax for origin.
231
232       -p type, --es-search-type=type
233            Elasticsearch search type for measurements.
234
235       -P precision, --precision=precision
236            Use precision for numeric non-integer output values.  The  default
237            is  to  use  3 decimal places (when applicable).  This option will
238            not override possible per-metric specifications.
239
240       -q scale, --count-scale=scale
241            Unit/scale for count metrics,  possible  values  include  count  x
242            10^-1, count, count x 10, count x 10^2, and so forth from 10^-8 to
243            10^7.  (These values are currently space-sensitive.)  This  option
244            will  not  override  possible per-metric specifications.  See also
245            pmParseUnitsStr(3).
246
247       -Q scale, --count-scale-force=scale
248            Like -q but this option will override per-metric specifications.
249
250       -r, --raw
251            Output raw metric values, do not convert  cumulative  counters  to
252            rates.   This  option will override possible per-metric specifica‐
253            tions.
254
255       -R, --raw-prefer
256            Like -r but this option will not  override  per-metric  specifica‐
257            tions.
258
259       -s samples, --samples=samples
260            The samples argument defines the number of samples to be retrieved
261            and reported.  If samples is 0 or -s is not  specified,  pcp2elas‐
262            ticsearch  will sample and report continuously (in real time mode)
263            or until the end of the set of PCP  archives  (in  archive  mode).
264            See also -T.
265
266       -S starttime, --start=starttime
267            When  reporting archived metrics, the report will be restricted to
268            those records logged at or after starttime.  Refer to  PCPIntro(1)
269            for a complete description of the syntax for starttime.
270
271       -t interval, --interval=interval
272            Set  the  reporting interval to something other than the default 1
273            second.  The interval argument follows  the  syntax  described  in
274            PCPIntro(1),  and  in the simplest form may be an unsigned integer
275            (the implied units in this case are seconds).   See  also  the  -T
276            option.
277
278       -T endtime, --finish=endtime
279            When  reporting archived metrics, the report will be restricted to
280            those records logged before or at endtime.  Refer  to  PCPIntro(1)
281            for a complete description of the syntax for endtime.
282
283            When  used  to  define  the  runtime before pcp2elasticsearch will
284            exit, if no samples is given (see -s) then the number of  reported
285            samples  depends  on  interval (see -t).  If samples is given then
286            interval will be adjusted to allow  reporting  of  samples  during
287            runtime.   In case all of -T, -s, and -t are given, endtime deter‐
288            mines the actual time pcp2elasticsearch will run.
289
290       -v, --omit-flat
291            Omit single-valued ``flat'' metrics from reporting, only  consider
292            set-valued  metrics  (i.e.,  metrics  with  multiple  values)  for
293            reporting.  See -i and -I.
294
295       -V, --version
296            Display version number and exit.
297
298       -x index, --es-index=index
299            Elasticsearch index for metric names.
300
301       -X hostid, --es-hostid=hostid
302            Elasticsearch hostid for measurements.
303
304       -y scale, --time-scale=scale
305            Unit/scale for time metrics, possible values include nanosec,  ns,
306            microsec,  us,  millisec,  ms,  and so forth up to hour, hr.  This
307            option will not override possible per-metric specifications.   See
308            also pmParseUnitsStr(3).
309
310       -Y scale, --time-scale-force=scale
311            Like -y but this option will override per-metric specifications.
312
313       -?, --help
314            Display usage message and exit.
315

OUTPUT FORMAT

317       First, pcp2elasticsearch sends this type definition to Elasticsearch:
318
319               {
320                   "mappings": {
321                       "pcp-metric": {
322                           "properties": {
323                               "@timestamp": {
324                                   "type": "date"
325                               },
326                               "host-id": {
327                                   "type": "string"
328                               }
329                           }
330                       }
331                   }
332               }
333
334       The actual metrics data document is similar to this example:
335
336               {
337                   "@host-id": "bozo",
338                   "mem": {
339                       "util": {
340                           "used": 4525820,
341                           "free": 3344000
342                       }
343                   },
344                   "@timestamp": 1507637520,
345                   "disk": {
346                       "partitions": {
347                           "@instances": [
348                               {
349                                   "read": 1121,
350                                   "@id": "sda1"
351                               },
352                               {
353                                   "read": 77306,
354                                   "@id": "sda2"
355                               },
356                               {
357                                   "read": 131,
358                                   "@id": "sda3"
359                               }
360                           ]
361                       }
362                   }
363               }
364

FILES

366       pcp2elasticsearch.conf
367            pcp2elasticsearch configuration file (see -c)
368

PCP ENVIRONMENT

370       Environment variables with the prefix PCP_ are used to parameterize the
371       file and directory names used by PCP.  On each installation,  the  file
372       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
373       $PCP_CONF variable may be used to specify an alternative  configuration
374       file, as described in pcp.conf(5).
375
376       For environment variables affecting PCP tools, see pmGetOptions(3).
377

SEE ALSO

379       mkaf(1),   PCPIntro(1),   pcp(1),   pcp2graphite(1),   pcp2influxdb(1),
380       pcp2json(1),  pcp2spark(1),  pcp2xlsx(1),  pcp2xml(1),   pcp2zabbix(1),
381       pmcd(1),   pminfo(1),  pmrep(1),  pmGetOptions(3),  pmSpecLocalPMDA(3),
382       pmLoadDerivedConfig(3), pmParseUnitsStr(3), pmRegisterDerived(3), LOGA‐
383       RCHIVE(5), pcp.conf(5), PMNS(5) and pmrep.conf(5).
384
385
386
387Performance Co-Pilot                  PCP                 PCP2ELASTICSEARCH(1)
Impressum