1PCP2ELASTICSEARCH(1)        General Commands Manual       PCP2ELASTICSEARCH(1)
2
3
4

NAME

6       pcp2elasticsearch - pcp-to-elasticsearch metrics exporter
7

SYNOPSIS

9       pcp2elasticsearch  [-5CGHIjLmnrRvV?]  [-4 action] [-8|-9 limit] [-a ar‐
10       chive] [-A align] [--archive-folio folio] [-b|-B space-scale] [-c  con‐
11       fig]  [--container  container]  [--daemonize] [-e derived] [-g URL] [-h
12       host] [-i instances] [-J rank] [-K spec] [-N predicate] [-O origin] [-p
13       type] [-P|-0 precision] [-q|-Q count-scale] [-s samples] [-S starttime]
14       [-t interval] [-T endtime] [-x index] [-X  hostid]  [-y|-Y  time-scale]
15       metricspec [...]
16

DESCRIPTION

18       pcp2elasticsearch  is  a customizable performance metrics exporter tool
19       from PCP to Elasticsearch.  Any available performance metric,  live  or
20       archived,  system and/or application, can be selected for exporting us‐
21       ing either command line arguments or a configuration file.
22
23       pcp2elasticsearch is a close relative of pmrep(1).  Refer  to  pmrep(1)
24       for  the  metricspec  description accepted on pcp2elasticsearch command
25       line.  See pmrep.conf(5) for description of the  pcp2elasticsearch.conf
26       configuration  file  overall  syntax.  This page describes pcp2elastic‐
27       search specific options and configuration  file  differences  with  pm‐
28       rep.conf(5).  pmrep(1) also lists some usage examples of which most are
29       applicable with pcp2elasticsearch as well.
30
31       Only the command line options listed on this page are supported,  other
32       options recognized by pmrep(1) are not supported.
33
34       Options  via environment values (see pmGetOptions(3)) override the cor‐
35       responding built-in default values (if any).   Configuration  file  op‐
36       tions  override the corresponding environment variables (if any).  Com‐
37       mand line options override the corresponding configuration file options
38       (if any).
39

CONFIGURATION FILE

41       pcp2elasticsearch  uses  a  configuration  file with overall syntax de‐
42       scribed in pmrep.conf(5).  The following options are  common  with  pm‐
43       rep.conf:  version,  source,  speclocal, derived, header, globals, sam‐
44       ples, interval, type, type_prefer, ignore_incompat,  names_change,  in‐
45       stances,   live_filter,  rank,  limit_filter,  limit_filter_force,  in‐
46       vert_filter, predicate, omit_flat,  include_labels,  precision,  preci‐
47       sion_force,      count_scale,      count_scale_force,      space_scale,
48       space_scale_force, time_scale, time_scale_force.  The output option  is
49       recognized but ignored for pmrep.conf compatibility.
50
51   pcp2elasticsearch specific options
52       es_server (string)
53           Specify  the  URL  of  Elasticsearch server to send the metrics to.
54           Corresponding command line option is -g.  Default is  http://local
55           host:9200/.
56
57       es_index (string)
58           Specify  the  Elasticsearch  index for metric names.  Corresponding
59           command line option is -x.  Default is pcp.
60
61       es_hostid (string)
62           Specify the Elasticsearch host-id for measurements.   Corresponding
63           command line option is -X.  Defaults to the metrics source host.
64
65       es_search_type (string)
66           Specify  the  Elasticsearch  search  type for measurements.  Corre‐
67           sponding command line option is -p.  Defaults to pcp-metric.
68

OPTIONS

70       The available command line options are:
71
72       -0 precision, --precision-force=precision
73            Like -P but this option will override per-metric specifications.
74
75       -4 action, --names-change=action
76            Specify which action to take on receiving a  metric  names  change
77            event  during  sampling.  These events occur when a PMDA discovers
78            new metrics sometime after starting up, and informs running client
79            tools  like pcp2elasticsearch.  Valid values for action are update
80            (refresh metrics being sampled), ignore (do nothing - the  default
81            behaviour) and abort (exit the program if such an event happens).
82
83       -5, --ignore-unknown
84            Silently ignore any metric name that cannot be resolved.  At least
85            one metric must be found for the tool to start.
86
87       -8 limit, --limit-filter=limit
88            Limit results to instances with values above/below limit.  A posi‐
89            tive  integer  will  include instances with values at or above the
90            limit in reporting.  A negative  integer  will  include  instances
91            with  values  at or below the limit in reporting.  A value of zero
92            performs no limit filtering.  This option will not override possi‐
93            ble per-metric specifications.  See also -J and -N.
94
95       -9 limit, --limit-filter-force=limit
96            Like -8 but this option will override per-metric specifications.
97
98       -a archive, --archive=archive
99            Performance  metric  values  are retrieved from the set of Perfor‐
100            mance Co-Pilot (PCP) archive log files identified by  the  archive
101            argument,  which is a comma-separated list of names, each of which
102            may be the base name of an archive or the name of a directory con‐
103            taining one or more archives.
104
105       -A align, --align=align
106            Force  the initial sample to be aligned on the boundary of a natu‐
107            ral time unit align.  Refer to PCPIntro(1) for a complete descrip‐
108            tion of the syntax for align.
109
110       --archive-folio=folio
111            Read  metric source archives from the PCP archive folio created by
112            tools like pmchart(1) or, less often, manually with mkaf(1).
113
114       -b scale, --space-scale=scale
115            Unit/scale for  space  (byte)  metrics,  possible  values  include
116            bytes, Kbytes, KB, Mbytes, MB, and so forth.  This option will not
117            override possible per-metric specifications.  See  also  pmParseU‐
118            nitsStr(3).
119
120       -B scale, --space-scale-force=scale
121            Like -b but this option will override per-metric specifications.
122
123       -c config, --config=config
124            Specify  the config file or directory to use.  In case config is a
125            directory all files under it ending .conf will be  included.   The
126            default   is   the   first   found  of:  ./pcp2elasticsearch.conf,
127            $HOME/.pcp2elasticsearch.conf,   $HOME/pcp/pcp2elasticsearch.conf,
128            and $PCP_SYSCONF_DIR/pcp2elasticsearch.conf.  For details, see the
129            above section and pmrep.conf(5).
130
131       --container=container
132            Fetch performance metrics from the specified container, either lo‐
133            cal or remote (see -h).
134
135       -C, --check
136            Exit before reporting any values, but after parsing the configura‐
137            tion and metrics and printing possible headers.
138
139       --daemonize
140            Daemonize on startup.
141
142       -e derived, --derived=derived
143            Specify derived performance metrics.  If  derived  starts  with  a
144            slash  (``/'')  or  with a dot (``.'') it will be interpreted as a
145            derived metrics configuration file, otherwise it  will  be  inter‐
146            preted  as  comma-  or  semicolon-separated derived metric expres‐
147            sions.  For details  see  pmLoadDerivedConfig(3)  and  pmRegister‐
148            Derived(3).
149
150       -g URL, --es-host=URL
151            Elasticsearch server URL to send the metrics to.
152
153       -G, --no-globals
154            Do not include global metrics in reporting (see pmrep.conf(5)).
155
156       -h host, --host=host
157            Fetch  performance  metrics from pmcd(1) on host, rather than from
158            the default localhost.
159
160       -H, --no-header
161            Do not print any headers.
162
163       -i instances, --instances=instances
164            Retrieve and report only the specified metric instances.   By  de‐
165            fault all instances, present and future, are reported.
166
167            Refer to pmrep(1) for complete description of this option.
168
169       -I, --ignore-incompat
170            Ignore  incompatible  metrics.   By  default  incompatible metrics
171            (that is, their type is unsupported or they cannot  be  scaled  as
172            requested) will cause pcp2elasticsearch to terminate with an error
173            message.  With this option all incompatible metrics  are  silently
174            omitted  from  reporting.   This may be especially useful when re‐
175            questing non-leaf nodes of the PMNS tree for reporting.
176
177       -j, --live-filter
178            Perform instance live filtering.  This allows capturing all  named
179            instances  even  if  processes are restarted at some point (unlike
180            without live filtering).  Performing live filtering  over  a  huge
181            number  of  instances  will add some internal overhead so a bit of
182            user caution is advised.  See also -n.
183
184       -J rank, --rank=rank
185            Limit results to highest/lowest  ranked  instances  of  set-valued
186            metrics.  A positive integer will include highest valued instances
187            in reporting.  A negative integer will include lowest  valued  in‐
188            stances in reporting.  A value of zero performs no ranking.  Rank‐
189            ing does not imply sorting, see -6.  See also -8.
190
191       -K spec, --spec-local=spec
192            When fetching metrics from a local context (see -L), the -K option
193            may  be used to control the DSO PMDAs that should be made accessi‐
194            ble.  The spec argument conforms to the syntax  described  in  pm‐
195            SpecLocalPMDA(3).  More than one -K option may be used.
196
197       -L, --local-PMDA
198            Use a local context to collect metrics from DSO PMDAs on the local
199            host without PMCD.  See also -K.
200
201       -m, --include-labels
202            Include metric labels in the output.
203
204       -n, --invert-filter
205            Perform ranking before live filtering.  By default  instance  live
206            filtering (when requested, see -j) happens before instance ranking
207            (when requested, see -J).  With this option the logic is  inverted
208            and ranking happens before live filtering.
209
210       -N predicate, --predicate=predicate
211            Specify  a comma-separated list of predicate filter reference met‐
212            rics.  By default ranking (see -J) happens for each  metric  indi‐
213            vidually.  With predicates, ranking is done only for the specified
214            predicate metrics.  When reporting, rest of  the  metrics  sharing
215            the  same  instance domain (see PCPIntro(1)) as the predicate will
216            include only the highest/lowest ranking instances  of  the  corre‐
217            sponding predicate.  Ranking does not imply sorting, see -6.
218
219            So  for  example,  using  proc.memory.rss (resident memory size of
220            process) as the predicate metric together with proc.io.total_bytes
221            and  mem.util.used  as  metrics to be reported, only the processes
222            using most/least (as per -J) memory will be included when  report‐
223            ing  total  bytes  written by processes.  Since mem.util.used is a
224            single-valued metric (thus not sharing the same instance domain as
225            the process related metrics), it will be reported as usual.
226
227       -O origin, --origin=origin
228            When  reporting archived metrics, start reporting at origin within
229            the time window (see -S and -T).  Refer to PCPIntro(1) for a  com‐
230            plete description of the syntax for origin.
231
232       -p type, --es-search-type=type
233            Elasticsearch search type for measurements.
234
235       -P precision, --precision=precision
236            Use  precision for numeric non-integer output values.  The default
237            is to use 3 decimal places (when applicable).   This  option  will
238            not override possible per-metric specifications.
239
240       -q scale, --count-scale=scale
241            Unit/scale  for  count  metrics,  possible  values include count x
242            10^-1, count, count x 10, count x 10^2, and so forth from 10^-8 to
243            10^7.   (These values are currently space-sensitive.)  This option
244            will not override possible per-metric  specifications.   See  also
245            pmParseUnitsStr(3).
246
247       -Q scale, --count-scale-force=scale
248            Like -q but this option will override per-metric specifications.
249
250       -r, --raw
251            Output  raw  metric  values, do not convert cumulative counters to
252            rates.  This option will override possible  per-metric  specifica‐
253            tions.
254
255       -R, --raw-prefer
256            Like  -r  but  this option will not override per-metric specifica‐
257            tions.
258
259       -s samples, --samples=samples
260            The samples argument defines the number of samples to be retrieved
261            and  reported.   If samples is 0 or -s is not specified, pcp2elas‐
262            ticsearch will sample and report continuously (in real time  mode)
263            or  until  the  end  of the set of PCP archives (in archive mode).
264            See also -T.
265
266       -S starttime, --start=starttime
267            When reporting archived metrics, the report will be restricted  to
268            those  records logged at or after starttime.  Refer to PCPIntro(1)
269            for a complete description of the syntax for starttime.
270
271       -t interval, --interval=interval
272            Set the reporting interval to something other than the  default  1
273            second.   The  interval  argument  follows the syntax described in
274            PCPIntro(1), and in the simplest form may be an  unsigned  integer
275            (the implied units in this case are seconds).  See also the -T op‐
276            tion.
277
278       -T endtime, --finish=endtime
279            When reporting archived metrics, the report will be restricted  to
280            those  records  logged before or at endtime.  Refer to PCPIntro(1)
281            for a complete description of the syntax for endtime.
282
283            When used to define  the  runtime  before  pcp2elasticsearch  will
284            exit,  if no samples is given (see -s) then the number of reported
285            samples depends on interval (see -t).  If samples  is  given  then
286            interval  will  be  adjusted  to allow reporting of samples during
287            runtime.  In case all of -T, -s, and -t are given, endtime  deter‐
288            mines the actual time pcp2elasticsearch will run.
289
290       -v, --omit-flat
291            Report only set-valued metrics with instances (e.g. disk.dev.read)
292            and omit single-valued ``flat'' metrics  without  instances  (e.g.
293            kernel.all.sysfork).  See -i and -I.
294
295       -V, --version
296            Display version number and exit.
297
298       -x index, --es-index=index
299            Elasticsearch index for metric names.
300
301       -X hostid, --es-hostid=hostid
302            Elasticsearch hostid for measurements.
303
304       -y scale, --time-scale=scale
305            Unit/scale  for time metrics, possible values include nanosec, ns,
306            microsec, us, millisec, ms, and so forth up to hour, hr.  This op‐
307            tion  will  not  override possible per-metric specifications.  See
308            also pmParseUnitsStr(3).
309
310       -Y scale, --time-scale-force=scale
311            Like -y but this option will override per-metric specifications.
312
313       -?, --help
314            Display usage message and exit.
315

OUTPUT FORMAT

317       First, pcp2elasticsearch sends this type definition to Elasticsearch:
318
319               {
320                   "mappings": {
321                       "pcp-metric": {
322                           "properties": {
323                               "@timestamp": {
324                                   "type": "date"
325                               },
326                               "host-id": {
327                                   "type": "string"
328                               }
329                           }
330                       }
331                   }
332               }
333
334       The actual metrics data document is similar to this example:
335
336               {
337                   "@host-id": "bozo",
338                   "mem": {
339                       "util": {
340                           "used": 4525820,
341                           "free": 3344000
342                       }
343                   },
344                   "@timestamp": 1507637520,
345                   "disk": {
346                       "partitions": {
347                           "@instances": [
348                               {
349                                   "read": 1121,
350                                   "@id": "sda1"
351                               },
352                               {
353                                   "read": 77306,
354                                   "@id": "sda2"
355                               },
356                               {
357                                   "read": 131,
358                                   "@id": "sda3"
359                               }
360                           ]
361                       }
362                   }
363               }
364

FILES

366       pcp2elasticsearch.conf
367            pcp2elasticsearch configuration file (see -c)
368

PCP ENVIRONMENT

370       Environment variables with the prefix PCP_ are used to parameterize the
371       file  and  directory names used by PCP.  On each installation, the file
372       /etc/pcp.conf contains the  local  values  for  these  variables.   The
373       $PCP_CONF  variable may be used to specify an alternative configuration
374       file, as described in pcp.conf(5).
375
376       For environment variables affecting PCP tools, see pmGetOptions(3).
377

SEE ALSO

379       mkaf(1),   PCPIntro(1),   pcp(1),   pcp2graphite(1),   pcp2influxdb(1),
380       pcp2json(1),   pcp2spark(1),  pcp2xlsx(1),  pcp2xml(1),  pcp2zabbix(1),
381       pmcd(1), pminfo(1), pmrep(1), pmGetOptions(3), pmSpecLocalPMDA(3),  pm‐
382       LoadDerivedConfig(3),  pmParseUnitsStr(3),  pmRegisterDerived(3), LOGA‐
383       RCHIVE(5), pcp.conf(5), PMNS(5) and pmrep.conf(5).
384
385
386
387Performance Co-Pilot                  PCP                 PCP2ELASTICSEARCH(1)
Impressum