1PMMGR(1)                    General Commands Manual                   PMMGR(1)
2
3
4

NAME

6       pmmgr - pcp daemon manager
7

SYNOPSIS

9       pmmgr [-v] [-c config-directory] [-p polling-interval] [-l log-file]
10

DESCRIPTION

12       pmmgr manages a collection of PCP daemons for a set of discovered local
13       and remote hosts running  the  Performance  Metrics  Collection  Daemon
14       (PMCD),  according to zero or more configuration directories.  It keeps
15       a matching set of pmie, pmlogger, pmrep and other daemons running,  and
16       their  archives/logs merged/rotated.  It provides an alternative to the
17       default pmlogger_daily and pmie_daily scripts that administer  pmlogger
18       and pmie ``farms'' using cron(3).
19
20       pmmgr  is largely self-configuring and perseveres despite most run-time
21       errors.  pmmgr runs in the foreground  until  interrupted.   When  sig‐
22       naled, it will stop its running daemons before exiting.
23

ALGORITHM

25       Each  poll  interval,  pmmgr  computes a list of possible targets for a
26       pmcd search.  This list is assembled from several configuration  files,
27       and  may  include  explicitly  listed  specifications, hosts discovered
28       through several different mechanisms, and/or individual containers run‐
29       ning within them.  Once the list is assembled, pmmgr attempts to make a
30       brief pmNewContext connection to each target, in order to check for the
31       existence  of an actual running pmcd instance, and to extract a hostid.
32       The hostid is treated as a unique identifier for the instance, so  that
33       redundant  connection  paths  to  the  same server can be filtered out.
34       Once the final list of live pmcd instances is  identified,  along  with
35       their  unique hostids, pmmgr ensures that any requested pcp client dae‐
36       mons are started (or restarted) for them.  If any pmcd instances disap‐
37       pear  from  the  list, its pcp client daemons are stopped.  This entire
38       cycle repeats every poll interval.
39

OPTIONS

41       A description of the command line options specific to pmmgr follows:
42
43       -c   directory adds a given configuration directory  to  pmmgr.   pmmgr
44            can  supervise multiple different configurations at the same time,
45            so this option may be repeated.  Errors in the  configuration  may
46            be  noted to standard error, but pmmgr will fill in missing infor‐
47            mation  with  built-in  defaults.   The   default   directory   is
48            $PCP_SYSCONF_DIR/pmmgr
49
50       -p   polling-interval  sets  the host-discovery polling interval to the
51            given number of seconds.  The default is 60.  Daemons for  a  par‐
52            ticular target host will be restarted no more frequently than this
53            interval.  There may be a  short-lived  thread  inside  pmmgr  for
54            startup and shutdown of each daemon for each target host.
55
56       -l   log-file  redirects  standard  output  and  error to the given log
57            file, which is created anew
58
59       -v   adds more verbose tracing to standard output.
60

CONFIGURATION

62   FILES
63       A pmmgr configuration identifies which hosts should be monitored, which
64       daemons  should  be maintained for them, and what options those daemons
65       should be run with.  pmmgr uses a small number of files in a configura‐
66       tion directory, instead of lines in a single text file.  The individual
67       files carry zero or more lines of 100% pure configuration text, and  no
68       comments.   (If  desired,  a configuration may be commented upon within
69       other files, such as a free-form README.)
70
71       Some of the configuration files are forked into pairs:  per-hostid  and
72       common.   This  permits numerous almost-identical-configuration targets
73       to be managed from the same configuration directory.  For these  files,
74       marked  with  *  below, pmmgr will concatenate a per-hostid file (if it
75       exists) and a common file (if it exists) in order to form the  complete
76       configuration item.
77
78       For  example,  for pmie configuration for target hostid foo, pmmgr will
79       search files named pmie.foo then pmie.  For  single-line  configuration
80       items,  the first file & line found will "win"; for multi-line configu‐
81       ration items, they all "win".
82
83   TARGET SELECTION
84       This set of configuration files identifies where  pmmgr  should  search
85       for pmcd instances, how to uniquely identify them, and where state such
86       as log files should be kept for each.  Ideally, a persistent and unique
87       hostid string is computed for each potential target pmcd from specified
88       metric values.  This hostid is also used as  a  subdirectory  name  for
89       locating daemon data.  The rare empty hostid is mapped to "-".
90
91       hostid-static
92              This  file  contains  one  or  more  lines specifying the static
93              string that should be used as multiple distinct hostids for  the
94              same target pmcd.  Treatment of the distinct hostids may be cus‐
95              tomized using per-hostid configuration files.  Specifying values
96              in  this file overrides the hostid-metrics file specified below.
97              It should be noted that using this option will cause all  target
98              pmcds  to  be  assigned  the same set of hostids.  Thus, this is
99              useful in monitoring single hosts or if each monitored host  has
100              its own configuration directory.
101
102       hostid-metrics
103              This file contains one or more lines of metric specifications in
104              the  format  accepted  by  pmParseMetricSpec.   Metrics  without
105              instance  specifiers  mean  all instances of that metric.  These
106              are used to generate the unique  hostid  string  for  each  pmcd
107              server  that  pmmgr  discovers.   Upon  discovery,  all the met‐
108              rics/instances named are queried,  string  values  fetched,  and
109              normalized/concatenated   into  a  single  hyphenated  printable
110              string.  The default is the single metric  pmcd.hostname,  which
111              is  sufficient  if  all  the  hosts discovered have unique host‐
112              name(2).  If they don't, you should add other pcp metric  speci‐
113              fications to set them apart at your site.  The more you add, the
114              longer the hostid string, but the more  likely  that  accidental
115              duplication is prevented.
116
117       However,  it  may  be  desirable for a hostid to also be persistent, so
118       that if the target host goes offline and later returns, the new  hostid
119       matches  the  previous  one,  because then old and new histories can be
120       joined.  This argues against using metrics whose values vary from  boot
121       to boot.
122
123       Some  candidate  metrics  to  consider: network.interface.hw_addr, net‐
124       work.interface.inet_addr["eth0"],   network.interface.ipv6_addr,   ker‐
125       nel.uname.nodename
126
127       log-directory
128              This  file  contains  the  path of a directory beneath which the
129              per-hostid subdirectories are to be created by pmmgr.  If it  is
130              not  a full path, it is implicitly relative to the configuration
131              directory itself.  The default is $PCP_LOG_DIR/pmmgr/.
132
133       target-host
134              This file contains one or more lines containing pmcd host speci‐
135              fications,  as  described  on  the PCPintro(1) manual page.  The
136              default is to target pmcd at local:.
137
138       target-discovery
139              This file contains one or more lines  containing  specifications
140              for  the  pmDiscoverServices  PMAPI  call, each of which may map
141              onto a fluctuating set of local or remote  pmcd  servers.   Each
142              poll interval, pmmgr will attempt to rerun discovery with all of
143              the given specifications.  Again, it is not a  problem  if  more
144              than  one  specification  matches the same actual pmcd: one con‐
145              firmed access path is arbitrarily selected.  The default  is  to
146              do  no discovery.  Consider including avahi,timeout=5 to rely in
147              pmcd self-announcements on the local network (searching  for  up
148              to    five    seconds    each    time).     Consider   including
149              probe=192.168.1.0/24 to quickly scan the given IP address range.
150
151       subtarget-containers
152              If this file exists, pmmgr will scan each host that is found for
153              running  containers.  For each running container, it will create
154              independent  subtargets  for  running  requested  daemons.   The
155              hostid  string for these subtargets is the host's hostid string,
156              followed by a double-hyphen,  then  the  full  unique  container
157              instance-name string.
158
159       target-threads
160              This  file  contains a limit on the number of concurrent threads
161              that analyze potential target pmcds  for  their  hostids  and/or
162              containers.  The default is a few dozen threads per CPU core, if
163              known.  Set this to zero if  remote  pmcds  should  be  analyzed
164              sequentially.   A  small number of threads is not a good idea if
165              any potential target pmcds  are  unreachable,  since  $PMCD_CON‐
166              NECT_TIMEOUT may be several seconds long each.
167
168       log-subdirectory-gc
169              This  file  may contain a time interval specification as per the
170              PCPintro(1) manual page.  All subdirectories of  the  log-direc‐
171              tory  are  presumed to contain data for pmmgr-monitored servers.
172              Those that have not been modified in at least that long, and not
173              associated  with  a  currently  monitored  target,  are  deleted
174              entirely.  This value should be longer than the longest interval
175              that  pmmgr  normally  recreates  archives (such as due to pmmgr
176              restarts, and  pmlogmerge  intervals).   The  default  value  is
177              90days.
178
179
180   PMLOGGER CONFIGURATION
181       This group of configuration options controls a pmlogger daemon for each
182       host.  This may include generating its configuration, and managing  its
183       archives.
184
185       pmlogger*
186              If  and only if this file exists, pmmgr will maintain a pmlogger
187              daemon for each targeted host.  This file contains one  line  of
188              additional  space-separated  options  for  the  pmlogger daemon.
189              (pmmgr already adds -h, -H, -f, -r, -l, and  perhaps  -c.)   The
190              default  is  to maintain no pmlogger (and no other configuration
191              in this section is processed).
192
193       pmlogger-timefmt*
194              Specify a time format to use in the archive-* name for  pmlogger
195              generated  archives. The default is "%Y%m%d.%H%M%S". Expected to
196              be in strftime(3) format.
197
198       pmlogconf*
199              If and only if this file exists, pmmgr  will  run  pmlogconf  to
200              generate  a  configuration  file for each target pmcd.  The file
201              contains one line of space-separated additional options for  the
202              pmlogconf  program.   pmlogconf's  generated output file will be
203              stored  under  the  log-directory/hostid  subdirectory.   (pmmgr
204              already  adds  -c, -r, and -h.)  The default is no pmlogconf, so
205              instead, the pmlogger file above should probably  contain  a  -c
206              option, to specify a fixed pmlogger configuration.
207
208   ARCHIVE LOG MANAGEMENT
209       Default  pmlogger  configurations can collect tens of megabytes of data
210       per day (possibly split into multiple archives), per target  host.   If
211       your  disk  space is less than infinite, or archive-splitting unwieldy,
212       this should be managed.  In the default,  unmanaged  case,  the  system
213       administrator  is  responsible  for  managing  the individual archive-*
214       files from the per-host logging subdirectories.  pmmgr  offers  several
215       other  options,  each  representing different performance and usability
216       tradeoffs.
217
218   ARCHIVE LOG MANAGEMENT - pmlogmerge
219       This style of archive log management regularly creates a single  merged
220       archive from prior archives for each target host, in effect lopping off
221       old data and appending the new.  A single merged archive can  be  rela‐
222       tively  large (defaults to approximately 100-400 MB per host), and puts
223       a corresponding I/O load on storage,  but  is  most  convenient  for  a
224       detailed  long-timeframe  analysis.   Once  pmlogger  is  restarted, it
225       always creates a new archive, so in the steady state, there will be one
226       merged  archive  of recent history, and one current archive being writ‐
227       ten-to by pmlogger.
228
229       pmlogmerge*
230              If this file exists, pmmgr will run pmlogextract to periodically
231              merge  together  preexisting  log  archives for each target pmcd
232              into a single large one.  Then, the preexisting log archives are
233              deleted  (including  any prior merged ones).  This configuration
234              file may contain a time interval specification as per the PCPin‐
235              tro(1) manual page, representing the period after which pmlogger
236              should be temporarily stopped, and archives merged.   It  repre‐
237              sents  the  maximum  amount of time that the merged archive lags
238              the present time.  The default is 24hours.
239
240       pmlogmerge-granular*
241              If this file also exists, pmmgr will merge only a subset of pre‐
242              existing  log archives into the new one, instead of all of them,
243              so as to approximate a granular, aligned set of merged archives.
244              The  subset  chosen  corresponds  to  the previous time interval
245              specified by the pmlogmerge control file.   The  default  is  no
246              granularity.
247
248       pmlogcheck-corrupt-gc*
249              Before  archives  are considered for merging, they are processed
250              through pmlogcheck to check for  corruption.   In  the  unlikely
251              case  of  a  problem,  such  archives are renamed out of the way
252              (named "corrupt-*"), and retained up to a  limited  time.   This
253              file  specifies  how  long.   If  this  file exists, it the time
254              interval it contains is the maximum age.  The default is 90days.
255              To  store  corrupt  archives  indefinitely,  set this to a large
256              quantity like "99999weeks".
257
258       pmlogmerge-rewrite*
259              If this file exists, pmmgr will run pmlogrewrite  -i  (plus  any
260              other  options listed in this file) on each input archive before
261              merging it.  This will naturally require  more  disk  I/O.   The
262              default is no rewriting.
263
264       pmlogmerge-retain*
265              pmmgr  reduces/deletes  any original-resolution archives after a
266              time period specified by this file,  as  measured  by  the  file
267              mtime.  The period will also be passed to pmlogextract as a neg‐
268              ative parameter to -S.  The default is  14days.   To  store  ar‐
269              chives   indefinitely,   set  this  to  a  large  quantity  like
270              "99999weeks".
271
272       pmlogreduce*
273              If this file exists, then prior to removing archives that expire
274              past the pmlogmerge-retain period, they are processed with pmlo‐
275              greduce to create reduced archives (named  reduced-*).   If  the
276              file  contains  space-separated  options,  they  are passed onto
277              pmlogreduce.   (By  default,  pmlogreduce  down-samples   to   a
278              600-second interval.)
279
280       pmlogreduce-retain*
281              If  this  file  exists, then reduced archives (identified by the
282              reduced-* pattern) are deleted after a time period specified  by
283              this  file, as measured from the file mtime.  Since this time is
284              likely that of the pmlogreduce run,  the  total  retention  time
285              will  be approximately the pmlogmerge-retain time plus the pmlo‐
286              greduce-retain time.  The default is 90days.  To  store  reduced
287              archives  indefinitely,  set  this  to  a  large  quantity  like
288              "99999weeks".
289
290       disk-full-threshold
291              If this file exists, then pmmgr will track the disk space avail‐
292              able  where pmlogger archives are kept.  If that partition fills
293              up past the configured percentage, pmmgr  will  linearly  reduce
294              the duration logs are kept via the disk-full-retention variable.
295
296              Values  must  be greater than zero, and expressed either a value
297              between 0 and 1, or decimal value between 1 and 100.
298
299       disk-full-retention
300              If expressed, this variable scales the rate at  which  logs  are
301              culled  when  disk-full-threshold  has  been surpassed.  A lower
302              percentage will cull logs more quickly (in favour of  preserving
303              disk  space),  while a higher percentage will opt to retain more
304              pcp archives.
305
306            ┌──────────────────────────┬────────────────┬────────────────────────┐
307            │Normalized Full Threshold │ Full Retention │ Final Retention Factor │
308            ├──────────────────────────┼────────────────┼────────────────────────┤
309            │            1             │      0.0       │          0.0           │
310            ├──────────────────────────┼────────────────┼────────────────────────┤
311            │          0.75            │      0.0       │          0.25          │
312            ├──────────────────────────┼────────────────┼────────────────────────┤
313            │           0.5            │      0.0       │          0.5           │
314            ├──────────────────────────┼────────────────┼────────────────────────┤
315            │           0.0            │      0.0       │          1.0           │
316            ├──────────────────────────┼────────────────┼────────────────────────┤
317            │            1             │      0.5       │          0.5           │
318            ├──────────────────────────┼────────────────┼────────────────────────┤
319            │          0.75            │      0.5       │         0.625          │
320            ├──────────────────────────┼────────────────┼────────────────────────┤
321            │           0.5            │      0.5       │          0.75          │
322            ├──────────────────────────┼────────────────┼────────────────────────┤
323            │           0.0            │      0.5       │          1.0           │
324            ├──────────────────────────┼────────────────┼────────────────────────┤
325            │            1             │      1.0       │          1.0           │
326            ├──────────────────────────┼────────────────┼────────────────────────┤
327            │          0.75            │      1.0       │          1.0           │
328            ├──────────────────────────┼────────────────┼────────────────────────┤
329            │           0.5            │      1.0       │          1.0           │
330            ├──────────────────────────┼────────────────┼────────────────────────┤
331            │           0.0            │      1.0       │          1.0           │
332            └──────────────────────────┴────────────────┴────────────────────────┘
333   PMIE CONFIGURATION
334       This group of configuration options controls a  pmie  daemon  for  each
335       host.  This may include generating a custom configuration.
336
337       pmie*  If and only if this file exists, pmmgr will maintain a pmie dae‐
338              mon for each targeted pmcd.  This  file  contains  one  line  of
339              additional  space-separated options for the pmie daemon.  (pmmgr
340              already adds -h, -f, -l, and perhaps -c.)   The  default  is  to
341              maintain  no pmie (and no other configuration in this section is
342              processed).
343
344       pmieconf*
345              If and only if this file exists, pmmgr will run pmieconf to gen‐
346              erate  a configuration file for each target pmcd.  The file con‐
347              tains one line of space-separated  additional  options  for  the
348              pmieconf  program.   pmieconf-  generated  output  file  will be
349              stored  under  the  log-directory/hostid  subdirectory.   (pmmgr
350              already  adds  -F,  -c, and -f.)  The default is no pmieconf, so
351              instead, the pmie  file  above  should  probably  contain  a  -c
352              option, to specify a fixed pmie configuration.
353
354   MONITOR DAEMON MANAGEMENT
355       pmmgr may be used to invoke arbitrary PCP client programs for each tar‐
356       get pmcd.  This can enable automated invocation of reporting or  relay‐
357       ing  tools, such as pmrep, pcp2graphite or pcp2influxdb without needing
358       a specialized system service.
359
360       monitor*
361              If this file exists, then for each line  in  this  file,  a  new
362              background  process  will  be  invoked.   (It is restarted if it
363              exits.)  The line specifies the beginning of  the  command  line
364              (including  the  program name); pmmgr appends a -h HOSTSPEC, and
365              arranges to collect the standard output and standard error  into
366              separate  monitor-NN.out  and monitor-NN.err files under the log
367              directory.  Errors messages in the  latter  are  transcribed  to
368              pmmgr's own logs.
369

FILES

371       $PCP_SYSCONF_DIR/pmmgr/
372                 default configuration directory
373       $PCP_LOG_DIR/pmmgr/
374                 default logging directory
375

PCP ENVIRONMENT

377       Environment  variables with the prefix PCP_ are used to parametrize the
378       file and directory names used by PCP.  On each installation,  the  file
379       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
380       $PCP_CONF variable may be used to specify an alternative  configuration
381       file, as described in pcp.conf(5).
382

SEE ALSO

384       PCPIntro(1),   cron(1),   pmcd(1),  pmlogconf(1),  pmlogger(1),  pmlog‐
385       ger_daily(1),   pmieconf(1),    pmie(1),    pmie_daily(1),    pmrep(1),
386       pcp2graphite(1),   pcp2influxdb(1),   pmlogreduce(1),  pcp.conf(5)  and
387       pcp.env(5).
388
389
390
391Performance Co-Pilot                  PCP                             PMMGR(1)
Impressum