1PMMGR(1) General Commands Manual PMMGR(1)
2
6 pmmgr - pcp daemon manager
7
9 pmmgr [-v] [-c config-directory] [-p polling-interval] [-l log-file]
10
12 pmmgr manages a collection of PCP daemons for a set of discovered local
13 and remote hosts running the Performance Metrics Collection Daemon
14 (PMCD), according to zero or more configuration directories. It keeps
15 a matching set of pmie, pmlogger, pmrep and other daemons running, and
16 their archives/logs merged/rotated. It provides an alternative to the
17 default pmlogger_daily and pmie_daily scripts that administer pmlogger
18 and pmie ``farms'' using cron(3).
19 pmmgr is largely self-configuring and perseveres despite most run-time
21 errors. pmmgr runs in the foreground until interrupted. When sig‐
22 naled, it will stop its running daemons before exiting.
23
25 Each poll interval, pmmgr computes a list of possible targets for a
26 pmcd search. This list is assembled from several configuration files,
27 and may include explicitly listed specifications, hosts discovered
28 through several different mechanisms, and/or individual containers run‐
29 ning within them. Once the list is assembled, pmmgr attempts to make a
30 brief pmNewContext connection to each target, in order to check for the
31 existence of an actual running pmcd instance, and to extract a hostid.
32 The hostid is treated as a unique identifier for the instance, so that
33 redundant connection paths to the same server can be filtered out.
34 Once the final list of live pmcd instances is identified, along with
35 their unique hostids, pmmgr ensures that any requested pcp client dae‐
36 mons are started (or restarted) for them. If any pmcd instances disap‐
37 pear from the list, its pcp client daemons are stopped. This entire
38 cycle repeats every poll interval.
39
41 A description of the command line options specific to pmmgr follows:
42 -c directory adds a given configuration directory to pmmgr. pmmgr
44 can supervise multiple different configurations at the same time,
45 so this option may be repeated. Errors in the configuration may
46 be noted to standard error, but pmmgr will fill in missing infor‐
47 mation with built-in defaults. The default directory is
48 $PCP_SYSCONF_DIR/pmmgr
49 -p polling-interval sets the host-discovery polling interval to the
51 given number of seconds. The default is 60. Daemons for a par‐
52 ticular target host will be restarted no more frequently than this
53 interval. There may be a short-lived thread inside pmmgr for
54 startup and shutdown of each daemon for each target host.
55 -l log-file redirects standard output and error to the given log
57 file, which is created anew
58 -v adds more verbose tracing to standard output.
60
62 FILES
63 A pmmgr configuration identifies which hosts should be monitored, which
64 daemons should be maintained for them, and what options those daemons
65 should be run with. pmmgr uses a small number of files in a configura‐
66 tion directory, instead of lines in a single text file. The individual
67 files carry zero or more lines of 100% pure configuration text, and no
68 comments. (If desired, a configuration may be commented upon within
69 other files, such as a free-form README.)
70 Some of the configuration files are forked into pairs: per-hostid and
72 common. This permits numerous almost-identical-configuration targets
73 to be managed from the same configuration directory. For these files,
74 marked with * below, pmmgr will concatenate a per-hostid file (if it
75 exists) and a common file (if it exists) in order to form the complete
76 configuration item.
77 For example, for pmie configuration for target hostid foo, pmmgr will
79 search files named pmie.foo then pmie. For single-line configuration
80 items, the first file & line found will "win"; for multi-line configu‐
81 ration items, they all "win".
82 TARGET SELECTION
84 This set of configuration files identifies where pmmgr should search
85 for pmcd instances, how to uniquely identify them, and where state such
86 as log files should be kept for each. Ideally, a persistent and unique
87 hostid string is computed for each potential target pmcd from specified
88 metric values. This hostid is also used as a subdirectory name for
89 locating daemon data. The rare empty hostid is mapped to "-".
90 hostid-static
92 This file contains one or more lines specifying the static
93 string that should be used as multiple distinct hostids for the
94 same target pmcd. Treatment of the distinct hostids may be cus‐
95 tomized using per-hostid configuration files. Specifying values
96 in this file overrides the hostid-metrics file specified below.
97 It should be noted that using this option will cause all target
98 pmcds to be assigned the same set of hostids. Thus, this is
99 useful in monitoring single hosts or if each monitored host has
100 its own configuration directory.
101 hostid-metrics
103 This file contains one or more lines of metric specifications in
104 the format accepted by pmParseMetricSpec. Metrics without
105 instance specifiers mean all instances of that metric. These
106 are used to generate the unique hostid string for each pmcd
107 server that pmmgr discovers. Upon discovery, all the met‐
108 rics/instances named are queried, string values fetched, and
109 normalized/concatenated into a single hyphenated printable
110 string. The default is the single metric pmcd.hostname, which
111 is sufficient if all the hosts discovered have unique host‐
112 name(2). If they don't, you should add other pcp metric speci‐
113 fications to set them apart at your site. The more you add, the
114 longer the hostid string, but the more likely that accidental
115 duplication is prevented.
116 However, it may be desirable for a hostid to also be persistent, so
118 that if the target host goes offline and later returns, the new hostid
119 matches the previous one, because then old and new histories can be
120 joined. This argues against using metrics whose values vary from boot
121 to boot.
122 Some candidate metrics to consider: network.interface.hw_addr, net‐
124 work.interface.inet_addr["eth0"], network.interface.ipv6_addr, ker‐
125 nel.uname.nodename
126 log-directory
128 This file contains the path of a directory beneath which the
129 per-hostid subdirectories are to be created by pmmgr. If it is
130 not a full path, it is implicitly relative to the configuration
131 directory itself. The default is $PCP_LOG_DIR/pmmgr/.
132 target-host
134 This file contains one or more lines containing pmcd host speci‐
135 fications, as described on the PCPintro(1) manual page. The
136 default is to target pmcd at local:.
137 target-discovery
139 This file contains one or more lines containing specifications
140 for the pmDiscoverServices PMAPI call, each of which may map
141 onto a fluctuating set of local or remote pmcd servers. Each
142 poll interval, pmmgr will attempt to rerun discovery with all of
143 the given specifications. Again, it is not a problem if more
144 than one specification matches the same actual pmcd: one con‐
145 firmed access path is arbitrarily selected. The default is to
146 do no discovery. Consider including avahi,timeout=5 to rely in
147 pmcd self-announcements on the local network (searching for up
148 to five seconds each time). Consider including
149 probe=192.168.1.0/24 to quickly scan the given IP address range.
150 subtarget-containers
152 If this file exists, pmmgr will scan each host that is found for
153 running containers. For each running container, it will create
154 independent subtargets for running requested daemons. The
155 hostid string for these subtargets is the host's hostid string,
156 followed by a double-hyphen, then the full unique container
157 instance-name string.
158 target-threads
160 This file contains a limit on the number of concurrent threads
161 that analyze potential target pmcds for their hostids and/or
162 containers. The default is a few dozen threads per CPU core, if
163 known. Set this to zero if remote pmcds should be analyzed
164 sequentially. A small number of threads is not a good idea if
165 any potential target pmcds are unreachable, since $PMCD_CON‐
166 NECT_TIMEOUT may be several seconds long each.
167 log-subdirectory-gc
169 This file may contain a time interval specification as per the
170 PCPintro(1) manual page. All subdirectories of the log-direc‐
171 tory are presumed to contain data for pmmgr-monitored servers.
172 Those that have not been modified in at least that long, and not
173 associated with a currently monitored target, are deleted
174 entirely. This value should be longer than the longest interval
175 that pmmgr normally recreates archives (such as due to pmmgr
176 restarts, and pmlogmerge intervals). The default value is
177 90days.
178 PMLOGGER CONFIGURATION
181 This group of configuration options controls a pmlogger daemon for each
182 host. This may include generating its configuration, and managing its
183 archives.
184 pmlogger*
186 If and only if this file exists, pmmgr will maintain a pmlogger
187 daemon for each targeted host. This file contains one line of
188 additional space-separated options for the pmlogger daemon.
189 (pmmgr already adds -h, -H, -f, -r, -l, and perhaps -c.) The
190 default is to maintain no pmlogger (and no other configuration
191 in this section is processed).
192 pmlogger-timefmt*
194 Specify a time format to use in the archive-* name for pmlogger
195 generated archives. The default is "%Y%m%d.%H%M%S". Expected to
196 be in strftime(3) format.
197 pmlogconf*
199 If and only if this file exists, pmmgr will run pmlogconf to
200 generate a configuration file for each target pmcd. The file
201 contains one line of space-separated additional options for the
202 pmlogconf program. pmlogconf's generated output file will be
203 stored under the log-directory/hostid subdirectory. (pmmgr
204 already adds -c, -r, and -h.) The default is no pmlogconf, so
205 instead, the pmlogger file above should probably contain a -c
206 option, to specify a fixed pmlogger configuration.
207 ARCHIVE LOG MANAGEMENT
209 Default pmlogger configurations can collect tens of megabytes of data
210 per day (possibly split into multiple archives), per target host. If
211 your disk space is less than infinite, or archive-splitting unwieldy,
212 this should be managed. In the default, unmanaged case, the system
213 administrator is responsible for managing the individual archive-*
214 files from the per-host logging subdirectories. pmmgr offers several
215 other options, each representing different performance and usability
216 tradeoffs.
217 ARCHIVE LOG MANAGEMENT - pmlogmerge
219 This style of archive log management regularly creates a single merged
220 archive from prior archives for each target host, in effect lopping off
221 old data and appending the new. A single merged archive can be rela‐
222 tively large (defaults to approximately 100-400 MB per host), and puts
223 a corresponding I/O load on storage, but is most convenient for a
224 detailed long-timeframe analysis. Once pmlogger is restarted, it
225 always creates a new archive, so in the steady state, there will be one
226 merged archive of recent history, and one current archive being writ‐
227 ten-to by pmlogger.
228 pmlogmerge*
230 If this file exists, pmmgr will run pmlogextract to periodically
231 merge together preexisting log archives for each target pmcd
232 into a single large one. Then, the preexisting log archives are
233 deleted (including any prior merged ones). This configuration
234 file may contain a time interval specification as per the PCPin‐
235 tro(1) manual page, representing the period after which pmlogger
236 should be temporarily stopped, and archives merged. It repre‐
237 sents the maximum amount of time that the merged archive lags
238 the present time. The default is 24hours.
239 pmlogmerge-granular*
241 If this file also exists, pmmgr will merge only a subset of pre‐
242 existing log archives into the new one, instead of all of them,
243 so as to approximate a granular, aligned set of merged archives.
244 The subset chosen corresponds to the previous time interval
245 specified by the pmlogmerge control file. The default is no
246 granularity.
247 pmlogcheck-corrupt-gc*
249 Before archives are considered for merging, they are processed
250 through pmlogcheck to check for corruption. In the unlikely
251 case of a problem, such archives are renamed out of the way
252 (named "corrupt-*"), and retained up to a limited time. This
253 file specifies how long. If this file exists, it the time
254 interval it contains is the maximum age. The default is 90days.
255 To store corrupt archives indefinitely, set this to a large
256 quantity like "99999weeks".
257 pmlogmerge-rewrite*
259 If this file exists, pmmgr will run pmlogrewrite -i (plus any
260 other options listed in this file) on each input archive before
261 merging it. This will naturally require more disk I/O. The
262 default is no rewriting.
263 pmlogmerge-retain*
265 pmmgr reduces/deletes any original-resolution archives after a
266 time period specified by this file, as measured by the file
267 mtime. The period will also be passed to pmlogextract as a neg‐
268 ative parameter to -S. The default is 14days. To store ar‐
269 chives indefinitely, set this to a large quantity like
270 "99999weeks".
271 pmlogreduce*
273 If this file exists, then prior to removing archives that expire
274 past the pmlogmerge-retain period, they are processed with pmlo‐
275 greduce to create reduced archives (named reduced-*). If the
276 file contains space-separated options, they are passed onto
277 pmlogreduce. (By default, pmlogreduce down-samples to a
278 600-second interval.)
279 pmlogreduce-retain*
281 If this file exists, then reduced archives (identified by the
282 reduced-* pattern) are deleted after a time period specified by
283 this file, as measured from the file mtime. Since this time is
284 likely that of the pmlogreduce run, the total retention time
285 will be approximately the pmlogmerge-retain time plus the pmlo‐
286 greduce-retain time. The default is 90days. To store reduced
287 archives indefinitely, set this to a large quantity like
288 "99999weeks".
289 disk-full-threshold
291 If this file exists, then pmmgr will track the disk space avail‐
292 able where pmlogger archives are kept. If that partition fills
293 up past the configured percentage, pmmgr will linearly reduce
294 the duration logs are kept via the disk-full-retention variable.
295 Values must be greater than zero, and expressed either a value
297 between 0 and 1, or decimal value between 1 and 100.
298 disk-full-retention
300 If expressed, this variable scales the rate at which logs are
301 culled when disk-full-threshold has been surpassed. A lower
302 percentage will cull logs more quickly (in favour of preserving
303 disk space), while a higher percentage will opt to retain more
304 pcp archives.
305 ┌──────────────────────────┬────────────────┬────────────────────────┐
307 │Normalized Full Threshold │ Full Retention │ Final Retention Factor │
308 ├──────────────────────────┼────────────────┼────────────────────────┤
309 │ 1 │ 0.0 │ 0.0 │
310 ├──────────────────────────┼────────────────┼────────────────────────┤
311 │ 0.75 │ 0.0 │ 0.25 │
312 ├──────────────────────────┼────────────────┼────────────────────────┤
313 │ 0.5 │ 0.0 │ 0.5 │
314 ├──────────────────────────┼────────────────┼────────────────────────┤
315 │ 0.0 │ 0.0 │ 1.0 │
316 ├──────────────────────────┼────────────────┼────────────────────────┤
317 │ 1 │ 0.5 │ 0.5 │
318 ├──────────────────────────┼────────────────┼────────────────────────┤
319 │ 0.75 │ 0.5 │ 0.625 │
320 ├──────────────────────────┼────────────────┼────────────────────────┤
321 │ 0.5 │ 0.5 │ 0.75 │
322 ├──────────────────────────┼────────────────┼────────────────────────┤
323 │ 0.0 │ 0.5 │ 1.0 │
324 ├──────────────────────────┼────────────────┼────────────────────────┤
325 │ 1 │ 1.0 │ 1.0 │
326 ├──────────────────────────┼────────────────┼────────────────────────┤
327 │ 0.75 │ 1.0 │ 1.0 │
328 ├──────────────────────────┼────────────────┼────────────────────────┤
329 │ 0.5 │ 1.0 │ 1.0 │
330 ├──────────────────────────┼────────────────┼────────────────────────┤
331 │ 0.0 │ 1.0 │ 1.0 │
332 └──────────────────────────┴────────────────┴────────────────────────┘
333 PMIE CONFIGURATION
334 This group of configuration options controls a pmie daemon for each
335 host. This may include generating a custom configuration.
336 pmie* If and only if this file exists, pmmgr will maintain a pmie dae‐
338 mon for each targeted pmcd. This file contains one line of
339 additional space-separated options for the pmie daemon. (pmmgr
340 already adds -h, -f, -l, and perhaps -c.) The default is to
341 maintain no pmie (and no other configuration in this section is
342 processed).
343 pmieconf*
345 If and only if this file exists, pmmgr will run pmieconf to gen‐
346 erate a configuration file for each target pmcd. The file con‐
347 tains one line of space-separated additional options for the
348 pmieconf program. pmieconf- generated output file will be
349 stored under the log-directory/hostid subdirectory. (pmmgr
350 already adds -F, -c, and -f.) The default is no pmieconf, so
351 instead, the pmie file above should probably contain a -c
352 option, to specify a fixed pmie configuration.
353 MONITOR DAEMON MANAGEMENT
355 pmmgr may be used to invoke arbitrary PCP client programs for each tar‐
356 get pmcd. This can enable automated invocation of reporting or relay‐
357 ing tools, such as pmrep, pcp2graphite or pcp2influxdb without needing
358 a specialized system service.
359 monitor*
361 If this file exists, then for each line in this file, a new
362 background process will be invoked. (It is restarted if it
363 exits.) The line specifies the beginning of the command line
364 (including the program name); pmmgr appends a -h HOSTSPEC, and
365 arranges to collect the standard output and standard error into
366 separate monitor-NN.out and monitor-NN.err files under the log
367 directory. Errors messages in the latter are transcribed to
368 pmmgr's own logs.
369
371 $PCP_SYSCONF_DIR/pmmgr/
372 default configuration directory
373 $PCP_LOG_DIR/pmmgr/
374 default logging directory
375
377 Environment variables with the prefix PCP_ are used to parametrize the
378 file and directory names used by PCP. On each installation, the file
379 /etc/pcp.conf contains the local values for these variables. The
380 $PCP_CONF variable may be used to specify an alternative configuration
381 file, as described in pcp.conf(5).
382
384 PCPIntro(1), cron(1), pmcd(1), pmlogconf(1), pmlogger(1), pmlog‐
385 ger_daily(1), pmieconf(1), pmie(1), pmie_daily(1), pmrep(1),
386 pcp2graphite(1), pcp2influxdb(1), pmlogreduce(1), pcp.conf(5) and
387 pcp.env(5).
388Performance Co-Pilot PCP PMMGR(1)