1PMMGR(1) General Commands Manual PMMGR(1)
2
6 pmmgr - PCP daemon manager
7
9 pmmgr [-v?] [-c config-directory] [-p polling-interval] [-l log-file]
10 [-U username]
11
13 pmmgr manages a collection of PCP daemons for a set of discovered local
14 and remote hosts running the Performance Metrics Collection Daemon
15 (PMCD, see pmcd(1)), according to zero or more configuration directo‐
16 ries. It keeps a matching set of pmie(1), pmlogger(1), pmrep(1) and
17 other daemons running, and their archives/logs merged/rotated. It pro‐
18 vides an alternative to the default pmlogger_daily(1) and pmie_daily(1)
19 scripts that administer pmlogger and pmie ``farms'' using cron(3).
20 pmmgr is largely self-configuring and perseveres despite most run-time
22 errors. pmmgr runs in the foreground until interrupted. When sig‐
23 naled, it will stop its running daemons before exiting.
24
26 Each poll interval, pmmgr computes a list of possible targets for a
27 pmcd search. This list is assembled from several configuration files,
28 and may include explicitly listed specifications, hosts discovered
29 through several different mechanisms, and/or individual containers run‐
30 ning within them. Once the list is assembled, pmmgr attempts to make a
31 brief pmNewContext connection to each target, in order to check for the
32 existence of an actual running pmcd instance, and to extract a hostid.
33 The hostid is treated as a unique identifier for the instance, so that
34 redundant connection paths to the same server can be filtered out.
35 Once the final list of live pmcd instances is identified, along with
36 their unique hostids, pmmgr ensures that any requested PCP client dae‐
37 mons are started (or restarted) for them. If any pmcd instances disap‐
38 pear from the list, its PCP client daemons are stopped. This entire
39 cycle repeats every poll interval.
40
42 The available command line options are:
43 -c directory, --config=directory
45 Adds a given configuration directory to pmmgr. pmmgr can super‐
46 vise multiple different configurations at the same time, so this
47 option may be repeated. Errors in the configuration may be noted
48 to standard error, but pmmgr will fill in missing information with
49 built-in defaults. The default directory is
50 $PCP_SYSCONF_DIR/pmmgr
51 -l logfile, --log=logfile
53 Redirect standard output and error to the given logfile, which is
54 created anew.
55 -p interval, --poll=interval
57 Set the host-discovery polling interval to the given number of
58 seconds. The default is 60. Daemons for a particular target host
59 will be restarted no more frequently than this interval. There
60 may be a short-lived thread inside pmmgr for startup and shutdown
61 of each daemon for each target host.
62 -U username, --username=username
64 Switch from the root to user username on startup. The default is
65 pcp.
66 -v Enable verbose mode.
68 -? Display usage message and exit.
70
72 FILES
73 A pmmgr configuration identifies which hosts should be monitored, which
74 daemons should be maintained for them, and what options those daemons
75 should be run with. pmmgr uses a small number of files in a configura‐
76 tion directory, instead of lines in a single text file. The individual
77 files carry zero or more lines of 100% pure configuration text, and no
78 comments. (If desired, a configuration may be commented upon within
79 other files, such as a free-form README.)
80 Some of the configuration files are forked into pairs: per-hostid and
82 common. This permits numerous almost-identical-configuration targets
83 to be managed from the same configuration directory. For these files,
84 marked with * below, pmmgr will concatenate a per-hostid file (if it
85 exists) and a common file (if it exists) in order to form the complete
86 configuration item.
87 For example, for pmie configuration for target hostid foo, pmmgr will
89 search files named pmie.foo then pmie. For single-line configuration
90 items, the first file & line found will "win"; for multi-line configu‐
91 ration items, they all "win".
92 TARGET SELECTION
94 This set of configuration files identifies where pmmgr should search
95 for pmcd instances, how to uniquely identify them, and where state such
96 as log files should be kept for each. Ideally, a persistent and unique
97 hostid string is computed for each potential target pmcd from specified
98 metric values. This hostid is also used as a subdirectory name for
99 locating daemon data. The rare empty hostid is mapped to "-".
100 hostid-static
102 This file contains one or more lines specifying the static
103 string that should be used as multiple distinct hostids for the
104 same target pmcd. Treatment of the distinct hostids may be cus‐
105 tomized using per-hostid configuration files. Specifying values
106 in this file overrides the hostid-metrics file specified below.
107 It should be noted that using this option will cause all target
108 pmcds to be assigned the same set of hostids. Thus, this is
109 useful in monitoring single hosts or if each monitored host has
110 its own configuration directory.
111 hostid-metrics
113 This file contains one or more lines of metric specifications in
114 the format accepted by pmParseMetricSpec. Metrics without
115 instance specifiers mean all instances of that metric. These
116 are used to generate the unique hostid string for each pmcd
117 server that pmmgr discovers. Upon discovery, all the met‐
118 rics/instances named are queried, string values fetched, and
119 normalized/concatenated into a single hyphenated printable
120 string. The default is the single metric pmcd.hostname, which
121 is sufficient if all the hosts discovered have unique hostname.
122 If they don't, you should add other pcp metric specifications to
123 set them apart at your site. The more you add, the longer the
124 hostid string, but the more likely that accidental duplication
125 is prevented.
126 However, it may be desirable for a hostid to also be persistent, so
128 that if the target host goes offline and later returns, the new hostid
129 matches the previous one, because then old and new histories can be
130 joined. This argues against using metrics whose values vary from boot
131 to boot.
132 Some candidate metrics to consider: network.interface.hw_addr, net‐
134 work.interface.inet_addr["eth0"], network.interface.ipv6_addr, ker‐
135 nel.uname.nodename.
136 log-directory
138 This file contains the path of a directory beneath which the
139 per-hostid subdirectories are to be created by pmmgr. If it is
140 not a full path, it is implicitly relative to the configuration
141 directory itself. The default is $PCP_LOG_DIR/pmmgr/.
142 target-host
144 This file contains one or more lines containing pmcd host speci‐
145 fications, as described on the PCPIntro(1) manual page. The
146 default is to target pmcd at local:.
147 target-discovery
149 This file contains one or more lines containing specifications
150 for the pmDiscoverServices PMAPI call, each of which may map
151 onto a fluctuating set of local or remote pmcd servers. Each
152 poll interval, pmmgr will attempt to rerun discovery with all of
153 the given specifications. Again, it is not a problem if more
154 than one specification matches the same actual pmcd: one con‐
155 firmed access path is arbitrarily selected. The default is to
156 do no discovery. Consider including avahi,timeout=5 to rely in
157 pmcd self-announcements on the local network (searching for up
158 to five seconds each time). Consider including
159 probe=192.168.1.0/24 to quickly scan the given IP address range.
160 subtarget-containers
162 If this file exists, pmmgr will scan each host that is found for
163 running containers. For each running container, it will create
164 independent subtargets for running requested daemons. The
165 hostid string for these subtargets is the host's hostid string,
166 followed by a double-hyphen, then the full unique container
167 instance-name string.
168 target-threads
170 This file contains a limit on the number of concurrent threads
171 that analyze potential target pmcds for their hostids and/or
172 containers. The default is a few dozen threads per CPU core, if
173 known. Set this to zero if remote pmcds should be analyzed
174 sequentially. A small number of threads is not a good idea if
175 any potential target pmcds are unreachable, since $PMCD_CON‐
176 NECT_TIMEOUT may be several seconds long each.
177 log-subdirectory-gc
179 This file may contain a time interval specification as per the
180 PCPIntro(1) manual page. All subdirectories of the log-direc‐
181 tory are presumed to contain data for pmmgr-monitored servers.
182 Those that have not been modified in at least that long, and not
183 associated with a currently monitored target, are deleted
184 entirely. This value should be longer than the longest interval
185 that pmmgr normally recreates archives (such as due to pmmgr
186 restarts, and pmlogmerge intervals). The default value is
187 90days.
188 PMLOGGER CONFIGURATION
190 This group of configuration options controls a pmlogger daemon for each
191 host. This may include generating its configuration, and managing its
192 archives.
193 pmlogger*
195 If and only if this file exists, pmmgr will maintain a pmlogger
196 daemon for each targeted host. This file contains one line of
197 additional space-separated options for the pmlogger daemon.
198 (pmmgr already adds -h, -H, -f, -r, -l, and perhaps -c.) The
199 default is to maintain no pmlogger (and no other configuration
200 in this section is processed).
201 pmlogger-timefmt*
203 Specify a time format to use in the archive-* name for pmlogger
204 generated archives. The default is %Y%m%d.%H%M%S. Expected to
205 be in strftime(3) format.
206 pmlogconf*
208 If and only if this file exists, pmmgr will run pmlogconf to
209 generate a configuration file for each target pmcd. The file
210 contains one line of space-separated additional options for the
211 pmlogconf program. pmlogconf's generated output file will be
212 stored under the log-directory/hostid subdirectory. (pmmgr
213 already adds -c, -r, and -h.) The default is no pmlogconf, so
214 instead, the pmlogger file above should probably contain a -c
215 option, to specify a fixed pmlogger configuration.
216 ARCHIVE LOG MANAGEMENT
218 Default pmlogger configurations can collect tens of megabytes of data
219 per day (possibly split into multiple archives), per target host. If
220 your disk space is less than infinite, or archive-splitting unwieldy,
221 this should be managed. In the default, unmanaged case, the system
222 administrator is responsible for managing the individual archive-*
223 files from the per-host logging subdirectories. pmmgr offers several
224 other options, each representing different performance and usability
225 tradeoffs.
226 ARCHIVE LOG MANAGEMENT - pmlogmerge
228 This style of archive log management regularly creates a single merged
229 archive from prior archives for each target host, in effect lopping off
230 old data and appending the new. A single merged archive can be rela‐
231 tively large (defaults to approximately 100-400 MB per host), and puts
232 a corresponding I/O load on storage, but is most convenient for a
233 detailed long-timeframe analysis. Once pmlogger is restarted, it
234 always creates a new archive, so in the steady state, there will be one
235 merged archive of recent history, and one current archive being writ‐
236 ten-to by pmlogger.
237 pmlogmerge*
239 If this file exists, pmmgr will run pmlogextract to periodically
240 merge together preexisting log archives for each target pmcd
241 into a single large one. Then, the preexisting log archives are
242 deleted (including any prior merged ones). This configuration
243 file may contain a time interval specification as per the PCPIn‐
244 tro(1) manual page, representing the period after which pmlogger
245 should be temporarily stopped, and archives merged. It repre‐
246 sents the maximum amount of time that the merged archive lags
247 the present time. The default is 24hours.
248 pmlogmerge-granular*
250 If this file also exists, pmmgr will merge only a subset of pre‐
251 existing log archives into the new one, instead of all of them,
252 so as to approximate a granular, aligned set of merged archives.
253 The subset chosen corresponds to the previous time interval
254 specified by the pmlogmerge control file. The default is no
255 granularity.
256 pmlogcheck-corrupt-gc*
258 Before archives are considered for merging, they are processed
259 through pmlogcheck to check for corruption. In the unlikely
260 case of a problem, such archives are renamed out of the way
261 (named "corrupt-*"), and retained up to a limited time. This
262 file specifies how long. If this file exists, it the time
263 interval it contains is the maximum age. The default is 90days.
264 To store corrupt archives indefinitely, set this to a large
265 quantity like 99999weeks.
266 pmlogmerge-rewrite*
268 If this file exists, pmmgr will run pmlogrewrite -i (plus any
269 other options listed in this file) on each input archive before
270 merging it. This will naturally require more disk I/O. The
271 default is no rewriting.
272 pmlogmerge-retain*
274 pmmgr reduces/deletes any original-resolution archives after a
275 time period specified by this file, as measured by the file
276 mtime. The period will also be passed to pmlogextract as a neg‐
277 ative parameter to -S. The default is 14days. To store ar‐
278 chives indefinitely, set this to a large quantity like
279 99999weeks.
280 pmlogreduce*
282 If this file exists, then prior to removing archives that expire
283 past the pmlogmerge-retain period, they are processed with pmlo‐
284 greduce to create reduced archives (named reduced-*). If the
285 file contains space-separated options, they are passed onto
286 pmlogreduce. (By default, pmlogreduce down-samples to a
287 600-second interval.)
288 pmlogreduce-retain*
290 If this file exists, then reduced archives (identified by the
291 reduced-* pattern) are deleted after a time period specified by
292 this file, as measured from the file mtime. Since this time is
293 likely that of the pmlogreduce run, the total retention time
294 will be approximately the pmlogmerge-retain time plus the pmlo‐
295 greduce-retain time. The default is 90days. To store reduced
296 archives indefinitely, set this to a large quantity like
297 99999weeks.
298 disk-full-threshold
300 If this file exists, then pmmgr will track the disk space avail‐
301 able where pmlogger archives are kept. If that partition fills
302 up past the configured percentage, pmmgr will linearly reduce
303 the duration logs are kept via the disk-full-retention variable.
304 Values must be greater than zero, and expressed either a value between
306 0 and 1, or decimal value between 1 and 100.
307 disk-full-retention
309 If expressed, this variable scales the rate at which logs are
310 culled when disk-full-threshold has been surpassed. A lower
311 percentage will cull logs more quickly (in favour of preserving
312 disk space), while a higher percentage will opt to retain more
313 pcp archives.
314 ┌──────────────────────────┬────────────────┬────────────────────────┐
316 │Normalized Full Threshold │ Full Retention │ Final Retention Factor │
317 ├──────────────────────────┼────────────────┼────────────────────────┤
318 │ 1 │ 0.0 │ 0.0 │
319 ├──────────────────────────┼────────────────┼────────────────────────┤
320 │ 0.75 │ 0.0 │ 0.25 │
321 ├──────────────────────────┼────────────────┼────────────────────────┤
322 │ 0.5 │ 0.0 │ 0.5 │
323 ├──────────────────────────┼────────────────┼────────────────────────┤
324 │ 0.0 │ 0.0 │ 1.0 │
325 ├──────────────────────────┼────────────────┼────────────────────────┤
326 │ 1 │ 0.5 │ 0.5 │
327 ├──────────────────────────┼────────────────┼────────────────────────┤
328 │ 0.75 │ 0.5 │ 0.625 │
329 ├──────────────────────────┼────────────────┼────────────────────────┤
330 │ 0.5 │ 0.5 │ 0.75 │
331 ├──────────────────────────┼────────────────┼────────────────────────┤
332 │ 0.0 │ 0.5 │ 1.0 │
333 ├──────────────────────────┼────────────────┼────────────────────────┤
334 │ 1 │ 1.0 │ 1.0 │
335 ├──────────────────────────┼────────────────┼────────────────────────┤
336 │ 0.75 │ 1.0 │ 1.0 │
337 ├──────────────────────────┼────────────────┼────────────────────────┤
338 │ 0.5 │ 1.0 │ 1.0 │
339 ├──────────────────────────┼────────────────┼────────────────────────┤
340 │ 0.0 │ 1.0 │ 1.0 │
341 └──────────────────────────┴────────────────┴────────────────────────┘
342 PMIE CONFIGURATION
343 This group of configuration options controls a pmie daemon for each
344 host. This may include generating a custom configuration.
345 pmie* If and only if this file exists, pmmgr will maintain a pmie dae‐
347 mon for each targeted pmcd. This file contains one line of
348 additional space-separated options for the pmie daemon. (pmmgr
349 already adds -h, -f, -l, and perhaps -c.) The default is to
350 maintain no pmie (and no other configuration in this section is
351 processed).
352 pmieconf*
354 If and only if this file exists, pmmgr will run pmieconf to gen‐
355 erate a configuration file for each target pmcd. The file con‐
356 tains one line of space-separated additional options for the
357 pmieconf program. pmieconf- generated output file will be
358 stored under the log-directory/hostid subdirectory. (pmmgr
359 already adds -F, -c, and -f.) The default is no pmieconf, so
360 instead, the pmie file above should probably contain a -c
361 option, to specify a fixed pmie configuration.
362 MONITOR DAEMON MANAGEMENT
364 pmmgr may be used to invoke arbitrary PCP client programs for each tar‐
365 get pmcd(1). This can enable automated invocation of reporting or
366 relaying tools, such as pmrep(1), pcp2graphite(1) or pcp2influxdb(1)
367 without needing a specialized system service.
368 monitor*
370 If this file exists, then for each line in this file, a new
371 background process will be invoked. (It is restarted if it
372 exits.) The line specifies the beginning of the command line
373 (including the program name); pmmgr appends a -h HOSTSPEC, and
374 arranges to collect the standard output and standard error into
375 separate monitor-NN.out and monitor-NN.err files under the log
376 directory. Errors messages in the latter are transcribed to
377 pmmgr's own logs.
378
380 $PCP_SYSCONF_DIR/pmmgr/
381 default configuration directory
382 $PCP_LOG_DIR/pmmgr/
384 default logging directory
385
387 Environment variables with the prefix PCP_ are used to parameterize the
388 file and directory names used by PCP. On each installation, the file
389 /etc/pcp.conf contains the local values for these variables. The
390 $PCP_CONF variable may be used to specify an alternative configuration
391 file, as described in pcp.conf(5).
392
394 PCPIntro(1), cron(1), pmcd(1), pmlogconf(1), pmlogger(1), pmlog‐
395 ger_daily(1), pmlogreduce(1), pmie(1), pmieconf(1), pmie_daily(1),
396 pmrep(1), pcp2graphite(1), pcp2influxdb(1), pcp.conf(5) and pcp.env(5).
397Performance Co-Pilot PCP PMMGR(1)