1PMLOGGER_CHECK(1) General Commands Manual PMLOGGER_CHECK(1)
2
6 pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot
7 archive log files
8
10 $PCP_BINADM_DIR/pmlogger_check [-CNpqsTV?] [-c control] [-l logfile]
11 $PCP_BINADM_DIR/pmlogger_daily [-EfKMNoprRV?] [-c control] [-k time]
12 [-l logfile] [-m addresses] [-s size] [-t want] [-x time] [-X program]
13 [-Y regex]
14
16 These shell scripts and associated control files may be used to create
17 a customized regime of administration and management for Performance
18 Co-Pilot (see PCPIntro(1)) archive log files.
19 pmlogger_check may be run at any time of the day and is intended to
21 check that a desired set of pmlogger(1) processes are running. If not,
22 it (re-)starts any missing logger processes.
23 pmlogger_daily is intended to be run once per day, preferably in the
25 early morning, as soon after midnight as practicable. Its task is to
26 aggregate, rotate and perform general housekeeping one or more sets of
27 PCP archives.
28 To accommodate the evolution of PMDAs and changes in production logging
30 environments, pmlogger_daily is integrated with pmlogrewrite(1) to al‐
31 low optional and automatic rewriting of archives before merging. If
32 there are global rewriting rules to be applied across all archives men‐
33 tioned in the control file(s), then create the directory
34 $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting
35 rules in this directory. For rewriting rules that are specific to only
36 one family of archives, use the directory name from the control file(s)
37 - i.e. the fourth field - and create a file, or a directory, or a sym‐
38 bolic link named pmlogrewrite within this directory and place the re‐
39 quired rewriting rule(s) in the pmlogrewrite file or in files within
40 the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting
41 rules from the archive directory if they exist, else rewriting rules
42 from $PCP_SYSCONF_DIR/pmlogrewrite if that directory exists, else no
43 rewriting is attempted.
44 As an alternate mechanism, if the file $PCP_LOG_DIR/pmlogger/.Nee‐
46 dRewrite exists when pmlogger_daily starts then this is treated the
47 same as specifying -R on the command line and $PCP_LOG_DIR/pmlog‐
48 ger/.NeedRewrite will be removed once all the rewriting has been done.
49
51 -c control, --control=control
52 Both pmlogger_check and pmlogger_daily are controlled by PCP log‐
53 ger control file(s) that specifies the pmlogger instances to be
54 managed. The default control file is $PCP_PMLOGGERCONTROL_PATH,
55 but an alternate may be specified using the -c option. If the di‐
56 rectory $PCP_PMLOGGERCONTROL_PATH.d (or control.d from the -c op‐
57 tion) exists, then the contents of any additional control files
58 therein will be appended to the main control file (which must ex‐
59 ist).
60 -C This option causes pmlogger_check to query the system service run‐
62 level information for pmlogger, and use that to determine whether
63 to start processes or not.
64 -E, --expunge
66 This option causes pmlogger_daily to pass the -E flag to pmlog‐
67 ger_merge in order to expunge metrics with metadata inconsisten‐
68 cies and continue rather than fail. This is intended for auto‐
69 mated daily log rotation where it is highly desirable for unat‐
70 tended daily archive merging, rewriting and compression to suc‐
71 ceed. For further details, see pmlogger_merge(1) and description
72 for the -x flag in pmlogextract(1).
73 -f, --force
75 This option causes pmlogger_daily to forces action. Using this
76 option in production is not recommended.
77 -k time, --discard=time
79 After some period, old PCP archives are discarded. time is a time
80 specification in the syntax of find-filter(1), so DD[:HH[:MM]].
81 The optional HH (hours) and MM (minutes) parts are 0 if not speci‐
82 fied. By default the time is 14:0:0 or 14 days, but may be
83 changed using this option.
84 Some special values are recognized for the time, namely 0 to keep
86 no archives beyond the the ones being currently written by pmlog‐
87 ger(1), and forever or never to prevent any archives being dis‐
88 carded.
89 The time can also be set using the $PCP_CULLAFTER variable, set in
91 either the environment or in a control file. If both $PCP_CUL‐
92 LAFTER and -k specify different values for time then the environ‐
93 ment variable value is used and a warning is issued. I.e., if
94 $PCP_CULLAFTER is set in the control file, it overrides -k given
95 on the command line.
96 Note that the semantics of time are that it is measured from the
98 time of last modification of each archive, and not from the origi‐
99 nal archive creation date. This has subtle implications for com‐
100 pression (see below) - the compression process results in the cre‐
101 ation of new archive files which have new modification times. In
102 this case, the time period (re)starts from the time of compres‐
103 sion.
104 -K When this option is specified for pmlogger_daily then only the
106 compression tasks are attempted, so no pmlogger rotation, no
107 culling, no rewriting, etc. When -K is used and a period of 0 is
108 in effect (from -x on the command line or $PCP_COMPRESSAFTER in
109 the environment or via the control file) this is intended for en‐
110 vironments where compression of archives is desired before the
111 scheduled daily processing happens. To achieve this, once pmlog‐
112 ger_check has completed regular processing, it calls pmlog‐
113 ger_daily with just the -K option. Provided $PCP_COMPRESSAFTER is
114 set to 0 along with any other required compression options to
115 match the scheduled invocation of pmlogger_daily, then this will
116 compress all volumes except the ones being currently written by
117 pmlogger(1). If $PCP_COMPRESSAFTER is set to a value greater than
118 zero, then manually running pmlogger_daily with the -x option may
119 be used to compress volumes that are younger than the $PCP_COM‐
120 PRESSAFTER time. This may be used to reclaim filesystem space by
121 compressing volumes earlier than they would have otherwise been
122 compressed. Note that since the default value of $PCP_COMPRES‐
123 SAFTER is 0 days, the -x option has no effect unless the control
124 file has been edited and $PCP_COMPRESSAFTER has been set to a
125 value greater than 0.
126 -l file, --logfile=file
128 In order to ensure that mail is not unintentionally sent when
129 these scripts are run from cron(8) diagnostics are always sent to
130 log files. By default, this file is $PCP_LOG_DIR/pmlogger/pmlog‐
131 ger_check.log or $PCP_LOG_DIR/pmlogger/pmlogger_daily.log but this
132 can be changed using the -l option. If this log file already ex‐
133 ists when the script starts, it will be renamed with a .prev suf‐
134 fix (overwriting any log file saved earlier) before diagnostics
135 are generated to the log file. The -l and -t options cannot be
136 used together.
137 -m addresses, --mail=addresses
139 Use of this option causes pmlogger_daily to construct a summary of
140 the ``notices'' file entries which were generated in the last 24
141 hours, and e-mail that summary to the set of space-separated ad‐
142 dresses. This daily summary is stored in the file
143 $PCP_LOG_DIR/NOTICES.daily, which will be empty when no new ``no‐
144 tices'' entries were made in the previous 24 hour period.
145 -M This option may be used to disable archive merging (or renaming)
147 and rewriting (-M implies -r). This is most useful in cases where
148 the archives are being incrementally copied to a remote reposi‐
149 tory, e.g. using rsync(1). Merging, renaming and rewriting all
150 risk an increase in the synchronization load, especially immedi‐
151 ately after pmlogger_daily has run, so -M may be useful in these
152 cases.
153 -N, --showme
155 This option enables a ``show me'' mode, where the programs actions
156 are echoed, but not executed, in the style of ``make -n''. Using
157 -N in conjunction with -V maximizes the diagnostic capabilities
158 for debugging.
159 -o By default all possible archives will be merged. This option re‐
161 instates the old behaviour in which only yesterday's archives will
162 be considered as merge candidates. In the special case where only
163 a single input archive needs to be merged, pmlogmv(1) is used to
164 rename the archive, otherwise pmlogger_merge(1) is used to merge
165 all of the archives for a single host and a single day into a new
166 PCP archive and the individual archives are removed.
167 -p, --skip-primary
169 If this option is specified for pmlogger_check then any line from
170 the control files for the primary pmlogger will be ignored. This
171 option is intended for environments where some system daemon, like
172 systemd(1), is responsible for controlling (starting, stopping,
173 restarting, etc.) the primary pmlogger.
174 -p If this option is specified for pmlogger_daily then the status of
176 the daily processing is polled and if the daily pmlogger(1) rota‐
177 tion, culling, rewriting, compressing, etc. has not been done in
178 the last 24 hours then it is done now. The intent is to have pm‐
179 logger_daily called regularly with the -p option (at 30 mins past
180 the hour, every hour in the default cron(8) set up) to ensure
181 daily processing happens as soon as possible if it was missed at
182 the regularly scheduled time (which is 00:10 by default), e.g. if
183 the system was down or suspended at that time. With this option
184 pmlogger_daily simply exits if the previous day's processing has
185 already been done. The -K and -p options to pmlogger_daily are
186 mutually exclusive.
187 -q, --quick
189 If this option is specified for pmlogger_check then the script
190 will ``quickstart'' avoiding any optional processing like file
191 compression.
192 -r, --norewrite
194 This command line option acts as an override and prevents all ar‐
195 chive rewriting with pmlogrewrite(1) independent of the presence
196 of any rewriting rule files or directories.
197 -R, --rewriteall
199 Sometimes PMDA changes require all archives to be rewritten, not
200 just the ones involved in any current merging. This is required
201 for example after a PCP upgrade where a new version of an existing
202 PMDA has revised metadata. The -R command line forces this uni‐
203 versal-style of rewriting. The -R option to pmlogger_daily is mu‐
204 tually exclusive with both the -r and -M options.
205 -s size, --rotate=size
207 If the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is larger than
208 20480 bytes, pmlogger_daily will rename the file with a ``.old''
209 suffix, and start a new ``notices'' file. The rotate threshold
210 may be changed from 20480 to size bytes using the -s option.
211 -s, --stop
213 Use of this option provides the reverse pmlogger_check functional‐
214 ity, allowing the set of pmlogger processes to be cleanly shut‐
215 down.
216 -t period
218 To assist with debugging or diagnosing intermittent failures the
219 -t option may be used. This will turn on very verbose tracing
220 (-VV) and capture the trace output in a file named
221 $PCP_LOG_DIR/pmlogger/daily.datestamp.trace, where datestamp is
222 the time pmlogger_daily was run in the format YYYYMMDD.HH.MM. In
223 addition, the period argument will ensure that trace files created
224 with -t will be kept for period days and then discarded.
225 -T, --terse
227 This option to pmlogger_check produces less verbose output than
228 the default. This is most suitable for a pmlogger ``farm'' where
229 many instances of pmlogger are expected to be running.
230 -V, --verbose
232 The output from the cron execution of the scripts may be extended
233 using the -V option to the scripts which will enable verbose trac‐
234 ing of their activity. By default the scripts generate no output
235 unless some error or warning condition is encountered. Using -N
236 in conjunction with -V maximizes the diagnostic capabilities for
237 debugging.
238 -x time, --compress-after=time
240 Archive data files can optionally be compressed after some period
241 to conserve disk space. This is particularly useful for large
242 numbers of pmlogger processes under the control of pmlogger_check.
243 time is a time specification in the syntax of find-filter(1), so
245 DD[:HH[:MM]]. The optional HH (hours) and MM (minutes) parts are
246 0 if not specified.
247 Some special values are recognized for the time, namely 0 to apply
249 compression as soon as possible, and forever or never to prevent
250 any compression being done.
251 If transparent_decompress is enabled when libpcp was built (can be
253 checked with the pmconfig(1) -L option), then the default behav‐
254 iour is compression ``as soon as possible''. Otherwise the de‐
255 fault behaviour is to not compress files (which matches the his‐
256 torical default behaviour in earlier PCP releases).
257 The time can also be set using the $PCP_COMPRESSAFTER variable,
259 set in either the environment or in a control file. If both
260 $PCP_COMPRESSAFTER and -x specify different values for time then
261 the environment variable value is used and a warning is issued.
262 For important other detailed notes concerning volume compression,
263 see the -K and -k options (above).
264 -X program, --compressor=program
266 This option specifies the program to use for compression - by de‐
267 fault this is xz(1). The environment variable $PCP_COMPRESS may
268 be used as an alternative mechanism to define program. If both
269 $PCP_COMPRESS and -X specify different compression programs then
270 the environment variable value is used and a warning is issued.
271 -Y regex, --regex=regex
273 This option allows a regular expression to be specified causing
274 files in the set of files matched for compression to be omitted -
275 this allows only the data file to be compressed, and also prevents
276 the program from attempting to compress it more than once. The
277 default regex is ".(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" - such
278 files are filtered using the -v option to egrep(1). The environ‐
279 ment variable $PCP_COMPRESSREGEX may be used as an alternative
280 mechanism to define regex. If both $PCP_COMPRESSREGEX and -Y
281 specify different values for regex then the environment variable
282 value is used and a warning is issued.
283 -?, --help
285 Display usage message and exit.
286
288 Warning: The $PCP_PMLOGGERCONTROL_PATH file and files within the
289 $PCP_PMLOGGERCONTROL_PATH.d directory must not be writable by any user
290 other than root.
291 The control file(s) should be customized according to the following
293 rules that define for the current version (1.1) of the control file
294 format.
295 1. Lines beginning with a ``#'' are comments. A special case is lines
297 beginning ``#!#''; these are control lines for a pmlogger(1) that
298 has been stopped using pmlogctl(1).
299 2. Lines beginning with a ``$'' are assumed to be assignments to envi‐
300 ronment variables in the style of sh(1), and all text following the
301 ``$'' will be eval'ed by the script reading the control file, and
302 the corresponding variable exported into the environment. This is
303 particularly useful to set and export variables into the environ‐
304 ment of the administrative scripts, e.g.
305 $ PMCD_CONNECT_TIMEOUT=20
306 3. There must be a version line in the initial control file of the
307 form:
308 $ version=1.1
309 4. There should be one line in the control file(s) for each pmlogger
310 instance of the form:
311 host y|n y|n directory args
313 5. Fields within a line of the control file(s) are usually separated
315 by one or more spaces or tabs (although refer to the description of
316 the directory field for some important exceptions).
317 6. The first field is the name of the host that is the source of the
318 performance metrics for this pmlogger instance.
319 7. The second field indicates if this is a primary pmlogger instance
320 (y) or not (n). Since the primary logger must run on the local
321 host, and there may be at most one primary logger for a particular
322 host, this field can be y for at most one pmlogger instance, in
323 which case the host name must be the name of the local host.
324 8. The third field indicates if this pmlogger instance needs to be
325 started under the control of pmsocks(1) to connect to a pmcd
326 through a firewall (y or n).
327 9. The fourth field is a directory name. All files associated with
328 this pmlogger instance will be created in this directory, and this
329 will be the current directory for the execution of any programs re‐
330 quired in the maintenance of those archives. A useful convention
331 is that primary logger archives for the local host with hostname
332 myhost are maintained in the directory $PCP_ARCHIVE_DIR/myhost
333 (this is where the default pmlogger start-up script in
334 $PCP_RC_DIR/pcp will create the archives), while archives for the
335 remote host mumble are maintained in $PCP_ARCHIVE_DIR/mumble.
336 10. The directory field may contain embedded shell syntax that will be
337 evaluated by sh(1) to produce the real directory name to be used.
338 The allowed constructs are:
339 • Any text (including white space) enclosed with $( and ).
340 • Any text (including white space) enclosed with ` and ` (back
341 quotes).
342 • Any text (including white space) enclosed with " and " (double
343 quotes).
344 • Any word containing a $ (assumed to introduce an environment
345 variable name).
346 11. All other fields are interpreted as arguments to be passed to pm‐
347 logger(1). Most typically this would be the -c option.
348 The following sample control lines specify a primary logger on the lo‐
350 cal host (bozo), and non-primary loggers to collect and log performance
351 metrics from the hosts wobbly and boing.
352 $version=1.1
354 bozo y n $PCP_ARCHIVE_DIR/bozo -c config.default
355 wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
356 boing n n $PCP_ARCHIVE_DIR/boing -c ./pmlogger.config
357 Typical crontab(5) entries for periodic execution of pmlogger_daily and
359 pmlogger_check are given in $PCP_SYSCONF_DIR/pmlogger/crontab (unless
360 installed by default in /etc/cron.d already) and shown below.
361 # daily processing of archive logs
363 14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
364 # every 30 minutes, check pmlogger instances are running
365 25,55 * * * * $PCP_BINADM_DIR/pmlogger_check
366 When using systemd(1) on Linux, no crontab entries are needed as the
368 timer mechanism provided by systemd is used instead.
369
371 $PCP_PMLOGGERCONTROL_PATH
372 the PCP logger control file. For a new installation this file
373 contains no pmlogger(1) control lines (the real control files are
374 all in the $PCP_PMLOGGERCONTROL_PATH.d directory), but this file
375 is still processed to support any legacy configurations therein
376 from earlier PCP releases.
377 Warning: this file must not be writable by any user other than
378 root.
379 $PCP_PMLOGGERCONTROL_PATH.d
381 optional directory containing additional PCP logger control files,
382 typically one per host
383 Warning: the files herein must not be writable by any user other
384 than root.
385 $PCP_SYSCONF_DIR/pmlogger/crontab
387 sample crontab for automated script execution by $PCP_USER (or
388 root). Exists only if the platform does not support the
389 /etc/cron.d mechanism.
390 $PCP_VAR_DIR/config/pmlogger/config.default
392 default pmlogger configuration file location for the local primary
393 logger, typically generated automatically by pmlogconf(1).
394 $PCP_ARCHIVE_DIR/<hostname>
396 default location for archives of performance information collected
397 from the host hostname
398 $PCP_ARCHIVE_DIR/<hostname>/lock
400 transient lock file to guarantee mutual exclusion during pmlogger
401 administration for the host hostname - if present, can be safely
402 removed if neither pmlogger_daily nor pmlogger_check are running
403 $PCP_ARCHIVE_DIR/<hostname>/Latest
405 PCP archive folio created by mkaf(1) for the most recently
406 launched archive containing performance metrics from the host
407 hostname
408 $PCP_LOG_DIR/NOTICES
410 PCP ``notices'' file used by pmie(1) and friends
411 $PCP_LOG_DIR/pmlogger/pmlogger_check.log
413 if the previous execution of pmlogger_check produced any output it
414 is saved here. The normal case is no output in which case the
415 file does not exist.
416 $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
418 if the previous execution of pmlogger_daily produced any output it
419 is saved here. The normal case is no output in which case the
420 file does not exist.
421 $PCP_ARCHIVE_DIR/<hostname>/SaveLogs
423 if this directory exists, then the log file from the -l argument
424 of a newly launched pmlogger(1) for hostname will be linked into
425 this directory with the name archive.log where archive is the
426 basename of the associated pmlogger(1) PCP archive files. This
427 allows the log file to be inspected at a later time, even if sev‐
428 eral pmlogger(1) instances for hostname have been launched in the
429 interim. Because the cron-driven PCP archive management scripts
430 run under the uid of the user ``pcp'', $PCP_ARCHIVE_DIR/host‐
431 name/SaveLogs typically needs to be owned by the user ``pcp''.
432 $PCP_LOG_DIR/pmlogger/.NeedRewrite
434 if this file exists, then this is treated as equivalent to using
435 -R on the command line and the file will be removed once all
436 rewriting has been done.
437
439 Environment variables with the prefix PCP_ are used to parameterize the
440 file and directory names used by PCP. On each installation, the file
441 /etc/pcp.conf contains the local values for these variables. The
442 $PCP_CONF variable may be used to specify an alternative configuration
443 file, as described in pcp.conf(5).
444 The default behaviour, when pmlogger(1) configuration comes from pmlog‐
446 conf(1), is to regenerate the configuration file and check for changes
447 whenever pmlogger(1) is started from pmlogger_check. If the PMDA con‐
448 figuration is stable, this is not necessary, and setting $PMLOG‐
449 GER_CHECK_SKIP_LOGCONF to yes disables the regeneration and checking.
450
452 Earlier versions of pmlogger_daily used find(1) to locate files for
453 compressing or culling and the -k and -x options took only integer val‐
454 ues to mean ``days''. The semantics of this was quite loose given that
455 find(1) offers different precision and semantics across platforms.
456 The current implementation of pmlogger_daily uses find-filter(1) which
458 provides high precision intervals and semantics that are relative to
459 the time of execution and are consistent across platforms.
460
462 egrep(1), find-filter(1), PCPIntro(1), pmconfig(1), pmlc(1), pmlog‐
463 conf(1), pmlogctl(1), pmlogger(1), pmlogger_daily_report(1), pmlog‐
464 ger_merge(1), pmlogextract(1), pmlogmv(1), pmlogrewrite(1), pmsocks(1),
465 systemd(1), xz(1) and cron(8).
466Performance Co-Pilot PCP PMLOGGER_CHECK(1)