1PMLOGGER_CHECK(1) General Commands Manual PMLOGGER_CHECK(1)
2
6 pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot
7 archive log files
8
10 $PCP_BINADM_DIR/pmlogger_check [-CNPpqsTV?] [-c control] [-l logfile]
11 $PCP_BINADM_DIR/pmlogger_daily [-EfKMNoprRV?] [-c control] [-k time]
12 [-l logfile] [-m addresses] [-s size] [-t want] [-x time] [-X program]
13 [-Y regex]
14
16 These shell scripts and associated control files may be used to create
17 a customized regime of administration and management for Performance
18 Co-Pilot (see PCPIntro(1)) archive log files.
19 pmlogger_check may be run at any time of the day and is intended to
21 check that a desired set of pmlogger(1) processes are running. If not,
22 it (re-)starts any missing logger processes.
23 pmlogger_daily is intended to be run once per day, preferably in the
25 early morning, as soon after midnight as practicable. Its task is to
26 aggregate, rotate and perform general housekeeping one or more sets of
27 PCP archives.
28 To accommodate the evolution of PMDAs and changes in production logging
30 environments, pmlogger_daily is integrated with pmlogrewrite(1) to al‐
31 low optional and automatic rewriting of archives before merging. If
32 there are global rewriting rules to be applied across all archives men‐
33 tioned in the control file(s), then create the directory
34 $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting
35 rules in this directory. For rewriting rules that are specific to only
36 one family of archives, use the directory name from the control file(s)
37 - i.e. the fourth field - and create a file, or a directory, or a sym‐
38 bolic link named pmlogrewrite within this directory and place the re‐
39 quired rewriting rule(s) in the pmlogrewrite file or in files within
40 the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting
41 rules from the archive directory if they exist, else rewriting rules
42 from $PCP_SYSCONF_DIR/pmlogrewrite if that directory exists, else no
43 rewriting is attempted.
44 As an alternate mechanism, if the file $PCP_LOG_DIR/pmlogger/.Nee‐
46 dRewrite exists when pmlogger_daily starts then this is treated the
47 same as specifying -R on the command line and $PCP_LOG_DIR/pmlog‐
48 ger/.NeedRewrite will be removed once all the rewriting has been done.
49
51 -c control, --control=control
52 Both pmlogger_check and pmlogger_daily are controlled by PCP log‐
53 ger control file(s) that specifies the pmlogger instances to be
54 managed. The default control file is $PCP_PMLOGGERCONTROL_PATH,
55 but an alternate may be specified using the -c option. If the di‐
56 rectory $PCP_PMLOGGERCONTROL_PATH.d (or control.d from the -c op‐
57 tion) exists, then the contents of any additional control files
58 therein will be appended to the main control file (which must ex‐
59 ist).
60 -C This option causes pmlogger_check to query the system service run‐
62 level information for pmlogger, and use that to determine whether
63 to start processes or not.
64 -E, --expunge
66 This option causes pmlogger_daily to pass the -E flag to pmlog‐
67 ger_merge in order to expunge metrics with metadata inconsisten‐
68 cies and continue rather than fail. This is intended for auto‐
69 mated daily log rotation where it is highly desirable for unat‐
70 tended daily archive merging, rewriting and compression to suc‐
71 ceed. For further details, see pmlogger_merge(1) and description
72 for the -x flag in pmlogextract(1).
73 -f, --force
75 This option causes pmlogger_daily to forces action. Using this
76 option in production is not recommended.
77 -k time, --discard=time
79 After some period, old PCP archives are discarded. time is a time
80 specification in the syntax of find-filter(1), so DD[:HH[:MM]].
81 The optional HH (hours) and MM (minutes) parts are 0 if not speci‐
82 fied. By default the time is 14:0:0 or 14 days, but may be
83 changed using this option.
84 Some special values are recognized for the time, namely 0 to keep
86 no archives beyond the the ones being currently written by pmlog‐
87 ger(1), and forever or never to prevent any archives being dis‐
88 carded.
89 The time can also be set using the $PCP_CULLAFTER variable, set in
91 either the environment or in a control file. If both $PCP_CUL‐
92 LAFTER and -k specify different values for time then the environ‐
93 ment variable value is used and a warning is issued. I.e., if
94 $PCP_CULLAFTER is set in the control file, it overrides -k given
95 on the command line.
96 Note that the semantics of time are that it is measured from the
98 time of last modification of each archive, and not from the origi‐
99 nal archive creation date. This has subtle implications for com‐
100 pression (see below) - the compression process results in the cre‐
101 ation of new archive files which have new modification times. In
102 this case, the time period (re)starts from the time of compres‐
103 sion.
104 -K When this option is specified for pmlogger_daily then only the
106 compression tasks are attempted, so no pmlogger rotation, no
107 culling, no rewriting, etc. When -K is used and a period of 0 is
108 in effect (from -x on the command line or $PCP_COMPRESSAFTER in
109 the environment or via the control file) this is intended for en‐
110 vironments where compression of archives is desired before the
111 scheduled daily processing happens. To achieve this, once pmlog‐
112 ger_check has completed regular processing, it calls pmlog‐
113 ger_daily with just the -K option. Provided $PCP_COMPRESSAFTER is
114 set to 0 along with any other required compression options to
115 match the scheduled invocation of pmlogger_daily, then this will
116 compress all volumes except the ones being currently written by
117 pmlogger(1). If $PCP_COMPRESSAFTER is set to a value greater than
118 zero, then manually running pmlogger_daily with the -x option may
119 be used to compress volumes that are younger than the $PCP_COM‐
120 PRESSAFTER time. This may be used to reclaim filesystem space by
121 compressing volumes earlier than they would have otherwise been
122 compressed. Note that since the default value of $PCP_COMPRES‐
123 SAFTER is 0 days, the -x option has no effect unless the control
124 file has been edited and $PCP_COMPRESSAFTER has been set to a
125 value greater than 0.
126 -l file, --logfile=file
128 In order to ensure that mail is not unintentionally sent when
129 these scripts are run from cron(8) diagnostics are always sent to
130 log files. By default, this file is $PCP_LOG_DIR/pmlogger/pmlog‐
131 ger_check.log or $PCP_LOG_DIR/pmlogger/pmlogger_daily.log but this
132 can be changed using the -l option. If this log file already ex‐
133 ists when the script starts, it will be renamed with a .prev suf‐
134 fix (overwriting any log file saved earlier) before diagnostics
135 are generated to the log file. The -l and -t options cannot be
136 used together.
137 -m addresses, --mail=addresses
139 Use of this option causes pmlogger_daily to construct a summary of
140 the ``notices'' file entries which were generated in the last 24
141 hours, and e-mail that summary to the set of space-separated ad‐
142 dresses. This daily summary is stored in the file
143 $PCP_LOG_DIR/NOTICES.daily, which will be empty when no new ``no‐
144 tices'' entries were made in the previous 24 hour period.
145 -M This option may be used to disable archive merging (or renaming)
147 and rewriting (-M implies -r). This is most useful in cases where
148 the archives are being incrementally copied to a remote reposi‐
149 tory, e.g. using rsync(1). Merging, renaming and rewriting all
150 risk an increase in the synchronization load, especially immedi‐
151 ately after pmlogger_daily has run, so -M may be useful in these
152 cases.
153 -N, --showme
155 This option enables a ``show me'' mode, where the programs actions
156 are echoed, but not executed, in the style of ``make -n''. Using
157 -N in conjunction with -V maximizes the diagnostic capabilities
158 for debugging.
159 -o By default all possible archives will be merged. This option re‐
161 instates the old behaviour in which only yesterday's archives will
162 be considered as merge candidates. In the special case where only
163 a single input archive needs to be merged, pmlogmv(1) is used to
164 rename the archive, otherwise pmlogger_merge(1) is used to merge
165 all of the archives for a single host and a single day into a new
166 PCP archive and the individual archives are removed.
167 -p, --skip-primary
169 If this option is specified for pmlogger_check then any line from
170 the control files for the primary pmlogger will be ignored. This
171 option is intended for environments where some system daemon, like
172 systemd(1), is responsible for controlling (starting, stopping,
173 restarting, etc.) the primary pmlogger.
174 -P, --only-primary
176 If this option is specified for pmlogger_check then only the pri‐
177 mary logger entry in the control files will be processed. This is
178 the logical opposite of the -p option described above and is in‐
179 tended for use by RC scripts that start only the primary logger,
180 such as the pmlogger.service unit. The -p and -P options to pm‐
181 logger_check are mutually exclusive.
182 -p If this option is specified for pmlogger_daily then the status of
184 the daily processing is polled and if the daily pmlogger(1) rota‐
185 tion, culling, rewriting, compressing, etc. has not been done in
186 the last 24 hours then it is done now. The intent is to have pm‐
187 logger_daily called regularly with the -p option (at 30 mins past
188 the hour, every hour in the default cron(8) set up) to ensure
189 daily processing happens as soon as possible if it was missed at
190 the regularly scheduled time (which is 00:10 by default), e.g. if
191 the system was down or suspended at that time. With this option
192 pmlogger_daily simply exits if the previous day's processing has
193 already been done. Note that this option is not used on platforms
194 supporting systemd(1) because the pmlogger_daily.timer service
195 unit specifies a timer setting with Persistent=true. The -K and
196 -p options to pmlogger_daily are mutually exclusive.
197 -q, --quick
199 If this option is specified for pmlogger_check then the script
200 will ``quickstart'' avoiding any optional processing like file
201 compression.
202 -r, --norewrite
204 This command line option acts as an override and prevents all ar‐
205 chive rewriting with pmlogrewrite(1) independent of the presence
206 of any rewriting rule files or directories.
207 -R, --rewriteall
209 Sometimes PMDA changes require all archives to be rewritten, not
210 just the ones involved in any current merging. This is required
211 for example after a PCP upgrade where a new version of an existing
212 PMDA has revised metadata. The -R command line forces this uni‐
213 versal-style of rewriting. The -R option to pmlogger_daily is mu‐
214 tually exclusive with both the -r and -M options.
215 -s size, --rotate=size
217 If the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is larger than
218 20480 bytes, pmlogger_daily will rename the file with a ``.old''
219 suffix, and start a new ``notices'' file. The rotate threshold
220 may be changed from 20480 to size bytes using the -s option.
221 -s, --stop
223 Use of this option provides the reverse pmlogger_check functional‐
224 ity, allowing the set of pmlogger processes to be cleanly shut‐
225 down.
226 -t period
228 To assist with debugging or diagnosing intermittent failures the
229 -t option may be used. This will turn on very verbose tracing
230 (-VV) and capture the trace output in a file named
231 $PCP_LOG_DIR/pmlogger/daily.datestamp.trace, where datestamp is
232 the time pmlogger_daily was run in the format YYYYMMDD.HH.MM. In
233 addition, the period argument will ensure that trace files created
234 with -t will be kept for period days and then discarded.
235 -T, --terse
237 This option to pmlogger_check produces less verbose output than
238 the default. This is most suitable for a pmlogger ``farm'' where
239 many instances of pmlogger are expected to be running.
240 -V, --verbose
242 The output from the cron execution of the scripts may be extended
243 using the -V option to the scripts which will enable verbose trac‐
244 ing of their activity. By default the scripts generate no output
245 unless some error or warning condition is encountered. Using -N
246 in conjunction with -V maximizes the diagnostic capabilities for
247 debugging.
248 -x time, --compress-after=time
250 Archive data files can optionally be compressed after some period
251 to conserve disk space. This is particularly useful for large
252 numbers of pmlogger processes under the control of pmlogger_check.
253 time is a time specification in the syntax of find-filter(1), so
255 DD[:HH[:MM]]. The optional HH (hours) and MM (minutes) parts are
256 0 if not specified.
257 Some special values are recognized for the time, namely 0 to apply
259 compression as soon as possible, and forever or never to prevent
260 any compression being done.
261 If transparent_decompress is enabled when libpcp was built (can be
263 checked with the pmconfig(1) -L option), then the default behav‐
264 iour is compression ``as soon as possible''. Otherwise the de‐
265 fault behaviour is to not compress files (which matches the his‐
266 torical default behaviour in earlier PCP releases).
267 The time can also be set using the $PCP_COMPRESSAFTER variable,
269 set in either the environment or in a control file. If both
270 $PCP_COMPRESSAFTER and -x specify different values for time then
271 the environment variable value is used and a warning is issued.
272 For important other detailed notes concerning volume compression,
273 see the -K and -k options (above).
274 -X program, --compressor=program
276 This option specifies the program to use for compression - by de‐
277 fault this is xz(1). The environment variable $PCP_COMPRESS may
278 be used as an alternative mechanism to define program. If both
279 $PCP_COMPRESS and -X specify different compression programs then
280 the environment variable value is used and a warning is issued.
281 -Y regex, --regex=regex
283 This option allows a regular expression to be specified causing
284 files in the set of files matched for compression to be omitted -
285 this allows only the data file to be compressed, and also prevents
286 the program from attempting to compress it more than once. The
287 default regex is ".(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" - such
288 files are filtered using the -v option to egrep(1). The environ‐
289 ment variable $PCP_COMPRESSREGEX may be used as an alternative
290 mechanism to define regex. If both $PCP_COMPRESSREGEX and -Y
291 specify different values for regex then the environment variable
292 value is used and a warning is issued.
293 -?, --help
295 Display usage message and exit.
296
298 Warning: The $PCP_PMLOGGERCONTROL_PATH file and files within the
299 $PCP_PMLOGGERCONTROL_PATH.d directory must not be writable by any user
300 other than root.
301 The control file(s) should be customized according to the following
303 rules that define for the current version (1.1) of the control file
304 format.
305 1. Lines beginning with a ``#'' are comments. A special case is lines
307 beginning ``#!#''; these are control lines for a pmlogger(1) that
308 has been stopped using pmlogctl(1).
309 2. Lines beginning with a ``$'' are assumed to be assignments to envi‐
310 ronment variables in the style of sh(1), and all text following the
311 ``$'' will be eval'ed by the script reading the control file, and
312 the corresponding variable exported into the environment. This is
313 particularly useful to set and export variables into the environ‐
314 ment of the administrative scripts, e.g.
315 $ PMCD_CONNECT_TIMEOUT=20
316 3. There must be a version line in the initial control file of the
317 form:
318 $ version=1.1
319 4. There should be one line in the control file(s) for each pmlogger
320 instance of the form:
321 host y|n y|n directory args
323 5. Fields within a line of the control file(s) are usually separated
325 by one or more spaces or tabs (although refer to the description of
326 the directory field for some important exceptions).
327 6. The first field is the name of the host that is the source of the
328 performance metrics for this pmlogger instance.
329 7. The second field indicates if this is a primary pmlogger instance
330 (y) or not (n). Since the primary logger must run on the local
331 host, and there may be at most one primary logger for a particular
332 host, this field can be y for at most one pmlogger instance, in
333 which case the host name must be the name of the local host.
334 8. The third field indicates if this pmlogger instance needs to be
335 started under the control of pmsocks(1) to connect to a pmcd
336 through a firewall (y or n).
337 9. The fourth field is a directory name. All files associated with
338 this pmlogger instance will be created in this directory, and this
339 will be the current directory for the execution of any programs re‐
340 quired in the maintenance of those archives. A useful convention
341 is that primary logger archives for the local host with hostname
342 myhost are maintained in the directory $PCP_ARCHIVE_DIR/myhost
343 (this is where the default pmlogger start-up script in
344 $PCP_RC_DIR/pcp will create the archives), while archives for the
345 remote host mumble are maintained in $PCP_ARCHIVE_DIR/mumble.
346 10. The directory field may contain embedded shell syntax that will be
347 evaluated by sh(1) to produce the real directory name to be used.
348 The allowed constructs are:
349 • Any text (including white space) enclosed with $( and ).
350 • Any text (including white space) enclosed with ` and ` (back
351 quotes).
352 • Any text (including white space) enclosed with " and " (double
353 quotes).
354 • Any word containing a $ (assumed to introduce an environment
355 variable name).
356 11. All other fields are interpreted as arguments to be passed to pm‐
357 logger(1). Most typically this would be the -c option.
358 The following sample control lines specify a primary logger on the lo‐
360 cal host (bozo), and non-primary loggers to collect and log performance
361 metrics from the hosts wobbly and boing.
362 $version=1.1
364 bozo y n $PCP_ARCHIVE_DIR/bozo -c config.default
365 wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
366 boing n n $PCP_ARCHIVE_DIR/boing -c ./pmlogger.config
367 Typical crontab(5) entries for periodic execution of pmlogger_daily and
369 pmlogger_check are given in $PCP_SYSCONF_DIR/pmlogger/crontab (unless
370 installed by default in /etc/cron.d already) and shown below.
371 # daily processing of archive logs
373 14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
374 # every 30 minutes, check pmlogger instances are running
375 25,55 * * * * $PCP_BINADM_DIR/pmlogger_check
376 When using systemd(1) on Linux, no crontab entries are needed as the
378 timer mechanism provided by systemd is used instead.
379
381 $PCP_PMLOGGERCONTROL_PATH
382 the PCP logger control file. For a new installation this file
383 contains no pmlogger(1) control lines (the real control files are
384 all in the $PCP_PMLOGGERCONTROL_PATH.d directory), but this file
385 is still processed to support any legacy configurations therein
386 from earlier PCP releases.
387 Warning: this file must not be writable by any user other than
388 root.
389 $PCP_PMLOGGERCONTROL_PATH.d
391 optional directory containing additional PCP logger control files,
392 typically one per host
393 Warning: the files herein must not be writable by any user other
394 than root.
395 $PCP_SYSCONF_DIR/pmlogger/crontab
397 sample crontab for automated script execution by $PCP_USER (or
398 root). Exists only if the platform does not support the
399 /etc/cron.d mechanism.
400 $PCP_VAR_DIR/config/pmlogger/config.default
402 default pmlogger configuration file location for the local primary
403 logger, typically generated automatically by pmlogconf(1).
404 $PCP_ARCHIVE_DIR/<hostname>
406 default location for archives of performance information collected
407 from the host hostname
408 $PCP_ARCHIVE_DIR/<hostname>/lock
410 transient lock file to guarantee mutual exclusion during pmlogger
411 administration for the host hostname - if present, can be safely
412 removed if neither pmlogger_daily nor pmlogger_check are running
413 $PCP_ARCHIVE_DIR/<hostname>/Latest
415 PCP archive folio created by mkaf(1) for the most recently
416 launched archive containing performance metrics from the host
417 hostname
418 $PCP_LOG_DIR/NOTICES
420 PCP ``notices'' file used by pmie(1) and friends
421 $PCP_LOG_DIR/pmlogger/pmlogger_check.log
423 if the previous execution of pmlogger_check produced any output it
424 is saved here. The normal case is no output in which case the
425 file does not exist.
426 $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
428 if the previous execution of pmlogger_daily produced any output it
429 is saved here. The normal case is no output in which case the
430 file does not exist.
431 $PCP_ARCHIVE_DIR/<hostname>/SaveLogs
433 if this directory exists, then the log file from the -l argument
434 of a newly launched pmlogger(1) for hostname will be linked into
435 this directory with the name archive.log where archive is the
436 basename of the associated pmlogger(1) PCP archive files. This
437 allows the log file to be inspected at a later time, even if sev‐
438 eral pmlogger(1) instances for hostname have been launched in the
439 interim. Because the cron-driven PCP archive management scripts
440 run under the uid of the user ``pcp'', $PCP_ARCHIVE_DIR/host‐
441 name/SaveLogs typically needs to be owned by the user ``pcp''.
442 $PCP_LOG_DIR/pmlogger/.NeedRewrite
444 if this file exists, then this is treated as equivalent to using
445 -R on the command line and the file will be removed once all
446 rewriting has been done.
447
449 Environment variables with the prefix PCP_ are used to parameterize the
450 file and directory names used by PCP. On each installation, the file
451 /etc/pcp.conf contains the local values for these variables. The
452 $PCP_CONF variable may be used to specify an alternative configuration
453 file, as described in pcp.conf(5).
454 The default behaviour, when pmlogger(1) configuration comes from pmlog‐
456 conf(1), is to regenerate the configuration file and check for changes
457 whenever pmlogger(1) is started from pmlogger_check. If the PMDA con‐
458 figuration is stable, this is not necessary, and setting $PMLOG‐
459 GER_CHECK_SKIP_LOGCONF to yes disables the regeneration and checking.
460
462 Earlier versions of pmlogger_daily used find(1) to locate files for
463 compressing or culling and the -k and -x options took only integer val‐
464 ues to mean ``days''. The semantics of this was quite loose given that
465 find(1) offers different precision and semantics across platforms.
466 The current implementation of pmlogger_daily uses find-filter(1) which
468 provides high precision intervals and semantics that are relative to
469 the time of execution and are consistent across platforms.
470
472 egrep(1), find-filter(1), PCPIntro(1), pmconfig(1), pmlc(1), pmlog‐
473 conf(1), pmlogctl(1), pmlogger(1), pmlogger_daily_report(1), pmlog‐
474 ger_merge(1), pmlogextract(1), pmlogmv(1), pmlogrewrite(1), pmsocks(1),
475 systemd(1), xz(1) and cron(8).
476Performance Co-Pilot PCP PMLOGGER_CHECK(1)