1PMLOGGER_CHECK(1) General Commands Manual PMLOGGER_CHECK(1)
2
6 pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot
7 archive log files
8
10 $PCP_BINADM_DIR/pmlogger_check [-CNpqsTV?] [-c control] [-l logfile]
11 $PCP_BINADM_DIR/pmlogger_daily [-EKMNoprRV?] [-c control] [-k time]
12 [-l logfile] [-m addresses] [-s size] [-t want] [-x time] [-X program]
13 [-Y regex]
14
16 These shell scripts and associated control files may be used to create
17 a customized regime of administration and management for Performance
18 Co-Pilot (see PCPIntro(1)) archive log files.
19 pmlogger_check may be run at any time of the day and is intended to
21 check that a desired set of pmlogger(1) processes are running. If not,
22 it (re-)starts any missing logger processes.
23 pmlogger_daily is intended to be run once per day, preferably in the
25 early morning, as soon after midnight as practicable. Its task is to
26 aggregate, rotate and perform general housekeeping one or more sets of
27 PCP archives.
28 To accommodate the evolution of PMDAs and changes in production logging
30 environments, pmlogger_daily is integrated with pmlogrewrite(1) to
31 allow optional and automatic rewriting of archives before merging. If
32 there are global rewriting rules to be applied across all archives men‐
33 tioned in the control file(s), then create the directory
34 $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting
35 rules in this directory. For rewriting rules that are specific to only
36 one family of archives, use the directory name from the control file(s)
37 - i.e. the fourth field - and create a file, or a directory, or a sym‐
38 bolic link named pmlogrewrite within this directory and place the
39 required rewriting rule(s) in the pmlogrewrite file or in files within
40 the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting
41 rules from the archive directory if they exist, else rewriting rules
42 from $PCP_SYSCONF_DIR/pmlogrewrite if that directory exists, else no
43 rewriting is attempted.
44 As an alternate mechanism, if the file $PCP_LOG_DIR/pmlogger/.Nee‐
46 dRewrite exists when pmlogger_daily starts then this is treated the
47 same as specifying -R on the command line and $PCP_LOG_DIR/pmlog‐
48 ger/.NeedRewrite will be removed once all the rewriting has been done.
49
51 -c control, --control=control
52 Both pmlogger_check and pmlogger_daily are controlled by PCP log‐
53 ger control file(s) that specifies the pmlogger instances to be
54 managed. The default control file is $PCP_PMLOGGERCONTROL_PATH,
55 but an alternate may be specified using the -c option. If the
56 directory $PCP_PMLOGGERCONTROL_PATH.d (or control.d from the -c
57 option) exists, then the contents of any additional control files
58 therein will be appended to the main control file (which must
59 exist).
60 -C This option causes pmlogger_check to query the system service run‐
62 level information for pmlogger, and use that to determine whether
63 to start processes or not.
64 -E This option causes pmlogger_daily to pass the -E flag to pmlog‐
66 ger_merge in order to expunge metrics with metadata inconsisten‐
67 cies and continue rather than fail. This is intended for auto‐
68 mated daily log rotation where it is highly desirable for unat‐
69 tended daily archive merging, rewriting and compression to suc‐
70 ceed. For further details, see pmlogger_merge(1) and description
71 for the -x flag in pmlogextract(1).
72 -k time, --discard=time
74 After some period, old PCP archives are discarded. time is a time
75 specification in the syntax of find-filter(1), so DD[:HH[:MM]].
76 The optional HH (hours) and MM (minutes) parts are 0 if not speci‐
77 fied. By default the time is 14:0:0 or 14 days, but may be
78 changed using this option.
79 Some special values are recognized for the time, namely 0 to keep
81 no archives beyond the the ones being currently written by pmlog‐
82 ger(1), and forever or never to prevent any archives being dis‐
83 carded.
84 The time can also be set using the $PCP_CULLAFTER variable, set in
86 either the environment or in a control file. If both $PCP_CUL‐
87 LAFTER and -k specify different values for time then the environ‐
88 ment variable value is used and a warning is issued. I.e., if
89 $PCP_CULLAFTER is set in the control file, it overrides -k given
90 on the command line.
91 Note that the semantics of time are that it is measured from the
93 time of last modification of each archive, and not from the origi‐
94 nal archive creation date. This has subtle implications for com‐
95 pression (see below) - the compression process results in the cre‐
96 ation of new archive files which have new modification times. In
97 this case, the time period (re)starts from the time of compres‐
98 sion.
99 -K When this option is specified for pmlogger_daily then only the
101 compression tasks are attempted, so no pmlogger rotation, no
102 culling, no rewriting, etc. When -K is used and a period of 0 is
103 in effect (from -x on the command line or $PCP_COMPRESSAFTER in
104 the environment or via the control file) this is intended for
105 environments where compression of archives is desired before the
106 scheduled daily processing happens. To achieve this, once pmlog‐
107 ger_check has completed regular processing, it calls pmlog‐
108 ger_daily with just the -K option. Provided $PCP_COMPRESSAFTER is
109 set to 0 along with any other required compression options to
110 match the scheduled invocation of pmlogger_daily, then this will
111 compress all volumes except the ones being currently written by
112 pmlogger(1). If $PCP_COMPRESSAFTER is set to a value greater than
113 zero, then manually running pmlogger_daily with the -x option may
114 be used to compress volumes that are younger than the $PCP_COM‐
115 PRESSAFTER time. This may be used to reclaim filesystem space by
116 compressing volumes earlier than they would have otherwise been
117 compressed. Note that since the default value of $PCP_COMPRES‐
118 SAFTER is 0 days, the -x option has no effect unless the control
119 file has been edited and $PCP_COMPRESSAFTER has been set to a
120 value greater than 0.
121 -l file, --logfile=file
123 In order to ensure that mail is not unintentionally sent when
124 these scripts are run from cron(8) diagnostics are always sent to
125 log files. By default, this file is $PCP_LOG_DIR/pmlogger/pmlog‐
126 ger_check.log or $PCP_LOG_DIR/pmlogger/pmlogger_daily.log but this
127 can be changed using the -l option. If this log file already
128 exists when the script starts, it will be renamed with a .prev
129 suffix (overwriting any log file saved earlier) before diagnostics
130 are generated to the log file. The -l and -t options cannot be
131 used together.
132 -m addresses, --mail=addresses
134 Use of this option causes pmlogger_daily to construct a summary of
135 the ``notices'' file entries which were generated in the last 24
136 hours, and e-mail that summary to the set of space-separated
137 addresses. This daily summary is stored in the file
138 $PCP_LOG_DIR/NOTICES.daily, which will be empty when no new
139 ``notices'' entries were made in the previous 24 hour period.
140 -M This option may be used to disable archive merging (or renaming)
142 and rewriting (-M implies -r). This is most useful in cases where
143 the archives are being incrementally copied to a remote reposi‐
144 tory, e.g. using rsync(1). Merging, renaming and rewriting all
145 risk an increase in the synchronization load, especially immedi‐
146 ately after pmlogger_daily has run, so -M may be useful in these
147 cases.
148 -N, --showme
150 This option enables a ``show me'' mode, where the programs actions
151 are echoed, but not executed, in the style of ``make -n''. Using
152 -N in conjunction with -V maximizes the diagnostic capabilities
153 for debugging.
154 -o By default all possible archives will be merged. This option
156 reinstates the old behaviour in which only yesterday's archives
157 will be considered as merge candidates. In the special case where
158 only a single input archive needs to be merged, pmlogmv(1) is used
159 to rename the archive, otherwise pmlogger_merge(1) is used to
160 merge all of the archives for a single host and a single day into
161 a new PCP archive and the individual archives are removed.
162 -p If this option is specified for pmlogger_check then any line from
164 the control files for the primary pmlogger will be ignored. This
165 option is intended for environments where some system daemon, like
166 systemd(1), is responsible for controlling (starting, stopping,
167 restarting, etc.) the primary pmlogger.
168 -p If this option is specified for pmlogger_daily then the status of
170 the daily processing is polled and if the daily pmlogger(1) rota‐
171 tion, culling, rewriting, compressing, etc. has not been done in
172 the last 24 hours then it is done now. The intent is to have
173 pmlogger_daily called regularly with the -p option (at 30 mins
174 past the hour, every hour in the default cron(8) set up) to ensure
175 daily processing happens as soon as possible if it was missed at
176 the regularly scheduled time (which is 00:10 by default), e.g. if
177 the system was down or suspended at that time. With this option
178 pmlogger_daily simply exits if the previous day's processing has
179 already been done. The -K and -p options to pmlogger_daily are
180 mutually exclusive.
181 -q If this option is specified for pmlogger_check then the script
183 will ``quickstart'' avoiding any optional processing like file
184 compression.
185 -r, --norewrite
187 This command line option acts as an override and prevents all ar‐
188 chive rewriting with pmlogrewrite(1) independent of the presence
189 of any rewriting rule files or directories.
190 -R, --rewriteall
192 Sometimes PMDA changes require all archives to be rewritten, not
193 just the ones involved in any current merging. This is required
194 for example after a PCP upgrade where a new version of an existing
195 PMDA has revised metadata. The -R command line forces this uni‐
196 versal-style of rewriting. The -R option to pmlogger_daily is
197 mutually exclusive with both the -r and -M options.
198 -s size, --rotate=size
200 If the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is larger than
201 20480 bytes, pmlogger_daily will rename the file with a ``.old''
202 suffix, and start a new ``notices'' file. The rotate threshold
203 may be changed from 20480 to size bytes using the -s option.
204 -s, --stop
206 Use of this option provides the reverse pmlogger_check functional‐
207 ity, allowing the set of pmlogger processes to be cleanly shut‐
208 down.
209 -t period
211 To assist with debugging or diagnosing intermittent failures the
212 -t option may be used. This will turn on very verbose tracing
213 (-VV) and capture the trace output in a file named
214 $PCP_LOG_DIR/pmlogger/daily.datestamp.trace, where datestamp is
215 the time pmlogger_daily was run in the format YYYYMMDD.HH.MM. In
216 addition, the period argument will ensure that trace files created
217 with -t will be kept for period days and then discarded.
218 -T, --terse
220 This option to pmlogger_check produces less verbose output than
221 the default. This is most suitable for a pmlogger ``farm'' where
222 many instances of pmlogger are expected to be running.
223 -V, --verbose
225 The output from the cron execution of the scripts may be extended
226 using the -V option to the scripts which will enable verbose trac‐
227 ing of their activity. By default the scripts generate no output
228 unless some error or warning condition is encountered. Using -N
229 in conjunction with -V maximizes the diagnostic capabilities for
230 debugging.
231 -x time, --compress-after=time
233 Archive data files can optionally be compressed after some period
234 to conserve disk space. This is particularly useful for large
235 numbers of pmlogger processes under the control of pmlogger_check.
236 time is a time specification in the syntax of find-filter(1), so
238 DD[:HH[:MM]]. The optional HH (hours) and MM (minutes) parts are
239 0 if not specified.
240 Some special values are recognized for the time, namely 0 to apply
242 compression as soon as possible, and forever or never to prevent
243 any compression being done.
244 If transparent_decompress is enabled when libpcp was built (can be
246 checked with the pmconfig(1) -L option), then the default behav‐
247 iour is compression ``as soon as possible''. Otherwise the
248 default behaviour is to not compress files (which matches the his‐
249 torical default behaviour in earlier PCP releases).
250 The time can also be set using the $PCP_COMPRESSAFTER variable,
252 set in either the environment or in a control file. If both
253 $PCP_COMPRESSAFTER and -x specify different values for time then
254 the environment variable value is used and a warning is issued.
255 For important other detailed notes concerning volume compression,
256 see the -K and -k options (above).
257 -X program, --compressor=program
259 This option specifies the program to use for compression - by
260 default this is xz(1). The environment variable $PCP_COMPRESS may
261 be used as an alternative mechanism to define program. If both
262 $PCP_COMPRESS and -X specify different compression programs then
263 the environment variable value is used and a warning is issued.
264 -Y regex, --regex=regex
266 This option allows a regular expression to be specified causing
267 files in the set of files matched for compression to be omitted -
268 this allows only the data file to be compressed, and also prevents
269 the program from attempting to compress it more than once. The
270 default regex is ".(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" - such
271 files are filtered using the -v option to egrep(1). The environ‐
272 ment variable $PCP_COMPRESSREGEX may be used as an alternative
273 mechanism to define regex. If both $PCP_COMPRESSREGEX and -Y
274 specify different values for regex then the environment variable
275 value is used and a warning is issued.
276 -?, --help
278 Display usage message and exit.
279
281 Warning: The $PCP_PMLOGGERCONTROL_PATH file and files within the
282 $PCP_PMLOGGERCONTROL_PATH.d directory must not be writable by any user
283 other than root.
284 The control file(s) should be customized according to the following
286 rules that define for the current version (1.1) of the control file
287 format.
288 1. Lines beginning with a ``#'' are comments. A special case is lines
290 beginning ``#!#''; these are control lines for a pmlogger(1) that
291 has been stopped using pmlogctl(1).
292 2. Lines beginning with a ``$'' are assumed to be assignments to envi‐
293 ronment variables in the style of sh(1), and all text following the
294 ``$'' will be eval'ed by the script reading the control file, and
295 the corresponding variable exported into the environment. This is
296 particularly useful to set and export variables into the environ‐
297 ment of the administrative scripts, e.g.
298 $ PMCD_CONNECT_TIMEOUT=20
299 3. There must be a version line in the initial control file of the
300 form:
301 $ version=1.1
302 4. There should be one line in the control file(s) for each pmlogger
303 instance of the form:
304 host y|n y|n directory args
306 5. Fields within a line of the control file(s) are usually separated
308 by one or more spaces or tabs (although refer to the description of
309 the directory field for some important exceptions).
310 6. The first field is the name of the host that is the source of the
311 performance metrics for this pmlogger instance.
312 7. The second field indicates if this is a primary pmlogger instance
313 (y) or not (n). Since the primary logger must run on the local
314 host, and there may be at most one primary logger for a particular
315 host, this field can be y for at most one pmlogger instance, in
316 which case the host name must be the name of the local host.
317 8. The third field indicates if this pmlogger instance needs to be
318 started under the control of pmsocks(1) to connect to a pmcd
319 through a firewall (y or n).
320 9. The fourth field is a directory name. All files associated with
321 this pmlogger instance will be created in this directory, and this
322 will be the current directory for the execution of any programs
323 required in the maintenance of those archives. A useful convention
324 is that primary logger archives for the local host with hostname
325 myhost are maintained in the directory $PCP_ARCHIVE_DIR/myhost
326 (this is where the default pmlogger start-up script in
327 $PCP_RC_DIR/pcp will create the archives), while archives for the
328 remote host mumble are maintained in $PCP_ARCHIVE_DIR/mumble.
329 10. The directory field may contain embedded shell syntax that will be
330 evaluated by sh(1) to produce the real directory name to be used.
331 The allowed constructs are:
332 · Any text (including white space) enclosed with $( and ).
333 · Any text (including white space) enclosed with ` and ` (back
334 quotes).
335 · Any text (including white space) enclosed with " and " (double
336 quotes).
337 · Any word containing a $ (assumed to introduce an environment
338 variable name).
339 11. All other fields are interpreted as arguments to be passed to
340 pmlogger(1). Most typically this would be the -c option.
341 The following sample control lines specify a primary logger on the
343 local host (bozo), and non-primary loggers to collect and log perfor‐
344 mance metrics from the hosts wobbly and boing.
345 $version=1.1
347 bozo y n $PCP_ARCHIVE_DIR/bozo -c config.default
348 wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
349 boing n n $PCP_ARCHIVE_DIR/boing -c ./pmlogger.config
350 Typical crontab(5) entries for periodic execution of pmlogger_daily and
352 pmlogger_check are given in $PCP_SYSCONF_DIR/pmlogger/crontab (unless
353 installed by default in /etc/cron.d already) and shown below.
354 # daily processing of archive logs
356 14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
357 # every 30 minutes, check pmlogger instances are running
358 25,55 * * * * $PCP_BINADM_DIR/pmlogger_check
359 When using systemd(1) on Linux, no crontab entries are needed as the
361 timer mechanism provided by systemd is used instead.
362
364 $PCP_PMLOGGERCONTROL_PATH
365 the PCP logger control file. For a new installation this file
366 contains no pmlogger(1) control lines (the real control files are
367 all in the $PCP_PMLOGGERCONTROL_PATH.d directory), but this file
368 is still processed to support any legacy configurations therein
369 from earlier PCP releases.
370 Warning: this file must not be writable by any user other than
371 root.
372 $PCP_PMLOGGERCONTROL_PATH.d
374 optional directory containing additional PCP logger control files,
375 typically one per host
376 Warning: the files herein must not be writable by any user other
377 than root.
378 $PCP_SYSCONF_DIR/pmlogger/crontab
380 sample crontab for automated script execution by $PCP_USER (or
381 root). Exists only if the platform does not support the
382 /etc/cron.d mechanism.
383 $PCP_VAR_DIR/config/pmlogger/config.default
385 default pmlogger configuration file location for the local primary
386 logger, typically generated automatically by pmlogconf(1).
387 $PCP_ARCHIVE_DIR/<hostname>
389 default location for archives of performance information collected
390 from the host hostname
391 $PCP_ARCHIVE_DIR/<hostname>/lock
393 transient lock file to guarantee mutual exclusion during pmlogger
394 administration for the host hostname - if present, can be safely
395 removed if neither pmlogger_daily nor pmlogger_check are running
396 $PCP_ARCHIVE_DIR/<hostname>/Latest
398 PCP archive folio created by mkaf(1) for the most recently
399 launched archive containing performance metrics from the host
400 hostname
401 $PCP_LOG_DIR/NOTICES
403 PCP ``notices'' file used by pmie(1) and friends
404 $PCP_LOG_DIR/pmlogger/pmlogger_check.log
406 if the previous execution of pmlogger_check produced any output it
407 is saved here. The normal case is no output in which case the
408 file does not exist.
409 $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
411 if the previous execution of pmlogger_daily produced any output it
412 is saved here. The normal case is no output in which case the
413 file does not exist.
414 $PCP_ARCHIVE_DIR/<hostname>/SaveLogs
416 if this directory exists, then the log file from the -l argument
417 of a newly launched pmlogger(1) for hostname will be linked into
418 this directory with the name archive.log where archive is the
419 basename of the associated pmlogger(1) PCP archive files. This
420 allows the log file to be inspected at a later time, even if sev‐
421 eral pmlogger(1) instances for hostname have been launched in the
422 interim. Because the cron-driven PCP archive management scripts
423 run under the uid of the user ``pcp'', $PCP_ARCHIVE_DIR/host‐
424 name/SaveLogs typically needs to be owned by the user ``pcp''.
425 $PCP_LOG_DIR/pmlogger/.NeedRewrite
427 if this file exists, then this is treated as equivalent to using
428 -R on the command line and the file will be removed once all
429 rewriting has been done.
430
432 Environment variables with the prefix PCP_ are used to parameterize the
433 file and directory names used by PCP. On each installation, the file
434 /etc/pcp.conf contains the local values for these variables. The
435 $PCP_CONF variable may be used to specify an alternative configuration
436 file, as described in pcp.conf(5).
437 The default behaviour, when pmlogger(1) configuration comes from pmlog‐
439 conf(1), is to regenerate the configuration file and check for changes
440 whenever pmlogger(1) is started from pmlogger_check. If the PMDA con‐
441 figuration is stable, this is not necessary, and setting $PMLOG‐
442 GER_CHECK_SKIP_LOGCONF to yes disables the regeneration and checking.
443
445 Earlier versions of pmlogger_daily used find(1) to locate files for
446 compressing or culling and the -k and -x options took only integer val‐
447 ues to mean ``days''. The semantics of this was quite loose given that
448 find(1) offers different precision and semantics across platforms.
449 The current implementation of pmlogger_daily uses find-filter(1) which
451 provides high precision intervals and semantics that are relative to
452 the time of execution and are consistent across platforms.
453
455 egrep(1), find-filter(1), PCPIntro(1), pmconfig(1), pmlc(1), pmlog‐
456 conf(1), pmlogctl(1), pmlogger(1), pmlogger_daily_report(1), pmlog‐
457 ger_merge(1), pmlogextract(1), pmlogmv(1), pmlogrewrite(1), pmsocks(1),
458 systemd(1), xz(1) and cron(8).
459Performance Co-Pilot PCP PMLOGGER_CHECK(1)