1PMLOGGER_CHECK(1) General Commands Manual PMLOGGER_CHECK(1)
2
6 pmlogger_check, pmlogger_daily, pmlogger_merge - administration of Per‐
7 formance Co-Pilot archive log files
8
10 $PCP_BINADM_DIR/pmlogger_check [-CNsTV] [-c control] [-l logfile]
11 $PCP_BINADM_DIR/pmlogger_daily [-KMNoprV] [-c control] [-k discard] [-l
12 logfile] [-m addresses] [-s size] [-t want] [-x compress] [-X program]
13 [-Y regex]
14 $PCP_BINADM_DIR/pmlogger_merge [-fNV] [input-basename ... output-name]
15
17 This series of shell scripts and associated control files may be used
18 to create a customized regime of administration and management for Per‐
19 formance Co-Pilot (see PCPintro(1)) archive log files.
20 pmlogger_daily is intended to be run once per day, preferably in the
22 early morning, as soon after midnight as practicable. Its task is to
23 aggregate and rotate one or more sets of PCP archives. After some
24 period, old PCP archives are discarded. This period is 14 days by
25 default, but may be changed using the -k option. Some special values
26 are recognized for the period (discard), namely 0 to keep no archives
27 beyond the current one, and forever or never to prevent any archives
28 being discarded. Note that the semantics of discard are that it is
29 measured from the time of last modification of each archive, and not
30 from the current day. This has subtle implications for compression
31 (see below) - the compression process results in the creation of new
32 archive files which have new modification times. In this case, the
33 discard period (re)starts from the time of compression.
34 Archive data files can optionally be compressed after some period to
36 conserve disk space. This is particularly useful for large numbers of
37 pmlogger processes under the control of pmlogger_check. If transpar‐
38 ent_decompress is enabled when libpcp was built (can be checked with
39 pmconfig -L), then the default behaviour is compression ``as soon as
40 possible'' otherwise the default behaviour is to not compress files
41 (which matches the historical default behaviour in earlier PCP
42 releases).
43 The -x option controls compression and compress specifies the number of
45 days after which to compress archive data files and metadata files. If
46 compress is 0 then compression will be applied as soon as possible. If
47 compress is never or forever then no compression will be done. The
48 environment variable PCP_COMPRESSAFTER may be used as an alternative
49 mechanism to define compress. If both PCP_COMPRESSAFTER and -x specify
50 different values for compress then the environment variable value is
51 used and a warning is issued.
52 The -X option specifies the program to use for compression - by default
54 this is xz(1). The environment variable PCP_COMPRESS may be used as an
55 alternative mechanism to define program. If both PCP_COMPRESS and -X
56 specify different compression programs then the environment variable
57 value is used and a warning is issued.
58 Use of the -Y option allows a regular expression to be specified caus‐
60 ing files in the set of files matched for compression to be omitted -
61 this allows only the data file to be compressed, and also prevents the
62 program from attempting to compress it more than once. The default
63 regex is ".(meta|index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" - such files are
64 filtered using the -v option to egrep(1). The environment variable
65 PCP_COMPRESSREGEX may be used as an alternative mechanism to define
66 regex. If both PCP_COMPRESSREGEX and -Y specify different values for
67 regex then the environment variable value is used and a warning is
68 issued.
69 To accommodate the evolution of PMDAs and changes in production logging
71 environments, pmlogger_daily is integrated with pmlogrewrite(1) to
72 allow optional and automatic rewriting of archives before merging. If
73 there are global rewriting rules to be applied across all archives men‐
74 tioned in the control file(s), then create the directory
75 $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting
76 rules in this directory. For rewriting rules that are specific to only
77 one family of archives, use the directory name from the control file(s)
78 - i.e. the fourth field - and create a file, or a directory, or a sym‐
79 bolic link named pmlogrewrite within this directory and place the
80 required rewriting rule(s) in the pmlogrewrite file or in files within
81 the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting
82 rules from the archive directory if they exist, else rewriting rules
83 from $PCP_SYSCONF_DIR/pmlogrewrite if that directory exists, else no
84 rewriting is attempted.
85 The -r command line option acts as an over-ride and prevents all ar‐
87 chive rewriting with pmlogrewrite(1) independent of the presence of any
88 rewriting rule files or directories.
89 By default all possible archives will be merged. The -o option rein‐
91 states the old behaviour in which only yesterday's archives will be
92 considered as merge candidates.
93 In the special case where only a single input archive needs to be
95 merged, pmlogmv(1) is used to rename the archive, rather than copy the
96 input archive using pmlogger_merge.
97 The -M option may be used to disable archive merging (or renaming) and
99 rewriting (-M implies -r). This is most useful in cases where the ar‐
100 chives are being incrementally copied to a remote repository, e.g.
101 using rsync(1). Merging, renaming and rewriting all risk an increase
102 in the synchronization load, especially immediately after pmlog‐
103 ger_daily has run, so -M may be useful in these cases.
104 To assist with debugging or diagnosing intermittent failures the -t
106 option may be used. This will turn on very verbose tracing (-VV) and
107 capture the trace output in a file named $PCP_LOG_DIR/pmlog‐
108 ger/daily.datestamp.trace, where datestamp is the time pmlogger_daily
109 was run in the format YYYYMMDD.HH.MM. In addition, the want argument
110 will ensure that trace files created with -t will be kept for want days
111 and then discarded.
112 In addition, if the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is
114 larger than 20480 bytes, pmlogger_daily will rename the file with a
115 ``.old'' suffix, and start a new ``notices'' file. The rotate thresh‐
116 old may be changed from 20480 to size bytes using the -s option.
117 Use of the -m option causes pmlogger_daily to construct a summary of
119 the ``notices'' file entries which were generated in the last 24 hours,
120 and e-mail that summary to the set of space-separated addresses. This
121 daily summary is stored in the file $PCP_LOG_DIR/NOTICES.daily, which
122 will be empty when no new ``notices'' entries were made in the previous
123 24 hour period.
124 If the -K option is specified for pmlogger_daily then only the compres‐
126 sion tasks are attempted, so no pmlogger(1) rotation, no culling, no
127 rewriting, etc. When -K is used and a compress value of 0 is in effect
128 (from -x on the command line or PCP_COMPRESSAFTER in the environment or
129 via the control file) this is intended for environments where compres‐
130 sion of archives is desired before the scheduled daily processing hap‐
131 pens. To achieve this, once pmlogger_check has completed regular pro‐
132 cessing, it calls pmlogger_daily with just the -K option. Provided
133 PCP_COMPRESSAFTER is set to 0 along with any other required compression
134 options to match the scheduled invocation of pmlogger_daily, then this
135 will compress all volumes except the ones being currently written by
136 pmlogger(1).
137 If the -p option is specified for pmlogger_daily then the status of the
139 daily processing is polled and if the daily pmlogger(1) rotation,
140 culling, rewriting, compressing, etc. has not been done in the last 24
141 hours then it is done now. The intent is to have pmlogger_daily called
142 regularly with the -p option (at 30 mins past the hour, every hour in
143 the default cron(8) set up) to ensure daily processing happens as soon
144 as possible if it was missed at the regularly scheduled time (which is
145 00:10 by default), e.g. if the system was down or suspended at that
146 time.
147 With the -p option, pmlogger_daily simply exits if the previous day's
149 processing has already been done.
150 The -K and -p options to pmlogger_daily are mutually exclusive.
152 The script $PCP_BINADM_DIR/pmlogger_daily could be copied and modified
154 to implement a site-specific procedure for end-of-week and/or end-of-
155 month management for a set of PCP archives.
156 pmlogger_check may be run at any time, and is intended to check that
158 the desired set of pmlogger(1) processes are running, and if not to re-
159 launch any failed loggers. Use of the -s option provides the reverse
160 functionality, allowing the set of pmlogger processes to be cleanly
161 shutdown. Use of the -C option queries the system service runlevel
162 information for pmlogger, and uses that to determine whether to start
163 processes.
164 The -T option provides a terser form of output for pmlogger_check that
166 is most suitable for a pmlogger ``farm'' where many instances of pmlog‐
167 ger are expected to be running.
168 pmlogger_merge is a wrapper script for pmlogextract(1) that merges all
170 of the archive logs matching the input-basename arguments, and creates
171 a new archive using output-name as the base name for the physical files
172 that constitute an archive log. The input-basename arguments may con‐
173 tain meta characters in the style of sh(1). If specified, the -f
174 option causes all of the input files to be removed once the output ar‐
175 chive has been created.
176 pmlogger_merge is used by pmlogger_daily.
178 Both pmlogger_daily and pmlogger_check are controlled by PCP logger
180 control file(s) that specifies the pmlogger instances to be managed.
181 The default control file is $PCP_PMLOGGERCONTROL_PATH, but an alternate
182 may be specified using the -c option. If the directory $PCP_PMLOGGER‐
183 CONTROL_PATH.d (or control.d from the -c option) exists, then the con‐
184 tents of any additional control files therein will be appended to the
185 main control file (which must exist).
186 Warning: The $PCP_PMLOGGERCONTROL_PATH and $PCP_PMLOGGERCONTROL_PATH.d
188 files must not be writable by any user other than root.
189 The control file(s) should be customized according to the following
191 rules that define for the current version (1.1) of the control file
192 format.
193 1. Lines beginning with a ``#'' are comments.
195 2. Lines beginning with a ``$'' are assumed to be assignments to envi‐
196 ronment variables in the style of sh(1), and all text following the
197 ``$'' will be eval'ed by the script reading the control file, and
198 the corresponding variable exported into the environment. This is
199 particularly useful to set and export variables into the environ‐
200 ment of the administrative scripts, e.g.
201 $ PMCD_CONNECT_TIMEOUT=20
202 3. There must be a version line in the initial control file of the
203 form:
204 $ version=1.1
205 4. There should be one line in the control file(s) for each pmlogger
206 instance of the form:
207 host y|n y|n directory args
209 5. Fields within a line of the control file(s) are usually separated
211 by one or more spaces or tabs (although refer to the description of
212 the directory field for some important exceptions).
213 6. The first field is the name of the host that is the source of the
214 performance metrics for this pmlogger instance.
215 7. The second field indicates if this is a primary pmlogger instance
216 (y) or not (n). Since the primary logger must run on the local
217 host, and there may be at most one primary logger for a particular
218 host, this field can be y for at most one pmlogger instance, in
219 which case the host name must be the name of the local host.
220 8. The third field indicates if this pmlogger instance needs to be
221 started under the control of pmsocks(1) to connect to a pmcd
222 through a firewall (y or n).
223 9. The fourth field is a directory name. All files associated with
224 this pmlogger instance will be created in this directory, and this
225 will be the current directory for the execution of any programs
226 required in the maintenance of those archives. A useful convention
227 is that primary logger archives for the local host with hostname
228 myhost are maintained in the directory $PCP_LOG_DIR/pmlogger/myhost
229 (this is where the default pmlogger start-up script in
230 $PCP_RC_DIR/pcp will create the archives), while archives for the
231 remote host mumble are maintained in $PCP_LOG_DIR/pmlogger/mumble.
232 10. The directory field may contain embedded shell syntax that will be
233 evaluated by sh(1) to produce the real directory name to be used.
234 The allowed constructs are:
235 · Any text (including white space) enclosed with $( and ).
236 · Any text (including white space) enclosed with ` and ` (back
237 quotes).
238 · Any text (including white space) enclosed with " and " (double
239 quotes).
240 · Any word containing a $ (assumed to introduce an environment
241 variable name).
242 11. All other fields are interpreted as arguments to be passed to
243 pmlogger(1) and/or pmnewlog(1). Most typically this would be the
244 -c option.
245 The following sample control lines specify a primary logger on the
247 local host (bozo), and non-primary loggers to collect and log perfor‐
248 mance metrics from the hosts wobbly and boing.
249 $version=1.1
251 bozo y n $PCP_LOG_DIR/pmlogger/bozo -c config.default
252 wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
253 boing n n $PCP_LOG_DIR/pmlogger/boing -c ./pmlogger.config
254 Typical crontab(5) entries for periodic execution of pmlogger_daily and
256 pmlogger_check are given in $PCP_SYSCONF_DIR/pmlogger/crontab (unless
257 installed by default in /etc/cron.d already) and shown below.
258 # daily processing of archive logs
260 14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
261 # every 30 minutes, check pmlogger instances are running
262 25,55 * * * * $PCP_BINADM_DIR/pmlogger_check
263 In order to ensure that mail is not unintentionally sent when these
265 scripts are run from cron(8) diagnostics are always sent to a log file.
266 By default, this file is $PCP_LOG_DIR/pmlogger/pmlogger_daily.log or
267 $PCP_LOG_DIR/pmlogger/pmlogger_check.log but this can be changed using
268 the -l option. If this log file already exists when the script starts,
269 it will be renamed with a .prev suffix (overwriting any log file saved
270 earlier) before diagnostics are generated to the log file. The -l and
271 -t options cannot be used together.
272 The output from the cron execution of the scripts may be extended using
274 the -V option to the scripts which will enable verbose tracing of their
275 activity. By default the scripts generate no output unless some error
276 or warning condition is encountered.
277
280 $PCP_PMLOGGERCONTROL_PATH
281 the PCP logger control file
282 Warning: this file must not be writable by any user other
283 than root.
284 $PCP_PMLOGGERCONTROL_PATH.d
286 optional directory containing additional PCP logger control
287 files, typically one per host
288 Warning: the files herein must not be writable by any user
289 other than root.
290 $PCP_SYSCONF_DIR/pmlogger/crontab
292 sample crontab for automated script execution by $PCP_USER
293 (or root). Exists only if the platform does not support the
294 /etc/cron.d mechanism.
295 $PCP_VAR_DIR/config/pmlogger/config.default
297 default pmlogger configuration file location for the local
298 primary logger, typically generated automatically by pmlog‐
299 conf(1).
300 $PCP_LOG_DIR/pmlogger/hostname
302 default location for archives of performance information col‐
303 lected from the host hostname
304 $PCP_LOG_DIR/pmlogger/hostname/lock
306 transient lock file to guarantee mutual exclusion during
307 pmlogger administration for the host hostname - if present,
308 can be safely removed if neither pmlogger_daily nor pmlog‐
309 ger_check are running
310 $PCP_LOG_DIR/pmlogger/hostname/Latest
312 PCP archive folio created by mkaf(1) for the most recently
313 launched archive containing performance metrics from the host
314 hostname
315 $PCP_LOG_DIR/NOTICES
317 PCP ``notices'' file used by pmie(1) and friends
318 $PCP_LOG_DIR/pmlogger/pmlogger_check.log
320 if the previous execution of pmlogger_check produced any out‐
321 put it is saved here. The normal case is no output in which
322 case the file does not exist.
323 $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
325 if the previous execution of pmlogger_daily produced any out‐
326 put it is saved here. The normal case is no output in which
327 case the file does not exist.
328 $PCP_LOG_DIR/pmlogger/hostname/SaveLogs
330 if this directory exists, then the log file from the -l argu‐
331 ment of a newly launched pmlogger(1) for hostname will be
332 linked into this directory with the name archive.log where
333 archive is the basename of the associated pmlogger(1) PCP ar‐
334 chive files. This allows the log file to be inspected at a
335 later time, even if several pmlogger(1) instances for host‐
336 name have been launched in the interim. Because the cron-
337 driven PCP archive management scripts run under the uid of
338 the user ``pcp'', $PCP_LOG_DIR/pmlogger/hostname/SaveLogs
339 typically needs to be owned by the user ``pcp''.
340
342 Environment variables with the prefix PCP_ are used to parameterize the
343 file and directory names used by PCP. On each installation, the file
344 /etc/pcp.conf contains the local values for these variables. The
345 $PCP_CONF variable may be used to specify an alternative configuration
346 file, as described in pcp.conf(5).
347
349 egrep(1), PCPIntro(1), pmconfig(1), pmlc(1), pmlogconf(1), pmlogex‐
350 tract(1), pmlogger(1), pmlogger_daily_report(1), pmlogmv(1), pmlo‐
351 grewrite(1), pmnewlog(1), pmsocks(1), xz(1) and cron(8).
352Performance Co-Pilot PCP PMLOGGER_CHECK(1)