1PMLOGGER_CHECK(1) General Commands Manual PMLOGGER_CHECK(1)
2
6 pmlogger_check, pmlogger_daily, pmlogger_merge - administration of Per‐
7 formance Co-Pilot archive log files
8
10 $PCP_BINADM_DIR/pmlogger_check [-CNsTV] [-c control] [-l logfile]
11 $PCP_BINADM_DIR/pmlogger_daily [-NorV] [-c control] [-k discard] [-l
12 logfile] [-m addresses] [-s size] [-t want] [-x compress] [-X program]
13 [-Y regex]
14 $PCP_BINADM_DIR/pmlogger_merge [-fNV] [input-basename ... output-name]
15
17 This series of shell scripts and associated control files may be used
18 to create a customized regime of administration and management for Per‐
19 formance Co-Pilot (see PCPintro(1)) archive log files.
20 pmlogger_daily is intended to be run once per day, preferably in the
22 early morning, as soon after midnight as practicable. Its task is to
23 aggregate and rotate one or more sets of PCP archives. After some
24 period, old PCP archives are discarded. This period is 14 days by
25 default, but may be changed using the -k option. Two special values are
26 recognized for the period (discard), namely 0 to keep no archives
27 beyond the current one, and forever to prevent any archives being dis‐
28 carded.
29 Archive data files can optionally be compressed after some period to
31 conserve disk space. This is particularly useful for large numbers of
32 pmlogger processes under the control of pmlogger_check. By default no
33 compression is done. The -x option enables compression and specifies
34 the number of days after which to compress archive data files, and the
35 -X option specifies the program to use for compression - by default
36 this is xz(1). Use of the -Y option allows a regular expression to be
37 specified causing files in the set of files matched for compression to
38 be omitted - this allows only the data file to be compressed, and also
39 prevents the program from attempting to compress it more than once.
40 The default regex is ".(meta|index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$" -
41 such files are filtered using the -v option to egrep(1).
42 To accommodate the evolution of PMDAs and changes in production logging
44 environments, pmlogger_daily is integrated with pmlogrewrite(1) to
45 allow optional and automatic rewriting of archives before merging. If
46 there are global rewriting rules to be applied across all archives men‐
47 tioned in the control file(s), then create the directory
48 $PCP_SYSCONF_DIR/pmlogrewrite and place any pmlogrewrite(1) rewriting
49 rules in this directory. For rewriting rules that are specific to only
50 one family of archives, use the directory name from the control file(s)
51 - i.e. the fourth field - and create a file, or a directory, or a sym‐
52 bolic link named pmlogrewrite within this directory and place the
53 required rewriting rule(s) in the pmlogrewrite file or in files within
54 the pmlogrewrite subdirectory. pmlogger_daily will choose rewriting
55 rules from the archive directory if they exist, else rewriting rules
56 from $PCP_SYSCONF_DIR/pmlogrewrite if that directory exists, else no
57 rewriting is attempted.
58 The -r command line option acts as an over-ride and prevents all ar‐
60 chive rewriting with pmlogrewrite(1) independent of the presence of any
61 rewriting rule files or directories.
62 By default all possible archives will be merged. The -o option rein‐
64 states the old behaviour in which only yesterday's archives will be
65 considered as merge candidates.
66 In the special case where only a single input archive needs to be
68 merged, pmlogmv(1) is used to rename the archive, rather than copy the
69 input archive using pmlogger_merge.
70 The -M option may be used to disable archive merging (or renaming) and
72 rewriting (-M implies -r). This is most useful in cases where the ar‐
73 chives are being incrementally copied to a remote repository, e.g.
74 using rsync(1). Merging, renaming and rewriting all risk an increase
75 in the synchronization load, especially immediately after pmlog‐
76 ger_daily has run, so -M may be useful in these cases.
77 To assist with debugging or diagnosing intermittent failures the -t
79 option may be used. This will turn on very verbose tracing (-VV) and
80 capture the trace output in a file named $PCP_LOG_DIR/pmlog‐
81 ger/daily.datestamp.trace, where datestamp is the time pmlogger_daily
82 was run in the format YYYYMMDD.HH.MM. In addition, the want argument
83 will ensure that trace files created with -t will be kept for want days
84 and then discarded.
85 In addition, if the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is
87 larger than 20480 bytes, pmlogger_daily will rename the file with a
88 ``.old'' suffix, and start a new ``notices'' file. The rotate thresh‐
89 old may be changed from 20480 to size bytes using the -s option.
90 Use of the -m option causes pmlogger_daily to construct a summary of
92 the ``notices'' file entries which were generated in the last 24 hours,
93 and e-mail that summary to the set of space-separated addresses. This
94 daily summary is stored in the file $PCP_LOG_DIR/NOTICES.daily, which
95 will be empty when no new ``notices'' entries were made in the previous
96 24 hour period.
97 The script $PCP_BINADM_DIR/pmlogger_daily could be copied and modified
99 to implement a site-specific procedure for end-of-week and/or end-of-
100 month management for a set of PCP archives.
101 pmlogger_check may be run at any time, and is intended to check that
103 the desired set of pmlogger(1) processes are running, and if not to re-
104 launch any failed loggers. Use of the -s option provides the reverse
105 functionality, allowing the set of pmlogger processes to be cleanly
106 shutdown. Use of the -C option queries the system service runlevel
107 information for pmlogger, and uses that to determine whether to start
108 or stop processes.
109 The -T option provides a terser form of output for pmlogger_check that
111 is most suitable for a pmlogger ``farm'' where many instances of pmlog‐
112 ger are expected to be running.
113 pmlogger_merge is a wrapper script for pmlogextract(1) that merges all
115 of the archive logs matching the input-basename arguments, and creates
116 a new archive using output-name as the base name for the physical files
117 that constitute an archive log. The input-basename arguments may con‐
118 tain meta characters in the style of sh(1). If specified, the -f
119 option causes all of the input files to be removed once the output ar‐
120 chive has been created.
121 pmlogger_merge is used by pmlogger_daily.
123 Both pmlogger_daily and pmlogger_check are controlled by PCP logger
125 control file(s) that specifies the pmlogger instances to be managed.
126 The default control file is $PCP_PMLOGGERCONTROL_PATH, but an alternate
127 may be specified using the -c option. If the directory $PCP_PMLOGGER‐
128 CONTROL_PATH.d (or control.d from the -c option) exists, then the con‐
129 tents of any additional control files therein will be appended to the
130 main control file (which must exist).
131 Warning: The $PCP_PMLOGGERCONTROL_PATH and $PCP_PMLOGGERCONTROL_PATH.d
133 files must not be writable by any user other than root.
134 The control file(s) should be customized according to the following
136 rules that define for the current version (1.1) of the control file
137 format.
138 1. Lines beginning with a ``#'' are comments.
140 2. Lines beginning with a ``$'' are assumed to be assignments to envi‐
141 ronment variables in the style of sh(1), and all text following the
142 ``$'' will be eval'ed by the script reading the control file, and
143 the corresponding variable exported into the environment. This is
144 particularly useful to set and export variables into the environ‐
145 ment of the administrative scripts, e.g.
146 $ PMCD_CONNECT_TIMEOUT=20
147 3. There must be a version line in the initial control file of the
148 form:
149 $ version=1.1
150 4. There should be one line in the control file(s) for each pmlogger
151 instance of the form:
152 host y|n y|n directory args
154 5. Fields within a line of the control file(s) are usually separated
156 by one or more spaces or tabs (although refer to the description of
157 the directory field for some important exceptions).
158 6. The first field is the name of the host that is the source of the
159 performance metrics for this pmlogger instance.
160 7. The second field indicates if this is a primary pmlogger instance
161 (y) or not (n). Since the primary logger must run on the local
162 host, and there may be at most one primary logger for a particular
163 host, this field can be y for at most one pmlogger instance, in
164 which case the host name must be the name of the local host.
165 8. The third field indicates if this pmlogger instance needs to be
166 started under the control of pmsocks(1) to connect to a pmcd
167 through a firewall (y or n).
168 9. The fourth field is a directory name. All files associated with
169 this pmlogger instance will be created in this directory, and this
170 will be the current directory for the execution of any programs
171 required in the maintenance of those archives. A useful convention
172 is that primary logger archives for the local host with hostname
173 myhost are maintained in the directory $PCP_LOG_DIR/pmlogger/myhost
174 (this is where the default pmlogger start-up script in
175 $PCP_RC_DIR/pcp will create the archives), while archives for the
176 remote host mumble are maintained in $PCP_LOG_DIR/pmlogger/mumble.
177 10. The directory field may contain embedded shell syntax that will be
178 evaluated by sh(1) to produce the real directory name to be used.
179 The allowed constructs are:
180 · Any text (including white space) enclosed with $( and ).
181 · Any text (including white space) enclosed with ` and ` (back
182 quotes).
183 · Any text (including white space) enclosed with " and " (double
184 quotes).
185 · Any word containing a $ (assumed to introduce an environment
186 variable name).
187 11. All other fields are interpreted as arguments to be passed to
188 pmlogger(1) and/or pmnewlog(1). Most typically this would be the
189 -c option.
190 The following sample control lines specify a primary logger on the
192 local host (bozo), and non-primary loggers to collect and log perfor‐
193 mance metrics from the hosts wobbly and boing.
194 $version=1.1
196 bozo y n $PCP_LOG_DIR/pmlogger/bozo -c config.default
197 wobbly n n "/store/wobbly/$(date +%Y)" -c ./wobbly.config
198 boing n n $PCP_LOG_DIR/pmlogger/boing -c ./pmlogger.config
199 Typical crontab(5) entries for periodic execution of pmlogger_daily and
201 pmlogger_check are given in $PCP_SYSCONF_DIR/pmlogger/crontab (unless
202 installed by default in /etc/cron.d already) and shown below.
203 # daily processing of archive logs
205 14 0 * * * $PCP_BINADM_DIR/pmlogger_daily
206 # every 30 minutes, check pmlogger instances are running
207 25,55 * * * * $PCP_BINADM_DIR/pmlogger_check
208 In order to ensure that mail is not unintentionally sent when these
210 scripts are run from cron(8) diagnostics are always sent to a log file.
211 By default, this file is $PCP_LOG_DIR/pmlogger/pmlogger_daily.log or
212 $PCP_LOG_DIR/pmlogger/pmlogger_check.log but this can be changed using
213 the -l option. If this log file already exists when the script starts,
214 it will be renamed with a .prev suffix (overwriting any log file saved
215 earlier) before diagnostics are generated to the log file. The -l and
216 -t options cannot be used together.
217 The output from the cron execution of the scripts may be extended using
219 the -V option to the scripts which will enable verbose tracing of their
220 activity. By default the scripts generate no output unless some error
221 or warning condition is encountered.
222
225 $PCP_PMLOGGERCONTROL_PATH
226 the PCP logger control file
227 Warning: this file must not be writable by any user other
228 than root.
229 $PCP_PMLOGGERCONTROL_PATH.d
231 optional directory containing additional PCP logger control
232 files, typically one per host
233 Warning: the files herein must not be writable by any user
234 other than root.
235 $PCP_SYSCONF_DIR/pmlogger/crontab
237 sample crontab for automated script execution by $PCP_USER
238 (or root). Exists only if the platform does not support the
239 /etc/cron.d mechanism.
240 $PCP_VAR_DIR/config/pmlogger/config.default
242 default pmlogger configuration file location for the local
243 primary logger, typically generated automatically by pmlog‐
244 conf(1).
245 $PCP_LOG_DIR/pmlogger/hostname
247 default location for archives of performance information col‐
248 lected from the host hostname
249 $PCP_LOG_DIR/pmlogger/hostname/lock
251 transient lock file to guarantee mutual exclusion during
252 pmlogger administration for the host hostname - if present,
253 can be safely removed if neither pmlogger_daily nor pmlog‐
254 ger_check are running
255 $PCP_LOG_DIR/pmlogger/hostname/Latest
257 PCP archive folio created by mkaf(1) for the most recently
258 launched archive containing performance metrics from the host
259 hostname
260 $PCP_LOG_DIR/NOTICES
262 PCP ``notices'' file used by pmie(1) and friends
263
265 Environment variables with the prefix PCP_ are used to parameterize the
266 file and directory names used by PCP. On each installation, the file
267 /etc/pcp.conf contains the local values for these variables. The
268 $PCP_CONF variable may be used to specify an alternative configuration
269 file, as described in pcp.conf(5).
270
272 egrep(1), PCPIntro(1), pmlc(1), pmlogconf(1), pmlogger(1), pmlogex‐
273 tract(1), pmlogmv(1), pmlogrewrite(1), pmnewlog(1), pmsocks(1), xz(1)
274 and cron(8).
275Performance Co-Pilot PCP PMLOGGER_CHECK(1)