1PMLOGGER_CHECK(1)           General Commands Manual          PMLOGGER_CHECK(1)
2
3
4

NAME

6       pmlogger_check, pmlogger_daily - administration of Performance Co-Pilot
7       archive log files
8

SYNOPSIS

10       $PCP_BINADM_DIR/pmlogger_check [-CNPpqsTV?]  [-c control] [-l logfile]
11       $PCP_BINADM_DIR/pmlogger_daily [-EfKMNoprRV?]  [-c control]  [-k  time]
12       [-l  logfile] [-m addresses] [-s size] [-t want] [-x time] [-X program]
13       [-Y regex]
14

DESCRIPTION

16       These shell scripts and associated control files may be used to  create
17       a  customized  regime  of administration and management for Performance
18       Co-Pilot (see PCPIntro(1)) archive log files.
19
20       pmlogger_check may be run at any time of the day  and  is  intended  to
21       check that a desired set of pmlogger(1) processes are running.  If not,
22       it (re-)starts any missing logger processes.
23
24       pmlogger_daily is intended to be run once per day,  preferably  in  the
25       early  morning,  as soon after midnight as practicable.  Its task is to
26       aggregate, rotate and perform general housekeeping one or more sets  of
27       PCP archives.
28
29       To accommodate the evolution of PMDAs and changes in production logging
30       environments, pmlogger_daily is integrated with pmlogrewrite(1) to  al‐
31       low  optional  and  automatic rewriting of archives before merging.  If
32       there are global rewriting rules to be applied across all archives men‐
33       tioned   in   the   control   file(s),   then   create   the  directory
34       $PCP_SYSCONF_DIR/pmlogrewrite and place any  pmlogrewrite(1)  rewriting
35       rules in this directory.  For rewriting rules that are specific to only
36       one family of archives, use the directory name from the control file(s)
37       -  i.e. the fourth field - and create a file, or a directory, or a sym‐
38       bolic link named pmlogrewrite within this directory and place  the  re‐
39       quired  rewriting  rule(s)  in the pmlogrewrite file or in files within
40       the pmlogrewrite subdirectory.  pmlogger_daily  will  choose  rewriting
41       rules  from  the  archive directory if they exist, else rewriting rules
42       from $PCP_SYSCONF_DIR/pmlogrewrite if that directory  exists,  else  no
43       rewriting is attempted.
44
45       As  an  alternate  mechanism,  if  the file $PCP_LOG_DIR/pmlogger/.Nee‐
46       dRewrite exists when pmlogger_daily starts then  this  is  treated  the
47       same  as  specifying  -R  on  the  command line and $PCP_LOG_DIR/pmlog‐
48       ger/.NeedRewrite will be removed once all the rewriting has been done.
49

OPTIONS

51       -c control, --control=control
52            Both pmlogger_check and pmlogger_daily are controlled by PCP  log‐
53            ger  control  file(s)  that specifies the pmlogger instances to be
54            managed.  The default control file  is  $PCP_PMLOGGERCONTROL_PATH,
55            but an alternate may be specified using the -c option.  If the di‐
56            rectory $PCP_PMLOGGERCONTROL_PATH.d (or control.d from the -c  op‐
57            tion)  exists,  then  the contents of any additional control files
58            therein will be appended to the main control file (which must  ex‐
59            ist).
60
61       -C   This option causes pmlogger_check to query the system service run‐
62            level information for pmlogger, and use that to determine  whether
63            to start processes or not.
64
65       -E, --expunge
66            This  option  causes  pmlogger_daily to pass the -E flag to pmlog‐
67            ger_merge in order to expunge metrics with  metadata  inconsisten‐
68            cies  and  continue  rather than fail.  This is intended for auto‐
69            mated daily log rotation where it is highly  desirable  for  unat‐
70            tended  daily  archive  merging, rewriting and compression to suc‐
71            ceed.  For further details, see pmlogger_merge(1) and  description
72            for the -x flag in pmlogextract(1).
73
74       -f, --force
75            This  option  causes  pmlogger_daily to forces action.  Using this
76            option in production is not recommended.
77
78       -k time, --discard=time
79            After some period, old PCP archives are discarded.  time is a time
80            specification  in  the  syntax of find-filter(1), so DD[:HH[:MM]].
81            The optional HH (hours) and MM (minutes) parts are 0 if not speci‐
82            fied.   By  default  the  time  is  14:0:0  or 14 days, but may be
83            changed using this option.
84
85            Some special values are recognized for the time, namely 0 to  keep
86            no  archives beyond the the ones being currently written by pmlog‐
87            ger(1), and forever or never to prevent any  archives  being  dis‐
88            carded.
89
90            The time can also be set using the $PCP_CULLAFTER variable, set in
91            either the environment or in a control file.   If  both  $PCP_CUL‐
92            LAFTER  and -k specify different values for time then the environ‐
93            ment variable value is used and a warning  is  issued.   I.e.,  if
94            $PCP_CULLAFTER  is  set in the control file, it overrides -k given
95            on the command line.
96
97            Note that the semantics of time are that it is measured  from  the
98            time of last modification of each archive, and not from the origi‐
99            nal archive creation date.  This has subtle implications for  com‐
100            pression (see below) - the compression process results in the cre‐
101            ation of new archive files which have new modification times.   In
102            this  case,  the  time period (re)starts from the time of compres‐
103            sion.
104
105       -K   When this option is specified for  pmlogger_daily  then  only  the
106            compression  tasks  are  attempted,  so  no  pmlogger rotation, no
107            culling, no rewriting, etc.  When -K is used and a period of 0  is
108            in  effect  (from  -x on the command line or $PCP_COMPRESSAFTER in
109            the environment or via the control file) this is intended for  en‐
110            vironments  where  compression  of  archives is desired before the
111            scheduled daily processing happens.  To achieve this, once  pmlog‐
112            ger_check  has  completed  regular  processing,  it  calls  pmlog‐
113            ger_daily with just the -K option.  Provided $PCP_COMPRESSAFTER is
114            set  to  0  along  with  any other required compression options to
115            match the scheduled invocation of pmlogger_daily, then  this  will
116            compress  all  volumes  except the ones being currently written by
117            pmlogger(1).  If $PCP_COMPRESSAFTER is set to a value greater than
118            zero,  then manually running pmlogger_daily with the -x option may
119            be used to compress volumes that are younger  than  the  $PCP_COM‐
120            PRESSAFTER  time.  This may be used to reclaim filesystem space by
121            compressing volumes earlier than they would  have  otherwise  been
122            compressed.   Note  that  since the default value of $PCP_COMPRES‐
123            SAFTER is 0 days, the -x option has no effect unless  the  control
124            file  has  been  edited  and  $PCP_COMPRESSAFTER has been set to a
125            value greater than 0.
126
127       -l file, --logfile=file
128            In order to ensure that mail  is  not  unintentionally  sent  when
129            these  scripts are run from cron(8) diagnostics are always sent to
130            log files.  By default, this file is  $PCP_LOG_DIR/pmlogger/pmlog‐
131            ger_check.log or $PCP_LOG_DIR/pmlogger/pmlogger_daily.log but this
132            can be changed using the -l option.  If this log file already  ex‐
133            ists  when the script starts, it will be renamed with a .prev suf‐
134            fix (overwriting any log file saved  earlier)  before  diagnostics
135            are  generated  to  the log file.  The -l and -t options cannot be
136            used together.
137
138       -m addresses, --mail=addresses
139            Use of this option causes pmlogger_daily to construct a summary of
140            the  ``notices''  file entries which were generated in the last 24
141            hours, and e-mail that summary to the set of  space-separated  ad‐
142            dresses.    This   daily   summary   is   stored   in   the   file
143            $PCP_LOG_DIR/NOTICES.daily, which will be empty when no new  ``no‐
144            tices'' entries were made in the previous 24 hour period.
145
146       -M   This  option  may be used to disable archive merging (or renaming)
147            and rewriting (-M implies -r).  This is most useful in cases where
148            the  archives  are  being incrementally copied to a remote reposi‐
149            tory, e.g. using rsync(1).  Merging, renaming  and  rewriting  all
150            risk  an  increase in the synchronization load, especially immedi‐
151            ately after pmlogger_daily has run, so -M may be useful  in  these
152            cases.
153
154       -N, --showme
155            This option enables a ``show me'' mode, where the programs actions
156            are echoed, but not executed, in the style of ``make -n''.   Using
157            -N  in  conjunction  with -V maximizes the diagnostic capabilities
158            for debugging.
159
160       -o   By default all possible archives will be merged.  This option  re‐
161            instates the old behaviour in which only yesterday's archives will
162            be considered as merge candidates.  In the special case where only
163            a  single  input archive needs to be merged, pmlogmv(1) is used to
164            rename the archive, otherwise pmlogger_merge(1) is used  to  merge
165            all  of the archives for a single host and a single day into a new
166            PCP archive and the individual archives are removed.
167
168       -p, --skip-primary
169            If this option is specified for pmlogger_check then any line  from
170            the  control files for the primary pmlogger will be ignored.  This
171            option is intended for environments where some system daemon, like
172            systemd(1),  is  responsible  for controlling (starting, stopping,
173            restarting, etc.) the primary pmlogger.
174
175       -P, --only-primary
176            If this option is specified for pmlogger_check then only the  pri‐
177            mary logger entry in the control files will be processed.  This is
178            the logical opposite of the -p option described above and  is  in‐
179            tended  for  use by RC scripts that start only the primary logger,
180            such as the pmlogger.service unit.  The -p and -P options  to  pm‐
181            logger_check are mutually exclusive.
182
183       -p   If  this option is specified for pmlogger_daily then the status of
184            the daily processing is polled and if the daily pmlogger(1)  rota‐
185            tion,  culling, rewriting, compressing, etc.  has not been done in
186            the last 24 hours then it is done now.  The intent is to have  pm‐
187            logger_daily  called regularly with the -p option (at 30 mins past
188            the hour, every hour in the default  cron(8)  set  up)  to  ensure
189            daily  processing  happens as soon as possible if it was missed at
190            the regularly scheduled time (which is 00:10 by default), e.g.  if
191            the  system  was down or suspended at that time.  With this option
192            pmlogger_daily simply exits if the previous day's  processing  has
193            already been done.  Note that this option is not used on platforms
194            supporting systemd(1)  because  the  pmlogger_daily.timer  service
195            unit  specifies  a timer setting with Persistent=true.  The -K and
196            -p options to pmlogger_daily are mutually exclusive.
197
198       -q, --quick
199            If this option is specified for  pmlogger_check  then  the  script
200            will  ``quickstart''  avoiding  any  optional processing like file
201            compression.
202
203       -r, --norewrite
204            This command line option acts as an override and prevents all  ar‐
205            chive  rewriting  with pmlogrewrite(1) independent of the presence
206            of any rewriting rule files or directories.
207
208       -R, --rewriteall
209            Sometimes PMDA changes require all archives to be  rewritten,  not
210            just  the  ones involved in any current merging.  This is required
211            for example after a PCP upgrade where a new version of an existing
212            PMDA  has  revised metadata.  The -R command line forces this uni‐
213            versal-style of rewriting.  The -R option to pmlogger_daily is mu‐
214            tually exclusive with both the -r and -M options.
215
216       -s size, --rotate=size
217            If  the PCP ``notices'' file ($PCP_LOG_DIR/NOTICES) is larger than
218            20480 bytes, pmlogger_daily will rename the file with  a  ``.old''
219            suffix,  and  start  a new ``notices'' file.  The rotate threshold
220            may be changed from 20480 to size bytes using the -s option.
221
222       -s, --stop
223            Use of this option provides the reverse pmlogger_check functional‐
224            ity,  allowing  the  set of pmlogger processes to be cleanly shut‐
225            down.
226
227       -t period
228            To assist with debugging or diagnosing intermittent  failures  the
229            -t  option  may  be  used.  This will turn on very verbose tracing
230            (-VV)  and  capture   the   trace   output   in   a   file   named
231            $PCP_LOG_DIR/pmlogger/daily.datestamp.trace,  where  datestamp  is
232            the time pmlogger_daily was run in the format YYYYMMDD.HH.MM.   In
233            addition, the period argument will ensure that trace files created
234            with -t will be kept for period days and then discarded.
235
236       -T, --terse
237            This option to pmlogger_check produces less  verbose  output  than
238            the  default.  This is most suitable for a pmlogger ``farm'' where
239            many instances of pmlogger are expected to be running.
240
241       -V, --verbose
242            The output from the cron execution of the scripts may be  extended
243            using the -V option to the scripts which will enable verbose trac‐
244            ing of their activity.  By default the scripts generate no  output
245            unless  some  error or warning condition is encountered.  Using -N
246            in conjunction with -V maximizes the diagnostic  capabilities  for
247            debugging.
248
249       -x time, --compress-after=time
250            Archive  data files can optionally be compressed after some period
251            to conserve disk space.  This is  particularly  useful  for  large
252            numbers of pmlogger processes under the control of pmlogger_check.
253
254            time  is  a time specification in the syntax of find-filter(1), so
255            DD[:HH[:MM]].  The optional HH (hours) and MM (minutes) parts  are
256            0 if not specified.
257
258            Some special values are recognized for the time, namely 0 to apply
259            compression as soon as possible, and forever or never  to  prevent
260            any compression being done.
261
262            If transparent_decompress is enabled when libpcp was built (can be
263            checked with the pmconfig(1) -L option), then the  default  behav‐
264            iour  is  compression  ``as soon as possible''.  Otherwise the de‐
265            fault behaviour is to not compress files (which matches  the  his‐
266            torical default behaviour in earlier PCP releases).
267
268            The  time  can  also be set using the $PCP_COMPRESSAFTER variable,
269            set in either the environment or  in  a  control  file.   If  both
270            $PCP_COMPRESSAFTER  and  -x specify different values for time then
271            the environment variable value is used and a  warning  is  issued.
272            For  important other detailed notes concerning volume compression,
273            see the -K and -k options (above).
274
275       -X program, --compressor=program
276            This option specifies the program to use for compression - by  de‐
277            fault  this  is xz(1).  The environment variable $PCP_COMPRESS may
278            be used as an alternative mechanism to define  program.   If  both
279            $PCP_COMPRESS  and  -X specify different compression programs then
280            the environment variable value is used and a warning is issued.
281
282       -Y regex, --regex=regex
283            This option allows a regular expression to  be  specified  causing
284            files  in the set of files matched for compression to be omitted -
285            this allows only the data file to be compressed, and also prevents
286            the  program  from  attempting to compress it more than once.  The
287            default regex is ".(index|Z|gz|bz2|zip|xz|lzma|lzo|lz4)$"  -  such
288            files  are filtered using the -v option to egrep(1).  The environ‐
289            ment variable $PCP_COMPRESSREGEX may be  used  as  an  alternative
290            mechanism  to  define  regex.   If  both $PCP_COMPRESSREGEX and -Y
291            specify different values for regex then the  environment  variable
292            value is used and a warning is issued.
293
294       -?, --help
295            Display usage message and exit.
296

CONFIGURATION

298       Warning:  The  $PCP_PMLOGGERCONTROL_PATH  file  and  files  within  the
299       $PCP_PMLOGGERCONTROL_PATH.d directory must not be writable by any  user
300       other than root.
301
302       The  control  file(s)  should  be customized according to the following
303       rules that define for the current version (1.1)  of  the  control  file
304       format.
305
306       1.  Lines beginning with a ``#'' are comments.  A special case is lines
307           beginning ``#!#''; these are control lines for a  pmlogger(1)  that
308           has been stopped using pmlogctl(1).
309       2.  Lines beginning with a ``$'' are assumed to be assignments to envi‐
310           ronment variables in the style of sh(1), and all text following the
311           ``$''  will  be eval'ed by the script reading the control file, and
312           the corresponding variable exported into the environment.  This  is
313           particularly  useful  to set and export variables into the environ‐
314           ment of the administrative scripts, e.g.
315               $ PMCD_CONNECT_TIMEOUT=20
316       3.  There must be a version line in the initial  control  file  of  the
317           form:
318               $ version=1.1
319       4.  There  should  be one line in the control file(s) for each pmlogger
320           instance of the form:
321
322               host y|n y|n directory args
323
324       5.  Fields within a line of the control file(s) are  usually  separated
325           by one or more spaces or tabs (although refer to the description of
326           the directory field for some important exceptions).
327       6.  The first field is the name of the host that is the source  of  the
328           performance metrics for this pmlogger instance.
329       7.  The  second  field indicates if this is a primary pmlogger instance
330           (y) or not (n).  Since the primary logger must  run  on  the  local
331           host,  and there may be at most one primary logger for a particular
332           host, this field can be y for at most  one  pmlogger  instance,  in
333           which case the host name must be the name of the local host.
334       8.  The  third  field  indicates  if this pmlogger instance needs to be
335           started under the control  of  pmsocks(1)  to  connect  to  a  pmcd
336           through a firewall (y or n).
337       9.  The  fourth  field  is a directory name.  All files associated with
338           this pmlogger instance will be created in this directory, and  this
339           will be the current directory for the execution of any programs re‐
340           quired in the maintenance of those archives.  A  useful  convention
341           is  that  primary  logger archives for the local host with hostname
342           myhost are  maintained  in  the  directory  $PCP_ARCHIVE_DIR/myhost
343           (this   is   where   the   default   pmlogger  start-up  script  in
344           $PCP_RC_DIR/pcp will create the archives), while archives  for  the
345           remote host mumble are maintained in $PCP_ARCHIVE_DIR/mumble.
346       10. The  directory field may contain embedded shell syntax that will be
347           evaluated by sh(1) to produce the real directory name to  be  used.
348           The allowed constructs are:
349           • Any text (including white space) enclosed with $( and ).
350           • Any  text  (including  white  space)  enclosed with ` and ` (back
351             quotes).
352           • Any text (including white space) enclosed with "  and  "  (double
353             quotes).
354           • Any  word  containing  a  $  (assumed to introduce an environment
355             variable name).
356       11. All other fields are interpreted as arguments to be passed  to  pm‐
357           logger(1).  Most typically this would be the -c option.
358
359       The  following sample control lines specify a primary logger on the lo‐
360       cal host (bozo), and non-primary loggers to collect and log performance
361       metrics from the hosts wobbly and boing.
362
363       $version=1.1
364       bozo   y  n  $PCP_ARCHIVE_DIR/bozo   -c config.default
365       wobbly n  n  "/store/wobbly/$(date +%Y)"  -c ./wobbly.config
366       boing  n  n  $PCP_ARCHIVE_DIR/boing  -c ./pmlogger.config
367
368       Typical crontab(5) entries for periodic execution of pmlogger_daily and
369       pmlogger_check are given in  $PCP_SYSCONF_DIR/pmlogger/crontab  (unless
370       installed by default in /etc/cron.d already) and shown below.
371
372       # daily processing of archive logs
373       14      0       *       *       *       $PCP_BINADM_DIR/pmlogger_daily
374       # every 30 minutes, check pmlogger instances are running
375       25,55   *       *       *       *       $PCP_BINADM_DIR/pmlogger_check
376
377       When  using  systemd(1)  on Linux, no crontab entries are needed as the
378       timer mechanism provided by systemd is used instead.
379

FILES

381       $PCP_PMLOGGERCONTROL_PATH
382            the PCP logger control file.  For a  new  installation  this  file
383            contains  no pmlogger(1) control lines (the real control files are
384            all in the $PCP_PMLOGGERCONTROL_PATH.d directory), but  this  file
385            is  still  processed  to support any legacy configurations therein
386            from earlier PCP releases.
387            Warning: this file must not be writable by  any  user  other  than
388            root.
389
390       $PCP_PMLOGGERCONTROL_PATH.d
391            optional directory containing additional PCP logger control files,
392            typically one per host
393            Warning: the files herein must not be writable by any  user  other
394            than root.
395
396       $PCP_SYSCONF_DIR/pmlogger/crontab
397            sample  crontab  for  automated  script execution by $PCP_USER (or
398            root).   Exists  only  if  the  platform  does  not  support   the
399            /etc/cron.d mechanism.
400
401       $PCP_VAR_DIR/config/pmlogger/config.default
402            default pmlogger configuration file location for the local primary
403            logger, typically generated automatically by pmlogconf(1).
404
405       $PCP_ARCHIVE_DIR/<hostname>
406            default location for archives of performance information collected
407            from the host hostname
408
409       $PCP_ARCHIVE_DIR/<hostname>/lock
410            transient  lock file to guarantee mutual exclusion during pmlogger
411            administration for the host hostname - if present, can  be  safely
412            removed if neither pmlogger_daily nor pmlogger_check are running
413
414       $PCP_ARCHIVE_DIR/<hostname>/Latest
415            PCP  archive  folio  created  by  mkaf(1)  for  the  most recently
416            launched archive containing  performance  metrics  from  the  host
417            hostname
418
419       $PCP_LOG_DIR/NOTICES
420            PCP ``notices'' file used by pmie(1) and friends
421
422       $PCP_LOG_DIR/pmlogger/pmlogger_check.log
423            if the previous execution of pmlogger_check produced any output it
424            is saved here.  The normal case is no output  in  which  case  the
425            file does not exist.
426
427       $PCP_LOG_DIR/pmlogger/pmlogger_daily.log
428            if the previous execution of pmlogger_daily produced any output it
429            is saved here.  The normal case is no output  in  which  case  the
430            file does not exist.
431
432       $PCP_ARCHIVE_DIR/<hostname>/SaveLogs
433            if  this  directory exists, then the log file from the -l argument
434            of a newly launched pmlogger(1) for hostname will be  linked  into
435            this  directory  with  the  name  archive.log where archive is the
436            basename of the associated pmlogger(1) PCP  archive  files.   This
437            allows  the log file to be inspected at a later time, even if sev‐
438            eral pmlogger(1) instances for hostname have been launched in  the
439            interim.   Because  the cron-driven PCP archive management scripts
440            run under the uid  of  the  user  ``pcp'',  $PCP_ARCHIVE_DIR/host‐
441            name/SaveLogs typically needs to be owned by the user ``pcp''.
442
443       $PCP_LOG_DIR/pmlogger/.NeedRewrite
444            if  this  file exists, then this is treated as equivalent to using
445            -R on the command line and the  file  will  be  removed  once  all
446            rewriting has been done.
447

PCP ENVIRONMENT

449       Environment variables with the prefix PCP_ are used to parameterize the
450       file and directory names used by PCP.  On each installation,  the  file
451       /etc/pcp.conf  contains  the  local  values  for  these variables.  The
452       $PCP_CONF variable may be used to specify an alternative  configuration
453       file, as described in pcp.conf(5).
454
455       The default behaviour, when pmlogger(1) configuration comes from pmlog‐
456       conf(1), is to regenerate the configuration file and check for  changes
457       whenever  pmlogger(1) is started from pmlogger_check.  If the PMDA con‐
458       figuration is stable,  this  is  not  necessary,  and  setting  $PMLOG‐
459       GER_CHECK_SKIP_LOGCONF to yes disables the regeneration and checking.
460

COMPATIBILITY ISSUES

462       Earlier  versions  of  pmlogger_daily  used find(1) to locate files for
463       compressing or culling and the -k and -x options took only integer val‐
464       ues to mean ``days''.  The semantics of this was quite loose given that
465       find(1) offers different precision and semantics across platforms.
466
467       The current implementation of pmlogger_daily uses find-filter(1)  which
468       provides  high  precision  intervals and semantics that are relative to
469       the time of execution and are consistent across platforms.
470

SEE ALSO

472       egrep(1), find-filter(1),  PCPIntro(1),  pmconfig(1),  pmlc(1),  pmlog‐
473       conf(1),  pmlogctl(1),  pmlogger(1),  pmlogger_daily_report(1),  pmlog‐
474       ger_merge(1), pmlogextract(1), pmlogmv(1), pmlogrewrite(1), pmsocks(1),
475       systemd(1), xz(1) and cron(8).
476
477
478
479Performance Co-Pilot                  PCP                    PMLOGGER_CHECK(1)
Impressum