1PMLOGGER(1) General Commands Manual PMLOGGER(1)
2
6 pmlogger - create archive log for performance metrics
7
9 pmlogger [-c configfile] [-h host] [-K spec] [-l logfile] [-L] [-m
10 note] [-n pmnsfile] [-o] [-p pid] [-P] [-r] [-s endsize] [-t interval]
11 [-T endtime] [-u] [-U username] [-v volsize] [-V version] [-x fd] [-y]
12 archive
13
15 pmlogger creates the archive logs of performance metric values that may
16 be ``played back'' by other Performance Co-Pilot (see PCPIntro(1))
17 tools. These logs form the basis of the VCR paradigm and retrospective
18 performance analysis services common to the PCP toolkit.
19 The mandatory argument archive is the base name for the physical files
21 that constitute an archive log.
22 The -V option specifies the version for the archive that is generated.
24 By default a version 2 archive is generated, and the only value cur‐
25 rently supported for version is 2.
26 Unless directed to another host by the -h option or when directly using
28 PMDAs via the -o option, pmlogger will contact the Performance Metrics
29 Collector Daemon (PMCD) on the local host and use that as the source of
30 the metric values to be logged.
31 To support the required flexibility and control over what is logged and
33 when, pmlogger maintains an independent two level logging state for
34 each instance of each performance metric. At the first (mandatory)
35 level, logging is allowed to be on (with an associated interval between
36 samples), or off or maybe. In the latter case, the second (advisory)
37 level logging is allowed to be on (with an associated interval between
38 samples), or off.
39 The mandatory level allows universal specification that some metrics
41 must be logged, or must not be logged. The default state for all
42 instances of all metrics when pmlogger starts is mandatory maybe and
43 advisory off.
44 Use pmlc(1) to interrogate and change the logging state once pmlogger
46 is running.
47 If a metric's state is mandatory (on or off) and a request is made to
49 change it to mandatory maybe, the new state is mandatory maybe and
50 advisory off. If a metric's state is already advisory (on or off) and
51 a request is made to change it to mandatory maybe, the current state is
52 retained.
53 It is not possible for pmlogger to log specific instances of a metric
55 and all instances of the same metric concurrently. If specific
56 instances are being logged and a request to log all instances is made,
57 then all instances of the metric will be logged according to the new
58 request, superseding any prior logging request for the metric. A
59 request to log all instances of a metric will supersede any previous
60 request to log all instances. A request to log specific instances of a
61 metric when all instances are already being logged is refused. To do
62 this one must turn off logging for all instances of the metric first.
63 In each case, the validity of the request is checked first; for example
64 a request to change a metric's logging state to advisory on when it is
65 currently mandatory off is never permitted (it is necessary to change
66 the state to mandatory maybe first).
67 Optionally, each system running pmcd(1) may also be configured to run a
69 ``primary'' pmlogger instance. This pmlogger instance is launched by
70 $PCP_RC_DIR/pmlogger, and is affected by the files
71 $PCP_SYSCONF_DIR/pmlogger/control, $PCP_SYSCONF_DIR/pmlogger/control.d
72 (use chkconfig(8) or similar platform-specific commands to activate or
73 disable the primary pmlogger instance), $PCP_SYSCONFIG_DIR/pmlogger
74 (environment variable settings for the primary pmlogger)
75 $PCP_SYSCONF_DIR/pmlogger/pmlogger.options (command line options passed
76 to the primary pmlogger) and $PCP_VAR_LIB/config/pmlogger/con‐
77 fig.default (the default initial configuration file for the primary
78 pmlogger).
79 The primary pmlogger instance is identified by the -P option. There
81 may be at most one ``primary'' pmlogger instance on each system. The
82 primary pmlogger instance (if any) must be running on the same host as
83 the pmcd(1) to which it connects (if any), so the -h and -P options are
84 mutually exclusive.
85 Logging of some metrics is possible even in the absence of a local
87 pmcd(1), using the "local context" mode of operation. This is acti‐
88 vated using the -o option, and causes pmlogger to make use of local DSO
89 PMDAs instead of communicating with pmcd(1). When operating using a
90 local context, the -K option may be used to control the DSO PMDAs that
91 should be made accessible. The spec argument conforms to the syntax
92 described in __pmSpecLocalPMDA(3). More than one -K option may be
93 used.
94 When launched as a non-primary instance, pmlogger will exit immediately
96 if the configuration file causes no metric logging to be scheduled.
97 The -L option overrides this behavior, and causes a non-primary pmlog‐
98 ger instance to ``linger'', presumably pending some future dynamic re-
99 configuration and state change via pmlc(1). pmlogger will also linger
100 without the -L option being used if all the metrics to be logged are
101 logged as once only metrics. When the once only metrics have been
102 logged, a warning message will be generated stating that the event
103 queue is empty and no more events will be scheduled.
104 By default all diagnostics and errors from pmlogger are written to the
106 file pmlogger.log in the directory where pmlogger is launched. The -l
107 option may be used to override the default behavior. If the log file
108 cannot be created or is not writable, output is written to standard
109 error instead.
110 If specified, the -s option instructs pmlogger to terminate after a
112 certain size in records, bytes or time units has been accumulated. If
113 endsize is an integer then endsize records will be written to the log.
114 If endsize is an integer suffixed by b or bytes then endsize bytes of
115 the archive data will be written out (note, however, that archive log
116 record boundaries will not be broken and so this limit may be slightly
117 surpassed). Other viable file size units include: K, Kb, Kbyte, Kilo‐
118 byte for kilobytes and M, Mb, Mbyte, Megabyte for megabytes and G, Gb,
119 Gbyte, Gigabyte for gigabytes. These units may be optionally suffixed
120 by an s and may be of mixed case. Alternatively endsize may be an
121 integer or a floating point number suffixed using a time unit as
122 described in PCPIntro(1) for the interval argument (to the standard PCP
123 -t command line option).
124 Some examples of different formats:
125 -s 100
126 -s 100bytes
127 -s 100K
128 -s 100Mb
129 -s 10Gbyte
130 -s 10mins
131 -s 1.5hours
132 The default is for pmlogger to run forever.
133 The -r option causes the size of the physical record(s) for each group
135 of metrics and the expected contribution of the group to the size of
136 the PCP archive for one full day of collection to be reported in the
137 log file. This information is reported the first time each group is
138 successfully written to the archive.
139 The -U option specifies the user account under which to run pmlogger.
141 The default is the current user account for interactive use. When run
142 as a daemon, the unprivileged "pcp" account is used in current versions
143 of PCP, but in older versions the superuser account ("root") was used
144 by default.
145 The log file is potentially a multi-volume data set, and the -v option
147 causes pmlogger to start a new volume after a certain size in records,
148 bytes, or time units has been accumulated for the current volume. The
149 format of this size specification is identical to that of the -s option
150 (see above). The default is for pmlogger to create a single volume
151 log. Additional volume switches can also be forced asynchronously by
152 either using pmlc(1) or sending pmlogger a SIGHUP signal (see below).
153 Note, if a scheduled volume switch is in operation due to the -v
154 option, then its counters will be reset after an asynchronous switch.
155 Independent of any -v option, each volume of an archive is limited to
157 no more than 2^31 bytes, so pmlogger will automatically create a new
158 volume for the archive before this limit is reached.
159 Normally pmlogger operates on the distributed Performance Metrics Name
161 Space (PMNS), however if the -n option is specified an alternative
162 local PMNS is loaded from the file pmnsfile.
163 Under normal circumstances, pmlogger will run forever (except for a -s
165 option or a termination signal). The -T option may be used to limit
166 the execution time using the format of time as prescribed by PCPIn‐
167 tro(1). The time is interpreted within the time zone of the PMCD
168 server, unless the -y option is given, within which case the time zone
169 at this logger host is used.
170 Some examples of different formats:
171 -T 10mins
172 -T '@ 11:30'
173 From this it can be seen that -T 10mins and -s 10mins perform identical
174 actions.
175 Alternatively, pmlogger runtime may be limited to the lifetime of
177 another process by using the -p or --PID option to nominate the PID of
178 the process of interest. In this case the pmlogger will exit when the
179 other process no longer exists.
180 When pmlogger receives a SIGHUP signal, the current volume of the log
182 is closed, and a new volume is opened. This mechanism (or the alterna‐
183 tive mechanism via pmlc(1)) may be used to manage the growth of the log
184 files - once a log volume is closed, that file may be archived without
185 ill-effect on the continued operation of pmlogger. See also the -v
186 option above.
187 Historically the buffers for the current log may be flushed to disk
189 using the flush command of pmlc(1), or by sending pmlogger a SIGUSR1
190 signal or by using the -u option. The current version of pmlogger and
191 the libpcp routines that underpin pmlogger unconditionally use
192 unbuffered writes and a single fwrite(3) for each logical record writ‐
193 ten, and so ``flushing'' does not force any additional data to be writ‐
194 ten to the file system. The -u option, the SIGUSR1 handling and the
195 pmlc(1) flush command are retained for backwards compatibility.
196 When launched with the -x option, pmlogger will accept asynchronous
198 control requests on the file descriptor fd. This option is only
199 expected to be used internally by PCP applications that support ``live
200 record mode''.
201 The -m option allows the string note to be appended to the map file for
203 this instance of pmlogger in the $PCP_TMP_DIR/pmlogger directory. This
204 is currently used internally to document the file descriptor (fd) when
205 the -x option is used, or to indicate that this pmlogger instance was
206 started under the control of pmlogger_check(1).
207
209 The configuration file may be specified with the -c option. If it is
210 not, configuration specifications are read from standard input.
211 If configfile does not exist, then a search is made in the directory
213 $PCP_VAR_LIB/config/pmlogger for a file of the same name, and if found
214 that file is used, e.g. if config.mumble does not exist in the current
215 directory and the file $PCP_VAR_LIB/config/pmlogger/config.mumble does
216 exist, then -c config.mumble and -c $PCP_VAR_LIB/config/pmlogger/con‐
217 fig.mumble are equivalent.
218 The syntax for the configuration file is as follows.
220 1. Words are separated by white space (space, tab or newline).
222 2. The symbol ``#'' (hash) introduces a comment, and all text up to
224 the next newline is ignored.
225 3. Keywords (shown in bold below) must appear literally (i.e. in
227 lower case).
228 4. Each specification begins with the optional keyword log, followed
230 by one of the states mandatory on, mandatory off, mandatory maybe,
231 advisory on or advisory off.
232 5. For the on states, a logging interval must follow using the syntax
234 ``once'', or ``default'', or ``every N timeunits'', or simply ``N
235 timeunits'' - N is an unsigned integer, and timeunits is one of
236 the keywords msec, millisecond, sec, second, min, minute, hour or
237 the plural form of one of the above.
238 Internal limitations require the interval to be smaller than
239 (approximately) 74 hours. An interval value of zero is a synonym
240 for once. An interval of default means to use the default logging
241 interval of 60 seconds; this default value may be changed to
242 interval with the -t command line option.
243 The interval argument follows the syntax described in PCPIntro(1),
245 and in the simplest form may be an unsigned integer (the implied
246 units in this case are seconds).
247 6. Following the state and possible interval specifications comes a
249 ``{'', followed by a list of one or more metric specifications and
250 a closing ``}''. The list is white space (or comma) separated.
251 If there is only one metric specification in the list, the braces
252 are optional.
253 7. A metric specification consists of a metric name optionally fol‐
255 lowed by a set of instance names. The metric name follows the
256 standard PCP naming conventions, see pmns(5), and if the metric
257 name is a non-leaf node in the PMNS (see pmns(5)), then pmlogger
258 will recursively descend the PMNS and apply the logging specifica‐
259 tion to all descendent metric names that are leaf nodes in the
260 PMNS. The set of instance names is a ``['', followed by a list of
261 one or more space (or comma) separated names, numbers or strings,
262 and a closing ``]''. Elements in the list that are numbers are
263 assumed to be internal instance identifiers, other elements are
264 assumed to be external instance identifiers - see pmGetInDom(3)
265 for more information.
266 If no instances are given, then the logging specification is
268 applied to all instances of the associated metric.
269 8. There may be an arbitrary number of logging specifications.
271 9. Following all of the logging specifications, there may be an
273 optional access control section, introduced by the literal token
274 [access]. Thereafter come access control rules that allow or dis‐
275 allow operations from particular hosts or groups of hosts.
276 The operations may be used to interrogate or control a running
278 pmlogger using pmlc(1) and fall into the following classes:
279 enquire interrogate the status of pmlogger and the metrics
281 it is logging
282 advisory Change advisory logging.
283 mandatory Change mandatory logging.
284 all All of the above.
285 Access control rules are of the form ``allow hostlist : opera‐
287 tionlist ;'' and ``disallow hostlist : operationlist ;''.
288 The hostlist follows the syntax and semantics for the access con‐
290 trol mechanisms used by PMCD and are fully documented in pmcd(1).
291 An operationslist is a comma separated list of the operations
292 advisory, mandatory, enquire and all.
293 A missing [access] section allows all access and is equivalent to
295 allow * : all;.
296 The configuration (either from standard input or configfile) is ini‐
298 tially scanned by pmcpp(1) with the options -rs and -I
299 $PCP_VAR_LIB/config/pmlogger. This extends the configuration file syn‐
300 tax with include file processing (%include), a common location to
301 search for include files ($PCP_VAR_LIB/config/pmlogger), macro defini‐
302 tions (%define), macro expansion (%name and %{name}) and conditional
303 inclusion of lines (%ifdef name ... %else ... %endif and %ifndef name
304 ... %else ... %endif).
305
307 For each PCP utility, there is a sample pmlogger configuration file
308 that could be used to create an archive log suitable for replaying with
309 that tool (i.e. includes all of the performance metrics used by the
310 tool). For a tool named foo this configuration file is located in
311 $PCP_VAR_LIB/config/pmlogger/config.foo.
312 The following is a simple default configuration file for a primary
314 pmlogger instance, and demonstrates most of the capabilities of the
315 configuration specification language.
316 log mandatory on once { hinv.ncpu hinv.ndisk }
318 log mandatory on every 10 minutes {
319 disk.all.write
320 disk.all.read
321 network.interface.in.packets [ "et0" ]
322 network.interface.out.packets [ "et0" ]
323 nfs.server.reqs [ "lookup" "getattr" "read" "write" ]
324 }
325 log advisory on every 30 minutes {
327 environ.temp
328 pmcd.pdu_in.total
329 pmcd.pdu_out.total
330 }
331 %include "macros.default"
333 %ifdef %disk_detail
335 log mandatory on %disk_detail_freq {
336 disk.dev
337 }
338 %endif
339 [access]
341 disallow * : all except enquire;
342 allow localhost : mandatory, advisory;
343
345 archive.meta
346 metadata (metric descriptions, instance domains, etc.) for
347 the archive log
348 archive.0 initial volume of metrics values (subsequent volumes have
349 suffixes 1, 2, ...)
350 archive.index
351 temporal index to support rapid random access to the other
352 files in the archive log
353 $PCP_TMP_DIR/pmlogger
354 pmlogger maintains the files in this directory as the map
355 between the process id of the pmlogger instance and the IPC
356 port that may be used to control each pmlogger instance (as
357 used by pmlc(1))
358 $PCP_VAR_LIB/config/pmlogger/config.default
359 default configuration file for the primary logger instance
360 launched from $PCP_RC_DIR/pmlogger
361 $PCP_VAR_LIB/config/pmlogger/config.*
362 assorted configuration files suitable for creating logs that
363 may be subsequently replayed with the PCP visualization and
364 monitoring tools
365 $PCP_LOG_DIR/pmlogger/hostname
366 Default directory for PCP archive files for performance met‐
367 ric values collected from the host hostname.
368 $PCP_SYSCONFIG_DIR/pmlogger
369 additional environment variables that will be set when the
370 primary pmlogger instance executes. Only settings of the
371 form "PMLOGGER_VARIABLE=value" will be honoured.
372 ./pmlogger.log
373 (or $PCP_LOG_DIR/pmlogger/hostname/pmlogger.log when started
374 automatically by either $PCP_RC_DIR/pmlogger or one of the
375 pmlogger(1) monitoring scripts such as pmlogger_check(1))
376 all messages and diagnostics are directed here
377
379 Normally pmlogger creates a socket to receive control messages from
380 pmlc(1) on the first available TCP/IP port numbered 4330 or higher.
381 The environment variable PMLOGGER_PORT may be used to specify an alter‐
382 native starting port number.
383 If set to the value 1, the PMLOGGER_LOCAL environment variable will
385 cause pmlogger to run in a localhost-only mode of operation, where it
386 binds only to the loopback interface.
387 The PMLOGGER_MAXPENDING variable can be set to indicate the maximum
389 length to which the queue of pending pmlc connections may grow.
390
392 Environment variables with the prefix PCP_ are used to parameterize the
393 file and directory names used by PCP. On each installation, the file
394 /etc/pcp.conf contains the local values for these variables. The
395 $PCP_CONF variable may be used to specify an alternative configuration
396 file, as described in pcp.conf(5).
397
399 PCPIntro(1), pmcd(1), pmdumplog(1), pmlc(1), pmlogger_check(1),
400 __pmSpecLocalPMDA(3), pcp.conf(5), pcp.env(5), pmns(5) and chkcon‐
401 fig(8).
402
404 The archive logs are sufficiently precious that pmlogger will not trun‐
405 cate an existing physical file. A message of the form
406 __pmLogNewFile: "foo.index" already exists, not over-written
407 __pmLogCreate: File exists
408 indicates this situation has arisen. You must explicitly remove the
409 files and launch pmlogger again.
410 There may be at most one primary pmlogger instance per monitored host;
412 attempting to bend this rule produces the error:
413 pmlogger: there is already a primary pmlogger running
414 Various other messages relating to the creation and/or deletion of
416 files in $PCP_TMP_DIR/pmlogger suggest a permission problem on this
417 directory, or some feral files have appeared therein.
418Performance Co-Pilot PCP PMLOGGER(1)