1PMLOGGER(1) General Commands Manual PMLOGGER(1)
2
3
4
6 pmlogger - create archive log for performance metrics
7
9 pmlogger [-CLoPruy?] [-c conffile] [-h host] [-H hostname] [-K spec]
10 [-l logfile] [-m note] [-n pmnsfile] [-p pid] [-s endsize] [-t inter‐
11 val] [-T endtime] [-U username] [-v volsize] [-V version] [-x fd] ar‐
12 chive
13
15 pmlogger creates the archive logs of performance metric values that may
16 be ``played back'' by other Performance Co-Pilot (see PCPIntro(1))
17 tools. These logs form the basis of the VCR paradigm and retrospective
18 performance analysis services common to the PCP toolkit.
19
20 The mandatory argument archive is the base name for the physical files
21 that constitute an archive log.
22
23 The -V option specifies the version for the archive that is generated.
24 By default a version 2 archive is generated, and the only value cur‐
25 rently supported for version is 2.
26
27 Unless directed to another host by the -h option or when directly using
28 PMDAs via the -o option, pmlogger will contact the Performance Metrics
29 Collector Daemon (PMCD) on the local host and use that as the source of
30 the metric values to be logged.
31
32 To support the required flexibility and control over what is logged and
33 when, pmlogger maintains an independent two level logging state for
34 each instance of each performance metric. At the first (mandatory)
35 level, logging is allowed to be on (with an associated interval between
36 samples), or off or maybe. In the latter case, the second (advisory)
37 level logging is allowed to be on (with an associated interval between
38 samples), or off.
39
40 The mandatory level allows universal specification that some metrics
41 must be logged, or must not be logged. The default state for all
42 instances of all metrics when pmlogger starts is mandatory maybe and
43 advisory off.
44
45 Use pmlc(1) to interrogate and change the logging state once pmlogger
46 is running.
47
48 If a metric's state is mandatory (on or off) and a request is made to
49 change it to mandatory maybe, the new state is mandatory maybe and
50 advisory off. If a metric's state is already advisory (on or off) and
51 a request is made to change it to mandatory maybe, the current state is
52 retained.
53
54 It is not possible for pmlogger to log specific instances of a metric
55 and all instances of the same metric concurrently. If specific
56 instances are being logged and a request to log all instances is made,
57 then all instances of the metric will be logged according to the new
58 request, superseding any prior logging request for the metric. A
59 request to log all instances of a metric will supersede any previous
60 request to log all instances. A request to log specific instances of a
61 metric when all instances are already being logged is refused. To do
62 this one must turn off logging for all instances of the metric first.
63 In each case, the validity of the request is checked first; for example
64 a request to change a metric's logging state to advisory on when it is
65 currently mandatory off is never permitted (it is necessary to change
66 the state to mandatory maybe first).
67
68 Optionally, each system running pmcd(1) may also be configured to run a
69 ``primary'' pmlogger instance. This pmlogger instance is launched by
70 $PCP_RC_DIR/pmlogger, and is affected by the files
71 $PCP_SYSCONF_DIR/pmlogger/control, $PCP_SYSCONF_DIR/pmlogger/control.d
72 (use chkconfig(8), systemctl(1) or similar platform-specific commands
73 to activate or disable the primary pmlogger instance), $PCP_SYSCON‐
74 FIG_DIR/pmlogger (environment variable settings for the primary pmlog‐
75 ger) $PCP_SYSCONF_DIR/pmlogger/pmlogger.options (command line options
76 passed to the primary pmlogger) and $PCP_VAR_DIR/config/pmlogger/con‐
77 fig.default (the default initial configuration file for the primary
78 pmlogger).
79
80 The primary pmlogger instance is identified by the -P option. There
81 may be at most one ``primary'' pmlogger instance on each system. The
82 primary pmlogger instance (if any) must be running on the same host as
83 the pmcd(1) to which it connects (if any), so the -h and -P options are
84 mutually exclusive.
85
86 Logging of some metrics is possible even in the absence of a local
87 pmcd(1), using the "local context" mode of operation. This is acti‐
88 vated using the -o option, and causes pmlogger to make use of local DSO
89 PMDAs instead of communicating with pmcd(1). When operating using a
90 local context, the -K option may be used to control the DSO PMDAs that
91 should be made accessible. The spec argument conforms to the syntax
92 described in pmSpecLocalPMDA(3). More than one -K option may be used.
93
94 When launched as a non-primary instance, pmlogger will exit immediately
95 if the configuration file causes no metric logging to be scheduled.
96 The -L option overrides this behavior, and causes a non-primary pmlog‐
97 ger instance to ``linger'', presumably pending some future dynamic re-
98 configuration and state change via pmlc(1). pmlogger will also linger
99 without the -L option being used if all the metrics to be logged are
100 logged as once only metrics. When the once only metrics have been
101 logged, a warning message will be generated stating that the event
102 queue is empty and no more events will be scheduled.
103
104 By default all diagnostics and errors from pmlogger are written to the
105 file pmlogger.log in the directory where pmlogger is launched. The -l
106 option may be used to override the default behavior. If the log file
107 cannot be created or is not writable, output is written to standard
108 error instead.
109
110 If specified, the -s option instructs pmlogger to terminate after a
111 certain size in records, bytes or time units has been accumulated. If
112 endsize is an integer then endsize records will be written to the log.
113 If endsize is an integer suffixed by b or bytes then endsize bytes of
114 the archive data will be written out (note, however, that archive log
115 record boundaries will not be broken and so this limit may be slightly
116 surpassed). Other viable file size units include: K, Kb, KiB, Kbyte,
117 Kilobyte for kilobytes and M, Mb, MiB, Mbyte, Megabyte for megabytes
118 and G, Gb, GiB, Gbyte, Gigabyte for gigabytes. These units may be
119 optionally suffixed by an s and may be of mixed case. Alternatively
120 endsize may be an integer or a floating point number suffixed using a
121 time unit as described in PCPIntro(1) for the interval argument (to the
122 standard PCP -t command line option).
123 Some examples of different formats:
124 -s 100
125 -s 100bytes
126 -s 100K
127 -s 100Mb
128 -s 10Gbyte
129 -s 10mins
130 -s 1.5hours
131 The default is for pmlogger to run forever.
132
133 The -r option causes the size of the physical record(s) for each group
134 of metrics and the expected contribution of the group to the size of
135 the PCP archive for one full day of collection to be reported in the
136 log file. This information is reported the first time each group is
137 successfully written to the archive.
138
139 The -U option specifies the user account under which to run pmlogger.
140 The default is the current user account for interactive use. When run
141 as a daemon, the unprivileged "pcp" account is used in current versions
142 of PCP, but in older versions the superuser account ("root") was used
143 by default.
144
145 The log file is potentially a multi-volume data set, and the -v option
146 causes pmlogger to start a new volume after a certain size in records,
147 bytes, or time units has been accumulated for the current volume. The
148 format of this size specification is identical to that of the -s option
149 (see above). The default is for pmlogger to create a single volume
150 log. Additional volume switches can also be forced asynchronously by
151 either using pmlc(1) or sending pmlogger a SIGHUP signal (see below).
152 Note, if a scheduled volume switch is in operation due to the -v
153 option, then its counters will be reset after an asynchronous switch.
154
155 Independent of any -v option, each volume of an archive is limited to
156 no more than 2^31 bytes, so pmlogger will automatically create a new
157 volume for the archive before this limit is reached.
158
159 Normally pmlogger operates on the distributed Performance Metrics Name
160 Space (PMNS), however if the -n option is specified an alternative
161 local PMNS is loaded from the file pmnsfile.
162
163 Under normal circumstances, pmlogger will run forever (except for a -s
164 option or a termination signal). The -T option may be used to limit
165 the execution time using the format of time as prescribed by PCPIn‐
166 tro(1). The time is interpreted within the time zone of the PMCD
167 server, unless the -y option is given, within which case the time zone
168 at this logger host is used.
169 Some examples of different formats:
170 -T 10mins
171 -T '@ 11:30'
172 From this it can be seen that -T 10mins and -s 10mins perform identical
173 actions.
174
175 Alternatively, pmlogger runtime may be limited to the lifetime of
176 another process by using the -p or --PID option to nominate the PID of
177 the process of interest. In this case the pmlogger will exit when the
178 other process no longer exists.
179
180 When pmlogger receives a SIGHUP signal, the current volume of the log
181 is closed, and a new volume is opened. This mechanism (or the alterna‐
182 tive mechanism via pmlc(1)) may be used to manage the growth of the log
183 files - once a log volume is closed, that file may be archived without
184 ill-effect on the continued operation of pmlogger. See also the -v
185 option above.
186
187 Historically the buffers for the current log may be flushed to disk
188 using the flush command of pmlc(1), or by sending pmlogger a SIGUSR1
189 signal or by using the -u option. The current version of pmlogger and
190 the libpcp routines that underpin pmlogger unconditionally use
191 unbuffered writes and a single fwrite(3) for each logical record writ‐
192 ten, and so ``flushing'' does not force any additional data to be writ‐
193 ten to the file system. The -u option, the SIGUSR1 handling and the
194 pmlc(1) flush command are retained for backwards compatibility.
195
196 When launched with the -x option, pmlogger will accept asynchronous
197 control requests on the file descriptor fd. This option is only
198 expected to be used internally by PCP applications that support ``live
199 record mode''.
200
201 The -m option allows the string note to be appended to the map file for
202 this instance of pmlogger in the $PCP_TMP_DIR/pmlogger directory. This
203 is currently used internally to document the file descriptor (fd) when
204 the -x option is used, or to indicate that this pmlogger instance was
205 started under the control of pmlogger_check(1).
206
207 The -H option allows the hostname written into the archive label to be
208 overridden. This mirrors the -H option of pmcd(1) , but allows it to
209 be specified on the pmlogger process. Without this option, the value
210 returned from the logged pmcd(1) is used.
211
212 The -C option will cause the configuration file to be parsed and pmlog‐
213 ger will then exit without creating an output archive, so when -C is
214 specified, the archive command line argument is not required. Any
215 errors in the configuration file are reported.
216
218 The configuration file may be specified with the -c option. If it is
219 not, configuration specifications are read from standard input.
220
221 If conffile does not exist, then a search is made in the directory
222 $PCP_VAR_DIR/config/pmlogger for a file of the same name, and if found
223 that file is used, e.g. if config.mumble does not exist in the current
224 directory and the file $PCP_VAR_DIR/config/pmlogger/config.mumble does
225 exist, then -c config.mumble and -c $PCP_VAR_DIR/config/pmlogger/con‐
226 fig.mumble are equivalent.
227
228 The syntax for the configuration file is as follows.
229
230 1. Words are separated by white space (space, tab or newline).
231
232 2. The symbol ``#'' (hash) introduces a comment, and all text up to
233 the next newline is ignored.
234
235 3. Keywords (shown in bold below) must appear literally (i.e. in
236 lower case).
237
238 4. Each specification begins with the optional keyword log, followed
239 by one of the states mandatory on, mandatory off, mandatory maybe,
240 advisory on or advisory off.
241
242 5. For the on states, a logging interval must follow using the syntax
243 ``once'', or ``default'', or ``every N timeunits'', or simply ``N
244 timeunits'' - N is an unsigned integer, and timeunits is one of
245 the keywords msec, millisecond, sec, second, min, minute, hour or
246 the plural form of one of the above.
247 Internal limitations require the interval to be smaller than
248 (approximately) 74 hours. An interval value of zero is a synonym
249 for once. An interval of default means to use the default logging
250 interval of 60 seconds; this default value may be changed to
251 interval with the -t command line option.
252
253 The interval argument follows the syntax described in PCPIntro(1),
254 and in the simplest form may be an unsigned integer (the implied
255 units in this case are seconds).
256
257 6. Following the state and possible interval specifications comes a
258 ``{'', followed by a list of one or more metric specifications and
259 a closing ``}''. The list is white space (or comma) separated.
260 If there is only one metric specification in the list, the braces
261 are optional.
262
263 7. A metric specification consists of a metric name optionally fol‐
264 lowed by a set of instance names. The metric name follows the
265 standard PCP naming conventions, see PMNS(5), and if the metric
266 name is a non-leaf node in the PMNS (see PMNS(5)), then pmlogger
267 will recursively descend the PMNS and apply the logging specifica‐
268 tion to all descendent metric names that are leaf nodes in the
269 PMNS. The set of instance names is a ``['', followed by a list of
270 one or more space (or comma) separated names, numbers or strings,
271 and a closing ``]''. Elements in the list that are numbers are
272 assumed to be internal instance identifiers, other elements are
273 assumed to be external instance identifiers - see pmGetInDom(3)
274 for more information.
275
276 If no instances are given, then the logging specification is
277 applied to all instances of the associated metric.
278
279 8. There may be an arbitrary number of logging specifications.
280
281 9. As of PCP version 4.0 and later, any metric name specification
282 that does not resolve to a leaf node in the PMNS is added to an
283 internal list of possible dynamic subtree roots. PMDAs can dynam‐
284 ically create new metrics below a dynamic root node in their PMNS,
285 and send a notification to clients that the PMNS has changed, see
286 pmdaExtSetFlags(3) and in particular the METRIC CHANGES section
287 for API details. This mechanism is currently supported by
288 pmdaprometheus(1) and pmdammv(1). When a fetch issued by pmlogger
289 returns with the PMDA_EXT_NAMES_CHANGE flag set, pmlogger will
290 traverse the internal list of possible dynamic subtree nodes and
291 dynamically discover any new metrics. In effect, as of PCP ver‐
292 sion 4.0 and later, pmlogger can be configured to dynamically log
293 new metrics that appear in the future, after the configuration
294 file is initially parsed.
295
296 10. Following all of the logging specifications, there may be an
297 optional access control section, introduced by the literal token
298 [access]. Thereafter come access control rules that allow or dis‐
299 allow operations from particular hosts or groups of hosts.
300
301 The operations may be used to interrogate or control a running
302 pmlogger using pmlc(1) and fall into the following classes:
303
304 enquire interrogate the status of pmlogger and the metrics
305 it is logging
306 advisory Change advisory logging.
307 mandatory Change mandatory logging.
308 all All of the above.
309
310 Access control rules are of the form ``allow hostlist : opera‐
311 tionlist ;'' and ``disallow hostlist : operationlist ;''.
312
313 The hostlist follows the syntax and semantics for the access con‐
314 trol mechanisms used by PMCD and are fully documented in pmcd(1).
315 An operationslist is a comma separated list of the operations
316 advisory, mandatory, enquire and all.
317
318 A missing [access] section allows all access and is equivalent to
319 allow * : all;.
320
321 The configuration (either from standard input or conffile) is initially
322 scanned by pmcpp(1) with the options -rs and -I $PCP_VAR_DIR/con‐
323 fig/pmlogger. This extends the configuration file syntax with include
324 file processing (%include), a common location to search for include
325 files ($PCP_VAR_DIR/config/pmlogger), macro definitions (%define),
326 macro expansion (%name and %{name}) and conditional inclusion of lines
327 (%ifdef name ... %else ... %endif and %ifndef name ... %else ...
328 %endif).
329
331 The available command line options are:
332
333 -c conffile, --config=conffile
334 Specify the conffile file to use.
335
336 -C, --check
337 Parse configuration and exit.
338
339 -h host, --host=host
340 Fetch performance metrics from pmcd(1) on host, rather than from
341 the default localhost.
342
343 -l logfile, --log=logfile
344 Write all diagnostics to logfile instead of the default pmlog‐
345 ger.log.
346
347 -L, --linger
348 Run even if not the primary logger instance and nothing to log.
349
350 -K spec, --spec-local=spec
351 When fetching metrics from a local context (see -o), the -K option
352 may be used to control the DSO PMDAs that should be made accessi‐
353 ble. The spec argument conforms to the syntax described in
354 pmSpecLocalPMDA(3). More than one -K option may be used.
355
356 -m note, --note=note
357 Append note to the port map file for this instance.
358
359 -o, --local-PMDA
360 Use a local context to collect metrics from DSO PMDAs on the local
361 host without PMCD. See also -K.
362
363 -n pmnsfile, --namespace=pmnsfile
364 Load an alternative Performance Metrics Name Space (PMNS(5)) from
365 the file pmnsfile.
366
367 -p PID, --PID=PID
368 Log specified metrics for the lifetime of the pid PID.
369
370 -P, --primary
371 Run as primary logger instance. See above for more detailed
372 description of this.
373
374 -r, --report
375 Report record sizes and archive growth rate.
376
377 -s endsize, --size=endsize
378 Terminate after log size exceeds endsize.
379
380 -t interval, --interval=interval
381 Specify the logging interval. The default value is 60 seconds.
382
383 -T endtime, --finish=endtime
384 Specify the endtime when to end logging.
385
386 -u Use unbuffered output. This is the default (so this option does
387 nothing).
388
389 -U username, --username=username
390 When in daemon mode, run as user username.
391
392 -v volsize, --volsize=volsize
393 Switch log volumes after reaching log volume size volsize.
394
395 -V version, --version=version
396 Specify log archive version. The default and the only accepted
397 value is 2.
398
399 -x fd
400 Allow asynchronous control requests on the file descriptor fd.
401
402 -y Use local timezone instead of the timezone from the pmcd(1) host.
403
404 -?, --help
405 Display usage message and exit.
406
408 For each PCP utility, there is a sample pmlogger configuration file
409 that could be used to create an archive log suitable for replaying with
410 that tool (i.e. includes all of the performance metrics used by the
411 tool). For a tool named foo this configuration file is located in
412 $PCP_VAR_DIR/config/pmlogger/config.foo.
413
414 The following is a simple default configuration file for a primary
415 pmlogger instance, and demonstrates most of the capabilities of the
416 configuration specification language.
417
418 log mandatory on once { hinv.ncpu hinv.ndisk }
419 log mandatory on every 10 minutes {
420 disk.all.write
421 disk.all.read
422 network.interface.in.packets [ "et0" ]
423 network.interface.out.packets [ "et0" ]
424 nfs.server.reqs [ "lookup" "getattr" "read" "write" ]
425 }
426
427 log advisory on every 30 minutes {
428 environ.temp
429 pmcd.pdu_in.total
430 pmcd.pdu_out.total
431 }
432
433 %include "macros.default"
434
435 %ifdef %disk_detail
436 log mandatory on %disk_detail_freq {
437 disk.dev
438 }
439 %endif
440
441 [access]
442 disallow * : all except enquire;
443 allow localhost : mandatory, advisory;
444
446 The archive logs are sufficiently precious that pmlogger will not trun‐
447 cate an existing physical file. A message of the form
448 ...: "foo.index" already exists, not over-written
449 ...: File exists
450 indicates this situation has arisen. You must explicitly remove the
451 files and launch pmlogger again.
452
453 There may be at most one primary pmlogger instance per monitored host;
454 attempting to bend this rule produces the error:
455 pmlogger: there is already a primary pmlogger running
456
457 Various other messages relating to the creation and/or deletion of
458 files in $PCP_TMP_DIR/pmlogger suggest a permission problem on this
459 directory, or some feral files have appeared therein.
460
462 archive.meta
463 metadata (metric descriptions, instance domains, etc.) for
464 the archive log
465 archive.0 initial volume of metrics values (subsequent volumes have
466 suffixes 1, 2, ...)
467 archive.index
468 temporal index to support rapid random access to the other
469 files in the archive log
470 $PCP_TMP_DIR/pmlogger
471 pmlogger maintains the files in this directory as the map
472 between the process id of the pmlogger instance and the IPC
473 port that may be used to control each pmlogger instance (as
474 used by pmlc(1))
475 $PCP_VAR_DIR/config/pmlogger/config.default
476 default configuration file for the primary logger instance
477 launched from $PCP_RC_DIR/pmlogger
478 $PCP_VAR_DIR/config/pmlogger/config.*
479 assorted configuration files suitable for creating logs that
480 may be subsequently replayed with the PCP visualization and
481 monitoring tools
482 $PCP_LOG_DIR/pmlogger/<hostname>
483 Default directory for PCP archive files for performance met‐
484 ric values collected from the host <hostname>.
485 $PCP_SYSCONFIG_DIR/pmlogger
486 additional environment variables that will be set when the
487 primary pmlogger instance executes. Only settings of the
488 form "PMLOGGER_VARIABLE=value" will be honoured.
489 ./pmlogger.log
490 (or $PCP_LOG_DIR/pmlogger/<hostname>/pmlogger.log when
491 started automatically by either $PCP_RC_DIR/pmlogger or one
492 of the pmlogger(1) monitoring scripts such as pmlog‐
493 ger_check(1))
494 all messages and diagnostics are directed here
495
497 Normally pmlogger creates a socket to receive control messages from
498 pmlc(1) on the first available TCP/IP port numbered 4330 or higher.
499 The environment variable PMLOGGER_PORT may be used to specify an alter‐
500 native starting port number.
501
502 If set to the value 1, the PMLOGGER_LOCAL environment variable will
503 cause pmlogger to run in a localhost-only mode of operation, where it
504 binds only to the loopback interface.
505
506 The PMLOGGER_MAXPENDING variable can be set to indicate the maximum
507 length to which the queue of pending pmlc connections may grow.
508
509 The default sampling interval used by pmlogger can be set using the
510 PMLOGGER_INTERVAL variable (if not set, 60 seconds will be used). Both
511 the command line and directives in the configuration file will override
512 this value. It is an integer in units of seconds.
513
515 Environment variables with the prefix PCP_ are used to parameterize the
516 file and directory names used by PCP. On each installation, the file
517 /etc/pcp.conf contains the local values for these variables. The
518 $PCP_CONF variable may be used to specify an alternative configuration
519 file, as described in pcp.conf(5).
520
522 PCPIntro(1), pmcd(1), pmdumplog(1), pmlc(1), pmlogger_check(1), system‐
523 ctl(1), pmSpecLocalPMDA(3), pcp.conf(5), pcp.env(5), pmlogger(5),
524 PMNS(5) and chkconfig(8).
525
526
527
528Performance Co-Pilot PCP PMLOGGER(1)