1LOGARCHIVE(5)                 File Formats Manual                LOGARCHIVE(5)
2
3
4

NAME

6       LOGARCHIVE - performance metrics archive format
7

DESCRIPTION

9       Performance Co-Pilot (PCP) archives store historical values about arbi‐
10       trary metrics recorded from a single host.  Archives are machine  inde‐
11       pendent and self-contained - all metadata required for off-line or off-
12       site analysis is stored.
13
14       The format is stable in order to allow long-term historical storage and
15       processing by PMAPI client tools.
16
17       Archives  may  be read by most PCP client tools, using the -a/--archive
18       NAME option, or dumped raw by pmdumplog(1).  Archives are created  pri‐
19       marily  by  pmlogger(1),  however  they  can  also be created using the
20       LOGIMPORT(3) programming interface.
21
22       Archives may be merged, analyzed, modified and subsampled  using  pmlo‐
23       greduce(1),  pmlogsummary(1),  pmlogrewrite(1) and pmlogextract(1).  In
24       addition, PCP archives may be examined in sets or grouped together into
25       "archive  folios",  which  are  created  and managed by the mkaf(1) and
26       pmafm(1) tools.
27
28       Archives consist of several physical files that share  a  common  arbi‐
29       trary prefix, e.g.  myarchive.
30
31       myarchive.0, myarchive.1, ...
32              One  or  more  data volumes containing the metric values and any
33              error codes encountered during metric sampling.   Typically  the
34              largest of the files and may grow very rapidly, depending on the
35              pmlogger sampling interval(s) being used.
36
37       myarchive.meta
38              Information  for  PMAPI  functions  such   as   pmLookupDesc(3),
39              pmLookupLabels(3)  and  pmLookupInDom(3).  The metadata file may
40              grow sporadically as logged metrics, instance domains and labels
41              vary over time.
42
43       myarchive.index
44              A  temporal  index,  mapping  timestamps to offsets in the other
45              files.
46

COMMON FEATURES

48       All three types of files have a similar record-based structure, a  con‐
49       vention  of network-byte-order (big-endian) encoding, and 32-bit fields
50       for tagging/padding for those records.  Strings  are  stored  as  8-bit
51       characters  without  assuming  a  specific encoding, so normally ASCII.
52       See also the __pmLog* types in libpcp.h.
53
54   RECORD FRAMING
55       The volume and meta files are divided into self-identifying records.
56
57       ┌───────┬────────┬─────────────────────────────────────────────────────┐
58       │Offset │ Length │                        Name                         │
59       ├───────┼────────┼─────────────────────────────────────────────────────┤
60       │  0    │   4    │ N, length of record, in bytes, including this field │
61       │  4    │  N-8   │ record payload, usually starting with a 32-bit tag  │
62       │ N-4   │   4    │ N, length of record (again)                         │
63       └───────┴────────┴─────────────────────────────────────────────────────┘
64
65   ARCHIVE LOG LABEL
66       All three types of files begin with a "log label" header, which identi‐
67       fies the host name, the time interval covered, and a time zone.
68
69       ┌───────┬────────┬────────────────────────────────────────────────────┐
70       │Offset │ Length │                        Name                        │
71       ├───────┼────────┼────────────────────────────────────────────────────┤
72       │  0    │   4    │ tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602       │
73       │  4    │   4    │ pid of pmlogger process that wrote file            │
74       │  8    │   4    │ log start time, seconds part (past UNIX epoch)     │
75       │  12   │   4    │ log start time, microseconds part                  │
76       │  16   │   4    │ current log volume number (or -1=.meta, -2=.index) │
77       │  20   │   64   │ name of collection host                            │
78       │  80   │   40   │ time zone string ($TZ environment variable)        │
79       └───────┴────────┴────────────────────────────────────────────────────┘
80
81       All  fields,  except for the current log volume number field, match for
82       all archive-related files produced by a single run of the tool.
83

ARCHIVE VOLUME (.0, .1, ...) RECORDS

85   pmResult
86       After the archive log label record, an  archive  volume  file  contains
87       metric  values  corresponding to the pmResult set of one pmFetch opera‐
88       tion, which is almost identical to the form on disk.  The  record  size
89       may  vary  according  to  number  of PMIDs being fetched, the number of
90       instances for their domains.  File size is  limited  to  2GiB,  due  to
91       storage of 32-bit offsets within the temporal index.
92
93           ┌────────┬────────┬───────────────────────────────────────────┐
94           │Offset  │ Length │                   Name                    │
95           ├────────┼────────┼───────────────────────────────────────────┤
96           │   0    │   4    │ timestamp, seconds part (past UNIX epoch) │
97           │   4    │   4    │ timestamp, microseconds part              │
98           │   8    │   4    │ number of PMIDs with data following       │
99           │  12    │   M    │ pmValueSet #0                             │
100           │ 12+M   │   N    │ pmValueSet #1                             │
101           │12+M+N  │  ...   │ ...                                       │
102           │  NOP   │   X    │ pmValueBlock #0                           │
103           │ NOP+X  │   Y    │ pmValueBlock #1                           │
104           │NOP+X+Y │  ...   │ ...                                       │
105           └────────┴────────┴───────────────────────────────────────────┘
106
107       Records  with  a  number-of-PMIDs equal to zero are "mark records", and
108       represent interruptions, missing data, or time discontinuities in  log‐
109       ging.
110
111   pmValueSet
112       This subrecord represents the measurements for one metric.
113
114         ┌───────┬────────┬────────────────────────────────────────────────┐
115         │Offset │ Length │                      Name                      │
116         ├───────┼────────┼────────────────────────────────────────────────┤
117         │  0    │   4    │ PMID                                           │
118         │  4    │   4    │ number of values                               │
119         │  8    │   4    │ storage mode, PM_VAL_INSITU=0 or PM_VAL_DPTR=1 │
120         │  12   │   M    │ pmValue #0                                     │
121         │ 12+M  │   N    │ pmValue #1                                     │
122         │12+M+N │  ...   │ ...                                            │
123         └───────┴────────┴────────────────────────────────────────────────┘
124
125       The  metric-description metadata for PMIDs is found in the .meta files.
126       These entries are not timestamped, so the metadata  is  assumed  to  be
127       unchanging throughout the archiving session.
128
129   pmValue
130       This  subrecord represents one measurement for one instance of the met‐
131       ric.  It is a variant type, depending on the parent pmValueSet's value-
132       format  field.   This allows small numbers to be encoded compactly, but
133       retain flexibility for larger or  variable-length  data  to  be  stored
134       later in the pmResult record.
135
136          ┌───────┬────────┬───────────────────────────────────────────────┐
137          │Offset │ Length │                     Name                      │
138          ├───────┼────────┼───────────────────────────────────────────────┤
139          │  0    │   4    │ number in instance-domain (or PM_IN_NULL=-1)  │
140          │  4    │   4    │ value (INSITU) or
141          │       │        │ offset in pmResult to our pmValueBlock (DPTR) │
142          └───────┴────────┴───────────────────────────────────────────────┘
143
144       The  instance-domain  metadata  for  PMIDs is found in the .meta files.
145       Since the numeric mappings may change during the lifetime of  the  log‐
146       ging session, it is important to match up the timestamp of the measure‐
147       ment record with the corresponding instance-domain  record.   That  is,
148       the  instance-domain  corresponding  to a measurement at time T are the
149       records with largest timestamps T' <= T.
150
151   pmValueBlock
152       Instances of this subrecord are placed at the end  of  the  pmValueSet,
153       after  all  the  pmValue subrecords.  If (and only if) needed, they are
154       padded at the end to the next-higher 32-bit boundary.
155
156         ┌───────┬────────┬────────────────────────────────────────────────┐
157         │Offset │ Length │                      Name                      │
158         ├───────┼────────┼────────────────────────────────────────────────┤
159         │  0    │   1    │ value type (same as pmDesc.type)               │
160         │  1    │   3    │ 4 + N, the length of the subrecord             │
161         │  4    │   N    │ bytes that make up the raw value               │
162         │ 4+N   │  0-3   │ padding (not included in the 4+N length field) │
163         └───────┴────────┴────────────────────────────────────────────────┘
164
165       Note that for PM_TYPE_STRING, the length includes an explicit NULL ter‐
166       minator  byte.   For  PM_TYPE_EVENT,  the  value  bytestring is further
167       structured.
168

METADATA FILE (.meta) RECORDS

170       After the archive log label record, the metadata file  contains  inter‐
171       leaved  metric-description and timestamped instance-domain descriptors.
172       File size is limited to 2GiB, due to storage of 32-bit  offsets  within
173       the  temporal  index.   Unlike  the data volumes, these records are not
174       forced to 32-bit alignment.  See also libpcp/logmeta.c.
175
176   pmDesc
177       Instances of this record represent the  metric  description,  giving  a
178       name, type, instance-domain identifier, and a set of names to each PMID
179       used in the archive volume.
180
181        ┌───────┬────────┬───────────────────────────────────────────────────┐
182        │Offset │ Length │                       Name                        │
183        ├───────┼────────┼───────────────────────────────────────────────────┤
184        │  0    │   4    │ tag, TYPE_DESC=1                                  │
185        │  4    │   4    │ PMID                                              │
186        │  8    │   4    │ type (PM_TYPE_*)                                  │
187        │  12   │   4    │ instance domain number                            │
188        │  16   │   4    │ semantics of value (PM_SEM_*)                     │
189        │  20   │   4    │ units: bit-packed pmUnits                         │
190        │  4    │   4    │ number of alternative names for this PMID         │
191        │  28   │   4    │ N: number of bytes in this name                   │
192        │  32   │   N    │ bytes of the name, no NULL terminator nor padding │
193        │ 32+N  │   4    │ N2: number of bytes in next name                  │
194        │ 36+N  │   N2   │ bytes of the name, no NULL terminator nor padding │
195        │ ...   │  ...   │ ...                                               │
196        └───────┴────────┴───────────────────────────────────────────────────┘
197
198   pmLogIndom
199       Instances of this record represent the number-string mapping  table  of
200       an  instance domain.  The instance domain number will have already been
201       mentioned in a prior pmDesc record.  As new instances may appear over a
202       long  archiving run these records are timestamped, and must be searched
203       when decoding pmResult records from the archive data volumes.  Instance
204       names may be reused between instance numbers, so an offset-based string
205       table is used that facilitates sharing.
206
207         ┌─────────┬────────┬───────────────────────────────────────────────┐
208         │ Offset  │ Length │                     Name                      │
209         ├─────────┼────────┼───────────────────────────────────────────────┤
210         │   0     │   4    │ tag, TYPE_INDOM=2                             │
211         │   4     │   4    │ timestamp, seconds part (past UNIX epoch)     │
212         │   8     │   4    │ timestamp, microseconds part                  │
213         │   12    │   4    │ instance domain number                        │
214         │   16    │   4    │ N: number of instances in domain, normally >0 │
215         │   20    │   4    │ first instance number                         │
216         │   24    │   4    │ second instance number (if appropriate)       │
217         │  ...    │  ...   │ ...                                           │
218         │ 20+4*N  │   4    │ first offset into string table (see below)    │
219         │20+4*N+4 │   4    │ second offset into string table (etc.)        │
220         │  ...    │  ...   │ ...                                           │
221         │ 20+8*N  │   M    │ base of string table, containing              │
222         │         │        │ packed, NULL-terminated instance names        │
223         └─────────┴────────┴───────────────────────────────────────────────┘
224
225       Records of  this  form  replace  the  existing  instance-domain:  prior
226       records are not searched for resolving instance numbers in measurements
227       after this timestamp.
228
229   pmLogLabelSet
230       Instances of this record represent sets of name:value pairs  associated
231       with labels of the context, instance domains and individual performance
232       metrics - refer to pmLookupLabels(3) for further details.
233
234       Any instance domain number will have already been mentioned in a  prior
235       pmDesc  record.   As new labels can appear during an archiving session,
236       these records are timestamped and must be searched when decoding  pmRe‐
237       sult records from the archive data volumes.
238
239       ┌────────────┬────────┬────────────────────────────────────────────────┐
240       │  Offset    │ Length │                      Name                      │
241       ├────────────┼────────┼────────────────────────────────────────────────┤
242       │     0      │   4    │ tag, TYPE_LABEL=3                              │
243       │     4      │   4    │ timestamp, seconds part (past UNIX epoch)      │
244       │     8      │   4    │ timestamp, microseconds part                   │
245       │    12      │   4    │ label type (PM_LABEL_* type macros.)           │
246       │    16      │   4    │ numeric identifier - domain, PMID, etc         │
247       │            │        │ or PM_IN_NULL=-1 for context labels            │
248       │    20      │   4    │ N: number of label sets in this record,        │
249       │            │        │ usually 1 except in the case of instances      │
250       │    24      │   4    │ offset to the start of the JSONB labels string │
251       │    28      │   L1   │ first labelset array entry (see below)         │
252       │    ...     │  ...   │ ...                                            │
253       │   28+L1    │   LN   │ N-th labelset array entry (see below)          │
254       │    ...     │  ...   │ ...                                            │
255       │28+L1+...LN │   M    │ concatenated JSONB strings for all labelsets   │
256       └────────────┴────────┴────────────────────────────────────────────────┘
257
258       Records  of  this  form  replace  the existing labels for a given type:
259       prior records are not searched for resolving that  class  of  label  in
260       measurements after this timestamp.
261
262       The individual labelset array entries are variable length, depending on
263       the number of labels present within that set.   These  entries  contain
264       the  instance  identifiers  (in  the  case  of  type PM_LABEL_INSTANCES
265       labels), lengths and offsets of each label name and value, and also any
266       flags set for each label.
267
268            ┌───────┬────────┬───────────────────────────────────────────┐
269            │Offset │ Length │                   Name                    │
270            ├───────┼────────┼───────────────────────────────────────────┤
271            │  0    │   4    │ instance identifier (or PM_IN_NULL=-1)    │
272            │  4    │   4    │ length of JSONB label string              │
273            │  8    │   4    │ N: number of labels in this labelset      │
274            │  12   │   2    │ first label name offset                   │
275            │  14   │   1    │ first label name length                   │
276            │  15   │   1    │ first label flags (e.g. optionality)      │
277            │  16   │   2    │ first label value offset                  │
278            │  18   │   2    │ first label value length                  │
279            │  20   │   2    │ second label name offset (if appropriate) │
280            │ ...   │  ...   │ ...                                       │
281            └───────┴────────┴───────────────────────────────────────────┘
282
283   pmLogText
284       This  record  stores  help text associated with a metric or an instance
285       domain - as provided by pmLookupText(3) and pmLookupInDomText(3).
286
287       The metric identifier and instance domain number will have already been
288       mentioned in a prior pmDesc record.
289
290
291          ┌───────┬────────┬──────────────────────────────────────────────┐
292          │Offset │ Length │                     Name                     │
293          ├───────┼────────┼──────────────────────────────────────────────┤
294          │  0    │   4    │ tag, TYPE_TEXT=4                             │
295          │  4    │   4    │ text and identifier type (PM_TEXT_* macros.) │
296          │  8    │   4    │ numeric identifier - PMID or instance domain │
297          │  12   │   M    │ help text string, arbitrary text             │
298          └───────┴────────┴──────────────────────────────────────────────┘
299

INDEX FILE (.index) RECORDS

301       After  the archive log label record, the temporal index file contains a
302       plainly concatenated, unframed group of tuples, which relate timestamps
303       to 32-bit seek offsets in the volume and meta files.  These records are
304       fixed-size,  fixed-format,  and  are  not  enclosed  in  the   standard
305       length/payload/length wrapper: they take up the entire remainder of the
306       .index file.  See also libpcp/logutil.c.
307
308        ┌───────┬────────┬───────────────────────────────────────────────────┐
309        │Offset │ Length │                       Name                        │
310        ├───────┼────────┼───────────────────────────────────────────────────┤
311        │  0    │   4    │ event time, seconds part (past UNIX epoch)        │
312        │  4    │   4    │ event time, microseconds part                     │
313        │  8    │   4    │ archive volume number (0...N)                     │
314        │  12   │   4    │ byte offset in .meta file of pmDesc or pmLogIndom │
315        │  16   │   4    │ byte offset in archive volume file of pmResult    │
316        └───────┴────────┴───────────────────────────────────────────────────┘
317
318       Since the temporal index is optional, and exists only to speed up time-
319       based  random  access  to metrics and their metadata, the index records
320       are emitted only intermittently.  An archive reader program should  not
321       presume any particular rate of data flow into the index.  However, com‐
322       mon events that may trigger a new temporal-index record include changes
323       in instance-domains, switching over to a new archive volume, and start‐
324       ing or stopping logging.  One reliable invariant however is  that,  for
325       each  index  entry,  there  are to be no meta or archive-volume records
326       with a timestamp after that in the index,  but  physically  before  the
327       byte-offset in the index.
328

FILES

330       Several PCP tools create archives in standard locations:
331
332       $HOME/.pcp/pmlogger
333                 default  location for the interactive chart recording mode in
334                 pmchart(1)
335       $PCP_LOG_DIR/pmlogger
336                 default location for pmlogger_daily(1) and  pmlogger_check(1)
337                 scripts
338       $PCP_LOG_DIR/pmmgr
339                 default location for the PCP daemon manager pmmgr(1)
340

SEE ALSO

342       PCPIntro(1),  PMAPI(3),  pmLookupDesc(3), pmLookupInDom(3), pmLookupIn‐
343       DomText(3),  pmLookupLabels(3),  pmLookupText(3),  mkaf(1),   pmafm(1),
344       pmchart(1),   pmdumplog(1),   pmlogger(1),   pmlogger_check(1),  pmlog‐
345       ger_daily(1),   pmlogreduce(1),    pmlogrewrite(1),    pmlogsummary(1),
346       pmmgr(1), pcp.conf(5), and pcp.env(5).
347
348
349
350Performance Co-Pilot                                             LOGARCHIVE(5)
Impressum