1LOGARCHIVE(5) File Formats Manual LOGARCHIVE(5)
2
6 LOGARCHIVE - performance metrics archive format
7
9 Performance Co-Pilot (PCP) archives store historical values about arbi‐
10 trary metrics recorded from a single host. Archives are machine inde‐
11 pendent and self-contained - all metadata required for off-line or off-
12 site analysis is stored.
13 The format is stable in order to allow long-term historical storage and
15 processing by PMAPI client tools.
16 Archives may be read by most PCP client tools, using the -a/--archive
18 NAME option, or dumped raw by pmdumplog(1). Archives are created pri‐
19 marily by pmlogger(1), however they can also be created using the
20 LOGIMPORT(3) programming interface.
21 Archives may be merged, analyzed, modified and subsampled using pmlo‐
23 greduce(1), pmlogsummary(1), pmlogrewrite(1) and pmlogextract(1). In
24 addition, PCP archives may be examined in sets or grouped together into
25 "archive folios", which are created and managed by the mkaf(1) and
26 pmafm(1) tools.
27 Archives consist of several physical files that share a common arbi‐
29 trary prefix, e.g. myarchive.
30 myarchive.0, myarchive.1, ...
32 One or more data volumes containing the metric values and any
33 error codes encountered during metric sampling. Typically the
34 largest of the files and may grow very rapidly, depending on the
35 pmlogger sampling interval(s) being used.
36 myarchive.meta
38 Information for PMAPI functions such as pmLookupDesc(3),
39 pmLookupLabels(3) and pmLookupInDom(3). The metadata file may
40 grow sporadically as logged metrics, instance domains and labels
41 vary over time.
42 myarchive.index
44 A temporal index, mapping timestamps to offsets in the other
45 files.
46
48 All three types of files have a similar record-based structure, a con‐
49 vention of network-byte-order (big-endian) encoding, and 32-bit fields
50 for tagging/padding for those records. Strings are stored as 8-bit
51 characters without assuming a specific encoding, so normally ASCII.
52 See also the __pmLog* types in libpcp.h.
53 RECORD FRAMING
55 The volume and meta files are divided into self-identifying records.
56 ┌───────┬────────┬─────────────────────────────────────────────────────┐
58 │Offset │ Length │ Name │
59 ├───────┼────────┼─────────────────────────────────────────────────────┤
60 │ 0 │ 4 │ N, length of record, in bytes, including this field │
61 │ 4 │ N-8 │ record payload, usually starting with a 32-bit tag │
62 │ N-4 │ 4 │ N, length of record (again) │
63 └───────┴────────┴─────────────────────────────────────────────────────┘
64 ARCHIVE LOG LABEL
66 All three types of files begin with a "log label" header, which identi‐
67 fies the host name, the time interval covered, and a time zone.
68 ┌───────┬────────┬────────────────────────────────────────────────────┐
70 │Offset │ Length │ Name │
71 ├───────┼────────┼────────────────────────────────────────────────────┤
72 │ 0 │ 4 │ tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602 │
73 │ 4 │ 4 │ pid of pmlogger process that wrote file │
74 │ 8 │ 4 │ log start time, seconds part (past UNIX epoch) │
75 │ 12 │ 4 │ log start time, microseconds part │
76 │ 16 │ 4 │ current log volume number (or -1=.meta, -2=.index) │
77 │ 20 │ 64 │ name of collection host │
78 │ 80 │ 40 │ time zone string ($TZ environment variable) │
79 └───────┴────────┴────────────────────────────────────────────────────┘
80 All fields, except for the current log volume number field, match for
82 all archive-related files produced by a single run of the tool.
83
85 pmResult
86 After the archive log label record, an archive volume file contains
87 metric values corresponding to the pmResult set of one pmFetch opera‐
88 tion, which is almost identical to the form on disk. The record size
89 may vary according to number of PMIDs being fetched, the number of
90 instances for their domains. File size is limited to 2GiB, due to
91 storage of 32-bit offsets within the temporal index.
92 ┌────────┬────────┬───────────────────────────────────────────┐
94 │Offset │ Length │ Name │
95 ├────────┼────────┼───────────────────────────────────────────┤
96 │ 0 │ 4 │ timestamp, seconds part (past UNIX epoch) │
97 │ 4 │ 4 │ timestamp, microseconds part │
98 │ 8 │ 4 │ number of PMIDs with data following │
99 │ 12 │ M │ pmValueSet #0 │
100 │ 12+M │ N │ pmValueSet #1 │
101 │12+M+N │ ... │ ... │
102 │ NOP │ X │ pmValueBlock #0 │
103 │ NOP+X │ Y │ pmValueBlock #1 │
104 │NOP+X+Y │ ... │ ... │
105 └────────┴────────┴───────────────────────────────────────────┘
106 Records with a number-of-PMIDs equal to zero are "mark records", and
108 represent interruptions, missing data, or time discontinuities in log‐
109 ging.
110 pmValueSet
112 This subrecord represents the measurements for one metric.
113 ┌───────┬────────┬────────────────────────────────────────────────┐
115 │Offset │ Length │ Name │
116 ├───────┼────────┼────────────────────────────────────────────────┤
117 │ 0 │ 4 │ PMID │
118 │ 4 │ 4 │ number of values │
119 │ 8 │ 4 │ storage mode, PM_VAL_INSITU=0 or PM_VAL_DPTR=1 │
120 │ 12 │ M │ pmValue #0 │
121 │ 12+M │ N │ pmValue #1 │
122 │12+M+N │ ... │ ... │
123 └───────┴────────┴────────────────────────────────────────────────┘
124 The metric-description metadata for PMIDs is found in the .meta files.
126 These entries are not timestamped, so the metadata is assumed to be
127 unchanging throughout the archiving session.
128 pmValue
130 This subrecord represents one measurement for one instance of the met‐
131 ric. It is a variant type, depending on the parent pmValueSet's value-
132 format field. This allows small numbers to be encoded compactly, but
133 retain flexibility for larger or variable-length data to be stored
134 later in the pmResult record.
135 ┌───────┬────────┬───────────────────────────────────────────────┐
137 │Offset │ Length │ Name │
138 ├───────┼────────┼───────────────────────────────────────────────┤
139 │ 0 │ 4 │ number in instance-domain (or PM_IN_NULL=-1) │
140 │ 4 │ 4 │ value (INSITU) or │
141 │ │ │ offset in pmResult to our pmValueBlock (DPTR) │
142 └───────┴────────┴───────────────────────────────────────────────┘
143 The instance-domain metadata for PMIDs is found in the .meta files.
145 Since the numeric mappings may change during the lifetime of the log‐
146 ging session, it is important to match up the timestamp of the measure‐
147 ment record with the corresponding instance-domain record. That is,
148 the instance-domain corresponding to a measurement at time T are the
149 records with largest timestamps T' <= T.
150 pmValueBlock
152 Instances of this subrecord are placed at the end of the pmValueSet,
153 after all the pmValue subrecords. If (and only if) needed, they are
154 padded at the end to the next-higher 32-bit boundary.
155 ┌───────┬────────┬────────────────────────────────────────────────┐
157 │Offset │ Length │ Name │
158 ├───────┼────────┼────────────────────────────────────────────────┤
159 │ 0 │ 1 │ value type (same as pmDesc.type) │
160 │ 1 │ 3 │ 4 + N, the length of the subrecord │
161 │ 4 │ N │ bytes that make up the raw value │
162 │ 4+N │ 0-3 │ padding (not included in the 4+N length field) │
163 └───────┴────────┴────────────────────────────────────────────────┘
164 Note that for PM_TYPE_STRING, the length includes an explicit NULL ter‐
166 minator byte. For PM_TYPE_EVENT, the value bytestring is further
167 structured.
168
170 After the archive log label record, the metadata file contains inter‐
171 leaved metric-description and timestamped instance-domain descriptors.
172 File size is limited to 2GiB, due to storage of 32-bit offsets within
173 the temporal index. Unlike the data volumes, these records are not
174 forced to 32-bit alignment. See also libpcp/logmeta.c.
175 pmDesc
177 Instances of this record represent the metric description, giving a
178 name, type, instance-domain identifier, and a set of names to each PMID
179 used in the archive volume.
180 ┌───────┬────────┬───────────────────────────────────────────────────┐
182 │Offset │ Length │ Name │
183 ├───────┼────────┼───────────────────────────────────────────────────┤
184 │ 0 │ 4 │ tag, TYPE_DESC=1 │
185 │ 4 │ 4 │ PMID │
186 │ 8 │ 4 │ type (PM_TYPE_*) │
187 │ 12 │ 4 │ instance domain number │
188 │ 16 │ 4 │ semantics of value (PM_SEM_*) │
189 │ 20 │ 4 │ units: bit-packed pmUnits │
190 │ 4 │ 4 │ number of alternative names for this PMID │
191 │ 28 │ 4 │ N: number of bytes in this name │
192 │ 32 │ N │ bytes of the name, no NULL terminator nor padding │
193 │ 32+N │ 4 │ N2: number of bytes in next name │
194 │ 36+N │ N2 │ bytes of the name, no NULL terminator nor padding │
195 │ ... │ ... │ ... │
196 └───────┴────────┴───────────────────────────────────────────────────┘
197 pmLogIndom
199 Instances of this record represent the number-string mapping table of
200 an instance domain. The instance domain number will have already been
201 mentioned in a prior pmDesc record. As new instances may appear over a
202 long archiving run these records are timestamped, and must be searched
203 when decoding pmResult records from the archive data volumes. Instance
204 names may be reused between instance numbers, so an offset-based string
205 table is used that facilitates sharing.
206 ┌─────────┬────────┬───────────────────────────────────────────────┐
208 │ Offset │ Length │ Name │
209 ├─────────┼────────┼───────────────────────────────────────────────┤
210 │ 0 │ 4 │ tag, TYPE_INDOM=2 │
211 │ 4 │ 4 │ timestamp, seconds part (past UNIX epoch) │
212 │ 8 │ 4 │ timestamp, microseconds part │
213 │ 12 │ 4 │ instance domain number │
214 │ 16 │ 4 │ N: number of instances in domain, normally >0 │
215 │ 20 │ 4 │ first instance number │
216 │ 24 │ 4 │ second instance number (if appropriate) │
217 │ ... │ ... │ ... │
218 │ 20+4*N │ 4 │ first offset into string table (see below) │
219 │20+4*N+4 │ 4 │ second offset into string table (etc.) │
220 │ ... │ ... │ ... │
221 │ 20+8*N │ M │ base of string table, containing │
222 │ │ │ packed, NULL-terminated instance names │
223 └─────────┴────────┴───────────────────────────────────────────────┘
224 Records of this form replace the existing instance-domain: prior
226 records are not searched for resolving instance numbers in measurements
227 after this timestamp.
228 pmLogLabelSet
230 Instances of this record represent sets of name:value pairs associated
231 with labels of the context, instance domains and individual performance
232 metrics - refer to pmLookupLabels(3) for further details.
233 Any instance domain number will have already been mentioned in a prior
235 pmDesc record. As new labels can appear during an archiving session,
236 these records are timestamped and must be searched when decoding pmRe‐
237 sult records from the archive data volumes.
238 ┌────────────┬────────┬────────────────────────────────────────────────┐
240 │ Offset │ Length │ Name │
241 ├────────────┼────────┼────────────────────────────────────────────────┤
242 │ 0 │ 4 │ tag, TYPE_LABEL=3 │
243 │ 4 │ 4 │ timestamp, seconds part (past UNIX epoch) │
244 │ 8 │ 4 │ timestamp, microseconds part │
245 │ 12 │ 4 │ label type (PM_LABEL_* type macros.) │
246 │ 16 │ 4 │ numeric identifier - domain, PMID, etc │
247 │ │ │ or PM_IN_NULL=-1 for context labels │
248 │ 20 │ 4 │ N: number of label sets in this record, │
249 │ │ │ usually 1 except in the case of instances │
250 │ 24 │ 4 │ offset to the start of the JSONB labels string │
251 │ 28 │ L1 │ first labelset array entry (see below) │
252 │ ... │ ... │ ... │
253 │ 28+L1 │ LN │ N-th labelset array entry (see below) │
254 │ ... │ ... │ ... │
255 │28+L1+...LN │ M │ concatenated JSONB strings for all labelsets │
256 └────────────┴────────┴────────────────────────────────────────────────┘
257 Records of this form replace the existing labels for a given type:
259 prior records are not searched for resolving that class of label in
260 measurements after this timestamp.
261 The individual labelset array entries are variable length, depending on
263 the number of labels present within that set. These entries contain
264 the instance identifiers (in the case of type PM_LABEL_INSTANCES
265 labels), lengths and offsets of each label name and value, and also any
266 flags set for each label.
267 ┌───────┬────────┬───────────────────────────────────────────┐
269 │Offset │ Length │ Name │
270 ├───────┼────────┼───────────────────────────────────────────┤
271 │ 0 │ 4 │ instance identifier (or PM_IN_NULL=-1) │
272 │ 4 │ 4 │ length of JSONB label string │
273 │ 8 │ 4 │ N: number of labels in this labelset │
274 │ 12 │ 2 │ first label name offset │
275 │ 14 │ 1 │ first label name length │
276 │ 15 │ 1 │ first label flags (e.g. optionality) │
277 │ 16 │ 2 │ first label value offset │
278 │ 18 │ 2 │ first label value length │
279 │ 20 │ 2 │ second label name offset (if appropriate) │
280 │ ... │ ... │ ... │
281 └───────┴────────┴───────────────────────────────────────────┘
282 pmLogText
284 This record stores help text associated with a metric or an instance
285 domain - as provided by pmLookupText(3) and pmLookupInDomText(3).
286 The metric identifier and instance domain number will have already been
288 mentioned in a prior pmDesc record.
289 ┌───────┬────────┬──────────────────────────────────────────────┐
292 │Offset │ Length │ Name │
293 ├───────┼────────┼──────────────────────────────────────────────┤
294 │ 0 │ 4 │ tag, TYPE_TEXT=4 │
295 │ 4 │ 4 │ text and identifier type (PM_TEXT_* macros.) │
296 │ 8 │ 4 │ numeric identifier - PMID or instance domain │
297 │ 12 │ M │ help text string, arbitrary text │
298 └───────┴────────┴──────────────────────────────────────────────┘
299
301 After the archive log label record, the temporal index file contains a
302 plainly concatenated, unframed group of tuples, which relate timestamps
303 to 32-bit seek offsets in the volume and meta files. These records are
304 fixed-size, fixed-format, and are not enclosed in the standard
305 length/payload/length wrapper: they take up the entire remainder of the
306 .index file. See also libpcp/logutil.c.
307 ┌───────┬────────┬───────────────────────────────────────────────────┐
309 │Offset │ Length │ Name │
310 ├───────┼────────┼───────────────────────────────────────────────────┤
311 │ 0 │ 4 │ event time, seconds part (past UNIX epoch) │
312 │ 4 │ 4 │ event time, microseconds part │
313 │ 8 │ 4 │ archive volume number (0...N) │
314 │ 12 │ 4 │ byte offset in .meta file of pmDesc or pmLogIndom │
315 │ 16 │ 4 │ byte offset in archive volume file of pmResult │
316 └───────┴────────┴───────────────────────────────────────────────────┘
317 Since the temporal index is optional, and exists only to speed up time-
319 based random access to metrics and their metadata, the index records
320 are emitted only intermittently. An archive reader program should not
321 presume any particular rate of data flow into the index. However, com‐
322 mon events that may trigger a new temporal-index record include changes
323 in instance-domains, switching over to a new archive volume, and start‐
324 ing or stopping logging. One reliable invariant however is that, for
325 each index entry, there are to be no meta or archive-volume records
326 with a timestamp after that in the index, but physically before the
327 byte-offset in the index.
328
330 Several PCP tools create archives in standard locations:
331 $HOME/.pcp/pmlogger
333 default location for the interactive chart recording mode in
334 pmchart(1)
335 $PCP_LOG_DIR/pmlogger
336 default location for pmlogger_daily(1) and pmlogger_check(1)
337 scripts
338 $PCP_LOG_DIR/pmmgr
339 default location for the PCP daemon manager pmmgr(1)
340
342 PCPIntro(1), PMAPI(3), pmLookupDesc(3), pmLookupInDom(3), pmLookupIn‐
343 DomText(3), pmLookupLabels(3), pmLookupText(3), mkaf(1), pmafm(1),
344 pmchart(1), pmdumplog(1), pmlogger(1), pmlogger_check(1), pmlog‐
345 ger_daily(1), pmlogreduce(1), pmlogrewrite(1), pmlogsummary(1),
346 pmmgr(1), pcp.conf(5), and pcp.env(5).
347Performance Co-Pilot LOGARCHIVE(5)