1PCP-ARCHIVE(5) File Formats Manual PCP-ARCHIVE(5)
2
6 pcp-archive - Archive Files for Performance Co-Pilot
7
10 $PCP_LOG_DIR/pmlogger/*/*.{meta,index,0}
11 $PCP_LOG_DIR/pmmgr/*/*.{meta,index,0}
13
16 PCP log archives store volumes of historical values of arbitrary Per‐
17 formance Co-Pilot metrics recorded from a single host. Archives are
18 self-contained in the sense that they contain all the important meta‐
19 data that would be required for off-line or off-site analysis. The
20 format is intended to be stable in order to allow long-term historical
21 storage and processing by current tools. (Compatibility in the other
22 direction - new files, old tools - is not as fully assured.)
23 Archives may be read by most PCP client tools, using the -a ARCHIVE
25 option, or dumped raw by pmdumplog(1). Archives may be created by
26 pmlogger(1) and bulk-import tools. Archives may be merged, analyzed,
27 and subsampled using specialized tools such as pmlogsummary(1), pmlo‐
28 greduce(1), pmlogrewrite(1), and pmlogextract(1). In addition, PCP ar‐
29 chives may be grouped together into PCP "archive folios", which are
30 managed by the pmafm(1) tool.
31 PCP archives consist of several physical files that share a common
33 arbitrary prefix, e.g., myarchive.
34 myarchive.0, myarchive.1, ...
36 Metric values. May grow rapidly.
37 myarchive.meta
39 Information for PMAPI functions such as pmLookupDesc(3) and
40 pmGetInDom(3). May grow in fits and spurts, as logged instances
41 and instance domains vary.
42 myarchive.index
44 A temporal index, mapping timestamps to offsets in the other
45 files. Grows slowly.
46
49 All three types of files have a similar record-based structure, a con‐
50 vention of network-byte-order (big-endian) encoding, and 32-bit fields
51 for tagging/padding for those records. Strings are stored as 8-bit
52 characters without assuming a specific encoding, so normally ASCII.
53 See also the __pmLog* types in include/pcp/impl.h.
54 RECORD FRAMING
57 The volume and meta files are divided into self-identifying records.
58 ┌───────┬────────┬─────────────────────────────────────────────────────┐
60 │Offset │ Length │ Name │
61 ├───────┼────────┼─────────────────────────────────────────────────────┤
62 │ 0 │ 4 │ N, length of record, in bytes, including this field │
63 │ 4 │ N-8 │ record payload, usually starting with a 32-bit tag │
64 │ N-4 │ 4 │ N, length of record (again) │
65 └───────┴────────┴─────────────────────────────────────────────────────┘
66 ARCHIVE LOG LABEL
68 All three types of files begin with a "log label" header, which identi‐
69 fies the host name, the time interval covered, and a time zone.
70 ┌───────┬────────┬────────────────────────────────────────────────────┐
72 │Offset │ Length │ Name │
73 ├───────┼────────┼────────────────────────────────────────────────────┤
74 │ 0 │ 4 │ tag, PM_LOG_MAGIC | PM_LOG_VERS02=0x50052602 │
75 │ 4 │ 4 │ pid of pmlogger process that wrote file │
76 │ 8 │ 4 │ log start time, seconds part (past UNIX epoch) │
77 │ 12 │ 4 │ log start time, microseconds part │
78 │ 16 │ 4 │ current log volume number (or -1=.meta, -2=.index) │
79 │ 20 │ 64 │ name of collection host │
80 │ 80 │ 40 │ time zone string ($TZ environment variable) │
81 └───────┴────────┴────────────────────────────────────────────────────┘
82 All fields, except for the current log volume number field, match for
83 all archive-related files produced by a single run of the tool.
84
87 pmResult
88 After the archive log label record, an archive volume file contains
89 metric values corresponding to the pmResult set of one pmFetch opera‐
90 tion, which is almost identical to the form on disk. The record size
91 may vary according to number of PMIDs being fetched, the number of
92 instances for their domains. File size is limited to 2GB, due to stor‐
93 age of 32-bit offsets within the .index file.
94 ┌────────┬────────┬───────────────────────────────────────────┐
96 │Offset │ Length │ Name │
97 ├────────┼────────┼───────────────────────────────────────────┤
98 │ 0 │ 4 │ timestamp, seconds part (past UNIX epoch) │
99 │ 4 │ 4 │ timestamp, microseconds part │
100 │ 8 │ 4 │ number of PMIDs with data following │
101 │ 12 │ M │ pmValueSet #0 │
102 │ 12+M │ N │ pmValueSet #1 │
103 │12+M+N │ ... │ ... │
104 │ NOP │ X │ pmValueBlock #0 │
105 │ NOP+X │ Y │ pmValueBlock #1 │
106 │NOP+X+Y │ ... │ ... │
107 └────────┴────────┴───────────────────────────────────────────┘
108 Records with a number-of-PMIDs equal to zero are "markers", and may
109 represent interruptions, missing data, or time discontinuities in log‐
110 ging.
111 pmValueSet
114 This subrecord represents the measurements for one metric.
115 ┌───────┬────────┬────────────────────────────────────────────────┐
118 │Offset │ Length │ Name │
119 ├───────┼────────┼────────────────────────────────────────────────┤
120 │ 0 │ 4 │ PMID │
121 │ 4 │ 4 │ number of values │
122 │ 8 │ 4 │ storage mode, PM_VAL_INSITU=0 or PM_VAL_DPTR=1 │
123 │ 12 │ M │ pmValue #0 │
124 │ 12+M │ N │ pmValue #1 │
125 │12+M+N │ ... │ ... │
126 └───────┴────────┴────────────────────────────────────────────────┘
127 The metric-description metadata for PMIDs is found in the .meta files.
129 These entries are not timestamped, so the metadata is assumed to be
130 unchanging throughout the archiving session.
131 pmValue
134 This subrecord represents one measurement for one instance of the met‐
135 ric. It is a variant type, depending on the parent pmValueSet's value-
136 format field. This allows small numbers to be encoded compactly, but
137 retain flexibility for larger or variable-length data to be stored
138 later in the pmResult record.
139 ┌───────┬────────┬───────────────────────────────────────────────┐
142 │Offset │ Length │ Name │
143 ├───────┼────────┼───────────────────────────────────────────────┤
144 │ 0 │ 4 │ number in instance-domain (or PM_IN_NULL=-1) │
145 │ 4 │ 4 │ value (INSITU) or │
146 │ │ │ offset in pmResult to our pmValueBlock (DPTR) │
147 └───────┴────────┴───────────────────────────────────────────────┘
148 The instance-domain metadata for PMIDs is found in the .meta files.
150 Since the numeric mappings may change during the lifetime of the log‐
151 ging session, it is important to match up the timestamp of the measure‐
152 ment record with the corresponding instance-domain record. That is,
153 the instance-domain corresponding to a measurement at time T are the
154 records with largest timestamps T' <= T.
155 pmValueBlock
158 Instances of this subrecord are placed at the end of the pmValueSet,
159 after all the pmValue subrecords. Iff needed, they are padded at the
160 end to the next-higher 32-bit boundary.
161 ┌───────┬────────┬────────────────────────────────────────────────┐
164 │Offset │ Length │ Name │
165 ├───────┼────────┼────────────────────────────────────────────────┤
166 │ 0 │ 1 │ value type (same as pmDesc.type) │
167 │ 1 │ 3 │ 4 + N, the length of the subrecord │
168 │ 4 │ N │ bytes that make up the raw value │
169 │ 4+N │ 0-3 │ padding (not included in the 4+N length field) │
170 └───────┴────────┴────────────────────────────────────────────────┘
171 Note that for PM_TYPE_STRING, the length includes an explicit NUL ter‐
172 minator byte. For PM_TYPE_EVENT, the value bytestring is further
173 structured.
174 pmEventArray
177 (TBD)
178
181 After the archive log label record, the metadata file contains inter‐
182 leaved metric-description and timestamped instance-domain descriptors.
183 File size is limited to 2GB, due to storage of 32-bit offsets within
184 the .index file. Unlike the archive volumes, these records are not
185 forced to 32-bit alignment! See also src/libpcp/src/logmeta.c.
186 pmDesc
189 Instances of this record represent the metric description, giving a
190 name, type, instance-domain identifier, and a set of names to each PMID
191 used in the archive volume.
192 ┌───────┬────────┬──────────────────────────────────────────────────┐
195 │Offset │ Length │ Name │
196 ├───────┼────────┼──────────────────────────────────────────────────┤
197 │ 0 │ 4 │ tag, TYPE_DESC=1 │
198 │ 4 │ 4 │ pmid │
199 │ 8 │ 4 │ type (PM_TYPE_*) │
200 │ 12 │ 4 │ instance domain number │
201 │ 16 │ 4 │ semantics of value (PM_SEM_*) │
202 │ 20 │ 4 │ units: bit-packed pmUnits │
203 │ 4 │ 4 │ number of alternative names for this PMID │
204 │ 28 │ 4 │ N: number of bytes in this name │
205 │ 32 │ N │ bytes of the name, no NUL terminator nor padding │
206 │ 32+N │ 4 │ N2: number of bytes in next name │
207 │ 36+N │ N2 │ bytes of the name, no NUL terminator nor padding │
208 │ ... │ ... │ ... │
209 └───────┴────────┴──────────────────────────────────────────────────┘
210 pmLogIndom
212 Instances of this record represent the number-string mapping table of
213 an instance domain. The instance domain number will have already been
214 mentioned in a prior pmDesc record. Since new instances may appear
215 over a long archiving run, these records are timestamped, and must be
216 searched when decoding pmResult records from the main archive volumes.
217 Instance names may be reused between instance numbers, so an offset-
218 based string table is used that could permit sharing.
219 ┌─────────┬────────┬───────────────────────────────────────────────┐
222 │ Offset │ Length │ Name │
223 ├─────────┼────────┼───────────────────────────────────────────────┤
224 │ 0 │ 4 │ tag, TYPE_INDOM=2 │
225 │ 4 │ 4 │ timestamp, seconds part (past UNIX epoch) │
226 │ 8 │ 4 │ timestamp, microseconds part │
227 │ 12 │ 4 │ instance domain number │
228 │ 16 │ 4 │ N: number of instances in domain, normally >0 │
229 │ 20 │ 4 │ first instance number │
230 │ 24 │ 4 │ second instance number (if appropriate) │
231 │ ... │ ... │ ... │
232 │ 20+4*N │ 4 │ first offset into string table (see below) │
233 │20+4*N+4 │ 4 │ second offset into string table (etc.) │
234 │ ... │ ... │ ... │
235 │ 20+8*N │ M │ base of string table, containing │
236 │ │ │ packed, NUL-terminated instance names │
237 └─────────┴────────┴───────────────────────────────────────────────┘
238 Records of this form replace the existing instance-domain: prior
239 records are not searched for resolving instance numbers in measurements
240 after this timestamp.
241
244 After the archive log label record, the temporal index file contains a
245 plainly concatenated, unframed group of tuples, which relate timestamps
246 to 32-bit seek offsets in the volume and meta files. (This limits
247 those files to 2GB in size.) These records are fixed-size, fixed-for‐
248 mat, and are not enclosed in the standard length/payload/length wrap‐
249 per: they just take up the entire remainder of the .index file. See
250 also src/libpcp/src/logutil.c.
251 ┌───────┬────────┬───────────────────────────────────────────────────┐
254 │Offset │ Length │ Name │
255 ├───────┼────────┼───────────────────────────────────────────────────┤
256 │ 0 │ 4 │ event time, seconds part (past UNIX epoch) │
257 │ 4 │ 4 │ event time, microseconds part │
258 │ 8 │ 4 │ archive volume number (0...N) │
259 │ 12 │ 4 │ byte offset in .meta file of pmDesc or pmLogIndom │
260 │ 16 │ 4 │ byte offset in archive volume file of pmResult │
261 └───────┴────────┴───────────────────────────────────────────────────┘
262 Since temporal indexes are optional, and exist only to speed up time-
263 wise random access of metrics and their metadata, index records are
264 emitted only intermittently. An archive reader program should not pre‐
265 sume any particular rate of data flow into the index. However, common
266 events that may trigger a new temporal-index record include changes in
267 instance-domains, switching over to a new archive volume, just starting
268 or stopping logging. One reliable invariant however is that, for each
269 index entry, there are to be no meta or archive-volume records with a
270 timestamp after that in the index, but physically before the byte-off‐
271 set in the index.
272
275 PCPIntro(1), PMAPI(3), pmlogger(1), pmdumplog(1), pmafm(1),
276 pcp.conf(5), and pcp.env(5).
277Performance Co-Pilot PCP-ARCHIVE(5)