SYNOPSIS
$PCP_LOG_DIR/pmlogger/*/*.{meta,index,0}
$PCP_LOG_DIR/pmmgr/*/*.{meta,index,0}
DESCRIPTION
PCP log archives store volumes of historical values of arbitrary Performance Co-Pilot metrics recorded from a single host. Archives are self-contained in the sense that they contain all the important metadata that would be required for off-line or off-site analysis. The format is intended to be stable in order to allow long-term historical storage and processing by current tools. (Compatibility in the other direction - new files, old tools - is not as fully assured.)
Archives may be read by most PCP client tools, using the -a ARCHIVE option, or dumped raw by pmdumplog(1). Archives may be created by pmlogger(1) and bulk-import tools. Archives may be merged, analyzed, and subsampled using specialized tools such as pmlogsummary(1), pmlogreduce(1), pmlogrewrite(1), and pmlogextract(1). In addition, PCP archives may examined in sets or grouped together into PCP "archive folios", which are managed by the pmafm(1) tool.
PCP archives consist of several physical files that share a common arbitrary prefix, e.g., myarchive.
- myarchive.0, myarchive.1, ...
- Metric values. May grow rapidly.
- myarchive.meta
- Information for PMAPI functions such as pmLookupDesc(3) and pmGetInDom(3). May grow in fits and spurts, as logged instances and instance domains vary.
- myarchive.index
-
A temporal index, mapping timestamps to offsets in the other files.
Grows slowly.
COMMON FEATURES
All three types of files have a similar record-based structure, a convention of network-byte-order (big-endian) encoding, and 32-bit fields for tagging/padding for those records. Strings are stored as 8-bit characters without assuming a specific encoding, so normally ASCII. See also the __pmLog* types in include/pcp/impl.h.
RECORD FRAMING
The volume and meta files are divided into self-identifying records.
| |||||||||||||||
ARCHIVE LOG LABEL
All three types of files begin with a "log label" header, which identifies the host name, the time interval covered, and a time zone.
| |||||||||||||||||||||||||||
All fields, except for the current log volume number field, match for all archive-related files produced by a single run of the tool.
ARCHIVE VOLUME (.0, .1, ...) RECORDS
pmResult
After the archive log label record, an archive volume file contains metric values corresponding to the pmResult set of one pmFetch operation, which is almost identical to the form on disk. The record size may vary according to number of PMIDs being fetched, the number of instances for their domains. File size is limited to 2GB, due to storage of 32-bit offsets within the .index file.
| |||||||||||||||||||||||||||||||||
Records with a number-of-PMIDs equal to zero are "markers", and may represent interruptions, missing data, or time discontinuities in logging.
pmValueSet
This subrecord represents the measurements for one metric.
| ||||||||||||||||||||||||
The metric-description metadata for PMIDs is found in the .meta files. These entries are not timestamped, so the metadata is assumed to be unchanging throughout the archiving session.
pmValue
This subrecord represents one measurement for one instance of the metric. It is a variant type, depending on the parent pmValueSet's value-format field. This allows small numbers to be encoded compactly, but retain flexibility for larger or variable-length data to be stored later in the pmResult record.
| |||||||||||||||
The instance-domain metadata for PMIDs is found in the .meta files. Since the numeric mappings may change during the lifetime of the logging session, it is important to match up the timestamp of the measurement record with the corresponding instance-domain record. That is, the instance-domain corresponding to a measurement at time T are the records with largest timestamps T' <= T.
pmValueBlock
Instances of this subrecord are placed at the end of the pmValueSet, after all the pmValue subrecords. Iff needed, they are padded at the end to the next-higher 32-bit boundary.
| ||||||||||||||||||
Note that for PM_TYPE_STRING, the length includes an explicit NUL terminator byte. For PM_TYPE_EVENT, the value bytestring is further structured.
pmEventArray
(TBD)
METADATA FILE (.meta) RECORDS
After the archive log label record, the metadata file contains interleaved metric-description and timestamped instance-domain descriptors. File size is limited to 2GB, due to storage of 32-bit offsets within the .index file. Unlike the archive volumes, these records are not forced to 32-bit alignment! See also src/libpcp/src/logmeta.c.
pmDesc
Instances of this record represent the metric description, giving a name, type, instance-domain identifier, and a set of names to each PMID used in the archive volume.
| ||||||||||||||||||||||||||||||||||||||||||
pmLogIndom
Instances of this record represent the number-string mapping table of
an instance domain. The instance domain number will have already been
mentioned in a prior
pmDesc
record. Since new instances may appear over a long archiving run, these
records are timestamped, and must be searched when decoding
pmResult
records from the main archive volumes. Instance names may be reused
between instance numbers, so an offset-based string table is used that
could permit sharing.
| |||||||||||||||||||||||||||||||||||||||||||||
Records of this form replace the existing instance-domain: prior
records are not searched for resolving instance numbers in measurements
after this timestamp.
INDEX FILE (.index) RECORDS
After the archive log label record, the temporal index file contains a plainly concatenated, unframed group of tuples, which relate timestamps to 32-bit seek offsets in the volume and meta files. (This limits those files to 2GB in size.) These records are fixed-size, fixed-format, and are not enclosed in the standard length/payload/length wrapper: they just take up the entire remainder of the .index file. See also src/libpcp/src/logutil.c.
| |||||||||||||||||||||
Since temporal indexes are optional, and exist only to speed up time-wise random access of metrics and their metadata, index records are emitted only intermittently. An archive reader program should not presume any particular rate of data flow into the index. However, common events that may trigger a new temporal-index record include changes in instance-domains, switching over to a new archive volume, just starting or stopping logging. One reliable invariant however is that, for each index entry, there are to be no meta or archive-volume records with a timestamp after that in the index, but physically before the byte-offset in the index.