Further to the above description of name, the set of rules describing external instance names is provided
in the pmdaCacheStore(3) manual page.
Instance domains adds another dimension (set of values) to metrics. However, this may not suffice to de‐
scribe complex multi-dimensional instance domain situations. For this case the approach used by a number
of PMDAs is to structure the external instance names using a delimiter (``/'' or ``::'' are most commonly
used) to allow separation of the other dimensions. In this situation, instance domain labels should be
used to define names for each instance name component. This allows PMAPI(3) client tools to identify and
refine value fetches to specific dimensions of interest.
For example, some of the Linux kernel cgroup (control group) metric instance domains are multi-dimension‐
al. The instance domain represents individual values across both control groups and CPUs, making this a
two-dimensional instance domain. The instance names associated with this cgroup metrics indom have been
structured using the ``::'' delimiter to separate the two dimensions. The instance domain itself has
been labeled accordingly, as follows.
$pminfo--desc--fetch--labelscgroup.cpuacct.usage_percpu
cgroup.cpuacct.usage_percpu
Data Type: 64-bit unsigned int InDom: 3.22 0xc00016
Semantics: counter Units: nanosec
inst [0 or "/::cpu0"] value 713787
inst [1 or "/::cpu1"] value 353969
inst [2 or "/app::cpu0"] value 407816
inst [3 or "/app::cpu1"] value 202747
inst [0 or "/::cpu0"] labels {"device_type":"cpu","cgroup":"/","cpu":0}
inst [1 or "/::cpu1"] labels {"device_type":"cpu","cgroup":"/","cpu":1}
inst [2 or "/app::cpu0"] labels {"device_type":"cpu","cgroup":"/app","cpu":0}
inst [3 or "/app::cpu1"] labels {"device_type":"cpu","cgroup":"/app","cpu":1}
$pminfo--labels3.22
InDom: 3.22 0xc00016
labels {"device_type":"cpu"}
As shown above the individual instances inherit the labels from the instance domain, and the PMDA also
applies additional per-instance labels describing individual cgroup and CPU names. When this model has
been used by the PMDA, PMAPI clients are able to restrict their queries to the cgroup metric instances -
in the example, restricting to processor "cpu0" using the "cpu" label, perhaps, or to just the "/app"
cgroup metrics using the "cgroup" label.
Furthermore, using this labeling scheme client tools can also correlate related instances across differ‐
ent instance domains.
$pminfo--desc--fetch--labelskernel.percpu.cpu.irq.soft
kernel.percpu.cpu.irq.soft
Data Type: 64-bit unsigned int InDom: 60.0 0xf000000
Semantics: counter Units: millisec
inst [0 or "cpu0"] value 6770
inst [1 or "cpu1"] value 100
inst [0 or "cpu0"] labels {"device_type":"cpu"}
inst [1 or "cpu1"] labels {"device_type":"cpu"}
$pminfo--labels60.0
InDom: 60.0 0xf000000
labels {"device_type":"cpu"}
Although these two metrics have different instance domains (60.0 and 3.22 respectively) and are sourced
from different PMDAs, the "device_type" label identifies the common device to which these values relate.