Kubernetes metrics cardinality: cut what nobody reads

Kubernetes metrics cardinality drives your storage bill long before application traffic does. Every unique combination of labels creates a distinct time series, and many of those series are never displayed in a dashboard or used in an alert rule. They are still collected, stored and sized for like all the others.

This guide proposes four steps: measure, find the metrics nobody reads, cut at the source, then add guardrails. The examples use Prometheus and kube-state-metrics, the most common setup. The approach also applies to an OpenTelemetry Collector sending data to Dynatrace SaaS or Managed. It complements the module on Kubernetes metrics in Grail.

In short

• A series is a metric name plus a set of labels. One more label can multiply the number of series.
• The Prometheus /api/v1/status/tsdb API ranks the most expensive metrics and labels.
• You can cut at three levels: the exporter, the scrape (metric_relabel_configs) and the Collector.
• A sample_limit cap protects the platform, but fails the whole scrape when exceeded.

Why Kubernetes metrics cardinality explodes

According to the Prometheus naming documentation, every unique combination of label key-value pairs represents a new time series, which can greatly increase the stored data. The page advises against using labels for high-cardinality dimensions such as user IDs or email addresses.

Kubernetes produces this problem naturally. A pod has a unique name and identifier, and every deployment creates new ones. kube-state-metrics series carry the pod, namespace and uid labels, as the pod metrics documentation shows. The kube_pod_container_info metric adds image, image_id and container_id. At each redeployment these values change and new series appear.

The cost therefore comes not only from the number of pods, but also from churn: old series disappear, new ones are born, and the database must index them all. The topic differs from alerting on nodes and quotas: here, you reduce what is collected instead of deciding what to watch.

Step 1: measure with the Prometheus TSDB API

Before cutting, you need to know where the volume goes. The TSDB status API answers that. According to the HTTP API documentation, the endpoint is GET /api/v1/status/tsdb, added in version 3.6.0 according to that page. The optional limit parameter caps the number of items per statistic (10 by default, 10,000 at most).

curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=20' 

The response has five useful blocks:

• headStats: total number of series (numSeries) and chunks in the head block;
• seriesCountByMetricName: the metrics with the most series;
• labelValueCountByLabelName: the labels with the most distinct values;
• memoryInBytesByLabelName: memory used per label, computed by adding the length of all its values;
• seriesCountByLabelValuePair: the label=value pairs that concentrate the most series.

The page names the last field two ways: the list says seriesCountByLabelPair, the example says seriesCountByLabelValuePair. Check the exact name on your own server.

Step 2: find the metrics nobody reads

An unread metric appears in no alert rule, no dashboard and no on-call query. No API flags it automatically, so the inventory is manual, crossing two lists.

1. Export the ten to twenty largest names from seriesCountByMetricName.
2. Search each one in your alert rules, recording rules and dashboards (a plain grep on the configuration repository is enough).
3. For each metric you cannot find, ask the owning team who last used it, and for which incident.
4. Sort them: keep, keep without some labels, or delete.

Keep the signals that serve an SLI or a diagnosis, and remove the rest to cut toil. The text on observability versus monitoring explains why collecting more does not replace collecting well.

Step 3: cut at the source in kube-state-metrics

The cheapest option is not to produce the metric. According to the kube-state-metrics arguments documentation, three options matter:

• --resources: comma-separated list of enabled resources;
• --metric-denylist: metrics not to enable, by exact name or regular expression;
• --metric-labels-allowlist: extra Kubernetes label keys added to the resource’s labels metric.

--metric-allowlist and --metric-denylist are mutually exclusive. Pick one.

In the example above, only three resource types are exported, two families of experimental metrics are dropped, and only the app and team pod labels are copied. According to the pod metrics documentation, kube_pod_tolerations is marked EXPERIMENTAL, as are the detailed kube_pod_init_container_* status metrics. Read the page for your deployed version before copying this list.

Step 4: cut at scrape time with metric_relabel_configs

When the exporter cannot be changed, Prometheus can discard series before ingestion. The configuration reference states that metric relabelling applies to samples as the last step before ingestion, with the same format and actions as target relabelling. It lists excluding series that are too expensive to ingest as one use. It does not apply to automatically generated series such as up.

Two actions are used here:

• drop: removes series whose regex matches the concatenated source_labels (here the metric name, __name__);
• labeldrop: removes labels whose name matches the regex.

Be careful with labeldrop. The documentation asks you to make sure metrics remain uniquely labelled once labels are removed. If two series become identical, they collide. Start with labels that identify nothing useful, such as a container ID, and check the effect on numSeries after the change.

Step 5: add guardrails

Scrape limits turn a silent risk into a visible alert. According to the same reference, sample_limit sets a per-scrape limit on the number of accepted samples. The count is taken after metric relabelling, and above it the entire scrape is treated as failed. A value of 0 means no limit. label_limit caps the number of labels per sample, with the same behaviour when exceeded.

This has a cost: an overrun loses all the metrics of the target, not only the extra series. Set the limit with headroom, for example above the observed maximum, and alert on scrape failure (the up series drops to 0). A cap detects drift. It does not replace the inventory of steps 1 and 2.

OpenTelemetry side: filter in the Collector

If metrics go through a Collector, the filter processor discards data before export. The filter processor README gives the current key, metric_conditions, and says the older metrics.metric key is deprecated and will be removed. Its example drops the data points of a metric based on an OTTL condition:

processors: filter: error_mode: ignore metric_conditions: - metric.name == "k8s.pod.phase" and datapoint.value_int == 4 

This example comes from the documentation. Replace the condition with your own after testing in a non-critical environment. For a full Collector deployment, see the Collector guide for Kubernetes. To compare the two worlds, read OpenTelemetry and Prometheus. The Prometheus-to-Dynatrace path is covered in this article.

How to check that it works

1. Run curl 'http://localhost:9090/api/v1/status/tsdb?limit=20' again and compare numSeries before and after, over a comparable period.
2. Check that the removed metrics no longer appear in seriesCountByMetricName.
3. Confirm that critical alerts and dashboards still return data.
4. Watch target health: a sample_limit set too low shows up as a failed scrape.

Best practices

• Measure before cutting, and note the initial numSeries.
• Prefer an allowlist of resources to a denylist when the scope is known.
• Never use a label with unbounded values for your own metrics.

Frequently asked questions

What is the cardinality of a metric?

It is the number of distinct time series it produces, that is, the number of observed combinations of label values. The more possible values a label has, the higher the cardinality.

How do I find which metrics cost the most?

Query the Prometheus /api/v1/status/tsdb API: it ranks metrics by series count and labels by number of distinct values. According to the documentation, this API is available from version 3.6.0.

Can I lose useful information by dropping labels?

Yes. Removing a label with labeldrop can merge series or hide an analysis dimension. First check that no alert or dashboard uses it, and test on a single scrape job.

Conclusion

Reducing Kubernetes metrics cardinality is not a compression exercise: it is a sort by usage. Measuring with the TSDB API, finding metrics without readers, cutting at the source and capping the scrape is enough to regain control of the bill, without touching the signals that serve your SLOs.

span.observer is an independent blog, not affiliated with Dynatrace LLC. Product names belong to their respective owners; the configurations are examples to adapt and test before any production use.

Originally published on span.observer. French version: https://span.observer/fr/cardinalite-metriques-kubernetes/.