Skip to content
Notifications
Clear all

Guide: Reducing metric cardinality in your Kubernetes deployments.

3 Posts
3 Users
0 Reactions
76 Views
(@craigs)
Reputable Member
Joined: 4 months ago
Posts: 294
Topic starter   [#9024]

Everyone's pushing you to instrument everything. Then you get the bill. The cardinality explosion from k8s labels is the silent killer of observability budgets.

Most guides miss the practical cuts. Here’s what actually works:

* **Label whitelisting at the agent.** Don’t just drop pods. Configure your collector (e.g., Otel, Prometheus) to only ingest labels you actually use in alerts or dashboards. `kube_pod_labels` with 20 labels per pod is a trap.
* **Beware of "standard" Helm charts.** The default values for many community charts add a dozen labels like `chart`, `heritage`, `app.kubernetes.io/version`. Half of them are useless for querying. Strip them at source.
* **Instrumentation libraries are culprits.** A lot of auto-instrumentation adds dimensions (HTTP method, status code). You need to explicitly disable high-cardinality ones you don't need.
* **The hidden cost of "free" platforms.** Many SaaS vendors have soft limits that become hard limits the moment you need support. Your "unlimited" plan has a throttling clause. Read it.

The goal isn't to stop collecting. It's to stop paying for labels that never trigger an alert or inform a decision. What labels did you cut that actually mattered?


Read the contract


   
Quote
(@joshua73)
Eminent Member
Joined: 4 months ago
Posts: 24
 

The hidden cost of "free" platforms is a crucial point. Many vendors base their enterprise support tier pricing on ingested volume. Once you exceed their 'standard' tier's soft limits, the next quote can be a 300% jump. It shifts the conversation from engineering to procurement.

Has anyone quantified the cost per-label-per-pod in a specific vendor's pricing model? I've only seen per-sample or per-gigabyte.



   
ReplyQuote
(@integration_ian_2)
Honorable Member
Joined: 6 months ago
Posts: 525
 

Absolutely. The "stop paying for labels that never trigger an alert" is the perfect framing. It turns a technical problem into a cost-benefit one our finance folks actually understand.

One practical step we took was tagging our dashboards and alerts in Grafana with the specific labels they used. We then ran a script to scrape those and compare against the label allow-list in our Prometheus relabel config. Found a bunch of legacy `owner` and `feature-branch` labels still being ingested that hadn't been referenced in anything for over a year. Cut them and saw a noticeable dip in series churn almost immediately.

It feels backwards, but you have to start from your dashboards and work backwards to the source, not the other way around.


api first


   
ReplyQuote