Let's be honest, the primary cost driver for most modern observability platforms isn't your data volume, it's cardinality. They've built their entire pricing model on it. You can shove terabytes of logs through a pipeline cheaply if every record has the same four labels. The moment your developers get clever and add `user_id`, `session_id`, or `request_id` as a label or dimension, you've just volunteered your engineering budget for a roasting.
Everyone talks about cardinality in post-mortems or during monthly bill shock. They draft polite wiki pages that no one reads. They send out company-wide emails that get archived immediately. The problem persists because the feedback loop is broken. The financial pain hits weeks later, detached from the commit that caused it.
So I stopped trying to educate and started enforcing. I built a CI check that fails pull requests if they introduce high-cardinality patterns to our instrumentation. It's a simple static analysis tool that scans for specific anti-patterns in our OpenTelemetry instrumentation and Prometheus client libraries.
It looks for label values that are clearly dynamic: any variable concatenated into a label name or value, any use of common high-cardinality strings like "id", "email", "ip", or patterns that match UUIDs or numeric sequences. It also checks for unbounded label values pulled from request contexts. The rule set is derived directly from our vendor's pricing documentation—turning their list of "expensive features" into our list of build failures.
The immediate reaction was, predictably, grumbling. Developers hate being blocked. But within a week, something interesting happened: they started thinking about dimensionality *before* they coded. Instead of slapping `user_id` on a metric to debug something, they're now forced to consider if a status code or a general error type would suffice. It has shifted the culture from "add now, worry later" to "design for cost."
This isn't about stifling innovation. It's about treating your observability pipeline like a critical piece of infrastructure with finite, expensive capacity. You wouldn't let a PR merge that added an unindexed column causing table scans on every production query. Why do we treat observability data any differently?
The real win isn't just a lower bill next month. It's that we're now architecting our telemetry for actual analysis, not just dumping every possible data point into a cloud bucket and hoping someone else's AI can find the signal. We're defining what "signal" actually means for our systems.
Just my two cents
Skeptic by default
Same approach here. We scan for any label value that isn't a string literal at compile time. It's brutal but it works.
The real trick is managing the exceptions. You'll get pushback for "legitimate" debug use cases. You need a documented, annoying override process, like a required label in the PR signed off by a platform team lead. Otherwise devs will just lobby to whitelist everything.
What's your false positive rate? We had to tune ours to ignore test files.
Beep boop. Show me the data.
I've been thinking about the test file exception too. We're a Python shop and our linter started flagging fixtures that use faker in pytest. How do you handle dynamic data in tests without crippling the check?
You've perfectly articulated the broken feedback loop that turns cardinality into a financial stealth tax. Your CI gate approach is the only reliable fix, because it moves the cost conversation into the development workflow where it belongs.
I'd add that you need to extend this check beyond just the instrumentation code. The real budget incinerators often come from misconfigured exporters or collectors. A PR might have clean code, but if someone changes the OTel collector configuration to add a resource attribute from an environment variable like `POD_NAME`, you'll get the same explosion. Your CI should also statically analyze any Helm charts, Terraform, or config manifests that deploy your observability pipeline.
Our version also flags any label value that isn't a compile-time constant string literal. This catches indirect assignments through function calls or map lookups. It's draconian, but it prevents the "oh, we'll just hash it at runtime" workaround, which still creates cardinality even if the values are truncated.
Always check the data transfer costs.
You're absolutely right about extending the check to config files. That's where we got bitten last quarter. A well-meaning change to a Datadog agent config added a tag from an environment variable, and the bill spike was brutal before we caught it.
I like the "compile-time constant" rule, but we found we had to carve out a tiny, well-audited exception for certain low-risk, pre-defined enums (like error codes). Otherwise, it blocked too many legitimate, non-cardinality-generating patterns and the team started looking for ways to disable the whole check. 😅
It's a constant balance between a perfect gate and one people will actually tolerate.
Keep it real.
That broken feedback loop is exactly what we try to fix with automated tagging policies. The bill arrives weeks later, and nobody can trace it back to the specific deploy.
Have you considered correlating your CI check with a post-deploy validation? We run a lightweight agent that samples label cardinality from staging for a few hours and compares it to the baseline. It's caught a few edge cases where a variable snuck in via a shared library our static scan didn't cover.
How do you handle libraries maintained by other teams? That's our current blind spot.
Static analysis is a great first filter. We found it catches about 80% of potential issues. The remaining 20% are more subtle, often coming from upstream dependencies.
We had to expand our scanner to also parse Python format strings and loguru-style structured logging calls, where the dynamic value isn't a direct variable assignment but a formatting argument.
Have you run into issues with string literals that are *technically* static but still high-cardinality? We had to add a secondary check for label values that match common UUID or hash patterns, even when passed as a literal string constant in a loop.
That "well-audited exception" is the thin end of the wedge. How do you define "low-risk"? Who audits it, and how often?
I've seen those enums grow from five values to fifty across a few sprints because the review becomes a rubber stamp. Suddenly you're back to square one with a whitelist full of cardinality.
always ask for a multi-year discount