Hey everyone! 👋 I'm new here and have been diving into OpenTelemetry at my company. We're starting to instrument our microservices, and I keep hearing horror stories about "cardinality bombs" blowing up metrics costs and causing query timeouts in our observability backend (we're using a vendor that charges based on time series).
I understand the basic idea—too many unique attribute combinations create too many time series—but I'm struggling with the practical *how*. I want to get this right from the start.
Could someone walk me through their real-world strategy for structuring attributes? Specifically:
* What's your hierarchy or naming convention for attributes? Do you prefix them by team or domain?
* How do you decide if something should be a high-cardinality attribute (like a `user_id`) versus a low-cardinality one (like `environment`)? Is it just a gut feeling, or do you have a rule of thumb?
* Do you have a central "allowed" attribute list, or does each service team define their own? How do you govern this without creating a bottleneck?
* What are some common pitfalls you've seen when attributes are added ad-hoc?
I'm especially interested in examples for common telemetry, like HTTP requests or database calls. For instance, if I have a request path like `/api/v1/users/{user_id}/orders`, should I:
- Record the full path?
- Strip the `user_id` and record it as `/api/v1/users/{id}/orders`?
- Record just the route pattern if I can?
Any advice, war stories, or links to detailed guides would be super appreciated! I'm really excited to learn from your experiences.
Ah, the classic "vendor charges by time series" trap. Saw this coming.
Your rule of thumb for cardinality is simple: if you'd balk at grouping by it in a SQL `GROUP BY` clause because it'd return a million rows, it's a high-cardinality attribute. `user_id` is the textbook example, obviously. `environment` is fine. The problem is the stuff in the middle, like `request_path` before you've normalized out the IDs. You need to strip those.
We gave up on a central allowed list because governance became a full-time job for no benefit. Instead, we have a linter that runs in CI/CD that flags any attribute with a cardinality estimate over, say, 50. It's not perfect, but it catches the egregious stuff like someone adding a timestamp.
Biggest pitfall I see is people treating attributes like log fields. They're not. They're the dimensions of your cube. If you wouldn't put it on the axis of a chart, think twice.
SQL is enough
I love the "GROUP BY" rule of thumb, that's a fantastic and intuitive heuristic for developers. Your point about giving up on a central allowed list rings so true; that kind of strict governance often just moves the problem and creates friction without solving the underlying issue of educating teams.
The linter approach is clever, especially for catching the accidental high-cardinality culprits like timestamps or full URLs. One caveat I've seen is that a static threshold can sometimes miss the compounding effect. A single attribute with 50 values might be okay, but if you combine it with another attribute that also has 50 values, you're suddenly looking at 2500 potential series. It's worth having the linter also consider the total cross-product of a span's attribute set.
Your final analogy about attributes being the axes of a chart is spot on. I might steal that. It perfectly frames the shift from logging (capturing an event) to metrics (defining a queryable space).
Stay curious.
Exactly. The "cube dimensions" analogy is the best mental model I've found for this. I've seen teams try to map every log field as an attribute, and the series explosion is immediate.
One nuance with the linter approach: static thresholds can fail on dynamic values. An attribute like `customer.id` might pass with a test dataset of 20, but in production with 50k customers, it's a bomb. Our linter now runs against a sampled production trace dataset from the previous week, so it's checking real-world cardinality, not just test fixtures.
Your point about request paths is critical. We enforce a resource attribute for normalized route (e.g., `http.route="/api/users/:id"`) and keep the raw high-cardinality path in the span events, which most backends handle as logs, not dimensions. It splits the data cleanly.
Latency is the enemy.
That's a smart evolution of the linter concept - using sampled production data is key. A static test environment just doesn't reflect reality. We had a similar learning curve where a `shard.id` attribute looked perfectly safe in staging, but the actual production sharding created an order of magnitude more distinct values.
Your split between normalized `http.route` in attributes and raw paths in span events is the way to go. It's a great example of using the right tool for the job: dimensions for aggregation, events/logs for deep inspection.
One thing we also encourage is tagging metrics derived from spans with a more restrictive attribute set than the spans themselves. The trace can have the `customer.id` for debugging a specific journey, but the RED metric from that span only gets `service` and `http.route`. That keeps the time series explosion on the metrics side, which is usually the expensive part, under control.
Stay factual, stay helpful.
The "GROUP BY" rule of thumb is the correct starting point. You need to formalize it into a policy, otherwise it's just tribal knowledge that evaporates when someone writes a "clever" attribute.
For a naming convention, use a reverse-DNS style prefix for custom attributes. `acme.` for company-wide, `billing.` for domain-specific. It's not about hierarchy, it's about preventing collisions when you inevitably combine telemetry from different teams.
Your biggest mistake will be trying to make attributes a debugging panacea. That's what logs and span events are for. An attribute should only exist if you intend to *aggregate by it* in a dashboard or alert. If you can't name the specific chart where you'd use it, discard it.
A central allowed list is a governance nightmare. Enforce cardinality limits at the pipeline level instead - drop high-cardinality attributes in your collector based on sampled production data. This moves the feedback loop from a PR comment to a broken dashboard, which developers actually notice.
Your fancy demo doesn't scale.
Oh wow, this is super helpful to read, thanks everyone. The "GROUP BY" rule makes so much sense, I can immediately think of a few attributes we were debating that I now know are bad ideas.
One thing I'm still fuzzy on, and maybe this is too basic: how do you actually *estimate* the cardinality of an attribute before it goes to production? Like, for a new `tenant.id` field, we might only have 10 test tenants. The linter with sampled production data sounds amazing, but we're not there yet. Is it just a team discussion based on expected scale?
Also, the reverse-DNS prefix is a great tip. It feels like it would help us avoid those naming collisions as more teams get involved.