ELI5: Why does my bill spike when I add a new tag to my spans?

The perennial complaint: "I merely added a `customer_tier='premium'` tag to my spans for better analysis, and my observability bill doubled overnight." You'll be told this is due to "cardinality," which is correct, but the standard explanation often stops at "unique combinations," which is a surface-level abstraction. Let's dissect the actual mechanics, because your billing spike is a direct result of how modern distributed tracing systems are architecturally coupled to their underlying metrics pipelines.

Firstly, understand that most commercial observability platforms do not bill you purely on the volume of span bytes ingested. They bill on *metrics generated from your traces*. Every span that arrives is processed through a aggregation pipeline that creates metrics—latency, error rates, request counts—**dimensionalized by every tag on that span**. This is the hidden cost engine.

When you add a new tag like `customer_tier` with values `['free', 'basic', 'premium', 'enterprise']`, you are not just adding four new time series. You are **multiplying** the existing series. Consider:

You have an operation `POST /api/v1/order`. Its metrics are initially broken down by:
* `service_name` (1 value)
* `http.method` (1 value)
* `http.status_code` (3 values: 2xx, 4xx, 5xx)

That's a manageable number of time series for that operation. Now, add `customer_tier` with 4 values. The combinatorics explode:

```
Series count = (service_name) * (http.method) * (http.status_code) * (customer_tier)
= 1 * 1 * 3 * 4
= 12 series for this single operation
```

Now scale this to all your services, all HTTP methods, all status codes, and all your other tags (e.g., `datacenter`, `k8s.cluster.name`). The formula is multiplicative, not additive. A seemingly innocuous tag can increase your metric volume by an order of magnitude.

Secondly, the ingestion pipeline must now maintain and update this vastly larger set of series in real-time. This impacts their infrastructure costs, which they pass on to you. The platforms often implement **cardinality limits** precisely to protect their own systems from this multiplicative explosion, which they will then frame as a "usage limit" for your protection.

Here is a pragmatic configuration example from OpenTelemetry Collector that people often overlook, which can *accidentally* cause this. Sending all span attributes as metric dimensions:

```yaml
receivers:
otlp:
protocols:
grpc:

processors:
batch:

exporters:
# This exporter might be configured to extract metrics from spans
prometheusremotewrite:
endpoint: "https://my-obsv-platform.com/api/v1/prometheus/write"
# If this is tied to span attributes without careful filtering, you export everything.
# A new span attribute becomes a new Prometheus label.

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
metrics:
receivers: [otlp] # <-- This is key! The OTLP receiver for metrics might be ingesting span-generated metrics.
processors: [batch]
exporters: [prometheusremotewrite]
```

The mitigation strategy isn't simply "use fewer tags." It's to **audit your automatic metric generation settings**. You must explicitly define which span attributes are high-cardinality (e.g., `user_id`, `request_id`) and should never be used as metric dimensions, and which are low-cardinality, business-relevant attributes (e.g., `deployment.environment`, `customer_tier`) that are safe for aggregation. This is a data governance problem masquerading as a billing issue.

Plan for failure.