The recent outage and subsequent billing incident at Claw Corp provides a compelling case study for a critical, often overlooked tension in modern observability: the dissonance between the promised "pay for what you use" pricing model and the operational reality of high-cardinality, high-throughput systems.
Claw’s architecture, a globally distributed microservice mesh for payment processing, naturally generates immense telemetry. Their observability vendor’s pricing is based on ingested volume and indexed span/request cardinality. Under normal load, their controls—static sampling rates, basic log filters—kept costs predictable. However, during the cascade failure that caused the outage, three factors interacted catastrophically from a cost perspective:
* **Error-Driven Volume Explosion:** As retries and circuit-breaking logic engaged, the sheer number of error logs and high-latency traces increased by two orders of magnitude.
* **Cardinality Spike from Failover:** The failure of a primary regional cluster triggered automated failover mechanisms. This injected new `region`, `failover_attempt`, and `recovery_session` tags into virtually all spans, exploding indexed cardinality.
* **Sampling Inefficacy:** Their head-based sampling, configured at 10%, was applied before these tags were added (a common implementation detail). The 10% of requests that were sampled now each carried the new high-cardinality tags, which were fully indexed. The sampling controlled volume but not the cost driver of unique tag combinations.
The result was a billing event that reportedly exceeded their annual observability budget in a single day. This forces the question: if an anomalous, undesirable system state can trigger financially ruinous observability costs, are we truly paying for "value" or "usage"? Or are we paying for the privilege of debugging our own failures under a model that monetizes our distress?
The core tradeoff here is between resolution and cost predictability. To achieve true predictability, one must implement controls that are reactive not just to technical conditions, but to financial ones. This moves the complexity into your instrumentation pipeline. For example, a potential technical mitigation involves dynamic filtering at the collector level:
```yaml
# Pseudo-config for an OpenTelemetry Collector processor
processors:
attributes/conditional_drop:
actions:
- key: http.status_code
value: 500
action: delete
# Only apply this action if cardinality for 'failover_attempt' is above threshold
condition: cardinality["failover_attempt"] > 100
probabilistic_sampler/late_sampling:
sampling_percentage: 10
# Sample AFTER high-cardinality tags are potentially dropped
attribute_source: trace
```
This approach requires maintaining a feedback loop from your observability backend's billing API to your collector configuration—a non-trivial distributed systems problem in itself.
The deeper discussion I wish to prompt is whether the industry's volume-based model is fundamentally misaligned with the stochastic nature of complex systems. Should pricing be based on a smoothed, predictable commitment rather than raw ingestion, which is inherently spiky? Or does the onus fall entirely on engineering teams to build financial circuit breakers into their telemetry pipelines, treating cost as just another SLO? Claw's experience suggests that without such design, "pay for what you use" becomes "pay for what fails," which is a very different proposition.
brianh