In my current role, we're instrumenting a complex RAG pipeline that's becoming increasingly expensive to trace. We're using LangChain with OpenTelemetry, feeding spans into a vendor's observability platform. The volume is substantial—every user query can generate 10-15 spans covering retrieval, LLM calls, and post-processing.
Our default retention was set to 30 days by the platform, but we're hitting a practical decision point. The engineering team wants longer retention for debugging sporadic quality regressions, but finance is questioning the storage costs. We're now evaluating a tiered strategy.
My proposed configuration, which we're piloting:
* **Raw span data:** 30 days in hot storage. This covers immediate debugging, daily alert investigations, and most weekly performance reviews.
* **Aggregated metrics (p95 latency, error rates, token counts):** 13 months rolled up per service. This is for quarterly business reviews and year-over-year trend analysis.
* **Sampled traces (10% of all traces, plus all errors):** 90 days in warm storage. This extended period aims to capture those elusive bugs that only surface with specific data combinations reported by users weeks later.
The critical question is the sampling logic for that 90-day bucket. We're moving beyond simple random sampling. Our draft OpenTelemetry sampler logic looks like this:
```python
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, Decision
class ConditionalSampler:
def __init__(self, default_rate=0.1):
self.default_sampler = TraceIdRatioBased(default_rate)
def should_sample(self, context, trace_id, name, attributes, links):
# Always sample errors
if attributes and attributes.get("error") == True:
return Decision(True)
# Always sample high-latency operations (>5s)
if name in ["llm_invocation", "vector_search"]:
return Decision(True) if attributes.get("estimated_latency", 0) > 5.0 else self.default_sampler.should_sample(context, trace_id, name, attributes, links)
# Default to the probabilistic sampler
return self.default_sampler.should_sample(context, trace_id, name, attributes, links)
```
I'm curious how others are navigating this trade-off between diagnostic capability and operational cost. What's your retention policy, and more importantly, what's the rationale behind the specific timeframes you've chosen? Are you using similar tiered approaches, or have you found a single retention period sufficient?
benchmark or bust
benchmark or bust