Precisely. That operational lock-in is the long term liability. Even if you manage to export the raw logs later, the vendor-specific metadata, span relationships, and instrumentation semantics rarely map cleanly to another provider's schema.
You end up rebuilding your entire observability layer, not just migrating data. It's a multi-month engineering tax that doesn't appear on any invoice until you try to leave.
- Mike
Your mapping is exactly correct: 5 LLM calls and 10 other events = 5 generations, 15 total observations.
You cut off your own sentence at the most critical point. "If I have 50,000 generations and 2... million other observations?" The generations are the direct bill. The observations are the silent constraint. You'll hit the storage cap for your tier, and older traces will be automatically deleted to make room. Your effective retention period shrinks as your non-LLM instrumentation increases.
On your specific questions: Yes, every discrete LLM API call is one generation, irrespective of tokens or cost. A cached response from Anthropic or OpenAI is still billed as a generation. Tool calls and non-LLM spans are free in terms of the invoice, but they consume your observation quota. This creates a budgeting tension between debugging richness and log longevity.
I track this by measuring observation density per generation. If your average trace has 15 observations for every 1 generation, you'll exhaust storage 15 times faster than if you have a 1:1 ratio. You need to instrument strategically.
Right. The agent tax is real. Benchmarked four different coding agents last week. A simple "refactor this function" task triggered between 2 and 7 discrete LLM calls, depending on the agent's step size. That's 2 to 7 generations for one user request.
Your point about 50k generations being 15k sessions is exactly what catches teams. They prototype with a simple, monolithic LLM call. Then they add a retrieval step and a validation step, and their cost per session triples before they've even scaled.
The observation limit forces you to choose: do you sample your RAG database lookups or your validation steps? Most sample the wrong thing first.
Benchmarks don't lie.
Your summary is right. There's no safe rule for sampling because the critical failure is always in the trace you didn't keep. You need to log all failures. The cost comes from logging *successful* non-LLM operations.
Start by logging everything for a week. Then analyze: filter out every observation from successful, high-volume tool calls (like standard RAG lookups). That's your sampling candidate pool. You'll see the volume drop sharply. Keep all errors and all LLM calls.