Retail's "high traffic spikes" is just vendor-speak for "please pay for our most expensive tier." Most observability tools fall over when you actually need them or bill you for the phantom traffic you *might* have.
I need something that doesn't implode during a flash sale or bill me for a million sampled tokens I never queried. OpenTelemetry-based? Proprietary? Who actually handles the cardinality of user sessions hitting an LLM-powered recommender at 3 PM without requiring a second mortgage?
Looking at cost per million tokens tracked, not marketing slides.
Your stack is too complicated.
I'm a community lead for a midsize online retailer, part of a team that runs a mix of in-house and vendor LLM services for search and customer support. We handle millions of product queries daily, with very real spikes during promotions.
**Actual Cost Structure:** Look for token-based pricing that excludes embeddings. At my last shop, we paid ~$4 per million tokens for *in/out only*, not the embedded context. That saved 60% versus vendors who billed for the full prompt. Watch for egress fees if you're on a cloud vendor.
**Scalability vs. Fidelity:** Proprietary agents often sample heavily under load (1 in 100 requests) to stay up. OpenTelemetry-based tools let you control the sampling rate in your pipeline, so you can keep 100% on key user sessions. The trade-off is you'll need to manage the collector resource limits.
**Integration Friction:** SDK-based tools can add 50-100ms of latency per call if they're doing inline analysis. We switched to a sidecar model that taps our OpenTelemetry streams, which cut that to under 10ms. The setup took our platform team about two weeks.
**The Breaking Point:** For us, it was cardinality from user session IDs. Some tools index every unique value and fall over. The one we use now lets us drop high-cardinality labels from indexes but keep them in raw logs. It held at ~2.5k req/s per collector node during our last major sale.
I'd recommend leaning into an OpenTelemetry-native platform given your focus on cost control and traffic spikes. But my pick depends on two things: whether your LLM stack is mostly vendor APIs (like OpenAI) or custom models, and if you already have OpenTelemetry instrumentation in place.
Keep it civil, keep it real.
Interesting that your ~$4 per million tokens didn't include embeddings. Did your billing dashboards actually prove the vendor was excluding them, or did you just take their word for it? I've seen "input/output only" line items quietly balloon with "processing overhead" charges once volume hits a certain threshold.
The sidecar model sounds good in theory, but moving to OpenTelemetry streams often just shifts the cost. Now you're paying for the collector infra and storage instead of the vendor's premium. That two-week setup is just the start - wait for the month-end bill when your cloud provider itemizes the inter-AZ traffic for that sidecar.
And cardinality from session IDs? That's the classic observability tax. Most tools promise to handle it until you hit a real spike, then the per-time-series cost kicks in.
cost_observer_42