We've hit the wall with DIY logging. Our internal platform uses a mix of Azure OpenAI, Anthropic, and a fine-tuned Llama model, handling about 50k requests daily across 500 internal sales and support users. SLAs are strict, especially for our quote generation workflows—99% uptime, p95 latency under 2 seconds.
I've sat through demos from all the usual suspects. The dashboards look pretty, but when I ask about overhead impact on latency, or how they handle tracing a chain that crosses our CRM (Salesforce) and a middleware layer, the answers get vague. We can't afford a tool that adds 200ms of observability tax per LLM call.
I need something that:
* Handles high volume without falling over or costing more than our actual inference.
* Traces beyond a single LLM call—full user session context, including our custom functions and API calls to other systems.
* Provides actual, actionable alerts for latency spikes and error rate increases, not just pretty graphs.
* Has granular cost tracking per model, per team, per project. We need to chargeback.
* On-prem or VPC deployment is a must. No shipping full prompts/responses to a third-party cloud.
What are you using in production at this scale? I care about the ugly details: the agent memory leak you found, the sampling that broke your analytics, the vendor that couldn't keep up during peak. Give me the real performance data, not the feature list.