We're evaluating observability platforms for our LLM-powered application stack (approx 200 daily active users, mix of RAG and agentic workflows). The shortlist is down to LangSmith and Datadog's LLM Observability. I've been digging into the architectural and pricing models for both, and the TCO picture is less clear than I'd like.
From my analysis, LangSmith's strength is its deep integration with the LangChain ecosystem, offering granular tracing for chains, tools, and retrievers. Datadog provides a broader APM context, tying LLM spans to infrastructure metrics and traditional services. For a production system, this integration could be valuable, but it introduces complexity.
My primary concerns are cost predictability and scaling efficiency:
* **Pricing Models:** LangSmith uses a credit-based system (traces, tokens, evaluations). Datadog uses its standard data-ingested volume model for LLM traces, plus APM host/index costs.
* **Key Variables:** With 200 users, our cost drivers are trace volume (especially for long RAG sessions), token counts for LLM calls, and whether we need persistent dataset storage for evaluation.
* **Hidden Costs:** With Datadog, we must factor in the cost of retaining trace data for historical analysis. LangSmith's evaluation runs could become a significant credit consumer if we automate them.
Has anyone run a similar comparison for a mid-scale production deployment? I'm particularly interested in real-world data on monthly trace volumes per active user and how the cost curves behave as you scale from prototype to steady-state production. Which platform provided better tools for rightsizing—identifying wasteful LLM calls or redundant retrieval steps?
Show me the bill.
The hidden costs with Datadog's model are real, but they often show up in surprising places beyond just the APM host indexing. The real killer is how their LLM observability pricing interacts with their log management. If you're logging LLM request/response pairs at any detail to correlate with traces, you'll be paying the ingest rate twice, once for the trace span and once for the log event, for the same underlying data. Their tagging doesn't prevent that double dip.
You mentioned complexity as a concern, and that's the trade-off. Datadog's strength is also its weakness: you get the infrastructure correlation, but you're now managing sampling rules, retention policies, and cost allocation tags across three different products (APM, Logs, LLM Observability) just to see a coherent picture. For 200 users, that operational overhead might outweigh the integration benefit.
Have you run the numbers on what a single, particularly chatty RAG session would generate in terms of span count with each platform? That's usually where the credit-based model gets unpredictable. A user asking ten follow-up questions isn't ten traces, it's potentially hundreds of nested spans for retrievers, tool calls, and LLM invocations. One platform might count that as one "trace," the other might bill for every leaf node.
P99 or bust.