Okay, I need to vent about this because I just ran the numbers for my team's upcoming project and... yikes. 😬
I've been a big proponent of LangSmith for prototyping and testing our LLM features. The tracing, the dataset management, the eval suites—it's genuinely fantastic for the iterative design and validation phase. It feels like Figma for LLM pipelines, letting you really see what's happening and test variations quickly.
But here's the brutal reality: the pricing model seems to scale in a way that feels punitive for actual production use. That initial tier is great, but the jump to the next one is steep, and it's based on "evaluation runs" and "traces." When you're moving from prototyping to a live SaaS feature with real user volume, those numbers explode. Suddenly, you're looking at a cost that rivals or even exceeds your actual LLM provider costs, which is wild to me.
It creates this perverse incentive where the tool that's essential for building a reliable feature becomes something you might have to limit or work around once it's live. How does that make sense? For a team that's big on continuous testing and monitoring, the idea of rationing "evaluation runs" or sampling traces because of cost is a total dealbreaker. It feels like you're penalized for using the tool properly.
I really hope they reconsider this structure. Maybe a flat enterprise fee, or pricing based on seats + retained data, not per-action. As it stands, we're now scrambling to evaluate more custom-built (and honestly, less capable) alternatives for our production monitoring, which is a shame because the product itself is so good. Anyone else hitting this wall?
You've nailed the exact transition point where the financial model shifts from enabling you to being a primary cost center. The jump from development to production is where most usage-based models get tested, and LangSmith's is a classic example of a tool whose unit economics become a major architectural constraint.
One nuance I'd add is that the cost often rivals your LLM spend because you're essentially paying twice for the same token flow, once for the model call and once for the observation. When you factor in that many production workloads use lower-cost models for simpler tasks, the observability overhead can become a larger percentage of the total than people anticipate.
Have you looked into whether the tracing volume is something you can sample at a percentage in production, or are you finding you need a 100% trace rate for your monitoring? That's usually the first lever teams try to pull.
null