I'm evaluating LangGraph for a customer support chatbot that needs to handle a few hundred concurrent users. Our main requirements are keeping response times under 2 seconds and handling spikes in traffic.
Most reviews I see focus on prototyping. Has anyone run it in a production environment at scale? I'm particularly curious about:
- Actual latency when you add observability, auth, and rate-limiting.
- The operational overhead of running your own LangGraph server versus a managed cloud service.
- How the cost compares to something like Vercel AI SDK or a more traditional orchestration layer.
Great question. We've been running LangGraph in production for about six months on a similar workload - a support bot that handles around 500 concurrent sessions during peak.
>Actual latency when you add observability, auth, and rate-limiting
You'll likely see your 2-second target stretch to 2.5-3 seconds once you layer everything on. The biggest hit for us was observability, specifically tracing. If you're instrumenting each node in the graph with, say, OpenTelemetry, the overhead adds up. Auth checks at the edge are cheap, but if your graph logic needs to make internal API calls to validate something, that can introduce sequential delays. The key is to push as much of that (auth context, user metadata) into the graph state at the start, so it's just in-memory access during the flow.
>operational overhead of running your own LangGraph server
It's non-trivial. You're essentially managing a stateful service, because of the persistent graph state across interactions. This means thinking about Redis/Postgres for checkpointing, making the service resilient to restarts, and monitoring for memory leaks in long-running conversations. A managed cloud service (like LangChain's own) abstracts this, but you trade off deep customization. For us, the need to tightly integrate with our existing KV store and mesh made the DIY path worth the pain, but it's a solid 0.5 FTE in ongoing tuning and care.
On cost, it's cheaper than Vercel AI SDK at our scale, but more expensive than a pure, simple orchestration layer you'd build with something like Temporal. You're paying for the convenience of the graph abstraction. If your flows are stable and you're good with writing more code, a traditional orchestrator might save you 30-40% on compute. But if you're iterating on agent logic weekly, LangGraph's development speed probably justifies the cost.
Prod is the only environment that matters.