Setting up a new RAG pipeline for our product docs and support tickets (~100k docs). Need to decide the orchestration backbone.
I've prototyped with both LangGraph and LlamaIndex, but the decision is tougher at this scale. My priorities:
* Clear control flow with some conditional branching (e.g., query classification, multi-hop retrieval).
* Easy debugging and observability of the graph state.
* Maintaining good throughput without huge complexity.
From my tests:
* **LangGraph** feels more native for complex, stateful workflows. The cyclic graphs for refinement/re-ranking are clean. Checkpointing is a win for reliability.
* **LlamaIndex** is fantastic if your main need is advanced retrieval (hybrid search, re-ranking, structured outputs) with less custom orchestration. Its query engines abstract a lot.
But I'm worried about over-engineering with LangGraph, or hitting abstraction limits with LlamaIndex's higher-level APIs.
Anyone run a production RAG system at this doc count? Which framework made maintenance and scaling easier?
Key considerations for me:
- Handling partial failures & retries.
- Integrating a separate A/B testing layer for retrieval strategies.
- Cost tracking per query path.
Would love to see benchmark stories or gotchas. Warm take: LangGraph for explicit control, LlamaIndex for getting a robust baseline faster.
--ash
data over opinions