We just finished rolling out a LlamaIndex-powered internal knowledge agent to our ~500-person customer support and sales teams. The goal was a unified Q&A system over our product docs, sales playbooks, and past support tickets. The launch was... bumpy, but ultimately successful. I wanted to share what actually broke under load and what held up surprisingly well, since most reviews are from smaller-scale POCs.
**What Broke (or Nearly Did)**
* **Chunking Strategy Was Everything:** Our first naive chunking (by document section) was a disaster for complex queries. The agent would fetch irrelevant context because the semantic search couldn't find the *right* part of a long section. We had to switch to a hybrid approach: smaller, semantic chunks for retrieval, but we preserved parent chunk references for the LLM to pull in broader context if needed. This was the single biggest performance fix.
* **"Simple" Ingestion Pipelines Aren't Simple:** Our initial pipeline assumed documents were static. The moment we tried updating a source doc (like a pricing PDF), we had version chaos. We ended up building a simple versioning layer using metadata timestamps and a manual "refresh" trigger for critical docs. LlamaIndex's built-in tools didn't quite handle this out of the box for our scale.
* **Cost Control Went Out the Window Initially:** We didn't gate complex queries that would trigger recursive retrieval across multiple indexes. A few power users accidentally generated massive LLM context windows. We implemented a query routing layer (simple if/else logic) to direct basic questions to a cheaper, pre-summarized index and only used the heavy recursive retrieval for complex, analytical questions.
**What Held Up Surprisingly Well**
* **The Core Retrieval:** Once we fixed the chunking, the `VectorStoreIndex` retrieval was rock-solid and fast, even with 100k+ chunks. We used OpenAI's embeddings and Pinecone, and latency stayed under 2 seconds for 95% of queries.
* **The Agent Abstraction:** Using the high-level `OpenAIAgent` with our custom tools (for querying different data sources) made the code much cleaner than building from scratch. The team could reason about "tools" and "responses" easily.
* **Metadata Filtering:** This saved us. Being able to tag chunks by department (e.g., `sales`, `support`) and product line, then filter at query time, drastically reduced irrelevant answers. We enforce this on every query now.
**My Key Checklist for Scaling Beyond a Demo**
1. **Profile your retrieval.** Don't guess on chunk size/overlap—test with real user questions.
2. **Build ingestion idempotency and versioning from day one.** Assume documents will change.
3. **Implement query cost/control gates immediately.** Even a simple max chunk limit per query.
4. **Metadata, metadata, metadata.** You'll need it for filtering later. Be generous with tags during ingestion.
The platform is stable now and loved by the teams, but the path to "just working" at 500 users was all about anticipating scale in the data prep and query layers, not the LlamaIndex components themselves.
Cheers!
> "Simple Ingestion Pipelines Aren't Simple"
This is the part that always gets glossed over in vendor demos. They show you a clean upload and a magic answer. They don't show you the half-hour of manual metadata cleanup after someone re-uploads a revised contract with a different filename.
Your solution of versioning by metadata timestamps works for a single team. But if you're planning to scale this across multiple departments, that metadata layer becomes a governance nightmare. Who owns the refresh trigger? What happens when the sales team updates a playbook but the legal team hasn't approved the new version yet? You've just described a custom content management system on top of your RAG pipeline.
I've seen teams spend more time maintaining the ingestion glue than building the actual agent. And when you factor in the cost of that engineering time plus the compute for re-indexing on every refresh, your total cost of ownership starts looking a lot like a small ERP project. Did you cap your re-indexing frequency or did you let it run wild?
Trust but verify — especially the fine print.