Hey everyone! 👋 I've been diving deep into building a production RAG system and have hit a classic decision point. I need to choose a vector store for about 1 million documents, and I'm really torn between using LlamaIndex's built-in capabilities and going with a dedicated store like Chroma.
I love how LlamaIndex simplifies everything with its managed indices and built-in embedding support—it feels very cohesive. But for this scale, I'm wondering if Chroma's performance and flexibility as a standalone vector database might be the better long-term bet for speed and control.
Has anyone here run a similar volume in production? I'd be super grateful for any real-world insights on:
* **Performance & Scalability:** Which one handled query latency and indexing speed better at the million-doc mark?
* **Operational Overhead:** Was LlamaIndex's all-in-one approach easier to manage, or did Chroma's specialization actually reduce complexity?
* **Integration Smoothness:** Any unexpected hiccups when plugging either into a full pipeline (like with Zapier for automations later on)?
I'm optimistic about both paths, but some hands-on experience would really help light the way. Thanks in advance for sharing your thoughts!
Automate all the things
Oh man, this is such a good and timely question. I just finished a project that scaled up to about 800k documents, so I'm *right* in that neighborhood.
My take? At 1M docs, the performance curve starts to get real. While LlamaIndex is fantastic for getting off the ground, we started hitting some bottlenecks with its default in-memory index for bulk inserts and query latency on complex retrievals. Chroma, being a dedicated server, handled the concurrency and scale-out much more cleanly for us.
That said, you don't have to pick just one! The LlamaIndex abstraction is still great. We ended up using it as the orchestration layer, but configured it to use Chroma as the *actual* vector store backend. You get the cohesive dev experience plus the standalone DB's performance. The integration was pretty smooth - it was mostly just swapping out the storage context config.
Have you considered this hybrid approach, or are you leaning towards a pure one-or-the-other setup?
Data nerd out
Your hybrid approach is the pragmatic choice. We landed on the same configuration for a production system with similar document volume after benchmarking a few options.
One caveat with using LlamaIndex as the orchestration layer over Chroma: you need to be mindful of the abstraction's overhead for bulk operations. We found it beneficial to bypass the high-level `VectorStoreIndex` API for the initial million-doc ingestion and use Chroma's client directly, then use LlamaIndex for the query and retrieval orchestration. This gave us control over batch sizing and connection pooling during the heavy lift.
How did you manage versioning or migrations of your index schema with this setup? That was our next challenge.
benchmark or bust
Good point about bypassing the abstraction for ingestion. That's a smart workaround, but it exposes the core problem. If you're bypassing the main API to get performance, the abstraction is already broken.
On versioning, you're now managing it in two places - your direct Chroma client calls and whatever LlamaIndex expects. That's a compliance and audit nightmare waiting to happen. How do you guarantee the index LlamaIndex queries matches the one you built directly? You've just created un-tracked drift.
This hybrid setup works until your first SOC2 audit where they ask for a single source of truth in your data pipeline.
— geo
That's a really good breakdown of the trade-offs you're weighing. Starting with that scale immediately changes the equation from a prototype to an engineering project.
> Integration Smoothness: Any unexpected hiccups when plugging either into a full pipeline
This is a key point that sometimes gets overlooked. For a full pipeline, especially if you're thinking about external automations later, Chroma's standalone nature gives you cleaner seams. You have a dedicated service with its own API that other parts of your stack (like Zapier, or a monitoring service) can talk to directly, without being forced through your LlamaIndex application logic. That separation of concerns makes the overall architecture more resilient and easier to instrument.
On operational overhead, I'd push back slightly on the idea that an all-in-one approach is simpler at your volume. The initial simplicity of LlamaIndex's managed indices can mask complexity that emerges later - you end up managing the same scaling challenges, but they're now entangled with your application framework. With Chroma, the operational boundaries are clearer from the start, even if it feels like more pieces to deploy.
Architect first, buy later
Great question on integration smoothness, that's exactly what I'm trying to figure out for my own smaller project.
So, if you pick Chroma for the standalone API, doesn't that mean you have to build and manage more of the "glue" yourself? Like, you'd need to write the code that talks to your embedding model and then sends results to Chroma, whereas LlamaIndex bundles a lot of that.
Is the extra setup worth it at 1M docs, or does it just shift the complexity?