I've been testing LangChain for a few weeks to process our company's support documentation. I'm using a simple ingest and query chain.
It works fine with a couple of PDFs. But when I point it at our full document set (maybe 200+ files), everything slows to a crawl. Simple questions take over a minute.
I'm cautious about scaling this to our team if it's always this slow. Is this a known issue? Are there specific settings I should check first?
My setup is basic: Chroma vector store, OpenAI embeddings, and the standard retrieval chain. I haven't done any special optimization.
I hit this same wall last quarter with our HR policy library. Your description matches what I saw almost exactly - a handful of files works fine, then the whole thing sinks under its own weight.
The culprit for me was less about LangChain itself and more about how the standard retrieval chain behaves at scale. It's trying to pull relevant chunks from that massive vector store for *every step* of the process.
A couple of quick things to check:
- Are you chunking your documents reasonably? If your chunks are too small, you get way too many vectors, which kills search speed.
- What's your `k` value set to in the retriever? Pulling 10 documents versus 40 makes a huge difference in latency.
I'd also look at your Chroma setup. Are you using it in-memory or client/server mode? That can change the performance profile significantly. For 200+ files, you might be pushing the limits of a simple local setup.
Test early, test often.
Exactly! The standard retrieval chain can be a real bottleneck. I'd add that your `k` value advice is spot-on. You also need to check the parent document retriever settings if you're using one. It might be re-fetching huge source documents just to pull a small chunk.
>Are you using it in-memory or client/server mode?
This was key for us. Moving Chroma to a separate server process helped a lot, but we still saw latency spikes. We ended up switching to a dedicated vector database for anything over 100 docs. The overhead of the local setup just wasn't sustainable.
Also, embedding time on that initial 200-file ingest can be brutal if you're re-running it often. I've started caching embeddings locally to avoid re-hitting the API unless a document changes. Cuts down the "test cycle" pain significantly
automate the boring stuff
Yeah, this matches what I saw last month with our internal docs. The initial embed for 200 files took forever and made me think it was broken.
Are you running the ingest and query in the same script? I found separating them helped a lot. I run the vector store population as a separate job, maybe once a day, then the query chain just reads from it. Saves waiting on embeddings every single time.
Also, what's your chunk size? I started with 500 characters and it created way too many vectors, made everything slow. Bumping it to 1000-1200 made a big difference for us.
Containers are magic, but I want to know how the magic works.
Your point about Chroma's local overhead is exactly right, and it's something we've benchmarked extensively. The performance degradation isn't linear; it's more of a cliff once you exceed a working set that fits comfortably in memory.
We found the Chroma client/server separation itself isn't a silver bullet if the bottleneck is simply the similarity search algorithm's complexity with many dense vectors. The network hop adds a small penalty, but the real gain is freeing up resources from your application process. For our load, we observed diminishing returns past about 50,000 vectors in Chroma, regardless of mode.
Switching to a dedicated vector DB (we tested Pinecone and Qdrant) was the decisive fix for us, but it introduced pipeline complexity. The caching strategy you mentioned is crucial for the initial embedding cost, but you also need to consider cache invalidation and document versioning, which can become a problem at scale.