I've been helping a few teams implement LangChain for their internal knowledge bases, and a consistent pain point has emerged: processing speed seems to degrade almost exponentially as the document set grows. If you're running into this, you're not alone.
I think the "slowness" often isn't a single bug, but a cascade of architectural choices that compound. Let's break down the usual suspects. I'll start with the most common culprits I've seen in code reviews and postmortems.
* **Chunking Strategy:** The default recursive character text splitter is safe, but can create a huge number of small chunks from dense documents. This multiplies the number of LLM calls later for embeddings and retrieval.
* **Embedding Model Choice:** Using a local model like `all-MiniLM-L6-v2` is fast for prototyping, but may not handle scale. Conversely, calling OpenAI's `text-embedding-ada-002` for thousands of chunks synchronously will throttle you.
* **Vector Store Operations:** Performing a similarity search without an index on a large collection is an O(n) operation. Some vector stores (like Chroma in its default in-memory mode) aren't optimized for large, persistent datasets.
* **Chain Overhead:** Using a generic `RetrievalQA` chain can hide inefficiencies. Every query triggers the full retrieval pipeline, which might re-embed your query, search the entire store, and then pass a massive context window to the LLM.
My troubleshooting approach is usually methodical: isolate the bottleneck. Is it during ingestion (embedding and storing) or during querying/retrieval?
For ingestion slowness, check:
- Your chunk size and overlap. Adjust for your content type.
- Whether you're regenerating embeddings on every run instead of checking for existing ones.
- If you're using a batch embedding API call or firing off thousands of individual requests.
For query slowness, examine:
- The index type in your vector database (e.g., HNSW for Qdrant/Weaviate).
- The `k` value in your retriever—returning 20 docs instead of 5 increases LLM context and processing time.
- If you're using metadata filtering, ensure it's leveraging indexed fields.
What specific stage are you finding slow? Are you hitting this during the initial document load, or when users are asking questions? Sharing your stack (embedding model, vector store, chain type) would help us give more targeted advice.
gh2
ship early, test often
Your point about the default recursive splitter is critical. The "huge number of small chunks" problem gets even worse when you consider the token window waste in subsequent LLM calls. If you're using a 128-token chunk for a model with an 8k context, you're paying for all that unused capacity per call. I've measured throughput dropping by over 60% in these scenarios compared to using a semantic or hierarchical splitter that aims for larger, more contextually complete units.
The embedding model choice is another performance cliff. Teams often miss that even with async, many hosted services have per-minute request limits, not just per-second. So you can still hit a wall with 50,000 documents. You need to benchmark your specific batch size against the provider's real-world quotas, not the theoretical ones.
numbers don't lie
Exactly on the token window waste! I saw that same throughput drop when we first built our retrieval pipeline. We were naively using small, uniform chunks and the cost per retrieved fact was insane.
One related observation: the "contextually complete units" point is huge for quality, not just speed. I've found that larger, semantic chunks from a good splitter often reduce the number of follow-up queries in a conversation, because the initial retrieval just works better. You're not patching together fragments.
Your note about per-minute limits is a real gotcha. We learned that the hard way with one provider - the async calls would all succeed initially, then we'd get slammed with a 429 after exactly 60 seconds. Had to implement a much smarter queue with pacing.
Happy testing!