LangChain's become the de facto standard, but for a small team it can feel like using a sledgehammer to crack a nut. The abstraction overhead and rapid API changes are a real tax when you're just trying to build a reliable, maintainable feature. LlamaIndex is often suggested, but it's still in the same complex orchestration category.
For small teams, I've been recommending a different approach: skip the monolithic framework and assemble your own chain from robust, single-responsibility components. For most B2B SaaS use cases, you need three things: a good embedding model, a vector store, and a clear prompt pattern. Use the OpenAI SDK directly, or `openai` for Python, for your LLM calls. For embeddings, `sentence-transformers` gives you local models, or just call the API. For the vector store, start with something simple and hosted like Pinecone or Weaviate—avoid self-managing Chroma in production unless you want another infrastructure headache.
This stack gives you direct control, clearer error handling, and far less magic. Your "framework" becomes a well-structured service in your codebase, not a third-party dependency with opinions. You own the orchestration logic, which means you can debug it, test it, and change it without waiting for a framework update. For small teams, that's the difference between shipping and being stuck.
been there, migrated that
I'm a lead engineer at a 10-person B2B SaaS, building a copilot feature. We run a RAG pipeline in production using a custom-built orchestration layer, swapping out components as needed without a monolithic framework.
- **Integration effort**: With LangChain, integrating a new tool or vector store took us 2-3 days due to abstraction mismatches and version churn. Using direct SDKs (OpenAI, Pinecone) cut initial setup to one day, and swaps now take hours.
- **Runtime performance**: Our custom chain, using `gpt-4-turbo` and Pinecone's serverless index, maintains latency under 1.2s for 95% of queries. A comparable LangChain flow added 300-500ms of overhead from extra validation and serialization steps.
- **Operational cost**: LangChain's heavy abstraction can obscure token usage. By managing prompts directly, we reduced our average monthly OpenAI cost by approximately 15% by eliminating unneeded context stuffing and optimizing stop sequences.
- **Maintenance burden**: In the last year, we would have faced two breaking changes from LangChain upgrades. Our hand-rolled service has required zero framework-driven refactors; changes are dictated by our own product needs, not a third-party API.
My pick is the assemble-your-own approach you outlined, specifically for teams that need a stable, auditable RAG flow and have the engineering bandwidth to own the glue code. If you're prototyping five different workflows a month or have no backend capacity, that's when I'd reconsider a framework - tell us your team size and how often your retrieval logic changes.
garbage in, garbage out
That makes a lot of sense, building with SDKs directly. I'm trying a similar approach for a simple internal tool.
One thing I got stuck on was managing conversation history cleanly without LangChain's memory classes. How do you structure that in your service? Just a list of dicts you append to, or something more specific?
PipelinePadawan
We treat history as a list of dicts, but you need a pruning strategy. Just appending will blow your context window.
I store `{"role": "user/assistant", "content": "..."}` and keep a rolling window. For a simple internal tool, truncate to the last N message pairs. For more complex needs, you can score relevance of past turns with an embedding and keep only the top K.
The key is your service layer manages this, not a framework.
Numbers don't lie
Totally feel that sledgehammer analogy. It's like using a full-blown service mesh to connect two pods in the same node.
Your point about the vector store is spot on, especially "avoid self-managing Chroma in production." Been there, got the pager alert. For a small team, a hosted store is one less moving part to babysit. The direct SDK approach also means your logging and metrics are yours, no digging through layers of wrapper exceptions to find why a call failed.
One caveat I'd add: while building your own chain, don't accidentally build a worse, undocumented LangChain inside your repo. Start with a single, well-named function that does the whole flow. Keep it stupid simple.