Skip to content
Notifications
Clear all

Tutorial: Building a simple RAG pipeline with Le Chat and local embeddings.

6 Posts
6 Users
0 Reactions
53 Views
(@infra_ops_guru)
Honorable Member
Joined: 7 months ago
Posts: 397
Topic starter   [#26797]

Having observed numerous discussions on integrating large language models into proprietary data workflows, I've noticed a concerning trend towards over-engineered, vendor-locked solutions. Many assume a RAG (Retrieval-Augmented Generation) pipeline requires expensive, managed APIs for both embeddings and LLM inference. This tutorial demonstrates a pragmatic, cost-effective alternative using Le Chat's API for the LLM component, while keeping embeddings generation local and free, leveraging a simple but robust architecture. The goal is to maintain data sovereignty for sensitive documents and minimize recurring costs, without sacrificing functionality.

We'll construct a pipeline with the following components:
* **Document Ingestion:** A simple Python script to load and chunk text documents.
* **Local Embeddings:** Utilizing the `sentence-transformers` library to generate vector embeddings locally on your machine or server.
* **Vector Store:** A local ChromaDB instance for storing embeddings and performing similarity searches.
* **LLM Inference:** Le Chat (Mistral) API, specifically the `mistral-small-latest` model, to generate the final answer based on retrieved context.
* **Orchestration:** A final Python script to tie retrieval and generation together.

First, ensure you have the necessary Python packages installed. We'll avoid monolithic frameworks in favor of discrete, controllable libraries.

```bash
pip install sentence-transformers chromadb requests python-dotenv
```

The core of the local embedding generation is straightforward. We'll use a lightweight model like `all-MiniLM-L6-v2`.

```python
# embeddings.py
from sentence_transformers import SentenceTransformer
import chromadb
from chromadb.config import Settings

# Initialize the local embedding model
embed_model = SentenceTransformer('all-MiniLM-L6-v2')

def generate_embeddings(text_chunks):
"""Generate embeddings for a list of text strings."""
return embed_model.encode(text_chunks).tolist()

# Setup ChromaDB client
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection(name="docs")

# Assuming you have your document chunks in `documents`
embeddings = generate_embeddings(documents)
# Add to ChromaDB with IDs and metadata
collection.add(
embeddings=embeddings,
documents=documents,
ids=[f"id_{i}" for i in range(len(documents))]
)
```

The retrieval script queries the local ChromaDB and formats a prompt for Le Chat.

```python
# query_rag.py
import os
from dotenv import load_dotenv
import requests
import json
from embeddings import embed_model, chroma_client

load_dotenv()
MISTRAL_API_KEY = os.getenv("MISTRAL_API_KEY")

collection = chroma_client.get_collection(name="docs")

def retrieve(query, top_k=3):
query_embedding = embed_model.encode(query).tolist()
results = collection.query(
query_embeddings=[query_embedding],
n_results=top_k
)
return results['documents'][0]

def ask_mistral(context, question):
"""Send the prompt with context to Le Chat API."""
prompt = f"""Use the following context to answer the question. If the context doesn't contain the answer, state that you don't know.

Context:
{chr(10).join(context)}

Question: {question}
Answer:"""

headers = {
"Authorization": f"Bearer {MISTRAL_API_KEY}",
"Content-Type": "application/json"
}
data = {
"model": "mistral-small-latest",
"messages": [{"role": "user", "content": prompt}]
}
response = requests.post("https://api.mistral.ai/v1/chat/completions", headers=headers, json=data)
return response.json()['choices'][0]['message']['content']

if __name__ == "__main__":
user_query = input("Enter your question: ")
relevant_docs = retrieve(user_query)
answer = ask_mistral(relevant_docs, user_query)
print(f"nAnswer: {answer}")
```

**Architectural Critique & Considerations:**

* **Separation of Concerns:** This design decouples the embedding model from the LLM. You can upgrade the local embedding model independently of the LLM API choice, and vice-versa.
* **Cost & Latency:** Embedding generation is a local CPU/GPU operation, incurring no API cost. You pay only for the LLM tokens via Mistral's transparent pricing, which is significantly cheaper than using, for example, OpenAI's embedding API for large document sets.
* **Data Privacy:** Your source documents never leave your infrastructure until they are embedded and stored as vectors. Only the final, contextually relevant chunks and the query are sent to the external API.
* **Scalability Pain Points:** For massive document corpora, you would need to consider batch processing for embeddings and potentially move ChromaDB to a client-server mode. The local embedding step can become a bottleneck, but for many internal knowledge bases (under 10k documents), it is perfectly adequate.

This pipeline exemplifies the principle of using the right tool for the job: local compute for deterministic, high-volume embedding tasks, and a cost-effective, powerful external service for the generative step. It provides a solid foundation that can be extended with more sophisticated chunking, metadata filtering, or a frontend interface.

--from the trenches


infrastructure is code


   
Quote
(@consulting_contractor_mike)
Honorable Member
Joined: 8 months ago
Posts: 393
 

Your approach to separating the embeddings and LLM components is sound for cost control, but I'd stress the hardware requirements for local embeddings at any real scale. The `sentence-transformers` models are efficient, but generating embeddings for a large corpus will be CPU/GPU-bound and can become a bottleneck. You'll need to plan for that compute capacity, which shifts the cost from an API line item to an infrastructure one.

Also, while ChromaDB is great for simplicity, its persistence model can be brittle for production use. If you're going to rely on it, you must implement a rigorous backup and snapshot strategy for the `chroma` directory. I've seen teams lose their entire vector store because they treated it like a regular database.


Mike


   
ReplyQuote
(@annam)
Reputable Member
Joined: 4 months ago
Posts: 275
 

I agree with the architectural choice to separate the LLM API from local embeddings. This decoupling is a key principle for sustainable cost management, particularly as your document corpus grows and requires re-indexing. However, the assumption that local embeddings are "free" merits a closer examination.

The true cost shifts to operational overhead. You'll need to establish a monitoring and maintenance protocol for the local embedding service. Model updates, library dependency conflicts, and hardware degradation become your direct responsibility, unlike with a managed API. This introduces a latent, often underestimated, operational tax.

For teams without dedicated platform engineers, that tax can quickly offset the API savings. The decision point should be whether you have the in-house capacity to treat the embeddings layer as a production service, not just a one-off script.


Migrate slow, validate fast.


   
ReplyQuote
(@devops_barbarian_v3)
Honorable Member
Joined: 7 months ago
Posts: 403
 

You've nailed the hidden costs. Calling it an "operational tax" is perfect. Everyone forgets they're now running a data pipeline service, not just a script.

My rule of thumb: if you aren't already comfortable containerizing the embedding model, setting up health checks, and automating its rollouts with something like ArgoCD or Flux, you shouldn't host it yourself. The API cost isn't just for compute, it's for deleting that entire problem space from your backlog.

The tax comes due at 3 AM when a security patch breaks a CUDA dependency and your RAG pipeline is down.



   
ReplyQuote
(@cloud_cost_nerd)
Reputable Member
Joined: 7 months ago
Posts: 348
 

You're right about the operational tax, but we can quantify it. The break-even point isn't just about headcount. It's about your per-embedding API cost versus the amortized EC2 spot instance cost plus the engineering time.

If you're generating embeddings for a static 100k document corpus once a month, the API cost is predictable. The operational tax for a local service might be negligible. If you're re-indexing a constantly changing corpus of millions of documents daily, the API bill becomes astronomical, and the operational tax for a local GPU cluster becomes the cheaper option, even with a dedicated engineer.

The real analysis is modeling total cost of ownership: API calls * rate versus (instance hours * rate) + (estimated engineering hours * burdened labor cost). That's the math most tutorials skip.


Right-size or die


   
ReplyQuote
(@alexc)
Reputable Member
Joined: 4 months ago
Posts: 341
 

Interesting choice to split it this way. I've tested a similar setup using a different local embedder, but I found the real trick is tuning the chunking strategy for your specific documents. A naive split can break the context the embedder needs.

How are you planning to handle document updates? That's where my last prototype got messy, trying to manage incremental additions to the vector store without a full re-index.


Automate everything.


   
ReplyQuote