Another week, another batch of junior engineers asking how to know if their RAG pipeline is actually any good. They slap together some embeddings and a vector store, call it a day, and wonder why production is a tire fire of hallucinations and missed context. Evaluation isn't an afterthought; it's the blueprint. If you're not measuring, you're just guessing.
The "best" workflow I've seen is the one that's automated, versioned, and runs on every significant commit. It goes beyond simple cosine similarity. You need a battery of tests. Here's a skeleton of a decent setup, using LlamaIndex's own tools and some brutal honesty:
* **Synthetic Q&A Generation:** Don't rely on a static handful of questions. Use the LLM itself (or a dedicated model) to generate question-answer pairs *from your source documents*. LlamaIndex has `DatasetGenerator` for this. This creates a scalable, relevant test suite.
```python
from llama_index.core.evaluation import DatasetGenerator
# ... load your documents into nodes ...
dataset_generator = DatasetGenerator(nodes, llm=llm)
eval_questions = dataset_generator.generate_questions_from_nodes()
# Save this dataset with your code.
```
* **Multi-Metric Evaluation:** Run each generated question through your query engine and evaluate the response. Key metrics:
* **Faithfulness:** Is the answer grounded *only* in the provided context? Use `FaithfulnessEvaluator`.
* **Relevancy:** Is the retrieved context actually relevant to the question? Use `RelevancyEvaluator`.
* **Answer Correctness:** Compare the generated answer to a reference (gold) answer. Use `CorrectnessEvaluator` (often involves semantic similarity between answers).
* **Integration into CI:** This whole process should be a job in your pipeline. The goal is to get a pass/fail signal and, more importantly, a history of metrics. I use GitHub Actions. On any push to main, it:
1. Builds the latest index.
2. Loads the versioned evaluation dataset.
3. Runs the battery of evaluators.
4. Outputs a metrics report (JSON).
5. Fails the build if scores drop below a configured threshold (e.g., faithfulness < 0.95).
The pitfall everyone makes is thinking this is a one-time research task. It's not. It's continuous testing. Your data changes, your embedding model gets updated, your chunking strategy evolves—each can silently break everything. You need a regression suite for your RAG system, just like you do for your application code.
Without this, you're just shipping a pipeline that feels fast but is probably stupid. And I hate slow pipelines, but I despise broken ones even more.
fix the pipe
Speed up your build