We've been evaluating our RAG pipeline outputs for months using RAGAS. The metrics (faithfulness, answer relevancy) were useful during development, but in production monitoring they became a bottleneck. Each eval run took ~45 minutes on our sample set.
We built a minimal set of checks that run in <9 minutes and catch the same regressions. The key insight: we don't need a holistic score for every query in production. We need to know if the system is **broken**.
Our checks:
- **Answer Presence**: Does the pipeline return a non-empty string? (Catches total failures)
- **Citation Validity**: Do any returned chunk IDs correspond to actual documents in our index?
- **Keyword Match**: For a set of 20 known factual queries, do the answers contain required key terms? (We derived these from our golden set)
We run these daily. If any fail, we trigger a deeper RAGAS evaluation on the failing batch.
Example check (Python, using our pipeline client):
```python
def validate_citations(response_chunk_ids, document_store):
"""Return False if any chunk ID is not in store."""
for cid in response_chunk_ids:
if not document_store.exists(cid):
return False
return True
```
The result: We haven't missed a major quality drop. The last three incidents (stale index, embedding model version drift, prompt truncation) were all caught by these basic checks.
Trade-off: We lose granular "answer similarity" scores. But for production monitoring, a binary "something is wrong" is faster and forces immediate investigation instead of debating a 0.82 vs 0.85 score.
Has anyone else moved from comprehensive eval frameworks to simpler production guards? What checks did you keep?
-shift
shift left or go home