Just got paged because our new RAG pipeline started returning medieval poetry for customer support queries. Turns out our "golden" eval dataset was about as useful as a chocolate teapot. We rolled back, but now I'm staring at the test suite thinking... how do you *actually* know your eval scores are production-worthy?
Everyone talks about tracking LLM latency and token costs, but the real nightmare is a silent regression in answer quality. You can have perfect uptime and blazing speed while serving utter nonsense.
From my midnight-oil-burning sessions, I've cobbled together a sanity check list. It's not perfect, but it catches the big stuff before the pager goes off:
* **Synthetic doesn't cut it.** You need a **small, real-world sample** of recent production queries. Annotate them manually. If your fancy BERTScore or cosine similarity passes on synthetic Q/A but fails on actual user questions, your test is lying.
* **Test against the known ghosts.** Create a mini-regression suite for your specific failure modes. For us, that's:
* Hallucination of non-existent document IDs
* Overly vague "contact support" answers
* Code snippets in weird languages (looking at you, COBOL)
* **Shadow scoring.** Run your eval pipeline on a sample of *live* traffic in parallel for a week. Compare the scores you get there to your "pre-prod" scores. If there's a delta, your test environment isn't representative.
Here's a snippet of how we structure our "regression" test now—simple but effective. We use pytest and just assert the score is above a threshold we derived from the shadow run.
```python
def test_rag_quality_no_hallucination(rag_client, known_qa_regression_set):
"""Run against known good Q/A, fail if answer_score dips."""
for qa_pair in known_qa_regression_set:
response = rag_client.query(qa_pair["question"])
score = evaluate_answer_fidelity(response, qa_pair["expected_answer"])
assert score >= 0.92, f"Fidelity dropped for: {qa_pair['question']}"
```
What's everyone else doing? Are you just trusting the LLM-as-a-judge pattern, or have you found something more robust that doesn't require a PhD in linguistics to maintain?
Pager duty survivor.
NightOps
Hey, I've been the one getting that 3 a.m. page. I run a mid-sized fintech's AI platform team. In prod, we have a multi-stage RAG pipeline (LangChain -> Pinecone -> GPT-4/Gemini) serving a dozen internal tools, and we process around 50k queries daily. Our eval suite is what stands between me and sleep.
Here's my breakdown on what actually matters for pre-production evals, based on what burned me.
* **Real-Query Fidelity:** Your primary eval dataset must be a slice of recent, anonymized production traffic. For us, that's 500 Q/A pairs, manually vetted, refreshed monthly. Synthetic data is for coverage, not truth. If your RAGScore is 0.92 on synthetic but drops below 0.8 on this set, you do not ship.
* **Cost of Running Evals:** This is a hidden budget killer. Running a full eval suite (50 questions, 5 metrics each) using GPT-4 as the judge costs us ~$15 per run. We do it 20 times a day across branches. That's $9k/month. You need to budget for this, or use a cheaper model like Claude Haiku for a first pass, knowing you lose some nuance.
* **Integration & Orchestration Effort:** You can't just run a notebook. It needs to be in CI/CD. We built a GitHub Actions workflow that, on any PR to the RAG config, spins up a test pipeline, runs the eval suite, and posts a comment with a diff vs main. Took two dev-weeks to get right. If you're looking at vendors like Weights & Biases or Arize, add a week for integration and expect ongoing config churn.
* **The Silent Regression Problem:** Standard cosine similarity and even answer relevance scores can pass while your system fails in production. You must write and track custom, scenario-specific metrics. For us, that's "Document ID Hallucination Rate" (must be <2%) and "Code Snippet Appropriateness" (a binary check we run via a simple regex/classifier). Generic metrics won't catch your specific failure modes.
My pick is a hybrid approach. I'd recommend building a custom, code-driven pipeline in your CI/CD (using tools like LangSmith's SDK or Phoenix's APIs) if you have the dev hours. If you're strapped for time and need a dashboard for stakeholders, Arize is okay, but you'll still need to write your own critical metric checks. The deciding factor is your team size: tell us if you have more than one FTE to dedicate to this, and whether you need a business-facing dashboard or just engineer-only alerts.
Automate everything. Twice.