Switched from automated to hybrid human+AI eval. Here's the accuracy bump.

For the last 18 months, our team relied exclusively on automated evaluation frameworks (LLM-as-judge, BERTScore, ROUGE) to benchmark our production RAG pipeline's response quality. While the metrics were consistent and allowed for rapid A/B testing of retrieval strategies and prompt templates, a persistent gap existed between our "improving" scores and sporadic user complaints about factual inaccuracies or incomplete reasoning. The dissonance suggested our automated evals were optimizing for a local maximum, not genuine utility.

We designed a controlled experiment to quantify the delta. The core hypothesis was that a hybrid evaluation system—combining deterministic checks, LLM-as-judge, and targeted human review—would yield a more accurate and actionable diagnosis of system failures, ultimately leading to a more significant accuracy improvement than automated-only tuning.

**Methodology & Baseline**
We took a frozen snapshot of our pipeline (PGVector for retrieval, gpt-4-turbo for generation) and a curated evaluation set of 500 complex, multi-faceted user queries. We established two parallel evaluation tracks:
1. **Automated-Only (Control):** Used our existing suite: answer relevance (LLM-judge on 1-5 scale), retrieval precision/recall, and a custom faithfulness metric using sentence-transformers to compare claim triplets against source chunks.
2. **Hybrid (Experimental):** A three-tiered funnel:
* **Tier 1 - Automated:** Same as control, plus a new rule-based checker for obvious hallucinations (e.g., dates, numerical facts contradicting source).
* **Tier 2 - LLM-as-Judge with Chain-of-Thought:** For any response scoring below 4 on relevance or failing the rule-based check, we triggered a detailed CoT evaluation prompting the judge LLM to list each factual claim and label it as `SUPPORTED`, `UNSUPPORTED`, or `PARTIALLY_SUPPORTED`.
* **Tier 3 - Human Audit:** A random 20% sample from Tier 2, plus all responses flagged with any `UNSUPPORTED` claims, were reviewed by domain experts using a structured rubric.

**Results: The Accuracy Bump**
After one full iteration of tuning our system based on each evaluation track's findings, we re-ran the entire 500-query set through both the old (auto-tuned) and new (hybrid-tuned) pipelines. The final accuracy, judged by a separate panel of human evaluators blind to the methodology, showed a clear divergence.

| Evaluation-Driven Tuning Method | Final Human-Evaluated Accuracy (Strict) | Δ from Automated-Tuning Baseline |
| :--- | :--- | :--- |
| Automated-Only Metrics | 71.4% | (Baseline) |
| Hybrid Human+AI | **83.9%** | **+12.5 pp** |

More telling than the headline number was the *nature* of the improvements. The hybrid-tuning led to changes that automated metrics alone undervalued:
* **Increased Context Window Retrieval:** Automated relevance scores slightly *dropped* when we increased the number of retrieved chunks to provide broader context, as the LLM-judge penalized verbosity. Human evaluators, however, marked accuracy up significantly because the model made fewer extrapolations from insufficient data.
* **Prompt Engineering for Uncertainty:** We modified the system prompt to explicitly state when the provided context was insufficient to answer a sub-question. This hurt "completeness" scores in automated eval but drastically reduced `UNSUPPORTED` claims, which humans rewarded.
* **Hard Deletion of Contradictory Sources:** The rule-based checker and human audit identified a critical flaw: our vector search sometimes returned two chunks with directly contradictory facts. Automated metrics averaged this noise. The hybrid insight forced us to implement a simple contradiction filter pre-generation, which had a measurable positive impact.

**Implementation Snapshot: Our Tier 2 Check**
Here is the simplified CoT prompt we used for the LLM-judge in the hybrid track. The key is the structured output forcing enumeration of claims.

```json
{
"system_prompt": "You are an accuracy evaluator. Analyze the 'Response' based solely on the provided 'Source Context'. List every distinct factual claim. For each, output 'SUPPORTED' if the claim is directly and clearly stated in the context, 'UNSUPPORTED' if it contradicts or is absent, or 'PARTIALLY_SUPPORTED' if it is a reasonable inference but not explicit.",
"user_prompt_template": "## Source Contextn{context}nn## Responsen{response}nn## TasknList the claims. Use the JSON format: {"claims": [{"claim": "...", "verdict": "...", "citation": "chunk_id"}]}"
}
```

**Conclusion**
The experiment confirmed that automated evals are necessary for velocity but insufficient for optimizing towards true user-perceived accuracy. They create a lossy compression of the quality landscape. The hybrid model acts as a periodic calibration, identifying systematic failure modes (like contradiction handling) that automated scores smooth over. The cost is non-linear, but the ROI in our case was clear: a 12.5-point accuracy lift justified the manual audit cycle. We now run hybrid evaluation quarterly, or after any major component change, while relying on automated metrics for daily regression checks.