Hey folks, been a while! Wanted to share a recent journey our team went through. We were running a high-stakes customer support agent LLM for a major retailer, and our evaluation was built on a mountain of custom Python scripts. It worked, but it was a pain to maintain and scale.
Our old setup involved stitching together:
* A bunch of scripts to calculate metrics like answer relevance and hallucination rate.
* A separate pipeline for A/B testing between model versions.
* A custom dashboard that was always a bit... fragile.
We decided to move to a dedicated evaluation framework. After a PoC with a few options, we went with **RagaAI**. The main draw was its ability to handle the entire lifecycle—from test dataset management to running evaluations and providing a central dashboard.
The migration wasn't just a lift-and-shift. We rethought our metrics. Instead of ten custom scores, we mapped them to RagaAI's built-in ones where possible, which simplified things. For example, our old "context adherence" check is now mostly covered by their **Context Precision** and **Hallucination** metrics.
Here’s a snippet of what our new test suite configuration looks like for a simple correctness test:
```yaml
test_suite_name: "customer_support_correctness"
tests:
- test_type: "hallucination"
params:
language: "en"
threshold: 0.2
- test_type: "response_relevance"
params:
threshold: 0.7
- test_type: "toxicity"
```
The biggest win? **The experiment tracking.** We can now run a new model version, get scores on 15+ metrics side-by-side with production, and make a data-driven promote/demote decision in hours, not days. The security and toxicity tests also gave our compliance team a lot more confidence.
Has anyone else made a similar switch from a home-grown system to a framework like RagaAI, LangSmith, or others? Curious about your experience, especially around managing custom metrics that don't have a direct equivalent.
~CloudOps
Infrastructure as code is the only way