We're evaluating two support bots for internal use, and I'm tasked with picking an LLM evaluation framework to score their outputs consistently. Both bots use GPT-4 under the hood, but their fine-tuning and retrieval are different, leading to varied response quality.
I've narrowed the tooling down to EvalGPT and LLM-Eval. My primary need is **actionable scores**—I don't just want a number; I need to know *why* a score is low so our team can fix specific issues like incorrect product details or unhelpful tone.
From my initial tests:
* **EvalGPT** seems strong on pre-defined metrics (correctness, clarity), but its feedback can be vague ("response could be more helpful").
* **LLM-Eval** allows more custom rubric definitions, which is great, but I'm unsure if its scoring is consistent across large batches.
Has anyone run a detailed comparison for a similar use case (support/QA)? I'm particularly interested in:
* Which framework's scoring correlated better with actual user satisfaction metrics?
* How easy is it to integrate a custom rubric—for example, one that penalizes hallucinations about our API pricing?
* The setup complexity for running automated evals on a few hundred conversation samples.
If it helps, here's the kind of custom criterion I'd need to encode:
```python
{
"criterion": "pricing_accuracy",
"description": "Response must not hallucinate pricing tiers or unit costs.",
"weight": 0.3
}
```
Cost is a factor, but secondary to getting clear, diagnostic results. Any hands-on experiences or gotchas to share?
terraform and chill