I've been running a series of tests for a client where we're evaluating the output of their fine-tuned support chatbot, and I'm seeing something that's giving me pause. We set up a fairly detailed automated evaluation pipeline using a combination of LLM-as-judge (GPT-4 for grading) and some traditional NLP metrics (BLEU, ROUGE) for reference. At the same time, we had three human experts score the same 200 response pairs using a rubric we built together. The results are in, and the correlation between the human scores and the automated scores is… weaker than I expected.
Specifically, for "completeness of answer," the automated judge agreed with the human consensus only about 65% of the time. For "appropriateness of tone," it was even lower, around 55%. The automated scores tend to be more generous, rarely giving the lowest scores that humans sometimes assign.
This leads me to my core question for this group: **in your practical experience, what level of discrepancy between human evaluation and automated scoring frameworks is considered "normal" or acceptable?** Are we looking at a flaw in our rubric design, our prompts for the LLM-as-judge, or is this just the inherent noise we have to live with?
Here's a simplified version of the automated judge prompt we used, which might be a good starting point for discussion:
```json
{
"system_prompt": "You are an evaluator for a customer support AI. Analyze the following response.",
"user_prompt_template": "Question: {user_query}nnGold Standard Answer: {reference_answer}nnModel Answer: {model_answer}nnScore the Model Answer (1-5) on 'Completeness' and 'Tone'. Provide only a JSON output: {'completeness': score, 'tone': score, 'reasoning': 'brief note'}",
"temperature": 0
}
```
Some of the biggest gaps I've noticed occur in these scenarios:
* **Implied Context:** Human evaluators penalized the model for missing subtle context hinted at in previous messages (which was in the chat history but not in the single-turn prompt we fed to the auto-judge).
* **Overly Formulaic Praise:** The LLM judge often scored polite, generic acknowledgments highly, while humans marked them down for not advancing the conversation.
* **Minor Factual Omissions:** The automated system would give a 4/5 if most facts were present, whereas humans would sometimes give a 2/5 if a single critical piece of information was missing, calling the entire answer "incomplete."
I'm leaning towards the idea that we need to invest more in a hybrid approach—using automated scoring for rapid iteration during development, but accepting that a significant portion of the final evaluation *must* be human-in-the-loop. But I'd love to hear from others who have run similar comparisons.
What has your data shown? Have you found certain metrics (like faithfulness, relevance) to align better than others (like tone, completeness)? And more importantly, how do you *calibrate* or adjust your automated systems once you discover these discrepancies?
api first
api first