Alright folks, I’ve hit a snag with DeepEval that’s giving me flashbacks to a failed client implementation last year, and I need to see if anyone else is running into this. We were brought in to build a QA system for a customer support chatbot, and part of the acceptance criteria was a consistent, objective scoring mechanism for response quality. DeepEval seemed like the perfect fit—out-of-the-box metrics, easy to integrate. But during our validation phase, we started seeing something troubling: running the same evaluation dataset multiple times would yield noticeably different scores for the same metric.
I’m talking about running the exact same `evaluate()` call, back-to-back, on a static set of 50 Q&A pairs, with no changes to the LLM (GPT-4) or the metric parameters. For instance, the `Groundedness` score for a particular response would swing from 0.85 to 0.72 on another run. The `AnswerRelevancy` metric has shown similar volatility. This isn't just a decimal-point fluctuation; it’s enough to change a pass/fail decision on a response.
Here’s a breakdown of what we’ve already ruled out:
* **Non-determinism in the LLM:** We’re using the same GPT-4 configuration with a temperature of 0. We’ve also tried with `gpt-4-turbo-preview` and `gpt-3.5-turbo`—the inconsistency persists, which points to something in the evaluation logic itself.
* **Data or context leakage:** The test dataset is fixed, loaded from a local JSON file. No external API calls or dynamic data fetching is happening between runs.
* **Concurrency issues:** We’re running these evaluations sequentially in a controlled dev environment, not in parallel.
* **Metric parameters:** We’ve simplified to using default thresholds and parameters to eliminate any misconfiguration on our end.
My working theory is that the inconsistency might stem from one of two places in DeepEval’s architecture:
1. **The internal "evaluation LLM" calls:** Even if our main LLM is deterministic, is DeepEval possibly using a separate, non-deterministic model call (maybe with a non-zero temperature?) to perform the judgment/scoring? I haven’t dug deep enough into their `evaluation_model` defaults.
2. **The parsing of the evaluation model’s output:** The framework has to take the LLM’s textual judgment (e.g., "This answer is fully grounded") and parse it into a numeric score. If that parsing logic is fuzzy or relies on regex that can match slightly different outputs, scores could bounce around.
Has anyone else in the community done stress-testing on DeepEval’s consistency? I’m particularly concerned because we’re about to sign off on a service-level agreement with our client based on these scores. In my world, an inconsistent evaluation tool isn't just a bug—it’s a project risk that can unravel stakeholder trust. I’d love to hear your experiences, especially if you’ve found config tweaks or alternative patterns to stabilize the scores. If we can’t lock this down, we might have to fall back to a more deterministic, rule-based scoring system, which defeats the purpose of using a framework like this in the first place.
Implementation is 80% process, 20% tool.
Check your prompts. The default ones often include "think step by step" or other instructions that introduce non-determinism at the LLM layer, even with temperature=0.
Also, verify you're setting the same random seed for any internal vector operations if you're using retrieval-augmented metrics. Inconsistency that big usually points to an uncontrolled variable.
Least privilege is not a suggestion.