Just finished untangling a mess where a team was bleeding cloud credits on LLM evaluations. The culprit? They were meticulously tracking and tweaking a prompt variable that had **zero** impact on their actual output quality. They were optimizing for noise.
The setup looked sensible on the surface. They used Freeplay to run A/B tests between prompt templates, measuring results against a golden dataset. Their key metric was a custom score from a judge LLM. But for weeks, their improvements were flatlining, while their Azure OpenAI bill kept climbing.
The issue was in their evaluation configuration. They were passing a `user_tone` variable into their prompts, convinced it was the lever to pull. Meanwhile, the *real* variable that determined success—`response_format`—was hardcoded and never surfaced to their experiment.
Here's the kind of misconfiguration I saw (anonymized):
```yaml
# Their Freeplay test config snippet
prompt_templates:
- name: "Support Reply v1"
content: |
You are a support agent. The user's tone is {{user_tone}}.
Respond to: {{query}}
Provide a helpful solution.
variables:
- name: "user_tone"
values: ["frustrated", "neutral", "urgent"] # They tested this endlessly
# The CRITICAL hardcoded parameter was buried in the model config:
model_parameters:
response_format: "json_object" # This was never varied. The system REQUIRED JSON, but half their test prompts failed to produce valid JSON, causing retries and failures.
```
They were generating thousands of completions, tweaking the `user_tone`, while 30% of their calls failed or retried due to invalid JSON—a cost they weren't even tracking separately. The evaluation score was low because the judge model received parse errors, not because of the tone.
**The fix?**
* First, used Freeplay's trace inspection to compare latency and token usage between successful and errored runs. The spike was obvious.
* Moved `response_format` into the prompt template as a proper variable.
* Added a validation step in the evaluation suite to check for valid JSON before scoring.
Lesson: Before you spin up a thousand test runs, **audit your parameters**. Map which ones actually affect cost (retries, context length) and which affect quality. Your prompt variables should be the ones that move the needle, not just the ones that are easy to change.
- elle
- elle