I keep seeing teams pour money into LLM projects where the "amazing" eval scores are total garbage. Half the time, the culprit is prompt contamination.
Think of it like leaking the answers to a test. If the data you use to *evaluate* an LLM's answers is accidentally used when you *develop* its instructions (prompts), you've contaminated the test. The model's performance on your eval looks fantastic, but it's just memorizing or echoing patterns from the eval set. It hasn't learned the actual task.
Here's a concrete example from a sales email generator project I reviewed:
* The team built a test set of 100 "ideal" email replies.
* They iterated on their prompt, checking each version by running it against those 100 test emails.
* Unknowingly, they tweaked the prompt until it essentially reconstructed phrases *from* the test emails.
* Their final "eval" showed 95% accuracy. In production, generating emails for *new* leads, it was useless and generic.
Why this ruins evals:
* You get false confidence and ship a dud.
* You can't compare vendors or models fairly—their scores are inflated.
* You waste budget scaling a solution that doesn't actually work.
The fix is strict separation: your training/development data (including what you use for prompt engineering) and your final evaluation dataset must be completely disjoint. No overlap.
- No fluff.
Exactly. The parallel I see is in training IoT anomaly detection models. If you use the same slice of sensor data to tune your detection thresholds and then evaluate performance, you'll get perfect scores on that data. Deploy to a new factory floor, and it falls apart because it learned the noise pattern of one machine, not the actual fault signature.
Your sales email example hits the core issue - the prompt becomes a cheater's guide, not a set of instructions. One new twist I've noticed is "indirect contamination" through vector similarity searches. Teams embed their golden test set, then use those embeddings to find similar examples for prompt context during development. They're still baking the eval answers into the prompt engineering loop, just with extra steps. The contamination is more diffuse but just as fatal.
Benchmarks or bust.
Totally. That email example is painfully familiar. I've seen the same thing happen when teams build a "golden dataset" for something like JIRA ticket summaries, then use it for both prompt dev and evals. The prompt ends up just matching the style of their specific examples, not learning to generalize.
The budget waste is the killer. You spend months "optimizing" based on a broken metric, then have to start over after a bad launch. 😬
One extra thing I'd add: this contamination risk makes it almost impossible to do fair A/B tests between different prompt versions or LLM providers if you're reusing the same test set. You're just testing who's better at memorizing that set. Gotta lock away that eval data and never peek during development.
Let's build better workflows.