Alright, let's get this out there before I get distracted and switch to testing something else. I ran a little experiment because I’m tired of the “just write a better prompt” hand-waving advice. So I took a single, relatively straightforward evaluation task—summarizing a technical support ticket—and ran it through the same LLM, ten times, each with a different “optimized” system prompt I’ve collected from blogs, courses, and community templates.
I used a consistent rubric I built for this (accuracy, conciseness, actionability, tone). The results were… illuminating in a deeply annoying way.
Here’s the kicker: the variance in scores across the ten prompts was higher than the variance I’ve seen when switching between two different mid-tier LLMs using the *same* prompt. The best system prompt outperformed the worst by a 22-point margin on my aggregate score. The most common failure modes weren't about the content, but about the *formatting* and *assumed context* the prompt silently demanded.
* The “Harvard-style” detailed persona prompt (you know the one) scored high on tone but hallucinated extra steps the ticket never mentioned.
* The ultra-concise, five-word "be a helpful expert" prompt actually did fine on accuracy, but its outputs were wildly inconsistent in length.
* The prompt heavy on XML-style tags (, ) failed spectacularly when the model decided to literally include the tags in its summary.
* The worst performer was a popular "chain-of-thought" variant for summarization—it spent so many tokens explaining its process that the actual summary was truncated and missed key details.
The takeaway for anyone building evals? Your system prompt isn’t just a soft factor; it’s a hard, quantifiable variable. If you’re comparing LLM A to LLM B without locking down the system prompt first, you’re mostly testing your prompt engineering skills, not the models. My rubric is now useless unless I document *exactly* which system prompt I used alongside the model version.
So my question to the room: are we accounting for this properly? I see rubrics for output quality, but are we standardizing the *input* conditions—specifically the system prompt—as part of the benchmark? Or are we all just silently baking our own preferences into every evaluation?
That's really interesting about the formatting and assumed context being the failure points, not the core task. It makes sense, though. I've tried using a few "optimized" prompts from GitHub for writing AWS security policies, and half the time the model would get stuck trying to output a specific markdown table format that wasn't even in the original ask.
So did the simple, clear prompts end up doing better than the overly complex ones? I'd guess the variance comes from the extra "persona" stuff adding noise.
That last point about variance between prompts being greater than variance between models is a fascinating data point. It mirrors what we see in A/B testing for email subject lines - sometimes the specific phrasing causes more performance delta than the underlying offer or audience segment.
Your experiment makes me wonder if we need a "baseline" system prompt for evaluation, something like a plain language control group. Then every "optimized" prompt's value could be measured as its delta from that baseline, not just its absolute score.
Data never lies, but it can be misleading
Your finding about the variance between prompts exceeding the variance between models is a critical, and often overlooked, performance variable. It directly impacts the latency and cost side of the equation, which is my main focus. Every token in a system prompt consumes processing time and budget.
If I can offer a methodological addition: this is why I always benchmark prompts not just on quality scores, but on token length and average time-to-first-token. I've seen "optimized" prompts that add 300 tokens of preamble and yield a 2% quality bump, but increase total inference latency by 40%. That's a terrible trade-off. The simplest prompt often wins on a performance-per-token basis, which in a production system is the metric that actually matters.
A plain language baseline prompt is an excellent idea. It serves as the control for both quality *and* inference efficiency. Any fancier prompt must justify its overhead.
Love that you measured this. The variance point is brutal but tracks with my own quick tests.
The "Harvard-style" persona prompt is a real trap. I tried one for writing product descriptions and got beautiful, unusable prose packed with imaginary features. It scored high on "brand voice" but failed on "truthfulness." The model was so busy *being* the persona it forgot the actual product specs.
Your 22-point margin between best and worst? That's a bigger swing than I see swapping between some email service providers for deliverability. Makes you wonder if half the "model benchmarking" out there is actually just "prompt benchmarking" by accident.
Trial number 47 this year.