Hey folks, I wanted to share a recent deep dive I did that felt a lot like evaluating a new APM or logging tool—except the "tool" was a pair of LLMs. We were evaluating two potential AI vendors for generating marketing copy. The usual human A/B testing was taking forever and getting subjective, so I built a small "LLM-as-a-judge" framework to systematically compare them. It reminded me of setting up automated performance regression tests for a CI/CD pipeline.
Here's the basic flow I implemented:
1. **Dataset:** I used 50 of our past product announcements as prompts (e.g., "Write a short, engaging tweet for the release of our new observability feature X").
2. **Generation:** Sent each prompt to both Vendor A's API (GPT-4 based) and Vendor B's API (Claude 3 based).
3. **Evaluation:** I then had a third, more powerful LLM (a specific Judge model configuration) evaluate each pair of outputs. The key was crafting a detailed rubric in the system prompt.
The judge's system prompt was crucial. I treated it like configuring a monitoring rule—it had to be specific and measurable. Here's a condensed version:
```yaml
Role: Expert Marketing Copy Judge
Instructions:
- You will receive a prompt and two outputs (Output A and Output B).
- Evaluate based on:
1. **Adherence to Prompt** (0-5): Does it follow all instructions?
2. **Brand Voice Consistency** (0-5): Is it on-brand for our technical, enthusiastic style?
3. **Creativity & Engagement** (0-5): Is it clever and likely to generate clicks?
4. **Conciseness** (0-5): Is it tight and avoids fluff?
- Output MUST be JSON: {"winner": "A" | "B" | "Tie", "scores": {"A": {"total": X, "breakdown": {...}}, "B": {...}}, "reason": "string"}
```
I ran this through a Python script that logged every judgment, similar to how I'd collect metrics from a pipeline run. The results were fascinating.
**Key Findings & Gotchas:**
* **Vendor A** won on **Adherence to Prompt** 68% of the time. Their outputs were reliably literal.
* **Vendor B** won on **Creativity & Engagement** 72% of the time, but sometimes veered off-brand into a tone that was too casual.
* The "Tie" rate was only about 12%, which told me the judge was decisively differentiatin.
* **Major Gotcha:** Latency and cost. Running 50 prompts through three separate models (2 vendors + judge) wasn't cheap or fast. This needed to be a scheduled batch job, not a real-time eval. It felt like dealing with a slow integration test suite.
* **Another Gotcha:** Judge bias. I had to run a few "meta-checks" where I swapped the order of A and B in the prompt to see if the judge had a positional preference. I also spot-checked 20% of the judgments with human review, which acted as my "monitoring" for the evaluator itself.
In the end, the data clearly pointed to **Vendor B** for our needs, as creativity was our higher priority. But more importantly, we now have a repeatable evaluation pipeline. I'm thinking of hooking this into our release automation—imagine automatically evaluating a new model version before it's promoted to production, much like a performance test gate!
Has anyone else tried similar LLM-as-a-judge setups for vendor or model selection? I'm particularly curious about how you're handling the inherent bias in the judge model and if you've found good open-source tools to orchestrate this whole process. The manual scripting was a bit of a bear.
—jr
—jr
Your approach mirrors how we'd evaluate a database query optimizer, treating the judge model's system prompt as a crucial configuration file. You mentioned crafting a detailed rubric; I'd be interested in how you handled the judge's own potential biases or stylistic drift across 50 evaluations, similar to metric degradation in a monitoring system.
Did you implement any calibration steps, like scoring a known "golden" output pair at intervals to ensure the judge's evaluations remained consistent? Without that, you might be measuring variance in the judge's interpretation over time rather than the true difference between vendors.
Data over dogma
That's a great point about treating the system prompt like a configuration file. It's the single source of truth for your entire evaluation run. But just like a config, if it drifts or isn't monitored, your results can't be trusted.
I'm also curious about the judge's potential bias. Using a more powerful LLM as a judge can introduce a different kind of subjective slant, especially if the vendors are built on different base models. Did you notice any tendency for the judge to prefer outputs that matched its own stylistic tendencies, even slightly?
Read the guidelines before posting
Interesting experiment, but I hope you're also tracking the vendor API costs alongside the judge's API calls. What was the total compute spend for this evaluation run? Your judge model likely wasn't cheap.
The bigger issue is you're now trusting a single LLM's subjective scoring as your source of truth. How do you validate the judge? Did you have any human spot checks on its decisions, or are you just accepting its rubric as law? I've seen these "detailed" rubrics fail on edge cases more often than not.
Also, you built this to avoid subjective human A/B testing, but you've just replaced it with subjective LLM A/B testing. At least with humans you can argue about the feedback.
show me the bill
The setup is basically an automated feedback loop with the judge's bias hardcoded as the config. You can't measure drift if your own rubric is the source of it.
Just my two cents.