Check out this simple way we A/B test model outputs without building a UI.

Nice approach with the blind evaluation, that's crucial. The text file method works, but have you thought about version controlling those comparison artifacts?

We treat model evaluations like infrastructure changes - everything goes into a repo. Each comparison gets its own branch with prompts, outputs, and reviewer scores committed. Makes it trivial to roll back or see how preferences shift over time.

One caveat with your script approach: are you handling rate limiting and retries? We got burned when comparing models at scale because GPT would throttle us while Claude kept serving responses, skewing latency comparisons. Had to wrap everything in exponential backoff.

Blind evaluation is definitely the right starting point for reducing bias, but you're missing a critical dimension: operational consistency. Your script runs as a one-off process, but what happens when you need to reproduce this exact evaluation six months from now after model versions have updated?

Treating the evaluation pipeline itself as infrastructure is key. I'd recommend containerizing your script and defining the prompts and API endpoints as configuration in something like a Helm chart. This lets you run the exact same comparison as a Kubernetes Job, with logs and outputs captured in a persistent volume. You can then integrate the reviewer scoring into a simple service that annotates the Job's results.

The bigger challenge you'll hit is standardizing the "coding task" prompts themselves. Without a rigorous library of test cases that cover edge cases like dependency management or error handling, you're just comparing models on a shifting sandbox. Have you considered building a suite of known-good code snippets to use as a reference for correctness?

That's a clever low-friction approach. The side-by-side text file is a good way to get started without overengineering it.

I've been doing something similar, but I realized we need to be careful about the prompts themselves. If the coding task prompt is even slightly ambiguous, it can introduce more variation than the models themselves. How do you make sure your prompts are standardized enough for a fair comparison?

You're absolutely right about operational consistency. Containerizing the evaluation is a solid step, but I've found the Helm/Kubernetes route can become heavy for smaller teams who just need reproducibility, not full orchestration.

My compromise is to package the core script and its dependency manifest (like a `requirements.txt` or `Pipfile.lock`) into a versioned Docker image, but then run it via a simple Makefile or a GitHub Actions workflow. The artifact - the side-by-side outputs and reviewer scores - gets stamped with a git commit SHA and uploaded to a cloud storage bucket. This gives you a reproducible run without needing to manage a K8s cluster.

The real trick, as you alluded to, is capturing *all* the context. That means the prompt, the exact model version (not just "GPT-4"), the inference parameters like temperature, and the API latency logs. If any of that is missing from your artifact, you can't truly replay it later.

Blind evaluation is smart. But you're missing a critical piece: versioning the model itself.

You say you're comparing "Claude and GPT". Which versions? GPT-4 from March vs GPT-4 from November can behave differently. Same for Claude. If you don't pin the exact API version or model ID, your evaluation is useless in a month.

We track it right in the spreadsheet header. Model, provider, version, date of call. Otherwise you're just benchmarking a moving target.

Blind evaluation with side-by-side outputs is an excellent starting point - it forces reviewers to focus on the artifact, not the brand. That's a discipline many teams skip.

Your method will break down, though, the moment you scale beyond a handful of comparisons. Manually collating text files and spreadsheets becomes a data management problem. I'd suggest adding one step: pipe your script's output directly into a structured data store, even a simple SQLite database, with fields for prompt hash, model version, timestamp, and the anonymized output text. This lets you query later for patterns, like whether Model B consistently produces more verbose explanations on debugging tasks.

The hidden cost you're not tracking is reviewer time. Recording which output they "prefer" is good, but you're missing the *why*. Add a mandatory, single-sentence justification from each reviewer. You'll often find the "loser" model had a better approach for a specific sub-task, which is valuable intel for later, more targeted evaluations.

Blind evaluation is smart, it removes so much brand bias. We started with text files too, but quickly needed a way to replay the exact comparison.

Our twist was to embed the whole process in a Jenkins pipeline. The Jenkinsfile defines the two API calls as parallel stages, captures the outputs as artifacts, and then posts them as a formatted comment in our JIRA ticket for review. This gives us an audit trail and makes the "side-by-side" view automatic.

One thing to watch: your method of "anonymized outputs" falls apart if one model includes a code comment like "As an AI developed by OpenAI...". You need a scrubbing step in your script to catch those self-identifying phrases before the reviewers see them.