Alright, so I got tired of the dance. You know the one: write a prompt, copy it into three different playgrounds (or worse, wrangle three different API clients), paste the outputs into a spreadsheet, and then manually score them against some rubric. It's a mess, and it's not reproducible.
I hacked together a CLI tool over the weekend to automate my own workflow. It's called `tri-bench`. The premise is stupidly simple: you give it a prompt and a scoring rubric, and it fires that prompt at three configurable models (local or API), captures the outputs, and scores them based on your criteria. It spits out a tidy markdown table.
Here's the config file (`bench_config.yaml`):
```yaml
models:
- provider: "openai"
model: "gpt-4o"
api_key_env: "OPENAI_API_KEY"
- provider: "anthropic"
model: "claude-3-5-sonnet-latest"
api_key_env: "ANTHROPIC_API_KEY"
- provider: "ollama"
model: "llama3.1:70b"
base_url: "http://localhost:11434"
scoring_rubrics:
- name: "factual_accuracy"
description: "Is the core claim correct? (1-5)"
- name: "conciseness"
description: "Avoids fluff and tangential info? (1-5)"
- name: "instruction_following"
description: "Adhered to format/constraints? (1-5)"
```
You run it like this:
```bash
tri-bench run --prompt "Explain quantum entanglement in one paragraph." --rubric factual_accuracy,conciseness
```
And you get output like this:
| Model | Output | factual_accuracy | conciseness | Total |
|-------|--------|------------------|-------------|-------|
| gpt-4o | [truncated text...] | 5 | 4 | 9 |
| claude-3-5-sonnet-latest | [truncated text...] | 5 | 5 | 10 |
| llama3.1:70b | [truncated text...] | 4 | 3 | 7 |
The scoring is still manual—I'm not a fan of black-box AI evaluators for this. It pauses after each output for you to enter scores. But having everything in one terminal, structured, with the prompts and outputs logged as JSON for later review... it's cut my comparison time by like 70%.
The code's a bit janky but it works. Anyone else doing something similar? I feel like the existing eval frameworks are overkill when you just want a quick, dirty, but *consistent* comparison between a few models.
benchmarks or bust