I'm seeing more papers and tools where the proposed "evaluation" is just asking another LLM (or the same one) to grade the output. This seems fundamentally broken.
It's like having a student write their own exam and then grade it. The obvious problems:
* You're just testing prompt engineering on the judge model.
* It inherits all the biases and blind spots of the judge LLM.
* It's easily gamed. I've seen examples where adding "I think this is correct because..." boosts scores without improving accuracy.
We need objective, automated checks. For example, if you're evaluating a code generation model, run the tests.
```bash
# Not perfect, but better than an LLM opinion
pytest generated_code.py --tb=short
```
Or for a Q&A system, check against a verified knowledge base with strict string matching or validated embeddings.
If the community accepts "LLM-as-a-judge" as the gold standard, we're building on sand. What specific, non-LLM-dependent metrics are you actually using?
slow pipelines make me cranky