Everyone's doing these comparisons wrong. If you don't control the prompt, the results are useless. You're testing the tool's default behavior, not its capability.
You need a prompt that's specific, has constraints, and a clear success metric. No "write a blog post about Kubernetes." That's garbage. Here's a template I use for technical writing comparisons.
```markdown
**Task:** Write a concise procedural explanation for a senior sysadmin.
**Topic:** Rotating a database connection password stored in HashiCorp Vault using Ansible.
**Constraints:** 150-200 words. Use a neutral tone. Assume the reader knows Vault and Ansible basics. Must include these three specific steps: 1) generating a new secret, 2) updating the application configuration, 3) validating the change. Do not use markdown headings.
**Evaluation Criteria:** Adherence to word count, clarity of the three mandated steps, absence of introductory fluff.
```
Then you run that *exact* prompt through each tool. You edit nothing. You compare the outputs against the constraints you set. The tool that deviates least while staying coherent wins. Most "reviews" just pick the output that sounds nicest, which is biased and pointless.
Don't panic, have a rollback plan.
You're absolutely right about controlling the prompt being the only way to get a real comparison. The template approach is smart.
My only addition would be to consider the audience variable. Your prompt assumes a senior sysadmin. If I were comparing tools for a broader team, I'd run a second, identical prompt but change "senior sysadmin" to "junior developer" to see how each tool adapts its explanation. The one that adjusts its complexity appropriately without breaking the other constraints might be more versatile. It's another layer to the test.
That said, your core point stands: without a locked-down prompt, you're just sharing opinions.
Keep it constructive.