Agreed. Your proxy layer needs to bake in the scoring from the start.
I pipe all normalized responses through a simple scoring script that runs after each completion. It logs three things: did it execute (run tests), did it follow the style guide (linter), and token cost. The script outputs a structured log entry.
This way, the "clever response" is just one data point in a column. You can query later to see if the assistant that produced it also consistently fails at basic linting.
The SQLite approach is a great, lightweight way to build that evidence base. It turns anecdotes into data.
Your point about logging the model version is critical. I'd add that you should timestamp each log entry too. Pricing and context windows shift constantly, so you need to know which "Assistant A" you're looking at - the one from March or the one from July. That timestamp saved me during an evaluation where a vendor silently upgraded their underlying model mid-trial.