That's a really good point about the real cost being senior debug time. We're just starting to track vetting vs drafting time ourselves.
>the economics only work if the human time saved in *drafting* outweighs the human time spent in *vetting*
This is exactly where we're stuck. How do you even measure that cleanly? Our early data is all over the place because sometimes the draft is 90% right, other times it's a total rewrite. Makes the ROI super fuzzy.
So doubling the token cost for a sanity check might be worth it if it cuts down the vetting cliff. But you'd need solid metrics to know for sure.
Adding the linter and type checker to the loop is non-negotiable. The cost of a missed type constraint isn't just a failed build - it's the compute wasted running integration tests on invalid code.
But that CI feedback loop creates its own cost. You're now paying for those linter/checker cycles on every iteration, not just in the final CI stage. If your model needs 3 correction cycles, you've 3x'd that auxiliary compute. You need to meter it.
cost per transaction is the only metric
Metering it is the right call. But that auxiliary compute cost isn't just about the cycles - it's about the concurrency limits on your CI runners. A team hitting those limits waiting for linter feedback means real human delays, which nobody budgets for.
You're also assuming the static checks are cheap. If your model hallucinates something that triggers a pathological case in your linter, suddenly you're burning minutes, not seconds, per correction.
trust but verify