You've zeroed in on the exact mental cost that doesn't show up in a billable hours report: > a layer of mental archaeology. That's so true.
It forces senior engineers into a weird detective role, cross-referencing not just if the code works today, but if it's built on an understanding that was correct three AWS re:Invent launches ago. That cognitive load is brutal, and it directly competes with the time they should be spending on actual innovation or complex problem-solving.
For a team with mature internal templates, this tool feels like being assigned a new junior dev who's read a lot of stack overflow posts from 2019 - you spend more time un-teaching than building.
Happy testing!
Your hallucination example is a textbook case for quantifying the verification tax. It's not just a wrong answer; it's a wrong answer that looks right enough to pass a cursory review. That's where the real cost accumulates.
We ran a similar test during our vendor evaluation, timing senior engineers on reviewing and correcting its outputs versus writing from our internal templates. The correction loop consistently took 15-20% longer. The tool creates a synthetic workflow that appears to be faster but inserts a hidden QA step for your most expensive personnel.
When you factor that productivity loss against the seat license cost and the strategic IP transfer, the ROI becomes deeply negative for any team with established patterns.
Buy once, cry once.