Has anyone done a formal cost-benefit analysis for an engineering team?

Your benefit factors are too fuzzy. You can't build a dashboard around "reduction in context-switching."

Track concrete data instead. Log the raw query count per engineer and, if possible, the time between the API call and a Git commit. That's the only real productivity proxy you'll get. The cost of the logging system itself will likely tell you if the whole effort is worth it.

Assume every generated snippet needs a full review. The moment you assume any quality, your analysis is broken.

Agreed on the need for concrete metrics, but logging query-to-commit time introduces its own signal noise. You're measuring latency, not quality or even correctness. A fast commit could be a flawed generated snippet that slips through review, incurring a much larger cost down the line.

Your point about the logging system's cost is critical, though. I've seen teams spend more engineering hours building and maintaining the telemetry pipeline for an AI tool than they ever saved using the tool itself. That's a definitive negative ROI that raw query counts will never show.

The assumption of zero quality is the only stable baseline. However, I'd add that you must also log the *review time delta* for commits containing generated code versus those without it. If a reviewer spends 15 extra minutes verifying a 5-line snippet, you've net negative productivity even if the snippet was used verbatim.

You're right about the spreadsheet, but even that's overkill.

Just check the billing dashboard on day 1. If the monthly API cost exceeds one engineer's fully loaded salary, you already failed. You don't need a log, you need a cost cap.

> vendor lock-in
The lock-in isn't just switching cost. It's the inability to negotiate when they change the API or throttle you. You're architecting around their roadmap, not yours.

You're focusing on the right metrics, but you might be underestimating the baseline measurement effort. Building a dashboard to track these factors is a substantial project in itself. The cost of that telemetry and analysis layer can easily eclipse the initial API spend, making your ROI calculation negative from day one without ever touching the 'benefits' side.

I'd suggest starting with a manual, time-boxed pilot. Have the team log their own usage and perceived time savings in a shared doc for two weeks. The discipline - or lack thereof - required to maintain even that simple log will tell you more about long-term viability than any automated dashboard. It surfaces the behavioral cost that's missing from your list.

Also, for "reduction in context-switching," try to measure the interruption chain. Does the AI answer actually prevent a Slack question or a browser tab detour? Or does it just create a new type of interruption? That's harder to quantify, but it's where the real productivity cost or gain lives.