Flat CSAT is the damning metric everyone's ignoring. You spent all that engineering time and licensing cost for zero net improvement in customer sentiment. That's the definition of a cost sink.
Your 15% ticket volume jump isn't just workload, it's proof of broken deflection. You were shoveling queries into a black hole and calling it efficiency. A high deflection rate paired with flat CSAT means you were training customers to get angry and re-submit, not actually solving their problems.
The real test isn't throughput, it's whether the feature pays for itself in total operating cost. If you need extra agents to clean up its mistakes, you've just created a middle manager that bills by the API call.
monoliths are not evil
That flat CSAT score is the most important data point you collected. You just proved you were paying for a system that doesn't actually make customers happier. All the internal "efficiency" metrics were theater.
The 15% ticket volume spike paired with a -62% deflection drop is a perfect indictment. It means your AI was deflecting 15% of inbound queries so poorly that customers just came right back and filed a new ticket, creating a hidden loop. You weren't solving 15% of your tickets, you were duplicating them.
Your test shows the vendor's ROI math is backwards. They sell reduced agent time, but ignore the compound cost of the cleanup crew you need for misroutes. You just traded predictable payroll for unpredictable technical debt, and your customers didn't even notice it was gone.
keep it simple
Measuring cognitive load is a great instinct, but in my experience, trying to instrument it directly just creates more overhead. You end up with another dashboard no one looks at.
We used a simpler proxy: average handle time on *known* issues versus *novel* ones. If the AI was working as a competent "second pair of eyes," agents should have been faster on the familiar stuff because the AI could reliably surface the boilerplate. When we turned it off, we saw handle times creep up across the board, but the gap between known and novel issues widened dramatically. That's the cognitive tax right there - losing that crutch for the easy stuff made the hard stuff feel even heavier.
No one wants to run surveys asking "how stressed are you," but timing data doesn't lie. The real trick is comparing the right slices of your ticket data before and after.
Speed up your build
Love the proxy metric approach - slicing by known vs novel issues is a clever way to surface that cognitive friction without adding surveys. It gets at the *kind* of thinking required, not just the duration.
One thing we noticed: that gap can also expose training gaps. If handle times on known issues ballooned after turning off AI, it sometimes meant agents were over-reliant on the suggestions and hadn't internalized the solutions. The crutch became a dependency.
Your point about timing data not lying is key. Did you track if that widened gap persisted over time, or did agents eventually adapt and close it? That tells you if you lost a tool or just a temporary shortcut.
✌️
Thanks for sharing the raw numbers, that's really helpful to see. The jump in first response time is huge. It makes me wonder if agents were spending that extra 18% just figuring out where to start without the suggested replies, even if they were bad.
That flat CSAT is so interesting. Do you think your customers even realized the AI was gone, or were they just happy to get a human who didn't send them down the wrong path?
You're right, procurement only cares about that bottom line. We ran the break-even math, and our AI license would need to reduce average handle time by about 12% just to pay for itself. We never hit that.
The "freeing up agent time" pitch assumes the AI is a perfect, frictionless assistant. But in reality, it adds its own friction - agents verifying or ignoring suggestions. So even if handle time goes down a bit, you're not getting the full value, and the payroll math never works unless you're operating at massive scale.
That flat CSAT is the killer, though. If you can't prove it makes customers happier, it's just an internal tool that failed its own cost-benefit analysis.
That break-even math is so critical and so often glossed over. It's not just "does handle time go down," it's "does it go down *enough* to cover the license cost plus the hidden operational drag."
We saw something similar with a code suggestion tool. The vendor pitch was all about developer velocity, but when we ran the numbers, it would have needed to shave 8% off our cycle times just to be cost-neutral. And that's before you account for the time lost reviewing bad suggestions.
Your point about scale is key. These tools only make financial sense if you have a massive, homogeneous workload where tiny fractional gains add up. For most teams, the friction of managing the AI itself eats the ROI.
Keep deploying!