Just got the Q3 deflection-rate dashboard from our support platform. The big green arrow pointing up next to "AI Auto-Reply Resolutions" is giving me a headache. Management is thrilled. My team leads? Not so much.
The agents are starting to treat the auto-reply suggestions as a first draft, not a sanity check. I'm seeing tickets closed with the *exact* canned response the AI spat out, even when the log snippet clearly shows a different error code. It's like they've outsourced the "reading" part of the job. The playbook here is broken.
```json
{
"observed_agent_workflow": {
"pre_AI": "Read ticket -> Check logs -> Consult KB -> Draft response",
"post_AI": "Glance at ticket -> Click 'Generate Reply' -> Maybe skim -> Send"
}
}
```
We built these systems to handle the brain-dead, repetitive stuff so agents could focus on complex issues. Now it feels like we're training them to not think. The metrics look great—lower first response time, higher "resolution" rate—but we're just creating a nicer looking pile of technical debt. The re-open rate on AI-handled tickets is creeping up, but that's a lagging indicator.
Anyone else's teams showing signs of alert fatigue... but for their own jobs? What's the SLO for agent engagement? 😅
-shift
Pager duty is not a hobby
Oh, you are definitely not alone. The `post_AI` workflow you've mapped is spot on. I've seen the same creep happen with our sales team and auto-generated email drafts.
The metric that finally got our team's attention was a simple sentiment spot-check on replies that were AI-generated versus agent-crafted. The "resolved" tickets where the agent just sent the AI draft had a 40% higher rate of "cold" or "frustrated" in the customer's next contact, even if they didn't formally re-open the ticket. The AI was technically correct but tone-deaf.
We had to rebuild the playbook to gate the feature. Now, using the auto-reply requires a quick "confidence score" from the agent on why the suggestion fits *before* they can send it. It forces that extra cognitive step. It's not perfect, but it broke the "glance and click" habit.
That pre/post AI workflow you mapped is painfully accurate. It's the classic automation trap - we build a tool to handle the boring part, but the second it becomes frictionless, the muscle atrophies.
The lagging re-open rate is the real killer. Management sees the green arrow, you see the future backlog. I've watched the same thing happen with design system component suggestions. Junior designers just grab the first auto-layout suggestion without understanding the grid, then wonder why everything breaks at smaller breakpoints.
Forcing a "confidence score" before sending, like user645 mentioned, is a decent band-aid. But it feels like treating the symptom. The real playbook fix has to make *ignoring* the AI suggestion more painful than just blindly clicking send. Maybe tie a quality metric to tickets where they *deviate* from the auto-reply? Reward the thinking, not the clicking.
Otherwise you're just optimizing for empty calories. The dashboard looks full, but the customer feels the hunger.
Yep, that design system comparison hits home. It's the same learned helplessness. The AI becomes the grid they don't understand.
I like your idea of rewarding deviation, but you'd have to be super careful. If you just measure *any* deviation, you'll get agents making pointless edits just to game the metric. You'd need a parallel quality score on the *substance* of the change.
What if the system tagged tickets where the AI's confidence was low - maybe based on conflicting log data - and then specifically tracked agent performance on *those* tickets? That would highlight the critical thinking you actually want, without punishing them for efficiently using a good suggestion.
Keep iterating
That workflow map is exactly what I'm scared of setting up. Just starting with monitoring and I'm already thinking about what "good" metrics even look like for this.
> metrics look great... but we're just creating a nicer looking pile of technical debt
This is what I'm trying to avoid with my dashboards. If the re-open rate is a lagging indicator, what's a good leading one to track? Maybe the time between the AI suggestion being generated and the agent sending it? A really short delta might be a red flag.
Your question about leading indicators is the right one to ask. The time delta between suggestion generation and sending is a decent behavioral proxy, but it's noisy. You'll get false positives from agents who are simply fast, and false negatives from those who take a long time but still don't engage.
Consider measuring interaction with the tool itself. A stronger leading metric is the rate of AI suggestion *edits* per ticket, coupled with a sample audit of those edits. An unedited suggestion is a potential red flag. However, as user746 hinted, you need to audit whether the edits are substantive or superficial. Track the proportion of tickets where agents add a custom sentence from the knowledge base, insert a new troubleshooting step not in the original AI draft, or modify the tone significantly. This directly measures the cognitive layer you're trying to preserve.
The technical debt accumulates when you only monitor deflection rates. You end up with a system that looks efficient but is actually hollowing out your team's diagnostic skills. The leading indicator should be a composite: edit rate + substantive change rate + time spent in the knowledge base after a suggestion is generated. If that last metric trends to zero, your playbook is broken.
You've hit on the core tension between the metrics we report and the skills we're actually managing. That creeping re-open rate is the warning sign everyone ignores because the deflection graph looks so good.
The "lagging indicator" problem is real. I've seen teams try to compensate by tracking things like "average number of edits per AI-generated draft," but as you point out, that can just lead to performative tweaking. A more useful, if messier, approach is random spot checks where a lead reviews the ticket logs and the final response side by side, looking for that cognitive gap. It's not a scalable metric, but it trains leads to see the problem management's dashboard is hiding.
It's less about fixing the agents and more about fixing what we reward. When we celebrate "AI-assisted resolution rate" without a quality gate, we're telling them exactly what to do.
No receipts, no trust.