We've been piloting a new AI support agent on our main customer portal for about three months now. The vendor keeps saying deflection is "high," but their weekly reports are buried in PDFs and feel a bit... cherry-picked. I wanted something real-time and transparent for the actual team—something that would spark conversation, not just sit in an inbox.
So, I spent a weekend building a simple Slack bot. Every morning at 9 AM, it posts a message to our internal support channel with the previous day's raw stats:
* Total customer queries to the AI agent
* Deflected (meaning fully resolved, no human handoff): **412**
* Escalated to human agent: **88**
* Calculated deflection rate: **82.4%**
* Top three query topics that escalated (e.g., "custom report export," "invoice payment dispute")
The reaction was immediate. Our senior agents started pointing out patterns by lunchtime on the first day. They noticed that "payment dispute" escalations almost always came from a specific, poorly documented feature in our new finance module. That's not an AI training issue—that's a product knowledge gap we need to address.
It's also changed the way we look at the "deflection rate." An 82% sounds great to management, but the bot posting the absolute numbers (412 vs. 88) makes it tangible. The team sees the 88 escalations as 88 opportunities to either improve the AI's knowledge base or to identify a process that needs fixing. It's shifted the conversation from "is the AI working?" to "what are these 88 conversations teaching us?"
The bot itself is just a scheduled script pulling from the AI platform's API (they all have one, even if they don't advertise it well) and formatting it for Slack. The real value is putting the data where the people doing the work live every day. It's turned a vanity metric into a team tool for continuous improvement.
Happy deploying!
Test early, test often.
This is exactly the kind of feedback loop that static reporting fails to create. You've moved from passive data consumption to an active diagnostic tool. The immediate pattern recognition by your senior agents proves its value.
One caveat on the data source: you're pulling from the vendor's API, I assume. Just be mindful that your bot's "deflected" and "escalated" counts are only as good as their event taxonomy. I've seen APIs where a session timeout is logged as a deflection, which obviously skews the real resolution rate. Have you had to map or filter their raw events to match your internal definitions?
Adding the top escalated topics is the crucial piece. It shifts the conversation from "is the AI working?" to "what's breaking and why?" - which, as you noted, often points to product or documentation issues, not the bot itself. Are you considering adding a second, weekly summary that tracks the volume trend for those top escalation topics? It would give a clear measure of whether fixes are actually moving the needle.
- Mike