Hi everyone. Just joined the team that manages our help desk. We're looking at adding some AI features to our support platform soon.
I want to make sure we're measuring the right things from the start. Beyond just "deflection rate," what are the key metrics you track to see if AI-assist is actually helping? I'm especially curious about what improves agent workflow, not just what reduces ticket volume.
Deflection rate is the metric everyone talks about because it sounds good to management. It's also the easiest to game.
If you actually care about agent workflow, track the time between a ticket landing in the queue and the first meaningful human action. If your AI is giving agents better context or draft replies, that time should drop. Then watch if resolution time *after* that first action also goes down.
Otherwise you're just building a fancy bounce machine that annoys customers and makes your team's job harder. Seen it happen.
-- old school
> "what improves agent workflow, not just what reduces ticket volume"
You're asking the right question. Deflection rate is a vanity metric unless you pair it with audit trails. If you can't replay what the AI suggested and what the agent actually did with it, you're flying blind.
I'd add two things to what's already been said:
- Track the adoption rate of AI-generated suggestions per agent, but also log the modification delta. Did the agent accept the draft verbatim, edit it heavily, or ignore it? That tells you where the model is weak and where your training data is failing.
- Watch the escalations from AI-assisted interactions. If your AI is deflecting tickets but those same issues come back as escalated cases two days later, your deflection rate is just a delay metric. Your audit log should show the full chain: initial AI interaction, agent intervention, final resolution.
Are you planning to store the raw AI inputs and outputs somewhere for compliance? That's a SOX consideration if you're in a regulated industry.
Logs don't lie.