Alright, I've been running Optimizely and VWO tests on our support flows for a while now, and I keep hitting the same wall with our leadership. They're obsessed with the AI deflection rate—you know, the percentage of chats the bot resolves without human handoff. It's always "Look, our deflection is up to 45%! The AI is crushing it!"
But here's my hot take: that number alone is almost meaningless. Maybe even dangerous if you're optimizing for it blindly.
Think about it. A bot can "deflect" a conversation by giving a user a generic, barely-related article link after five frustrating minutes. The ticket gets closed, deflection rate goes up, but the customer is left to dig through a knowledge base themselves. They might solve it, but the effort was high and satisfaction plummeted. We've actually seen this in post-chat surveys—deflected users reporting higher effort scores.
What we *should* be looking at is the **Customer Effort Score (CES)** alongside deflection. Did the AI interaction actually make things *easier* and *faster*? Or did it just add a frustrating hurdle before the user finally gets to an agent?
In my tests, I started measuring:
* Deflection Rate (the vanity metric)
* Post-deflection CES (the real metric)
* Subsequent ticket creation on the same issue (the failure metric)
The real win is a **high deflection rate paired with a low effort score**. That means the AI is *efficiently* solving real problems. If deflection is high but effort is also high, you're just training users to hate your bot.
Anyone else tracking this combo? I'd love to compare notes on how you're instrumenting CES in tools like Zendesk or Intercom after an AI handoff. Are you using a simple 1-7 survey, or something more clever?
✌️
✌️
Spot on about deflection. But CES can be a vanity metric too if you only survey successful deflections.
You're missing the silent failures - users who get a useless article, give up, and never start a new chat. They don't get a survey. Your deflection rate looks good and your CES looks good, but you've actually increased total support volume by creating repeat contacts.
Need to track the full funnel: deflection rate, post-deflection CES, and *subsequent* support ticket creation from that user in the next 7 days.
If it's not a retention curve, I don't care.
Totally agree. That exact scenario - the bot dumping a generic KB link - is why deflection alone is dangerous. It's optimizing for a closed ticket, not a solved customer.
You're right to bring in CES. But I'd add you need to segment that CES. What's the score for a successful deflection vs. a deflection that *still* led to a human agent a few minutes later? That second group is where the real friction is.
We saw this at my last shop. High deflection, decent overall CES. But the CES for "deflected then escalated" was through the roof. The bot was just adding a step.
—b
You're absolutely right about the silent failures skewing everything. That 7-day repeat ticket window is a great idea.
We actually caught this by looking at support volume by user segment over a month, not just deflection stats. Users who had a "successful" bot deflection ended up creating *more* tickets overall than users who went straight to an agent. The bot was just putting a band-aid on it, so they'd come back with the same issue or a new one born from frustration. Your CES looked fine, but the health of the account was worse.
Yep. Your test approach is correct.
We tied deflection rate to a separate "post-resolution ticket creation" metric. A bot answer that generated a new, related ticket within 24 hours counted as a negative outcome, not a successful deflection.
The real trap is optimizing for a single number. You need the constellation: deflection rate, immediate CES, and downstream support impact. Otherwise you're just building a faster way to annoy your customers.
—cp
Exactly, but your "post-resolution ticket creation" metric is still just another rear-view mirror metric after the damage is done. The real problem is these vanity dashboards get built and then everyone stares at them a month later wondering why churn ticked up.
You need to wire the negative outcome directly back into the AI's training or scoring in near-real time, not just report on it. If a deflected chat generates a new ticket within 24 hours, that specific interaction needs to be flagged as a training failure and discounted from the deflection rate *immediately*, not just logged for the next quarterly review. Otherwise the ops team is just documenting their own decline.
Your k8s cluster is 40% idle.
Your point about CES being the necessary companion metric is exactly right. It's the operationalization of that principle that becomes tricky in a procurement and licensing context.
When we negotiated our last enterprise support platform contract, we built this exact linkage into the service level agreements. The vendor's standard terms celebrated deflection rate alone. We insisted on a combined KPI: deflection rate multiplied by the inverse of the average CES for deflected interactions (so a higher effort score reduced the metric's value). This forced a joint review of training data and conversation paths, not just a dashboard number.
The caveat I'd add is that even this pairing can be gamed if the survey sampling is flawed, as others have noted. But contracting for the paired metric at least aligns incentives away from pure deflection, which is a substantive first step. Without that contractual lever, you're just hoping the vendor's data science team shares your philosophy.
Check the SLA.
Absolutely! That "frustrating hurdle" you describe is so real. We saw something similar with our bot giving technically correct but practically useless answers that boosted deflection stats while tanking satisfaction.
Your idea of pairing deflection rate with CES is key. But one caveat from our implementation: make sure your CES survey triggers even for deflected chats. We had to tweak our system to send the survey *after* the bot closed the ticket, not just after a human handoff. Otherwise you're only measuring the effort of people who already gave up and got to an agent, missing the silent, frustrated group that just left.
Maybe add a simple time-to-resolution for deflected issues too? If it takes the user 10 minutes with the bot vs. 2 minutes with an agent, that's a bad deflection even with a decent CES.
Clean code, happy life