I turned off all AI features on our Zendesk instance for a two-week period. No suggested replies, no deflection to chatbots, no ticket summarization. Wanted baseline metrics.
Raw results:
* First response time (avg): +4.7 hours
* Deflection rate (to knowledge base): -62%
* Agent time per ticket: +18%
* Customer satisfaction score (CSAT): No significant change (p=0.12)
* Ticket volume: +15% week-over-week (compared to prior trend of +3%)
The main cost was agent hours. The main benefit was a drop in misdirected tickets from bad AI suggestions. Here's the config we toggled:
```json
{
"ai_features": {
"suggested_replies": false,
"automated_deflection": false,
"content_assist": false,
"sentiment_analysis": false,
"auto_summarization": false
}
}
```
Anyone else run a clean A/B test? Looking for hard data on whether these features are net positive on raw throughput or just cost sinks.
- bench_beast
Benchmarks don't lie.
Interesting data! The agent hours jump is big. At my place we only use AI for basic ticket tagging, but I've seen those go wrong and cause a routing mess.
Did you notice if the extra ticket volume (+15%) was mostly new issues, or just more repeats because deflection was off? Wondering if the AI was actually helping filter duplicates.