Our leadership team mandated a 90-day pilot of an "AI Agent" on our tier-1 support queue, promising increased deflection and consistent CSAT. After the initial 90 days, the metrics were, on paper, acceptable: a 22% deflection rate and a CSAT of 4.1/5. However, agent sentiment and my own review of sampled conversations revealed significant issues. I advocated for a controlled experiment: a two-week period where we disabled the AI feature entirely for a randomly selected 50% of incoming tickets, routing them to human agents only. The other 50% continued through the AI layer. The results were unambiguous.
**Key Metrics Comparison (2-Week Period)**
| Metric | AI-Enabled Cohort | Human-Only Cohort |
| :--- | :--- | :--- |
| **First-Contact Resolution** | 31% | 68% |
| **Average Handle Time** | 42 min | 28 min |
| **Deflection Rate** | 24% | 11% |
| **CSAT (5-point scale)** | 4.2 | 4.7 |
| **Agent Escalation Rate** | 15% | 8% |
The most telling data point is the inverse relationship between deflection and CSAT. The AI was successfully deflecting tickets by providing generic, often incomplete, solutions. These deflections generated low-quality "resolutions" that led to repeat contacts, which our tracking system logged as new tickets, artificially inflating deflection while burying the re-open rate in a separate dashboard.
Agent feedback was scathing. The AI's suggested replies, which they were pressured to use, often contained inaccuracies. The workflow disruption was significant:
```yaml
# Example of AI-suggested workflow vs. actual agent process
ai_workflow:
step_1: review_ai_suggestion
step_2: edit_suggestion # (often a complete rewrite)
step_3: send_to_customer
step_4: await_clarification # (high probability)
human_workflow:
step_1: diagnose_issue
step_2: craft_targeted_response
step_3: send_to_customer
step_4: resolve_ticket
```
The 11% deflection in the human-only cohort is more valuable; it represents true, actionable self-service (e.g., linking to a specific, known KB article for a common password reset). The AI's 24% deflection was largely "noise" – customers being given a procedural step that didn't resolve the core issue.
My conclusion is that for complex, technical support environments, current-generation AI assistants optimized for deflection actively degrade service quality. They trade measurable agent efficiency and customer satisfaction for a vanity metric. The pilot has now been terminated. We are reallocating the AI license budget towards enhancing our internal knowledge base structure and improving agent training, which has a proven ROI.
-- elliot
Data first, decisions later.
Your controlled experiment is exactly the right methodology. Too many teams accept the initial vanity metrics like deflection without looking at the resultant system state.
The inverse relationship you note is a classic systems tradeoff. You're measuring deflection as a simple throughput metric for the initial queue, but you're not accounting for the secondary queue it creates, which is the repeat contacts from incomplete resolutions. Those repeats increase load on later stages, often with higher complexity and more frustrated users, which your escalation rate and handle time increases reflect.
For a proper evaluation, you'd need to model the total cost across the entire support lifecycle, including the second and third contacts. The AI cohort likely looks efficient in a vacuum at first contact, but it's just shifting and amplifying the work downstream. This is similar to caching with a high invalidation rate; you get a fast read but a costly recomputation.
brianh
This is a fantastic write-up, and that 31% vs 68% FCR gap is brutal. The inverse relationship you're seeing reminds me a lot of security theater - you can inflate the "tickets blocked" number by configuring a WAF with overly aggressive rules, but that just means you're now dealing with angry customers and a stack of false positives that actually take longer to triage than a real incident would have.
The 42 min handle time on the AI cohort is telling. I'd bet a chunk of that is agents picking up the pieces after the AI handed off a half-baked solution. I've seen the same pattern in IAM where a "self-service" permission boundary tool deflected 80% of requests but created a two-week backlog of escalated tickets because the automated approvals were too permissive and had to be manually reversed.
One thing I'd be curious about: did you track the channel touchpoints? Like, did the AI deflected users come back via chat or phone, thus making the human agent handle time even longer because they had to re-verify context?
security by default