We recently completed a migration of our primary customer support platform from Intercom (utilizing their Answer Bot and Fin capabilities) to Zendesk's Advanced AI suite, specifically for automated ticket resolution and deflection. The anticipated outcome was an improvement in our first-contact resolution (FCR) and deflection rate metrics, given Zendesk's deeper integration with our knowledge base and ticketing workflow. The observed result, however, has been a significant and counterintuitive regression.
Our pre-migration baseline, measured over a 90-day period with Intercom, showed a consistent deflection rate (conversations resolved by AI without agent intervention) of **31-34%**. This was across a dataset of approximately 15,000 monthly conversations in a B2B SaaS context, involving technical integration questions, billing inquiries, and feature guidance.
Post-migration to Zendesk AI, after a 30-day training/learning period and a subsequent 60-day measurement window, our deflection rate has stabilized at **18-22%**. This represents a drop of roughly 12 percentage points, or a **~35% relative decrease** in deflection effectiveness. The drop is statistically significant (p < 0.01).
We've conducted a preliminary architectural and configuration analysis to isolate potential causes. The core implementation differences are as follows:
* **Knowledge Base Vectorization:** Intercom's model seemed to perform better with shorter, more fragmented documentation. Zendesk AI, with its claimed superior semantic understanding, appears to over-index on certain keywords, leading to irrelevant article suggestions for complex, multi-faceted queries.
* **Confidence Thresholds:** Zendesk's configuration for "certainty" before triggering an auto-reply is less transparent. We've had to adjust the `intent_confirmation_threshold` in the AI settings downward to even achieve the current deflection rate, which has increased the risk of incorrect answers.
* **Context Window Handling:** Our support tickets often include code snippets or error logs. We observed that Zendesk's pre-processing sometimes truncates or mishandles this structured data before feeding it to the LLM, stripping crucial context.
A sample of mis-handled ticket intents from our logs shows a pattern:
- **User Query:** "Getting a 429 rate limit error on the Events API, is this per endpoint or per API key?"
- **Intercom Response:** Correctly linked to article: "Rate Limits and Throttling" (Deflected).
- **Zendesk AI Response:** "Here's an article on how to get your API key." (Failed deflection, escalated to agent).
Has anyone else in the community undertaken a similar migration and observed comparable performance degradation? I am particularly interested in:
* Benchmark data from similar-scale B2B or technical support environments.
* Deep-dive configuration trade-offs you've made in Zendesk (e.g., AI model selection, context tuning, answer confidence settings).
* Whether you supplemented Zendesk's native AI with a custom model or a secondary routing layer to regain performance.
The promise of a tightly integrated AI suite is compelling, but the empirical results so far suggest the underlying retrieval and ranking mechanisms may not be as mature for complex technical domains. I am currently designing an A/B test framework to run both systems in parallel on routed traffic to gather more granular data on intent-classification failure modes.
throughput is truth