Hey everyone,
I've been quietly implementing and tweaking the AI support features on our platform (we use a heavily customized version of Zendesk with their Answer Bot and some homegrown logic) for about six months now. The promise of "deflection" is everywhere, but I was getting frustrated with the vanity metrics the platform itself provided. It was all "potential deflection rate" based on article suggestions, which told me nothing about what happened *after* the customer clicked a link.
So, I spent the last few weeks building a custom dashboard in our data warehouse (pulling from Zendesk APIs, our help center analytics, and session replay snippets) to try and separate true AI-driven self-service from what I'm calling "phantom deflection."
My goal was to track the complete journey: from the initial AI-suggested article, to the click, to the time spent on the article, and crucially, whether a ticket was still created within, say, 30 minutes.
Here’s a simplified view of the core metrics I'm now tracking side-by-side:
* **Platform's Reported Deflection Rate:** "Answers Bot suggested 1,200 articles this week. Deflection rate: 22%." (This just means 22% of the time, a suggested article was clicked).
* **True Self-Service Completion Rate:** Of those 1,200 suggestions, only 340 users spent >90 seconds on the article **and did not** create a ticket within 30 minutes. That's a **4.7% true self-service rate** from the initial interaction.
* **The "Bounce-Back" Rate:** This is the painful one. 510 users clicked the AI-suggested article, spent less than 15 seconds on it, and *immediately* created a ticket. That's 42.5% of all AI interactions just adding a step (and likely frustration) for the user.
Some early, concrete takeaways:
* The AI is fantastic at identifying keywords and surfacing *relevant* articles for common, simple queries like "how to reset password." True self-service is high here.
* It's dangerously bad at complex, multi-faceted issues. It will latch onto one keyword and suggest a totally unhelpful article, which leads directly to a high "bounce-back." For example, a query about "invoice discrepancy with early payment discount" will just pull up the general "how to view your invoice" article.
* This data has been gold for our content team. We can now see exactly which suggested articles have the worst bounce-back rates and prioritize rewriting them or creating new, more granular content.
I'm now working on a lead-scoring inspired model to classify incoming queries by their "self-service potential" based on historical data, to maybe route only the high-probability ones through the AI deflector initially.
Has anyone else tried to dig past the platform's surface-level metrics? I'd be really curious to know if you're seeing similar gaps between reported deflection and true resolution. What are you using to get that clearer picture?
Happy to help.
hannah
I'm on a small data team at a 150-person SaaS company, and we built a similar pipeline using Stitch for ELT and dbt to model our Zendesk and Mixpanel data in Snowflake.
Here's what I've seen while comparing platforms for this kind of workflow:
**Target audience fit:** Stitch is built for SMBs and teams new to data pipelines. It's excellent if you just need to pull from common SaaS APIs (like Zendesk) into your warehouse with minimal fuss. Fivetran is a better fit for mid-market to enterprise where you need more supported sources and reliability.
**Real pricing:** Stitch has a simpler model based on rows synced monthly. We started around $100/month. Fivetran's pricing is based on monthly active rows (MAR) and can scale quickly; at my last shop, a moderate volume was $2,000+ per month. Both have ingestion costs that can surprise you if you don't monitor volume.
**Deployment effort:** Stitch is almost zero. You connect sources and a destination in their UI, and it just runs. Fivetran has more knobs for tuning syncs and handling edge cases, which adds some initial config time.
**Where it breaks:** Stitch's API syncs can be brittle if the source API changes or has rate limits, and you have to wait for their team to update the connector. With Fivetran, you get more proactive support, but you pay for it. Stitch's transformation layer is very basic; you'll need dbt or something else for complex logic.
I'd recommend Stitch if your main goal is getting data into the warehouse quickly and cheaply, and you're okay managing transformations separately. For your use case, I'd choose Fivetran only if you know you'll be adding many more complex sources and need the uptime guarantees. Tell us if you're planning to add more than 5-10 sources, and what your monthly data volume looks like.
PipelinePadawan