We've been using the new "AI Answers" feature on our support widget for three months now. I was told the deflection rate was "over 30%," but that felt... fuzzy. What's being counted? A click on an AI-suggested article? The user closing the widget? I needed to see the actual flow.
So, I built a dashboard using the platform's webhooks and a few custom endpoints to track the *user's journey*, not just the widget's event. The goal: distinguish between **true self-service** (user finds and uses a solution) and **AI deflection** (user gets an answer and leaves, but was it correct?).
Here's the core logic I'm tracking via webhook payloads:
- `ai_suggestion_served` (based on the `support.ai_suggestion` webhook)
- `article_opened` & `article_scrolled_to_bottom` (for engagement)
- `widget_closed_without_contact` (potential deflection)
- `subsequent_ticket_same_user` (within 24h = likely failed deflection)
I pipe everything into a Postgres DB and use a simple Grafana frontend. The key metric is a "Confirmed Deflection" – where the AI provided a solution snippet *and* the user didn't open a linked article *and* didn't submit a ticket within a day.
**Early finding:** The platform's reported 32% deflection drops to about 18% when you filter for "confirmed." A lot of users are just clicking away from the AI answer and then searching manually or opening a ticket later.
Has anyone else tried to slice their deflection data like this? I'm curious about:
* How you're defining a "successful" deflection?
* Are you tracking if deflected users come back with the *same* issue?
* Any clever webhook payloads or fields I should be looking for?
My current aggregation query looks something like this:
```sql
SELECT
DATE(created_at) as day,
COUNT(*) as total_ai_interactions,
SUM(CASE WHEN article_opened = false AND ticket_created = false THEN 1 ELSE 0 END) as high_confidence_deflections
FROM ai_interaction_sessions
GROUP BY day
ORDER BY day DESC;
```
The data quality totally depends on the webhook reliability, of course. Had to add some retry logic with Make for the `ticket_created` event.
chloe
Webhooks or bust.