Our tier 1 deflection rate went from 38% to 22% after enabling the vendor's "Agent Assist" AI. This is the opposite of what the sales deck promised.
We rolled it out to the general support queue two weeks ago. Agents say the suggestions are either too generic or dangerously wrong. Example from our logs:
```sql
-- Suggested response for "pg_stat_statements shows high shared_blks_hit ratio"
SELECT * FROM pg_settings WHERE name LIKE '%work_mem%';
-- Completely irrelevant. Correct action is checking cache hit ratios per DB and index usage.
```
Key points from the team:
* Agents now ignore the AI panel, adding cognitive overhead.
* Simple, repetitive tickets that used to be deflected are now being escalated.
* The confidence score shown to agents is consistently above 80%, even for bad suggestions.
Has anyone else seen metrics get worse? What did you actually measure before/after besides the vendor's dashboard? I'm looking for real agent workflow changes, not ROI slides.
latency kills
Oh yeah, we saw this with a different vendor last year. That high confidence score on bad answers is the real killer, it trains agents to distrust the system entirely. We measured it by tracking the "AI suggestion used" click-through rate from the agent's UI - it plummeted to like 5% after the first week.
The fix that sort of worked for us was creating a parallel, "shadow mode" queue for a month. The AI made suggestions but they were hidden from the agent until after they resolved the ticket, then we could compare. Turns out for simple tickets, the time to find and dismiss the bad suggestion was longer than just solving it from memory. We ended up restricting it to tier 2 for complex tickets only, with a mandatory "thumbs down" feedback loop. Deflection came back up, but only because we stopped letting it near the easy stuff. 😅
What's your agent feedback button setup like? If they're hitting "not helpful" and nothing changes, the model's probably stuck in a bad initial training snapshot from the vendor.
That shadow mode idea is really smart, I haven't seen that approach before. It makes sense that you'd restrict it to complex tickets later.
> What's your agent feedback button setup like?
This is actually something I'm trying to figure out in our own setup. We have the standard thumbs up/down, but I suspect it's not actually feeding back to improve the model in real time. How did you make the "mandatory feedback loop" work without annoying your tier 2 agents? Was it just a required field before closing the ticket?