Ran a 6-month test feeding 5000 anonymized, historical tickets into a major vendor's new AI-assist for support agents.
Advertised as "draft answer generation." My team used it on a real queue, but we logged every single AI suggestion and whether it was usable.
**Bottom line: 68% accuracy.** That's being generous—it means the suggestion was *technically* correct but often incomplete or required heavy editing.
**Where it fails consistently:**
* Multi-step troubleshooting. Suggests step 1, ignores steps 2-4.
* Code snippets. Outdated API versions, wrong language.
* Reading attached logs. It mostly just summarizes the ticket description instead.
**Example of a bad suggestion:**
Ticket: "Client reports 502 error from ingress after deploying new service."
AI Suggestion:
```
Check the ingress configuration for syntax errors.
```
What was actually needed (and missing):
* Check pod readiness/liveness probes.
* Verify service selector matches new deployment labels.
* Review ingress controller logs for upstream errors.
**Cost:** The time spent verifying and correcting the AI output often negated the time saved. For simple password resets or FAQ-level tickets, it's fine. For actual technical issues, it's a distraction.
Platforms are pushing this as a deflection tool, but the accuracy isn't there yet. You're just shifting the workload onto your senior agents to fix the AI's mistakes.
pipeline_mechanic_99