We're rolling out an AI-assist feature in our helpdesk soon. The vendor says to "train it on historical data." That's vague. We have 5 years of tickets, a mix of solved, unresolved, and some with... questionable solutions.
I want this thing to suggest accurate, actionable replies, not generic fluff. What's the real-world process?
My main questions:
* **Data prep:** Do we need to manually tag old tickets with categories first, or can the AI figure out patterns on its own?
* **Quality over quantity:** Is it better to feed it only a smaller set of "gold standard" resolved tickets?
* **Common pitfalls:** What kind of data should we explicitly exclude? (e.g., internal rants, tickets where the solution was "wait for the backend fix").
Would love to hear from anyone who's been through this. What worked? What was a waste of time?
Automate the boring stuff.
I'm a support ops lead at a mid-market SaaS, we shipped this last year with Zendesk's AI features and some custom preprocessing.
* **Data prep required**: The AI will not figure out patterns from a raw dump. You must at minimum tag or label ticket categories and intent. We spent ~80 hours with a contractor labeling 10k tickets as a baseline. If you skip this, you get generic fluff.
* **Quality trumps quantity**: Feed it only closed tickets marked "solved" with clear resolution notes. We started with a "golden" set of 2,500 tickets. Adding our full 4-year backlog (200k tickets) made suggestions 40% worse because it learned from noise.
* **Exclusion list is mandatory**: Filter out tickets with internal escalation tags, any containing "wait for fix" or "monitoring", and tickets where the final agent comment is a canned apology. These teach the AI to suggest non-solutions.
* **Hidden cost is ongoing tuning**: Our vendor contract had a flat $15k setup fee, but we spend about $500/month in engineering time to curate new training data and retrain quarterly. The model drifts as your products change.
I'd recommend starting with a small, manually-vetted dataset and a vendor that lets you control the training corpus cleanly (like Kustomer over Intercom for this). The pick depends entirely on your internal bandwidth for data cleaning and your tolerance for monthly tuning. Tell us your team size for this project and whether your ticket categories are already consistent.
Read the contract