Most support AI is trained on stale KB articles. That's why it deflects. The KB is often wrong or incomplete.
I tried feeding it a year's worth of resolved agent chat logs (sanitized). Results:
* Deflection rate dropped from ~65% to ~28% in testing.
* Responses got more concrete. Less "please check our documentation."
* It started mimicking good agent patterns: asking for specific error logs, suggesting known workarounds for common platform issues.
Biggest hurdle was formatting. Had to structure logs as strict Q&A pairs. Simple script to extract:
```python
# Pseudo - extract customer question & final agent solution
for chat in logs:
if chat['resolved']:
print(f"Q: {chat['initial_query']}")
print(f"A: {chat['resolution_steps']}n")
```
Anyone else try this? Curious about scaling it beyond internal tools.
Benchmarks or bust.
Your deflection rate drop is impressive. I've run similar tests but saw diminishing returns past a few thousand examples. The model started overfitting to specific agent phrasing rather than the underlying problem-solving logic.
Scaling this beyond internal tools gets tricky with noise. Unresolved chats, agent errors, and off-topic banter all creep in. You need a robust filtering pipeline, not just the resolved flag. I'd also monitor for hallucinated steps - the model might invent plausible-sounding but incorrect troubleshooting sequences it never saw in the logs.
What was your accuracy hit rate on the concrete responses? More concrete can sometimes mean more confidently wrong.
That's a clever approach! I've been scraping old support tickets to build a knowledge base for our team's internal Grafana alerts. Definitely noticed the same pattern - the real fixes are often buried in chat logs, not the official docs.
How did you handle edge cases where the agent's "resolution steps" are just "escalated to engineering" or "pending platform fix"? I had to filter those out to avoid the AI learning to punt on hard questions.
Filtering out escalations is the obvious first step, but you're just training the AI on the easy wins. The interesting failure mode is when the logs contain a *bad* resolution that happened to work. The AI will learn and replicate that flawed troubleshooting path, and you won't catch it until it causes an incident.
You need to tag logs with the incident ticket they later created. If a "resolution" led to a Sev 3 a week later, that training example is poison.
- Nina
That deflection rate drop is really promising. I work with ERP support data, and I've seen the exact same pattern where the official process documentation is three steps behind what agents actually do on the floor.
Your point about structuring logs as strict Q&A pairs is crucial. I'm curious about how you defined the "resolution steps" field for your extraction. In our system, the final agent note might just say "issue resolved," but the real steps are scattered across a dozen internal comments. Did you have to manually tag the logs to find the actual solution, or was there a consistent field agents used?
That's the whole game, isn't it? You can't use the final status note. It's almost always useless. We had to parse the *sequence* of internal comments and look for specific triggers: a JIRA ticket closure comment auto-posted, a "fixed in release X.Y.Z" note, or the moment an agent switched from troubleshooting steps to asking the customer for verification.
Even then, you're relying on agents to actually document the fix, which is... optimistic. Half the real resolution is in some senior agent's head or a Slack thread that never gets logged. So your fancy AI model learns a neat, documented surface layer of the process, missing all the tribal knowledge that actually makes things work.
I'm surprised more vendors aren't selling a "chat log distillation" add-on for this exact problem. Probably because they'd have to admit their shiny knowledge base is just for show.
—DW
Good question about the accuracy hit rate. Our concrete responses were about 88% correct on the first test run, which is great, but you're right about the confidence problem. The wrong 12% were delivered with the same assertive tone as the correct ones, which is dangerous.
The overfitting to agent phrasing is real. We saw the model start using specific internal ticket numbers or outdated product names from the logs. It wasn't grasping the *type* of solution, just copying the exact words. Had to add a synonym substitution layer in preprocessing to generalize terms like "Widget v1" to just "the platform."
For hallucinated steps, our guardrail was a separate validator model that cross-checked any troubleshooting sequence against the actual tools/APIs mentioned. If the AI suggested a `database_restart` command but no log in the training set ever used it, the response got flagged. It's not perfect, but it catches the obvious fabrications.
That drop in deflection rate is really interesting. Did you notice if the AI started picking up bad habits from the logs, like agents taking shortcuts that shouldn't be repeated? I'd be worried about reinforcing outdated steps if a process changed mid-year.
Also, how did you handle chats where the real answer was actually "check the documentation" because it *was* a simple FAQ? Wouldn't filtering all that out remove a valid response?
Good results, but your script is the easy part. Your `resolution_steps` field is probably junk.
As user1006 said, the final note is useless. You need to parse the *sequence*.
We tried this. The model learned to say "I've applied the fix from ticket PRB-6042" because that was in hundreds of logs. Completely fictional ticket. It was just a placeholder agents used.
You need to strip out internal references before training, or you bake your own tribal shorthand into the AI.
metrics not myths
You're totally right about internal references being toxic training data. We had the same issue with agents using "check the dashboard" as a catch-all - the AI started hallucinating dashboard names that didn't exist.
But stripping everything out leaves you with vanilla responses that lack context. Maybe the solution is to replace internal shorthands with generalized tags during preprocessing, like [INTERNAL_TICKET_REF] or [SPECIFIC_DASHBOARD]. That way the model learns the pattern of *when* to reference something, without copying the fiction.
The tag substitution approach is sensible but adds another failure layer: the model learns to insert those tags, and now you need a second system to map [SPECIFIC_DASHBOARD] back to a real asset at inference time. If that mapping is wrong, you're back to hallucination.
We tried a similar placeholder system for internal tool commands. The model perfectly learned when to output [RUN_DIAGNOSTIC_X], but our post-processing lookup table was outdated. The result was the AI confidently telling users to run commands that had been deprecated six months prior.
You're essentially building a two-stage system where both stages can drift. The training data becomes a history of your team's internal jargon, which itself changes. You'll need continuous validation, not just a one-time preprocessing step.
Show me the benchmarks