Hey everyone, new here but been lurking while building out our customer support pipeline. 😅
We started using Intercom's new AI summary feature for tickets a few weeks ago. The idea is fantastic for triage, but we're finding the summaries are often... wrong? Like, they'll miss the core issue or attribute statements to the wrong person. One summary said a customer reported a "login bug," but they were actually asking about our API rate limits. This is causing some internal confusion.
Is anyone else seeing this? I'm curious what factors might be influencing the accuracy. We use a lot of custom attributes and have some webhook events flowing in—wondering if that's confusing their model. How are you all handling it—just turning it off for now, or is there a config trick?
Yeah, we've noticed the same thing. It's frustrating when you're trying to quickly scan.
Ours keep summarizing the customer's greeting or small talk instead of the actual problem. Had one where a customer said "Hope you're having a good week!" and then detailed a billing error, but the summary just said "Customer inquired about their week." 😅
Do you think the accuracy depends on how long the conversation is? Ours seem worse on short, simple tickets.
Your point about custom attributes and webhook data potentially confusing the model is a good one. We've observed similar noise interference in our own observability stack when feeding unstructured metadata into LLM-based summarization tools. The model might be giving disproportionate weight to structured event data over the actual conversational text.
For a temporary mitigation, you could try a pre-processing step if you have the capability. We routed our webhook payloads through a small script that strips out non-conversational JSON fields before the ticket is finalized, which improved summary relevance for us. It's a workaround, but it reduced the "attribute to the wrong person" errors significantly.
Are your webhook events primarily system-generated alerts, or do they contain user-supplied data as well? The semantic difference seems to be a key factor.
Measure everything.