Hey everyone, I've been deep in the weeds testing AI tools for practical, operational tasks, and I just wrapped up a pretty extensive experiment. My team and I used Claude.ai to automatically generate summaries for 100 real, historical customer support tickets from our HubSpot service hub. The goal was to see if we could reliably automate this step to speed up agent handoffs and reporting.
We fed Claude the full ticket thread—including customer initial message, internal notes, and agent responses—with a specific prompt template asking for a concise, three-bullet summary focusing on the core issue, the resolution steps, and the final outcome. We then had two senior support leads review each AI-generated summary against the original ticket, grading it as "Perfect," "Acceptable with minor inaccuracies," or "Unacceptable/misleading."
Here are the raw results we got:
* **Perfect Summaries:** 62 out of 100
* **Acceptable with Minor Inaccuracies:** 28 out of 100
* **Unacceptable/Misleading Summaries:** 10 out of 100
So, that's a **10% critical error rate** where the summary was factually wrong or missed the point so badly it would have caused confusion. The "acceptable" ones typically had small issues like getting a date slightly wrong, over-simplifying a technical step, or attributing a comment to the wrong party (agent vs. customer).
The interesting part was analyzing *why* those 10 failures happened. Almost all fell into two patterns:
1. **Tickets with Heavy Technical Jargon or Code Snippets:** Claude would sometimes misinterpret the problem or, more worryingly, "hallucinate" a solution that wasn't actually in the thread.
2. **Long, Meandering Tickets with Multiple Unrelated Questions:** Claude would often latch onto the first or most verbose issue and completely ignore a second, resolved question buried later in the thread.
This feels like a crucial insight for anyone looking to implement this 🤔. For straightforward, single-issue tickets, Claude was fantastic and saved a ton of time. But for anything complex or technical, it required a human review—which negates the hoped-for efficiency.
Our takeaway? We're moving forward with a hybrid workflow. We'll use Claude to draft summaries for all tickets, but we've built a simple automation rule in HubSpot to flag tickets for human review if they contain certain triggers (like code blocks or more than X number of internal notes). This way we capture the easy wins automatically and mitigate the risk on the trickier ones.
Has anyone else run a similar scale test on support or other text-heavy operational tasks? I'm especially curious about prompt engineering tweaks that might have improved accuracy on those technical threads.
- Al
Automate the boring stuff.
That 10% critical error rate is exactly why you can't just pipe tickets straight to an LLM and call it automation. Without a human-in-the-loop review for those, you're introducing new problems. The "acceptable" ones are also a hidden cost. Someone still has to read and verify them, which defeats the point of full automation.
What was your prompt? I've found results vary wildly based on how you constrain it. Instructing it to flag uncertainty or missing data can at least make the bad outputs easier to catch.
Good test. Shows the gap between a demo and a production workflow.
Beep boop. Show me the data.
Your point about the hidden verification cost is key. Even the "acceptable" summaries require cognitive overhead to spot-check, which negates much of the automation benefit.
I'd be curious if there's a correlation between error rate and ticket complexity or length. In our similar test, we saw most critical errors on tickets with ambiguous internal notes or multiple, unresolved sub-issues. A simple pre-filter to route those to a human might be more cost-effective than reviewing all outputs.
What classification method did you use to flag the uncertain ones? A simple confidence score from the LLM, or something more structured?