So we rolled out Relevance AI for our support ticket triage last quarter, hoping to finally tame the chaos. The pitch was solid: automatically categorize, prioritize, and route thousands of tickets based on content and sentiment. Let's just say... the results were a mixed bag, like a Kubernetes pod that sometimes schedules itself on a tainted node.
The good stuff first: their workflow builder is genuinely slick. Setting up the initial logic to parse ticket content, extract entities (like product names, error codes), and assign a severity score was surprisingly intuitive. We had a basic triage flow live in an afternoon. The accuracy on straightforward technical tickets—think "API timeout error 504"—was great, maybe 90% correct routing. It saved the team hours of manual sorting.
But here's where the wheels got a little wobbly 😅. When tickets got nuanced—like a customer vaguely describing a "slow dashboard" with five possible underlying causes—the confidence scores plummeted. We saw a lot of misrouting to the wrong engineering squad. The real kicker? The cost. The "pay per workflow execution" model got *spicy* at our volume. Our cloud bill for this tool started looking like we'd accidentally left a crypto miner in our CI pipeline.
We ended up having to add a bunch of custom logic and fallback rules, which made the workflow look like a spaghetti Terraform module no one wants to touch.
```yaml
# Example of the extra "safety net" we had to add
- step: check_ambiguity
condition: confidence_score < 0.7
action: reroute_to_human_queue
comment: "Basically, 'I have no idea, help'"
```
In the end, we're using it as a first-pass filter for the blatant stuff, but humans still handle the complex triage. It's a powerful tool, but don't expect a set-and-forget solution. If your tickets are highly structured and predictable, it's a win. If they're messy and require nuance, budget for tuning, oversight, and a surprisingly large line item.
Anyone else taken it for a spin? How did your tuning journey go?
- tm
Spot on about the nuance problem. I've seen similar issues with other workflow-based triage tools. They're fantastic for the 80% of tickets that follow a predictable pattern, but they choke on the edge cases where context or a bit of historical knowledge is key.
That "spicy" cloud bill line hits home, too. The per-execution model can get out of hand fast, especially when you start adding more complex logic to try and catch those ambiguous tickets. It turns into a race between spending on more AI calls or spending on human rework.
Did you find the misrouting caused any friction between support and the engineering squads? That's often the hidden cost that doesn't show up on the invoice.
ian
That "hidden cost" is such a real thing. In my experience, misrouting doesn't just cause inter-team friction - it actively erodes trust in the system itself. Once a few high-severity tickets get incorrectly binned as "low priority" and slip through, agents start second-guessing every automated assignment, effectively double-handling tickets and negating the efficiency gains.
A workaround we've seen is forcing a human-in-the-loop for any ticket with a low confidence score, but as you pointed out, that's where the per-execution pricing can really bite you. You're paying for the AI to tell you it's not sure, then paying a human to decide anyway.
Curious, have you found any tools that handle that historical context piece well? Like pulling in notes from a similar past ticket?
Keep it real