We've been using Relevance AI for a few months, mostly for internal doc search. Decided to push it into a real-time support workflow. Here's what we built:
Connected it to our Intercom via webhooks. Now, it automatically analyzes incoming customer conversation text (initial message + first few replies) and tags conversations that are likely high-value. We defined "high-value" as:
* Mentions specific enterprise-tier features
* Discusses annual contract terms
* Contains keywords from our "urgent escalation" list
* Shows strong purchase intent (vs. just troubleshooting)
The setup was straightforward. The actual ROI question: does it save time? Early results:
* Support leads get flagged conversations in ~30 seconds vs. manual triage.
* Sales team gets warm leads faster.
* False positives? About 15% – we're tuning the instructions.
Biggest pitfall: you have to be very specific in your instructions to Relevance. "High-value" is too vague. You need concrete examples.
Has anyone else plugged Relevance into a live comms channel? Curious about your use case and if the accuracy improved over time.
—CR
Ask me about hidden egress costs.
Interesting setup. The 15% false positives number is helpful to see. Did you find that tuning the instructions made a significant dent in that, or is it more about feeding it more example conversations over time?
I'm looking at a similar integration but for tagging support tickets by complexity, not value. Your point about being specific with instructions is something I'll definitely take. How long did it take you to land on a set of instructions that felt stable?
That's a clever use of webhooks. We tried something similar for tagging during our migration from Zendesk to Intercom, but we focused on sentiment flags. Your point about being very specific with instructions is spot on - we learned the hard way that "urgent" meant different things to support and engineering.
Has the 15% false positive rate changed as you've added more tagged conversations to the system, or is it purely an instruction-tuning issue right now?
Migration is never smooth.