Everyone's rushing to slap "AI-powered suggestions" on their support platforms, and I'm sitting here wondering who's actually running the cost-benefit analysis. We're told these features boost agent efficiency, but I've yet to see a convincing audit trail that proves it.
The core issue is the training data. These models are often trained on generic, public datasets, not your specific, nuanced ticket history. The suggestions are either laughably generic ("I understand you're having an issue, please try restarting") or, worse, confidently incorrect in a way that could breach compliance if an agent blindly clicks send. You're paying a premium per agent for a feature that requires constant supervision to be safe, negating the supposed efficiency gain. When was the last time you saw a vendor provide a detailed breakdown of false-positive rates for their reply suggestions?
Then there's the lock-in. These "smart" features bake you deeper into their ecosystem, making your historical data a hostage for future pricing. Good luck exporting the "learned" model to another platform. For the cost of these AI add-ons, you could fund a proper internal program to build a curated knowledge base of verified, compliant macros. That gives you actual control, auditability, and doesn't rely on a black box making suggestions that might violate GDPR right-to-explanation principles.
—Greg
Trust but verify
I'm a data lead for a mid-market e-commerce company, and we've been running Intercom for support with their AI features turned on for about 6 months, after a previous stint with Zendesk. Here's what I've seen.
**Cost vs. Measured Efficiency**: The AI add-on for Intercom added about $29 per agent per month to our plan. To justify it, I tracked suggestions for a month. Only about 15% were used verbatim; 60% required heavy editing due to being too generic or off-brand, and 25% were ignored entirely. The efficiency gain was a net 1-2 minutes saved per ticket, not the 5+ minutes suggested in sales demos.
**Training Data and Accuracy**: As you guessed, the generic training is a problem. For nuanced issues like refund eligibility or specific promo terms, the AI would often draft incorrect policy statements. We had to create a strict rule that agents must never send a suggestion without verifying against our internal wiki, which added a verification step.
**Vendor Transparency and Lock-in**: Neither vendor provided a false-positive or accuracy rate during sales. The "learning" is completely opaque and platform-bound. When we evaluated a switch to Freshdesk, we confirmed our years of historical ticket data used to *train* their suggestions couldn't be exported or migrated. You're paying to rent context.
**Alternative Investment Scope**: The annual cost for the AI add-on for our 20-agent team was nearly $7k. For that same budget, we built a set of internal macros and a streamlined knowledge base in Guru, which agents now use more reliably. The ROI was clearer and we own the assets.
My pick is to skip the AI add-ons for now, unless your use case is handling a very high volume of simple, repetitive queries like password resets. To make a cleaner call, tell us the complexity of your most common ticket types and what percentage of your total support volume they represent.
Stay grounded, stay skeptical.
That verification step you mentioned is the killer. If agents have to cross-check every suggestion against a wiki anyway, you're just moving the effort around, not reducing it. You've now got a two-step process: read the AI draft, then verify it, which might be slower than just writing from a template.
The platform-bound learning you described is a huge red flag. So the "feature" actually devalues your own ticket history as an asset? That feels predatory, like they're building a moat with your data.
Do you think there's any path for these features to work, or is the foundational model approach just wrong for this use case?
You've nailed it with the data hostage point. It's a classic vendor lock-in strategy dressed up as innovation. The real cost isn't just the per-agent premium, it's the accumulated value of your historical tickets you can't leverage elsewhere.
The false-positive rate question is key. In my work with internal platforms, vendors never lead with that metric because it's often embarrassingly high for nuanced domains. The verification burden you mentioned often shifts cognitive load instead of reducing it - an agent now has to critically parse an AI draft before they even start their own work.
There might be a narrow path for these features, but only with models fine-tuned on a *company's own* data, running in their own environment. Otherwise, you're paying to beta-test a generic model on your customers.
Prod is the only environment that matters.
> who's actually running the cost-benefit analysis
Right? I asked our vendor for the ROI model they used, and they sent a generic case study from a different industry. It's all about the sales demo, not real-world use.
Your point on the curated knowledge base is key. For the same monthly cost as an AI add-on for our 20-person team, we hired a contractor for 40 hours to audit and rebuild our internal wiki. The quality and clarity improvement was immediate, and agents save more time because they trust the source. It's a permanent asset, not a rented feature.
I wonder if the push for AI suggestions is a distraction from platforms under-investing in their core template and macro systems, which could provide more reliable, predictable gains.
Thanks for sharing those hard numbers. Seeing the breakdown of 15% used verbatim is really eye-opening.
Your strict rule about verifying against the internal wiki makes total sense, but it sounds like it just adds a new step. Did you find that extra verification step actually slowed agents down more at first, before they got used to it?
I've only trialed these features briefly, and the generic training data issue was the biggest letdown. Your point about it drafting incorrect policy statements is exactly the kind of risk that makes me nervous.
That verification step slowing people down is a great question. From what I've seen in our own trials, it absolutely did at first. Agents were spending more time being frustrated and editing than they would've just starting from scratch. It's like getting a bad first draft from a new hire.
It got a little faster once they learned to completely ignore certain suggestion types, but that feels like a weird skill to develop. You're training people to filter out a feature you pay for.
Your nervousness about incorrect policy statements is totally valid. That's the scariest part. Has anyone on your team ever missed an error during verification? I'm always worried about that risk when agents are rushing.
Your point about the training data being a core issue aligns with what I've observed in migration projects. The problem is often architectural: these platforms are using a single, general-purpose model that's been lightly fine-tuned, rather than building a domain-specific system from the ground up.
The cost of verifying suggestions against your own policies frequently outweighs the drafting benefit, especially in regulated industries. You end up with a more expensive, more complex workflow that introduces new points of potential failure. The real efficiency loss isn't just in the time spent editing; it's in the constant context-switching for the agent.
The lack of transparency on false-positive rates is telling. In a proper system, that would be a primary performance metric. Its absence suggests the underlying model isn't reliable enough for the promised use case.
Migrate slow, validate fast.
You hit on something I hadn't considered fully. The constant context-switching sounds exhausting for an agent. Going from reading a complex customer issue to then parsing an AI draft for errors seems mentally draining.
>The lack of transparency on false-positive rates is telling.
This is huge. If the model was reliable, wouldn't they lead with that stat? The fact they don't suggests it's a feature for the sales deck, not a core tool for agents.
Is the only solution a model trained entirely on a company's own data? That seems expensive for most teams.