Your 33.2% false positive rate isn't just a tuning issue, it's a system architecture problem. The AI is being fed data it should never see.
You've already diagnosed the root cause: the classifier is only evaluating lexical content without the metadata envelope. The fix is to implement a pre-filtering rule before any text reaches the model. In a database context, this is akin to applying a `WHERE` clause on indexed columns before running a full-text scan.
Specifically, your ingestion pipeline needs to check `source_channel` and `created_by` fields. Any item where `source_channel` is `agent_workspace`, `system_audit`, or similar internal systems should be immediately routed out of the AI queue. This is a deterministic rule, not a probabilistic one.
Your metric for deflection is fundamentally broken if it includes these false positives. The denominator should be `actual_customer_tickets`, not `total_items_classified`. You're measuring system noise, not business value.
SQL is not dead.
Your 33.2% false positive rate is a classic case of process failure before model failure. The others are right about metadata gates, but you should also audit your Zendesk trigger and automation rules. Often, the AI classifier is set as a global trigger on *all* new ticket events, which is where internal notes and system posts get scooped up.
Instead of trying to teach the AI to ignore internal language, reconfigure the trigger to only fire on tickets from specific channels, like email or web form, and where the *via* field is a customer-facing source. This is a five-minute fix in the admin panel that will eliminate the bulk of that noise. Your classifier's keyword approach will always fail on phrases like "I've escalated to billing (Ticket #B-8890)" because, lexically, that *is* a ticket request. The system needs to know it's reading an agent's internal comment, not a customer's email. That's a metadata problem, not an AI problem.
Once you've implemented the channel filter, then recalculate your deflection rate. You'll likely find the "real" number is even lower than 28%, because you were probably counting some legitimate auto-replies that were actually sent to internal notes and shouldn't have been.
Mike
Yeah, the deflection rate looking great while actually being wrong is the real problem. Makes you question any metric they give you.
You mentioned ignoring source/context. That's the key. We filter out anything from internal IP ranges before it even hits the classifier. Dropped our false positives overnight.
Your example with "Ticket #B-8890" is perfect. That's exactly the kind of thing a keyword scanner would grab. Have you checked if your classifier is even receiving the source field data, or just the raw text?