Skip to content
Notifications
Clear all

What is the real-world accuracy of their phishing category? False positives?

3 Posts
3 Users
0 Reactions
4 Views
(@julieh)
Estimable Member
Joined: 1 week ago
Posts: 52
Topic starter   [#12753]

Everyone cites the "industry-leading" threat intel. I need the operational truth, not the datasheet.

In our last evaluation:
* A critical internal tool's update domain got flagged as phishing. Took engineering hours to diagnose.
* Meanwhile, actual spear-phishing from a compromised partner account sailed right through. It was "Business Services" category.

What are you actually seeing?
* How many legitimate business or SaaS domains get mis-categorized weekly?
* Does tuning the "phishing" category just break something else?
* Is the accuracy consistent, or does it vary wildly by industry/vertical?

Pricing is one thing. The cost of my team chasing ghosts or missing real threats is the real metric.


Caveat emptor.


   
Quote
(@cloud_security_sera)
Estimable Member
Joined: 1 month ago
Posts: 134
 

Exactly your two points are why we ditched the default "Phishing" category for a custom one. The false positives on update domains are constant noise.

> actual spear-phishing from a compromised partner account sailed right through

This is the real failure mode. Their "Business Services" and "Webmail" categories are massive blind spots. Tuning the Phishing category didn't break detection for us, it just shifted the problem. We had to build additional logic on top, checking for anomalous sender geolocation and new domain age, to catch what they missed.

Accuracy is inconsistent. It's worse for any niche SaaS or regional business service. You end up managing an allow list that never shrinks.


Least privilege is not a suggestion.


   
ReplyQuote
(@crusty_pipeline_v2)
Estimable Member
Joined: 2 months ago
Posts: 94
 

Yeah, the update domain thing is a classic. We had a similar hit on a Jenkins plugin mirror. Took hours to trace because the domain was a CDN alias. The real problem is the "Business Services" black hole. We saw a fake Office 365 login page that was sitting in "Business Services" for three days before it got reclassified.

Tuning the phishing category is a trap. You just lower the threshold and shift the noise. The logic behind the categories is a black box. You can't really fix it without a separate allow list for known-good update domains, which you'll be maintaining forever.

The accuracy varies by vertical. For us, it's worse with any SaaS that uses a subdomain structure for customer tenants. Those get flagged as phishing constantly. I'd rather have a tool that lets me define my own scoring rules than trust their categories. What's your current false positive rate per thousand requests?


slow pipelines make me cranky


   
ReplyQuote