Skip to content
Notifications
Clear all

Just built a simple scorecard for our vendor review. Sharing the criteria.

4 Posts
4 Users
0 Reactions
0 Views
(@cost_optimizer_99)
Reputable Member
Joined: 3 months ago
Posts: 248
Topic starter   [#23211]

Everyone talks "comprehensive security" but nobody shows the bill. We ran Intercept X for 12 months on 500 endpoints. Here's the actual scorecard we used to decide not to renew.

**Scoring: 1-5 per category (5=best). Weighted total drives final decision.**

* **Efficacy (Weight: 30%):** Scored a 4. Crypto locker test blocks were solid. But the "predictive" AI detection flagged too many internal dev tools as malicious. Increased admin overhead.
* **Performance Impact (Weight: 25%):** Scored a 2. Our standardized benchmarks showed consistent 8-12% overhead on memory-bound applications. Not "lightweight" as advertised.
* **Operational Cost (Weight: 25%):** Scored a 3. License cost is one line. The real cost is in management. Integration with our existing SIEM required extra parsing logic (see below). Added ~0.1 FTE in labor we didn't budget for.
* **TCO/ROI (Weight: 20%):** Scored a 1. When we modeled the 3-year commitment they pushed, the numbers didn't justify the performance tax. Competing solutions offered similar block rates for ~35% less when you factor in the management overhead.

Example of the extra SIEM parsing needed for their alerts:

```sql
-- Their JSON alert payload nests critical data 3 levels deep.
-- This extra view was necessary for our dashboards.
CREATE VIEW parsed_interceptx_alerts AS
SELECT
JSON_EXTRACT(event_data, '$.user') as user,
JSON_EXTRACT(event_data, '$.threat_name') as threat,
-- ... more extraction logic
FROM raw_sophos_logs;
```

The sales deck is all about "next-gen" protection. The reality is hidden compute tax and management complexity. For our workload, the math said no.

show the math
**Final Weighted Score:** (4*0.3)+(2*0.25)+(3*0.25)+(1*0.2) = **2.65/5**


show the math


   
Quote
(@harryj)
Estimable Member
Joined: 2 weeks ago
Posts: 153
 

The SIEM parsing overhead is the killer. It's never in the sales deck. We had the same issue with a different vendor - their "open API" still required custom scripts to normalize every alert field, which added latency to our response playbooks.

Your performance impact score lines up with our tests on developer workstations. That 8-12% isn't trivial when you scale it across a team compiling code all day.


Automate the boring stuff.


   
ReplyQuote
(@carolinem)
Estimable Member
Joined: 2 weeks ago
Posts: 93
 

The emphasis on modeling the 3-year TCO/ROI is the critical piece here. Many teams stop at the license cost, but you've correctly factored in the compounded performance tax on developer productivity, which is an operational cost multiplier. The ~35% delta you found aligns with a pattern I've observed where efficacy plateaus but integration and performance overheads vary widely.

Your SQL snippet point about parsing is a great example of a hidden variable cost that breaks many ROI models. For a statistically sound comparison, you'd need to treat that extra parsing logic as a fixed implementation cost and then an ongoing marginal cost per alert. Did you consider running a controlled A/B test on a subset of endpoints for the competing solution to validate their actual overhead before committing, or was the decision based on the modeled projections alone?

The 8-12% overhead on memory-bound applications is significant. That could translate to a measurable increase in cloud compute costs if your workloads are elastic, which further degrades the ROI.


Nullius in verba


   
ReplyQuote
(@connork)
Estimable Member
Joined: 2 weeks ago
Posts: 77
 

That performance impact score is really interesting. We've been looking at similar tools and I keep hearing "lightweight" in demos. Do you find that 8-12% overhead is typical, or did you see it spike during specific tasks, like full system scans?



   
ReplyQuote