I'm deep in the marketing tech world, but our security team just greenlit an EDR evaluation and asked me to sit in on some demos. My brain works with KPIs and dashboards, so I'm trying to figure out what to actually ask for when they're showing off the platform.
What are the key operational and security metrics I should be requesting to see? I'm thinking less about the vendor's marketing claims and more about the tangible data points I can compare side-by-side in a spreadsheet later. For example:
* **Detection metrics:** Mean Time to Detect (MTTD) for common attack patterns in their own lab environment? Not just "we alert," but how *fast*?
* **Performance impact:** CPU/RAM overhead benchmarks—not just "lightweight," but average % impact on a standard workload (like a dev's laptop or a server).
* **Operational clarity:** What's the average number of alerts per endpoint per day after typical tuning? I need to know the potential signal-to-noise ratio.
* **Response efficacy:** For automated response actions (like isolating a host), what's the average execution time?
Basically, what numbers turn a flashy demo into an apples-to-apples comparison? I'll be building a scoring matrix for the shortlist.
— alex
Data > opinions
Your list is a solid start. Push for real data, not marketing slides.
On performance, ask for the 99th percentile CPU spike, not just average. I've seen agents that idle at 1% but hit 40% during a scan and freeze a developer's machine. Get their testing methodology - is it on a bare OS or with your typical workload (IDE, browser, Docker) running?
For alerts, the daily number is less useful than the *actionable* alert rate after 30 days of tuning. A good platform should learn and suppress noise. Ask for their average time to first false positive suppression.
Also add: deployment/rollback time. How long to push a new policy to all endpoints, or to revert if it breaks something? That's a key operational metric.
shift left or go home
Oh, you're thinking about this the right way. Your point about wanting numbers for a spreadsheet is key. I'd push them for their **false positive ratio during a live demo**. Ask them to run a detection on a clean, baseline Windows 10 or macOS VM with standard office apps while you watch. The alert count you see in that 15 minutes is worth more than a hundred slides about their ML models.
And for automated response times, don't just ask for the average. Ask for the *variance*. I've seen tools that isolate a host in 2 seconds most of the time, but when the network's having a bad day, it can take over a minute. That's the kind of thing that'll ruin your afternoon during a real incident. Good luck with the demos
it worked on my machine