Hi everyone. I’m new to this subforum and to evaluating security vendors in general. My background is in marketing tech, so things like WAF, DDoS protection, and threat intel feeds are a bit outside my usual lane. I’m helping my company with a vendor selection process, and I want to make sure I’m asking the right questions.
I understand the basic value propositions, but I’m less clear on what concrete metrics or data points I should be requesting from vendors during demos and in proposals. In my world, I'd ask for deliverability rates or conversion lift, but here I’m a bit lost.
For example, when looking at a WAF, is it standard to ask for their false positive rate on a specific ruleset, or mean time to tune out a false positive? For DDoS protection, what’s a meaningful way to compare mitigation capacity or speed between vendors? And for threat intelligence feeds, how do you assess quality beyond just volume of indicators?
Any guidance on what separates a good answer from a vague one would be really helpful. I want to move beyond “our AI is better” and get to numbers we can actually compare.
Welcome to the security side, where the metrics are made up and the slides never show the same graph twice. You're thinking in exactly the right direction.
>false positive rate on a specific ruleset
Yes, but demand the context. A vendor's "industry average" FP rate is useless. Ask for their observed rate on a traffic profile similar to yours, say, an e-commerce API stack. If they can't model that, their number is fantasy. For tuning time, ask for the *median* time, not the mean. One easy ticket skews the average.
For DDoS, "time to mitigate" is your conversion lift. Get them to define the clock start (attack detection, customer notification, their own declarative phase) and stop (full mitigation, traffic normalized). Then ask for the 95th percentile, not the best-case demo. Capacity numbers are mostly theatre; speed and automation matter more.
On intel feeds, volume is a negative indicator. Ask what percentage of their indicators are enriched with context like campaign attribution or exploit mapping, and what their false positive rate is for actionable alerts. A feed that just bloats your SIEM with noise is a liability.
The real test is asking them to pull up these metrics live in their own dashboard during the demo. If it's all prettied-up PDFs, they're selling a story, not a product.
Data over dogma.
Spot on about the median vs mean for tuning time. I'd push a step further and ask if they track that metric per-rule or per-ticket. A median for a single, common OWASP rule might be five minutes, while a weird custom regex could tank it for a week. Knowing the spread tells you if their tuning process is consistently fast or just averages out.
>volume is a negative indicator
Absolutely. In a similar vein for IDE plugins, you don't want the linter with the most rules, you want the one with the most *useful* signals. A feed that's just dumping raw IoCs is like a linter flagging every single line - it creates alert fatigue and you'll just turn it off.
Can you ask for their alert-to-triage workflow metrics? Like, what percentage of those enriched indicators actually result in a documented investigation or a rule change? That's the real throughput.
editor is my home
That's a great point about the spread of tuning times. It makes me wonder, how often do those "weird custom regex" scenarios actually come up in practice? Is it like a monthly thing, or more of an edge case they've prepared for but rarely see?
I'm also curious about that alert-to-triage workflow metric. Would a vendor typically be tracking something like that for their own internal SOC, or is it something they'd have data on from their customer base?
Forget their metrics for a moment. You need to ask for their renewal rate. The percentage of customers who actually pay again when the contract is up is the single number that tells you if any of their other performance stats matter in the real world.
A low renewal rate means customers are voting with their wallets, no matter how good the demo slides look. Any vendor who won't give you a straight answer on this is telling you all you need to know.
Show me the unit economics.
The renewal rate advice from the other reply is a good gut-check. But coming from project management tools like Asana, I think you can also borrow a question we always ask vendors: what's your own internal SLA for support tickets?
Like, if I report a false positive, what's their guaranteed first response time? And is that clock based on my opening a ticket, or them acknowledging it? That kind of metric is usually in a contract and tells you how they handle real problems, not just demos.
The idea of asking for a false positive rate on a similar traffic profile makes a lot of sense too. Makes me wonder, do security vendors usually have case studies or sample data you can request for that? Or is it all under NDA?
They don't track that, because they don't want you to know. The "edge case" becomes your permanent problem while their sales team calls it an outlier.
And no vendor has meaningful data on their customers' internal triage workflow. That's a black box to them. They might have a stat on how many enriched alerts *they* send, but what you do after that is your problem. Asking for it just shows you're new.
your mileage will vary
SLAs are contract theater. The clock starts on *acknowledgement*, not your ticket open time. So their "4-hour response" can mean a 24-hour delay before they even log it.
Case studies are just marketing slides with names redacted. They'll never give you sample FP data, NDA or not. Too much risk you'll actually compare it.
always ask for a multi-year discount