Complete newbie here - what metrics should I track when evaluating an AI SOC tool?

You've nailed the starting point. But I'd push on the "compare these to your pre-AI SOC baselines" bit. Most teams don't actually have clean MTTA/MTTR baselines, because their old tools were so noisy the data was garbage. You're comparing against a moving, ill-defined target.

Also, don't let them hide "AI SOC latency" inside MTTA. The tool's own processing and enrichment time needs to be its own metric. If it takes 90 seconds to "think" before it alerts, you've lost that time. Ask for p50/p95 alert generation latency post-event ingestion. If they can't provide it, they're measuring nothing useful.

You're right about needing to ask for metrics per alert type. A vendor's overall FPR might look decent, but it's often being dragged down by high-volume, easy-to-detect stuff while completely flubbing the sophisticated attack patterns you actually care about. Demand a confusion matrix for each major use case, not an aggregated average.

I'd add that you need to see how TPR/FPR changes over a long evaluation. A model can drift, or an adversary can adapt. A static benchmark they ran six months ago is borderline useless. Ask for their process for continuous validation and model retraining, and what performance degradation triggers an update.

And absolutely yes on irrelevant logs. That's a direct cost multiplier in cloud environments. Ask them for the average number of log entries or API calls their automated investigation consumes per alert, and what logic limits the scope. If they can't give you a number, they aren't measuring it.

Totally agree on the per-use-case confusion matrix. I've seen tools that ace brute force detection but go completely silent on lateral movement patterns. Vendors love to blend all the numbers together.

The point about model drift is critical and often hand-waved. A vendor told us "the model continuously learns," but couldn't define the feedback loop or a performance floor. You need to ask what specific drop in TPR or spike in FPR triggers a model retrain. If they say "it's automatic," that's a red flag - it means they don't know either.

On irrelevant logs, great call. I'd also ask about the cost of false positives *beyond* log ingestion. If the tool auto-creates a Jira ticket for every FP, that's a huge time sink. The "noise tax" isn't just cloud bills.

Yeah, that's a great point about the per-use-case matrix. It makes me wonder, what do you do if a vendor provides a dozen different confusion matrices? As a smaller team, we'd struggle to evaluate all of them. Is there a rule of thumb for which two or three use cases are the most important to scrutinize first? Like maybe credential access and data exfiltration?

That's an excellent question and a very real problem. When faced with a dozen matrices, you need a triage strategy. Start by mapping the use cases directly to your organization's top-priority MITRE ATT&CK tactics, as defined by your threat model and risk assessment. For most, this will indeed be **Initial Access (like credential stuffing)** and **Exfiltration**, because they represent a clear, tangible loss. A tool that fails at those is non-negotiable.

Then, add one more: look for the **Lateral Movement** matrix. This is often where behavioral AI models claim superiority over signature-based tools, but it's also a much harder pattern to detect with low false positives. If they perform poorly here, their "AI" might be just window dressing on simpler rules. Scrutinizing these three should give you a solid signal on whether the tool understands basic attacks versus sophisticated ones.

You can safely deprioritize matrices for high-volume, low-criticality alerts, like generic scanning, in the initial deep dive. Just note if they exist.

Good starting list, but you left out the biggest trap: vendors defining "false positive." I've seen them label analyst-dismissed alerts as "true positives" because the tool technically detected something. You need the exact definition they use before you compare any FPR numbers.

Also, Investigation Scope Precision is meaningless without context. Is it pulling 100 irrelevant logs per alert or 10,000? You need a raw number, not just "often." Ask for the average log volume per investigation and the p95.

Good points, but your MTTA/MTTR advice assumes the old baseline is worth a damn. Most teams I've seen have no coherent baseline because their previous alerting was a firehose of garbage. You're benchmarking against chaos.

And you've got to isolate the tool's own think time from the human response time. If their "AI" takes three minutes to generate an alert after ingesting an event, that's dead time they'll happily bury in your MTTA. Demand the p95 latency from event to actionable alert. If they can't measure that internally, they're selling a black box.

Solid starter list, but I'd push on the "compare these to your pre-AI SOC baselines" bit. Most teams don't actually have clean MTTA/MTTR baselines, because their old tools were so noisy the data was garbage. You're comparing against a moving, ill-defined target.

Also, don't let them hide "AI SOC latency" inside MTTA. The tool's own processing and enrichment time needs to be its own metric. If it takes 90 seconds to "think" before it alerts, you've lost that time. Ask for p50/p95 alert generation latency post-event ingestion. If they can't provide it, they're measuring nothing useful.

You're right to start with those core metrics. Building on your point about TPR/FPR per alert type, I'd press for how they segment "credential stuffing" versus "suspicious authentication." One vendor I evaluated counted any failed login cluster as a potential credential stuffing attempt, massively inflating their TPR on easy, high-volume noise while missing the actual low-and-slow attacks. You need their exact detection rule logic, not just the category label.

On MTTA/MTTR, comparing to a pre-AI baseline is essential but often flawed, as others noted. A more telling metric is to track the *change* in these times for validated true positives only, over the first 90 days of deployment. This isolates whether the tool's enrichment is genuinely speeding up analysis or if your team is just getting better at ignoring its noise.

This is super helpful, thanks for breaking down the basics like this. As someone new to this side of things, I have a rookie question about MTTA.

When you say "compare these to your pre-AI SOC baselines," what if we basically don't have any? We're a smaller shop and our current "baseline" is just our team reacting to a flood of generic alerts. Is it okay to start from zero, or does that make the comparison useless?

That's a very common and valid position to be in. Starting from zero doesn't make the comparison useless; it just changes what you're comparing.

Instead of comparing against a broken baseline, use the first 30-60 days after implementation *as* your new baseline. Track MTTA for the alerts the new tool surfaces. Then, look for improvement over the *next* 30-60 days. This measures the tool's (and your team's) ability to learn and accelerate, rather than comparing against noise.

Also, pressure the vendor on what I call "clarity latency." How long from their alert do you get enough context to decide to investigate? If it takes 15 minutes just to understand what the alert is even about, your effective MTTA is already sunk.

Precisely, and that's the core of the sales pitch. They'll give you the 'category label' and a dazzling matrix, while the actual logic is a fragile pile of regex and threshold counters that any moderately patient attacker can stroll around.

Your point about tracking the change over 90 days is smart, but assumes you can stomach the noise for that long. For a smaller team, that first month of inflated 'credential stuffing' alerts could be the death of the project before you even get to measure a trend.

That's the real risk. They sell you on the long-term trend while your team burns out on week two. For smaller teams, the deployment contract must include a service level for initial alert volume, with clear penalties if they exceed it by a set percentage. If their "AI" can't tune itself pre-deployment, they're selling you an intern.

Love this starting list, it really cuts through the vendor fluff. The point about comparing MTTA/MTTR to pre-AI baselines is so true. It's easy to get fooled by a dashboard that just shows "faster" without knowing what you're measuring against.

Can I add a newbie question from a different angle? You mentioned money with unnecessary log queries. How do you even track that cost practically? Is it just an API call count, or is there a standard way to tie the tool's investigation scope directly to your cloud bill? I'd hate to solve alert fatigue but create a budget crisis instead 😅