So they're letting you loose with the company credit card to buy an "AI SOC" tool. Congratulations, or perhaps condolences. Before you get dazzled by a vendor's dashboard of glowing orbs, you need to establish what you're actually measuring. It's not magic, it's just another piece of infrastructure that will eventually fail in interesting ways.
Track these, and not just the vendor's cherry-picked numbers:
* **False Positive Rate (FPR) & True Positive Rate (TPR):** The classic duo. Demand the baselines they used. A 99% detection rate is meaningless if it's on a curated dataset of ten known-bad IPs. Ask for these metrics per alert/incident *type* (e.g., credential stuffing vs. data exfiltration).
* **Mean Time to Acknowledge (MTTA) & Mean Time to Resolve (MTTR):** This is where the rubber meets the road. Did the AI's fancy narrative actually help your tier-1 analyst, or did they spend 20 minutes deciphering its hallucinated justification? Compare these to your pre-AI SOC baselines.
* **Investigation Scope Precision:** How often does the tool's automated investigation pull in irrelevant logs or events, creating noise and cost? Every unnecessary cloud log query is money.
```json
// Example of what you should be able to audit
{
"incident_id": "INC-2024-5678",
"ai_generated_scope": {
"suspicious_entities": ["user-a", "host-b"],
"time_window": "2h",
"data_sources_queried": ["cloudtrail", "vpcflow", "okta", "crowdstrike", "s3_access"] // Why S3?
},
"analyst_final_scope": {
"suspicious_entities": ["user-a"],
"time_window": "45m",
"data_sources_queried": ["cloudtrail", "okta"]
},
"noise_ratio": 0.6 // 60% of the AI's queries were irrelevant
}
```
* **Cost Per Investigated Incident:** Factor in the tool's license cost, plus the compute/storage cost of its data ingestion and the queries it runs autonomously. An AI that saves 5 analyst hours but runs $500 of BigQuery scans per alert is a net loss.
* **Explainability Audit Trail:** Can you trace *why* it made a specific correlation? Not a vague "anomaly score of 87%," but a lineage of the logic. You'll need this for compliance (GDPR, etc.) and for the inevitable postmortem when it flags your CEO's login as malicious.
And the most important metric? The one you prepare for now: **Mean Time to Failure (MTTF) and your recovery process.** Request their last three major incident postmortems related to model drift, false negative storms, or integration outages. If they won't share them, that's your first red flag.
- Nina
- Nina
You've nailed the starting point. But I'd push on the "compare these to your pre-AI SOC baselines" bit. Most teams don't actually have clean MTTA/MTTR baselines, because their old tools were so noisy the data was garbage. You're comparing against a moving, ill-defined target.
Also, don't let them hide "AI SOC latency" inside MTTA. The tool's own processing and enrichment time needs to be its own metric. If it takes 90 seconds to "think" before it alerts, you've lost that time. Ask for p50/p95 alert generation latency post-event ingestion. If they can't provide it, they're measuring nothing useful.
- Nina
You're right about needing to ask for metrics per alert type. A vendor's overall FPR might look decent, but it's often being dragged down by high-volume, easy-to-detect stuff while completely flubbing the sophisticated attack patterns you actually care about. Demand a confusion matrix for each major use case, not an aggregated average.
I'd add that you need to see how TPR/FPR changes over a long evaluation. A model can drift, or an adversary can adapt. A static benchmark they ran six months ago is borderline useless. Ask for their process for continuous validation and model retraining, and what performance degradation triggers an update.
And absolutely yes on irrelevant logs. That's a direct cost multiplier in cloud environments. Ask them for the average number of log entries or API calls their automated investigation consumes per alert, and what logic limits the scope. If they can't give you a number, they aren't measuring it.
Show me the benchmarks
Totally agree on the per-use-case confusion matrix. I've seen tools that ace brute force detection but go completely silent on lateral movement patterns. Vendors love to blend all the numbers together.
The point about model drift is critical and often hand-waved. A vendor told us "the model continuously learns," but couldn't define the feedback loop or a performance floor. You need to ask what specific drop in TPR or spike in FPR triggers a model retrain. If they say "it's automatic," that's a red flag - it means they don't know either.
On irrelevant logs, great call. I'd also ask about the cost of false positives *beyond* log ingestion. If the tool auto-creates a Jira ticket for every FP, that's a huge time sink. The "noise tax" isn't just cloud bills.