Hey everyone! 👋 This is a question that’s been on my mind a lot lately, especially as we’ve been piloting some new AI-assisted triage tools in our SOC. It’s easy to get excited about the potential for efficiency, but I keep having this nagging worry: are we actually enhancing our team’s capabilities, or are we just building a fancy crutch that might atrophy their critical skills over time?
I come from a cloud migration background, and I've seen a similar dynamic there. When you move from on-prem to AWS or Azure, there's a risk of teams just clicking buttons in the console without understanding the underlying infrastructure. The same feels true here. If an AI tool is just handing analysts a "likely verdict" without insight, we're not improving—we're just adding a layer of automation that makes them passive consumers.
So, how do we measure this? I think we need to look beyond just raw metrics like "tickets closed per hour." Here’s what I’ve been discussing with my team:
* **Skill-Based Metrics:** Track the *complexity* of alerts analysts are able to handle independently over time. If AI is doing the heavy lifting, you might see them plateau at handling only the AI-pre-digested simple alerts.
* **Investigation Depth:** Measure the number of unique data sources an analyst consults *before* reaching a conclusion. Is the AI tool their first and only stop, or a starting point for a deeper dive? A lazy pattern is taking the AI's "medium confidence" finding and rubber-stamping it.
* **"Explainability" Quotient:** Can your analysts clearly articulate *why* an alert was false positive or a true positive, beyond "the tool said so"? We do quick, informal peer reviews where an analyst has to walk through their reasoning.
* **Proactive Activity:** Are analysts using the time saved to do threat hunting, improve detection rules, or write post-mortems? Or is that newfound time just… disappearing? This one is cultural as much as technological.
From my Azure/AWS experience, a tool that just automates is a cost center; a tool that augments and educates becomes a force multiplier. I’m really curious—is anyone else tracking this? Have you found a specific metric or process that shines a light on whether your AI SOC investment is building a stronger team or just creating a more efficient dependency?
null
Principal security engineer, 450-ish employee SaaS company in healthcare. We run a hybrid SOC handling a mix of cloud and on-prem assets, and I've pushed three different AI-assisted tools through evaluation cycles in the last two years. I currently have a home-grown workflow using the OpenAI API and a commercial agent triage tool in production.
You're asking the right question, and the slick demos never address it. Here's how you measure impact without getting fooled by vanity metrics.
1. **Skill Decay Proxy: Escalation Rate.** Don't track what they handle, track what they *bounce*. Measure the rate of tier 1 tickets escalated to tier 2 before and after AI tool introduction. If tier 1 starts escalating more, especially for alerts *outside* the AI's perfect scope, that's atrophy. A real metric we saw: our escalation rate for non-malware network anomalies crept up from 12% to 19% in the first quarter of relying on a vendor's auto-classifier.
2. **Critical Thinking Metric: Counter-Prediction Rate.** Force analysts to submit their own verdict *before* they see the AI's suggestion. Then measure how often they disagreed with the AI and, crucially, how often they were *correct* when they disagreed. If the "counter-prediction" rate drops to near zero, they've become rubber stamps. We built a simple plugin to log this, and it was the single most revealing data point.
3. **Tool Dependency: Mean Time To Manual (MTTM).** This is your "crutch" metric. For a defined set of common, well-understood alert types (e.g., known-bad IP connections), measure the average time it takes an analyst to reach a resolution when the AI tool is *unavailable*. Run a scheduled, unannounced drill once a month. If MTTM climbs steadily, the skill is eroding. We saw a jump from ~8 minutes to over 22 minutes on phishing alerts after six months.
4. **Insight Generation, Not Just Triage: Net-New IOC Ratio.** Tracks if the AI is helping them *discover* vs. just *process*. Count the number of new indicators (IPs, hashes, domains) an analyst adds to the threat intel platform that were *not* provided by the AI tool. A healthy tool should give them more mental bandwidth to hunt, causing this to stay level or rise. If it plummets, the AI is just a faster assembly line. Our ratio dropped 40% with one tool; we killed the pilot.
My recommendation hinges on your team's maturity. If you have strong senior analysts drowning in volume, the commercial agent triage tools (like those from Darktrace or Vectra) can be tuned to focus on noise reduction, preserving brainpower for the complex stuff. If your team is junior or the tool is marketed as a "force multiplier," you must implement counter-prediction tracking and monthly MTTM drills from day one, or you are absolutely building a crutch.
Just my 2 cents
The escalation rate metric is solid. We saw something similar after automating our legacy data mapping - junior analysts lost the ability to diagnose basic schema mismatch errors because the tool always "just fixed it" for our primary sources.
Your counter-prediction idea is clever, but be careful with workflow friction. We tried mandatory pre-AI judgments and got rushed, low-quality entries just to unlock the tool's output. The sweet spot for us was weekly, ungraded scenario drills without the AI present - that showed real skill retention (or decline) separate from daily ticket pressure.
test the migration twice
That's a great insight about forced friction backfiring. We saw the same thing with mandatory "pre-query justifications" for our analytics AI, people just typed nonsense to get past the gate.
The weekly drill idea is solid. It reminds me of how we test our email automation logic by having someone manually build a segment once a month, just to keep the mental model fresh. It's not about the volume, it's about the process.
The weekly drills make a lot of sense, like a regular fire drill. It forces the thinking process.
I'm curious, how do you prevent those drills from feeling like a test or extra admin work? Do you make them a team thing, or just something an analyst does alone? I could see it feeling like a chore if it's not built into the culture right.
That cloud migration comparison really hits home. I've seen teams lose track of the underlying principles when the platform does too much.
I like your idea about tracking *complexity* instead of just speed. One thing I'm wondering is how you'd define complexity in a consistent way for those metrics. Is it based on the number of data sources, or something else?
Following the thread, I think pairing a complexity metric with those weekly drills could give a clearer picture. If someone can handle a complex alert with the AI, but fails a similar one in a drill, that's a red flag.
Still learning.