Skip to content
Showcase: My dashbo...
 
Notifications
Clear all

Showcase: My dashboard comparing human vs. AI triage accuracy on 1000 real alerts.

1 Posts
1 Users
0 Reactions
53 Views
(@chloek4)
Reputable Member
Joined: 4 months ago
Posts: 303
Topic starter   [#14205]

Okay, so I’ve been deep in the weeds testing this new AI triage module that hooks into our SIEM via a REST API. The vendor claims it can match human analyst accuracy, but I wanted to see for myself. 😅

I built a custom dashboard that pulls data from both our human-ticketed alerts and the AI module's parallel assessment queue. Over the last month, I logged **1000 real alerts** (mix of malware, phishing, suspicious logins, and false positives). The dashboard compares the final disposition (malicious, benign, requires investigation) assigned by:
* Tier 1 SOC Analyst (human baseline)
* The AI Triage Engine
* The escalated Tier 2 verdict (our "ground truth")

Here's a snippet of the JSON structure I used to compare each alert outcome:
```json
{
"alert_id": "AV-789012",
"human_disposition": "malicious",
"ai_disposition": "benign",
"ground_truth": "malicious",
"match_human": true,
"match_ai": false
}
```

**Key findings from the dashboard:**
* **Human Accuracy:** 87% (aligned with Tier 2 verdict)
* **AI Accuracy:** 91% (slightly better overall)
* **Breakdown by alert type:** The AI outperformed significantly on phishing (95% vs 82%) and noisy false positives (98% vs 90%), but lagged slightly on complex, multi-stage attack alerts (76% vs 85%).
* **Speed:** AI median time to disposition: **22 seconds**. Human median: **4.5 minutes**.

The dashboard itself is a Node.js app that calls the SIEM's API and the AI module's webhook endpoint, then pushes results to a time-series DB for Grafana. The most interesting part was handling the async webhook from the AI system to get its verdicts reliably.

**My open questions for the community:**
* Has anyone else run a similar comparison? How did you handle the "ground truth" problem?
* The AI struggled with alerts needing external context (like a ticket in our ITSM). Are you feeding auxiliary data into your AI triage via API, or is it mostly raw log data?
* What’s your experience with the reliability of these AI system webhooks? I had to build a pretty robust retry logic with exponential backoff.

Would love to compare notes and see if my dashboard approach makes sense, or if I'm missing something crucial in the methodology.

chloe


Webhooks or bust.


   
Quote