I've been experimenting with ways to introduce lightweight, intelligent triage into our alert flow without committing to a full vendor platform. Our team's primary notification channel is Slack, so I decided to see if I could route those alerts through a structured prompt to a custom GPT (using the API) for initial assessment.
The basic architecture is simple:
* A Slack webhook listener (a small Python Flask app on a free tier cloud instance) captures specific alert messages from our SIEM integration.
* It formats the alert text, along with some static context about our environment (e.g., "prioritize alerts from the finance VPC"), into a predefined prompt.
* This prompt is sent to a custom GPT configured with guidelines for security triage—asking it to classify urgency, suggest possible false positive indicators, and recommend initial containment steps.
* The GPT's response is then posted back to a dedicated Slack channel, prefixed with `[AI_Triage]`.
The results have been surprisingly coherent. For example, it correctly flagged a series of failed login alerts from a non-existent user as a likely scan (low urgency) and suggested checking the source IP against our threat intel feed, while a `CloudTrail` `StopLogging` alarm was immediately escalated as critical. It's not perfect—it sometimes hallucinates irrelevant details from its training—but as a first-pass filter to reduce alert fatigue, it's proving useful.
I'm curious if others are pursuing similar "poor man's" AI SOC integrations. Specifically:
* What prompt engineering techniques have you found most effective for security alerts?
* How are you handling the inherent risks of data leakage when sending potentially sensitive alerts to an external LLM API?
* Has anyone built a simple feedback loop to improve these models, perhaps by logging analyst overrides?
The cost is negligible so far, akin to a few reserved instances. The biggest hurdle is trust, not technology.
—A
Every dollar counts.
The coherence you're seeing aligns with my own experiments using LLMs for initial signal classification. The static context about prioritizing finance VPC alerts is a smart inclusion-it mitigates the biggest weakness, which is the model's lack of live environmental awareness.
Have you measured any latency from alert generation to the triage response appearing in Slack? I tried a similar pipeline and found the API call was the bottleneck, sometimes adding 8-12 seconds. That's fine for low-urgency triage but becomes problematic if you ever want to escalate its role.
Also, how are you handling the risk of the model getting "creative" with containment steps? I had to build a strict validation layer that cross-references recommended actions against a known, approved list because mine once suggested isolating a server by shutting down a core network service we don't even operate.
Data > opinions
Cool experiment for a weekend project. But you built a critical path dependency on a free tier instance and an opaque API.
What happens when your Flask app hits a memory leak? Or OpenAI's API has an outage during a real incident? You've now hidden your primary alert channel behind two new single points of failure.
Static context isn't enough. The model has no real-time state. It can't know if finance VPC is currently under maintenance, which makes its "prioritization" guesswork.
Simplicity is the ultimate sophistication