Hi everyone, I hope this is the right place to post this. I’m still pretty new to SRE and we just finished a 30-day proof of concept with Prisma Access for securing our remote workforce and cloud apps.
One thing really stood out in my monitoring dashboards: our overall alert volume went up by about 40% compared to the month before. I know more alerts isn’t inherently bad, but I’m struggling to figure out if this is “good” noise or “bad” noise. For example, we got a lot more alerts for “URL Category: High-Risk” and “Data Patterns Detected,” which *seem* important, but I’m not sure how to triage them without getting overwhelmed. Our old setup was much quieter, maybe too quiet.
I was wondering if others have seen this? How do you filter or prioritize these security-focused alerts so they’re actionable for an infra/SRE team? Also, did you find a way to correlate these with your existing platform SLIs, or do you keep them totally separate? Sorry if these are basic questions—I’m still learning how to balance security signals with operational reliability. Any advice on your workflow would be amazing.
Welcome to the world of actually seeing what's happening. Your old setup was "quieter" because it was blind, not because it was better.
You're asking how to triage. Don't. Not yet. First, you figure out if they're real.
* Is "High-Risk URL" someone clicking a shady link, or is it your security scanner doing its job?
* "Data Patterns Detected" - is that a dev uploading test data to a personal cloud bucket, or a real exfiltration attempt?
Correlate with existing SLIs? Usually a waste of time. These are different signals. You don't mix your "server is down" pager with your "someone might be doing something bad" alert. You build a separate playbook, or better yet, make it a sec team problem.
40% more alerts isn't a metric. How many resulted in a ticket? A block? An actual incident? Start counting that instead.