Guide: Avoiding alert fatigue by tuning the continuous monitoring thresholds.

Hey folks, kerneldev here. Been running Secureframe's continuous monitoring for a few months on our containerized workloads, and while the out-of-the-box alerts are great for initial coverage, I was getting absolutely flooded. Especially from the "Failed login attempts" and "Unusual process activity" monitors. It felt like debugging a noisy IRQ storm 😅

Turns out, the default thresholds are quite conservative (which makes sense for a security product). But if you're in a dev-heavy environment with CI/CD pipelines and automated tooling, you'll need to tune them to match your actual baseline. Otherwise, real issues just get lost in the noise.

Here’s what I did to reduce the noise by about 70%:

**1. Identify Your Noisiest Monitors**
First, go to `Settings > Continuous Monitoring > Policies`. Check the "Alerts" tab and sort by frequency. For me, the top offenders were:
* `failed_login_attempts_per_hour` (default threshold: 5)
* `unusual_process_count` (default threshold: 10 new processes/min on a host)
* `network_listener_change` (alerting on every ephemeral port from our job runners)

**2. Establish a Baseline Using Logs**
Don't guess. I used a combination of the Secureframe event log and our own telemetry (Prometheus for process counts, audit logs for auth) to figure out normal behavior for a 24-hour period on a standard worker node.

For example, our automated testing suite spawns many short-lived containers. The `unusual_process_count` alert was constantly firing. My data showed a baseline of ~25 new processes/minute during work hours, not 10.

**3. Adjust Thresholds Gradually**
You can edit the thresholds directly in the policy. I recommend incremental changes. For the process count, I did:
```yaml
# In the monitor's YAML configuration
metric: "process.count.new"
threshold: 25 # Was 10
operator: "gt"
```
I waited 48 hours, checked if alerts became more meaningful, and then bumped it to 30. The goal isn't to silence all alerts, but to surface anomalies *outside* your normal operational envelope.

**4. Use Exclusions Wisely**
For the `network_listener_change` monitor, instead of raising the threshold, I added exclusions for specific subnets or processes. Our CI system uses a known IP range and runs under a dedicated service account.
```yaml
exclusions:
- source_ip: "10.10.5.0/24"
- process_name: "jenkins-agent"
```
This prevented hundreds of alerts from known-good systems.

**5. Implement Alert Aggregation**
Some monitors, like failed logins, are inherently noisy. I changed the alert aggregation window for that specific monitor from "per event" to "per hour, grouped by source IP." This gives me one digestible alert per suspicious source, instead of a blast of individual events.

The key takeaway? Treat these monitors like any other observability system: you need to tune them based on *your* signal. The defaults are a starting point, not a finish line. What strategies are others using to cut down the noise? Anyone using eBPF or custom agents to feed more precise data into Secureframe's classification engine?