Walkthrough: reducing alert noise by tuning threshold parameters

Alright, listen up. I've been woken up at 3 AM more times than I can count because of garbage alerts. The number one culprit, bar none, is lazy threshold configuration. Everyone sets up their monitoring, slaps in the defaults from the vendor blog post, and calls it a day. Then they wonder why they're drowning in noise and missing the real fires.

Tuning thresholds isn't about making your graphs look pretty; it's about aligning your alerts with actual user pain and system failure modes. This is a process, not a one-time task. Here's how I do it, the boring way that actually works.

First, you need a baseline. You don't just guess. For a week, you let those noisy alerts fire, but you **tag them**. Every single one. Your alerting system needs to support this. I use a simple annotation in Alertmanager or a tag in the alert itself. The categories are:
* **Actionable & Correct:** The system is broken, and you need to do something.
* **Actionable but Wrong Severity:** It's a real issue, but a P4, not a P1.
* **Non-Actionable Noise:** The metric breached a theoretical threshold, but nothing is actually degraded. This is your target for elimination.

After that week, you analyze. If 80% of your alerts are "Non-Actionable Noise," your thresholds are worthless. This is where the real work starts.

Let's take a concrete example: CPU usage. The default alert is always something brain-dead like `cpu_usage > 80%`. This is useless. A modern system can spike to 100% for minutes during a batch job with zero user impact. Instead, you need to alert on *sustained* high CPU *combined* with a symptom of user impact.

Here's a Prometheus-style rule that's more thoughtful:

```yaml
groups:
- name: node.rules
rules:
- alert: HostHighCpuLoad
expr: |
(100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
and on(instance)
(rate(node_network_receive_bytes_total[5m]) < 1000)
and on(instance)
(rate(http_requests_total{code!~"5.."}[5m]) 80%) for 10 minutes, AND low network traffic, AND low successful HTTP requests. Might be a stalled process, not just load.'
```

See the difference? This only fires if high CPU is **paired with low traffic and low successful requests**. That suggests the system is choked and can't do work, not that it's busy. You tune the `80`, the `10m`, and the ancillary condition thresholds (`1000` bytes, `1` request) based on your baseline data.

The process is always the same:
1. **Measure the baseline** for the metric during known good and known bad states. Use percentiles (p95, p99), not averages.
2. **Set the threshold** between these observed bands, with a healthy buffer. If your app performs fine with latency at 500ms, but users complain at 1500ms, maybe alert at 1200ms.
3. **Add a duration clause (`for:`)**. Almost nothing should fire instantly. Spikes happen. 5 minutes is a good starting point for most system metrics.
4. **Correlate with a symptom.** High memory? Check for OOM kills or swap usage. High disk usage? Check the rate of growth, not the static percentage.
5. **Document the rationale.** In the runbook, state *why* this threshold exists and what the expected response is. If you can't write this, the alert shouldn't exist.

Finally, you make this part of your post-mortem process. Every incident review should ask: "Did our alerts fire correctly? Were they too noisy? Did we miss something?" Then you go back and adjust. It's a feedback loop. The goal is to get to a state where every page is a real page, and you can actually trust your dashboards.