How do you decide which alerts should page vs just send a notification?

The eternal question, one that seems to generate more philosophical debate than actionable policy. Every vendor's blog post on "alerting best practices" will smugly tell you to "only page on actionable alerts" as if they've discovered fire, while conveniently ignoring the fact that determining "actionability" is the entire art form. It's a tautology dressed up as insight.

So, let's get concrete. I've seen too many teams, especially those new to the on-call rodeo, fall into one of two traps. The first is the "everything is critical" panic-button approach, which burns out your engineers in six months. The second is the "we'll just notify for now" approach, which leads to a Slack channel so noisy it becomes digital wallpaper, and the real Sev-1 quietly simmers for hours because everyone muted it. The decision matrix can't just be a gut feeling; it needs to be codified, debated, and ruthlessly refined. For me, a page-worthy alert must pass a three-stage filter, and I'm not talking about some fluffy mental checklist. I'm talking about explicit, written criteria that your routing logic enforces.

First, is there a clear, documented, and immediate **customer impact**? Not "might affect latency," but "the checkout API is returning 5xx for >5% of traffic for 5 minutes." This requires your monitoring to actually understand what a customer journey is, which is rarer than you'd think. Second, does it **require human intervention right now**? If the system is designed to auto-heal, let it. Paging someone to watch a script run is a fantastic way to breed alert fatigue. Third, and most importantly, **is the runbook or playbook definitive**? If the page goes off and the engineer's first step is to open a dozen dashboards to figure out what's happening, you've failed. The alert payload must contain enough context to immediately begin remediation.

Here's a simplistic but illustrative example of how we encode some of this logic for a Kubernetes cluster. This isn't the whole solution, but it forces the conversation.

```yaml
# Example Alert Rule Snippet (Prometheus-like)
- alert: APIErrorBudgetBurn
expr: |
rate(api_http_requests_total{status=~"5.."}[5m]) / rate(api_http_requests_total[5m]) > 0.05
for: 5m
annotations:
impact: "Critical user-facing API degradation."
action: "Immediate investigation required. Check deployment health, downstream dependencies."
runbook_url: "https://runbook.internal/sev2-api-errors"
# This annotation is parsed by our routing tool (e.g., Opsgenie, PagerDuty)
severity: "critical" # This routes to a page
auto_heal_check: "none" # No automated remediation exists for this symptom.
```

Contrast that with a notification-only alert:

```yaml
- alert: NodeMemoryHigh
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.15
for: 30m
annotations:
impact: "Potential future node pressure. No current user impact."
action: "Review during next business window. May require node scaling."
severity: "warning" # This routes to a Slack channel
```

The real work isn't writing the YAML. It's the weekly, tedious process of reviewing what paged, why it paged, and whether the human response changed the outcome. If you're not constantly promoting alerts to pages and demoting pages to tickets, you're just accepting that your process is decaying. So, I'm curious: what are the specific, non-negotiable gates your team uses to decide what gets the dreaded 3 AM wake-up call, and how do you audit its effectiveness beyond just "we got fewer alerts this month"?

-- Cam