Finally got our alert routing to a good place. Here are the rules we use.

We wasted 18% of our cloud budget last year on over-provisioned monitoring and alert noise. Fixed it by overhauling our routing logic.

Goal: Route to the correct team, with the correct urgency, on the first try. No all-hands pages. No ignored Slack channels.

We use Opsgenie. The core logic is in priority and routing rules.

**1. Severity Definition (Ties to SLO)**
- **P1/Critical**: Breach of SLO *and* active revenue impact. Page immediately.
- **P2/High**: Breach of SLO, no immediate revenue impact. Page within 30 min.
- **P3/Moderate**: At risk of breaching SLO. Ticket, no page.
- **P4/Low**: Informational. Ticket.

**2. Key Routing Rules (Condensed)**
Rules are evaluated in this order. First match wins.

```yaml
rules:
- match: 'source = "synthetic" AND error_rate > 95%'
priority: P1
team: platform-eng
note: 'Full region outage likely.'

- match: 'source = "kubernetes" AND container_restarts > 50 in 5m'
priority: P2
team: app-owner-${labels.app}
note: 'Check deployment health.'

- match: 'cost_anomaly > 200%'
priority: P3
team: finops
note: 'Spike investigation.'

- match: 'alertname = "CPUHigh" AND utilization < 85%'
action: suppress
note: 'Noise reduction. Threshold raised.'
```

**3. The Filters That Made It Work**
- **Time-to-acknowledge escalation**: If a P2 isn't acknowledged in 15 min, escalate to team lead.
- **Weekend buffer**: For P3s, delay notification to weekday business hours unless tagged `[override]`.
- **Alert de-duplication**: Any alert that repeats 3+ times in 10 minutes becomes a single incident.

Result: Page volume down 70%. Actual incidents get attention faster. The on-call engineer isn't sifting through garbage.