We've been running Kubernetes in production for about five years now, and one of the first pain points we hit was alert routing. It's not enough to just send everything to a single on-call engineer anymore, especially with global teams and services that have different criticality windows.
The core challenge is balancing two needs: making sure the right person gets paged for a true emergency, and respecting team schedules to avoid burnout. A simple time-based rotation in your alerting tool often isn't enough. You need to consider:
* **Team-based schedules (Follow-the-Sun)**: Routing alerts to the team currently in their daytime working hours.
* **Individual overrides (PTO, sick leave)**: Ensuring someone covering is actually available.
* **Service ownership**: Alerts for Service A should always go to Team Alpha first, regardless of the time.
* **Escalation paths**: What happens if the primary doesn't acknowledge within X minutes?
We ended up implementing a two-layer approach using a common toolchain: Prometheus Alertmanager for routing logic, and Opsgenie for schedule management and actual notification delivery. Alertmanager handles the "what" (which alert) and Opsgenie handles the "who" (which human, based on a dynamic schedule).
Here's a simplified snippet of how we structure our Alertmanager config to route to different Opsgenie teams based on a label. The critical piece is that the `opsgenie_team` label is attached by our Prometheus rules, often based on the `service` label.
```yaml
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'default-opsgenie'
routes:
- match:
severity: critical
opsgenie_team: 'platform-emergency'
receiver: 'platform-emergency-opsgenie'
group_wait: 10s # Faster grouping for critical
- match:
opsgenie_team: 'data-team'
receiver: 'data-team-opsgenie'
```
The receivers then point to Opsgenie integration APIs. In Opsgenie, the `data-team-opsgenie` integration is tied to a **Team Routing Rule**, which checks the `Data Team Follow-the-Sun` schedule to see who is currently on-call and pushes the notification to them. This decoupling is key—it lets dev teams manage their own rotations in a human-friendly UI, while the infra team manages the alert routing logic as code.
What patterns have you all found effective? I'm particularly interested in how you handle escalations when the primary on-call is unresponsive—do you escalate within the same team schedule, or to a dedicated backup engineer?
I'm an SRE at a ~150 person SaaS company. We run EKS with Prometheus for alerting and have used both PagerDuty and Opsgenie in production for on-call routing.
**Cost at scale:** Opsgenie starts cheaper for small teams (free tier for 5 users). PagerDuty's entry point is higher but pricing becomes comparable around 25 users, roughly $20-$40/user/month for core on-call features. Opsgenie's hidden cost is needing the higher "Standard" tier ($29/user/mo) for advanced routing like alert deduplication.
**Integration and setup:** Opsgenie felt simpler to wire into Alertmanager with its native Prometheus integration. PagerDuty has more webhook configuration. Setting up follow-the-sun schedules was a similar 30-60 minute task in both.
**Where PagerDuty wins:** Escalation logic is more granular out of the box. You can build multi-layer escalations with different rules per service, which we needed for our tiered SLA products. Opsgenie needs more manual workflow configuration for that.
**Where Opsgenie wins:** The UI is easier for on-call engineers to manage overrides and swaps directly. PagerDuty tends to push more admin work to the schedule manager. Opsgenie's mobile app also had faster load times in my experience.
I'd recommend PagerDuty if your main concern is complex, reliable escalation paths for critical services. Pick Opsgenie if team usability and cost for a sub-25 person on-call rotation is the priority. To decide, tell us how many services have unique escalation rules and the size of your primary on-call team.
Still learning
Thanks for this breakdown, user169. Hearing from someone who's actually used both is super helpful.
I'm curious about your point on Opsgenie's UI being easier for engineers. Does that simplicity extend to the *setup* of the schedules themselves, or is it really just better for day-to-day swaps and overrides?
Totally feel the Prometheus/Opsgenie combo pain. That two-layer split is clever, but we found the sync between them can drift. You get a config change in Alertmanager and forget to update the Opsgenie schedule, or vice versa.
Our hack was to move the routing logic *into* Opsgenie entirely using its API-driven integrations. We feed raw alerts from Prometheus to Opsgenie, then use its own rules engine to handle the "Service A -> Team Alpha" logic based on tags. It's one less system to debug at 3am.
The gotcha is you lose some of Alertmanager's fancier grouping, but for us, the trade-off in simplicity was worth it.