Let me start by saying that if your engineering leadership believes "being on-call" is just a synonym for "working more hours for free," you have a fundamental process problem, not a tooling problem. Burnout isn't an inevitability; it's a direct result of poor signal-to-noise ratio in your alerting and a lack of quantitative rigor in managing the on-call load.
I approach this like any performance benchmarking problem. You need to instrument the *system*—which in this case is your on-call rotation—and establish baseline metrics before you can optimize. You can't fix what you don't measure.
First, you must instrument your alerting pipeline. Every single page should be logged with metadata that allows for retrospective analysis. I've seen teams do this with a simple script that annotates their PagerDuty/Opsgenie incidents. The goal is to answer these questions for every alert:
* **Alert Source & Routing:** Which monitoring system (e.g., Prometheus, Datadog) fired it?
* **Service/Team:** Which microservice or team does it belong to?
* **Severity:** Was it a real P1 incident or a low-priority warning that could have waited?
* **Action Taken:** Was the page actionable? Did it require immediate human intervention, or was it auto-remediated?
* **Time to Acknowledge/Resolve:** Basic latency metrics.
* **Noise Classification:** Was it a false positive? A duplicate? A non-actionable "informational" alert?
You can structure this data simply. Without proper tagging, you're flying blind.
```json
{
"incident_id": "PD-2023-5678",
"triggered_at": "2023-10-26T03:14:00Z",
"alert_name": "High_API_Latency_99th",
"service": "payment-processor",
"severity_assigned": "critical",
"severity_actual": "low",
"actionable": false,
"root_cause": "downstream provider scheduled maintenance",
"false_positive": true,
"time_to_ack": "5m",
"time_to_resolve": "15m",
"escalated": false
}
```
After collecting this data for at least one full rotation cycle, you run the analysis. This is your synthetic benchmark for on-call health. The key performance indicators (KPIs) you're looking for are:
* **Pager Fatigue Metric:** The percentage of pages that are false positives, non-actionable, or low-severity during off-hours (e.g., 10 PM to 7 AM, weekends). If this is above 15-20%, your alerting logic is broken and needs immediate tuning.
* **Average Weekly Page Volume per Engineer:** Break this down by hour-of-day and day-of-week. If one person is getting 10+ pages between midnight and 6 AM every Sunday, that's a systemic imbalance, not bad luck.
* **Mean Time to Acknowledge (MTTA) vs. Time of Day:** Graph this. If MTTA spikes dramatically at 3 AM, your runbooks are insufficient or your alert isn't clear enough for a sleep-addressed brain.
* **Cost per Incident:** Factor in the engineer's fully loaded cost per hour, multiplied by the time spent on the incident. A $1500 engineer spending 45 minutes on a false positive is a quantifiable waste that directly impacts product development velocity.
The tooling discussion is secondary to this data. However, based on these metrics, your tooling requirements become clear:
* **Runbooks are not Confluence documents.** They must be executable, context-rich, and integrated directly into the alert. Think tools that provide pre-populated query templates, one-click diagnostic scripts, or direct links to curated dashboards. A runbook that says "check the logs" is worthless at 2 AM.
* **Escalation policies must be data-driven.** If the metrics show the first responder can't resolve 80% of alerts for Service X within 15 minutes, the policy should auto-escalate sooner, or the training/runbooks need to be fixed.
* **Snooze/Suppression rules are critical.** For known maintenance windows or deployments, you need robust, time-bound suppression. Getting paged for something you're actively doing is the fastest path to alert fatigue.
The post-incident workflow is your model training phase. Every actionable incident must result in an action item to improve the system. This is non-negotiable. The goal is to either:
1. Fix the root cause so the alert never fires again for that issue.
2. Improve the alert's specificity (e.g., adjust thresholds, add conditions) to reduce false positives.
3. Automate the response if the remediation steps are deterministic.
If you're not systematically reducing the weekly page volume and improving the actionable alert rate over time, your process has failed. Treat on-call like a production service. You wouldn't tolerate a service with 95% error rates; don't tolerate an on-call rotation that is 95% burnout-inducing noise.
Show me the benchmarks