I built an internal dashboard to track our team's on-call response times over Q2. The primary goal was to move past gut feelings about our performance and get objective data. It pulls from our PagerDuty API and visualizes median and 95th percentile response times per service, along with acknowledgment and resolution times.
The initial data revealed a clear pattern: our median response is under five minutes, which is solid, but our 95th percentile spikes to nearly twenty minutes for two specific services. This points to a chronic issue, not random noise. The long tail correlates almost exclusively with alerts lacking clear, actionable runbooks.
I'm sharing this to ask: what metrics do you all track for on-call health beyond simple uptime? We're debating whether to add a metric for "time to effective engagement" (i.e., when the on-call engineer has the correct runbook or context in hand), but I'm wary of overcomplicating it. How do you balance actionable data with pager-fatigue for the team being measured?
—AF
—AF
Your approach of separating median from the 95th percentile is key. The long tail is where real operational debt accumulates. At my last role, we found that "time to effective engagement" became our most predictive metric for actual resolution time, but it's hard to instrument automatically.
We approximated it by logging the delay between first acknowledgment and the first access of a specific, tagged runbook in our internal wiki. It introduced some noise, but the correlation was strong enough to prioritize runbook updates over simply hammering response times.
The balance you're asking about is real. Adding metrics increases load on the team. We only introduced a new one after proving it could directly reduce toil, for example by showing that a 10-minute investment in a runbook cut the 95th percentile engagement time by half for that alert category. Are you tracking which specific alert titles or codes are generating those 20-minute outliers? That granularity often points to the exact documentation gap.
data is the product