Just built a small dashboard that shows our on-call response times over the last quarter

catherine9 · 2026-08-23T06:11:12Z

Over the last quarter, our platform team observed a subjective increase in reported "pager fatigue" and anecdotal concerns about escalating mean time to acknowledge (MTTA) for severity-1 incidents. To move beyond anecdotal evidence, I undertook an analysis to quantify our response performance by building a dedicated internal dashboard that aggregates and visualizes on-call response metrics. The primary goal was to correlate alert volume, time-of-day, and team composition with our MTTA and mean time to resolve (MTTR), thereby identifying systemic bottlenecks rather than isolated events. The dashboard ingests data from three primary sources: * **PagerDuty API** for incident timelines, including acknowledged_at and resolved_at timestamps. * **Our internal team roster service** (a simple REST API) to map engineer IDs to their respective team and on-call schedule. * **A custom event log** where we tag incidents with the affected service and hypothesized root cause category (e.g., "database-latency", "third-party-api-failure"). The transformation layer, built as a set of scheduled AWS Lambda functions, performs the following key operations: 1. Enriches raw PagerDuty incidents with team and service context. 2. Calculates MTTA (acknowledged_at - created_at) and MTTR (resolved_at - created_at) for each incident, filtering out low-severity alerts (severity 3 and 4). 3. Aggregates the data into daily and weekly summaries, stored in Amazon DynamoDB for the dashboard backend to query. A simplified version of the core aggregation function is as follows: ```python def calculate_daily_metrics(incidents): daily_summary = {} for inc in incidents: day = inc['created_at'].date() if day not in daily_summary: daily_summary[day] = {'total_incidents': 0, 'total_mtta_seconds': 0, 'total_mttr_seconds': 0} daily_summary[day]['total_incidents'] += 1 daily_summary[day]['total_mtta_seconds'] += inc['mtta_seconds'] daily_summary[day]['total_mttr_seconds'] += inc['mttr_seconds'] # Calculate averages for day, metrics in daily_summary.items(): metrics['avg_mtta_minutes'] = (metrics['total_mtta_seconds'] / metrics['total_incidents']) / 60 metrics['avg_mttr_hours'] = (metrics['total_mttr_seconds'] / metrics['total_incidents']) / 3600 return daily_summary ``` The dashboard itself (a React frontend) presents four core views: * **Trend Lines**: Weekly average MTTA/MTTR over the 13-week period. * **Heatmap**: Incident count and average MTTA by hour-of-day and day-of-week. * **Team Comparison**: Breakdown of average response times by responsible team. * **Service Correlation**: A scatter plot relating incident volume per service to its average MTTR. Preliminary findings from the first quarter of data indicate two significant patterns. First, our MTTA for incidents created between 02:00 and 05:00 local time is 4.7 times higher than the daily average, despite a lower overall incident volume. Second, incidents related to our payment service module have a 40% higher MTTR compared to the platform average, which appears correlated with more complex, multi-step runbooks. I am now interested in expanding this analysis and comparing methodologies. Specifically: * What other data sources or key performance indicators (KPIs) have you found valuable to integrate into such an internal monitoring dashboard (e.g., alert grouping effectiveness, notification channel latency)? * For teams that have implemented similar tooling, how have you structured the post-incident workflow to automatically feed findings from retrospectives back into these metrics, creating a closed feedback loop? * Are there established benchmarks or industry studies for expected MTTA/MTTR ratios across different severity levels in a microservices environment that we could use for context?

Page 3 / 3 Prev

Incident Response Tooling

Last Post by greentea 1 month ago

31 Posts

30 Users

0 Reactions

66 Views

RSS

greentea

(@greentea)

Reputable Member

Joined: 2 months ago

Posts: 241

24/08/2026 11:55 am

Versioning at the source is the right long term move. It shifts the burden of consistency back to the authoritative system.

A caveat, though: this assumes the roster service maintains a queryable history. If it doesn't, you're adding a major dependency. A pragmatic hybrid is to have your Lambda snapshot the mapping *and* capture the roster service's reported "last_updated" timestamp. That gives you a version anchor you can audit later, even if the service only adds history down the road.

Without that, you're stuck if the historical view endpoint is never built.

ReplyQuote

Page 3 / 3 Prev