After years of running a traditional weekly on-call rotation, my team has just completed a six-month experiment with an event-based scheduling model, and the results have fundamentally changed how I think about operational burden and alert ownership. The shift was driven by a familiar pain point: our weekly on-call engineer, regardless of how "quiet" the week was, would start the week already carrying a significant cognitive load and anticipation anxiety, which often impacted their project work. We were measuring pager fatigue in terms of weeks on duty, not actual incidents handled.
Our new model works like this: instead of a predetermined schedule, we have a primary and secondary on-call "pool." An engineer becomes the primary on-call from the moment they acknowledge and begin working on a Severity 1 or 2 incident. They remain the primary for the entire lifecycle of that incident—through resolution, documentation, and the initial post-mortem. Once the incident is fully closed, they return to the pool. If a second incident fires while they are still engaged, the secondary from the pool picks it up. We use the incident management tool itself (in our case, PagerDuty) to manage this state, leveraging its ability to change responders based on incident activity.
The most striking outcomes weren't just about fatigue metrics, but about behavioral and cultural shifts:
* **Incident Ownership Skyrocketed:** The engineer who picks up the alert sees it through to completion. This eliminated the frustrating handoffs we sometimes had near the end of a weekly shift and created a stronger sense of responsibility for root cause analysis and follow-up actions.
* **Alert Quality Improved Dramatically:** With no "scheduled week" to passively endure, the entire team became hyper-motivated to fix noisy alerts. If your poorly tuned alert goes off, you're just as likely as anyone else to get pulled in, even at 3 AM. This drove a 70% reduction in total alert volume within three months as teams instrumented better and refined thresholds.
* **Post-Incident Workflow Cohesion:** Because the same person drove the incident from detection to retrospective, the narrative in the post-mortem was more coherent and actionable. The context wasn't lost in a shift change.
However, this model isn't without its challenges. It requires a high degree of trust and a robust secondary/escalation path to prevent a single engineer from being overwhelmed by a cascading failure. We also had to carefully define what constitutes an "incident" that triggers the handoff—our low-severity alerts and scheduled tasks still go through a lightweight, voluntary rotation.
I'm now deeply curious if others have explored similar models. How do you handle coverage for vacations or planned time off in an event-based system? Has anyone tied this model to a FinOps practice, linking incident response directly to the cost-impact of the affected services? The move from time-based to event-based felt less like a scheduling tweak and more like aligning our operational processes with the actual event-driven nature of our cloud architecture.
~jason
~jason
This is a fascinating shift in metrics. You've moved from measuring simple calendar occupancy to tracking actual engagement time, which is far more meaningful. It reminds me of switching from measuring database uptime to tracking P99 latency during peak load. The vanity metric (weeks on call) gets replaced by the operational one (incident handling time).
Have you quantified the reduction in that anticipatory cognitive load? I'd be curious if you tracked something like the number of context switches for engineers in the week before their scheduled on-call duty versus under the new model. Without data, it's hard to know if the anxiety is just displaced rather than eliminated.
The lifecycle ownership piece is smart. It creates a closed feedback loop for the individual engineer, tying the pain of the page directly to the resolution and learning. Does this create a perverse incentive to close incidents quickly at the expense of thoroughness, though? How do you gate the "fully closed" state to prevent that?
-- bb42
That's a really clever way to structure it. I'm still new to all this, but the part about measuring pager fatigue in actual incident time instead of just calendar weeks makes so much sense. It seems way more fair.
How do you handle making sure the work is balanced? I worry someone might get stuck with a really long, complex incident and then another right after, while others have a quiet month. Is there a cap on handling time?