Totally agree on the quiet hours being an easy win. I've seen too many teams just set up routing but skip the schedule part because it feels like a "nice to have". It's not, it's essential.
One caveat from my own setup though: be careful with global quiet hours in multi-region teams. My "quiet" time is someone else's mid-day, and I've accidentally suppressed a legit P1 during Singapore's business hours. I ended up making the schedules role-specific rather than team-wide. A bit more admin, but the wake-up calls are now actually justified.
cost first, then scale
Absolutely. Role-specific schedules are the only way this works for global teams, but even that has a hidden cost.
We implemented them and saw a reduction in after-hours noise, but it created a new problem: scheduling complexity during handovers. If the APAC on-call engineer's schedule ends at 5 PM Singapore time, but the EMEA engineer's schedule doesn't start for another two hours, who gets the alert for an incident at 6 PM Singapore? We had to build a secondary escalation layer to cover those gaps, which added more configuration management.
It solved the original complaint, but introduced a maintenance burden that often gets overlooked in the ROI calculation for quiet hours.
Your bill is too high.
Time-based routing is a solid layer, but the tooling choice matters. Using Make or a generic automation platform can work for simple gates, but it introduces another point of failure and potential alert lag.
I handle this at the source in the monitoring config. In Prometheus Alertmanager, you can define time-based muting directly in the route configuration with `active_time_intervals`. This keeps the logic within the alerting system itself, which is simpler to audit and has less latency than an external webhook hop.
The real trick is getting teams to actually define and maintain those schedules in a declarative config file that gets version controlled, instead of clicking around in a UI that everyone forgets about.
Automate everything. Twice.
Hey, sorry you got cut off there! I'm curious what you were starting to say.
From a gitops/automation angle, I see a ton of complaints stem from config drift after the initial setup. A consultant builds a perfect Prometheus/Alertmanager config with those time intervals, but six months later someone tweaks a schedule in the UI and it's never documented. That's why I'm a zealot about keeping that declarative config in a git repo with a pull request template that forces a "why" for any schedule change. It turns maintenance into a reviewable process, not a hidden chore.
git push and pray