We all talk about pager fatigue, but I've found it's often discussed in a vacuum. We look at alert volume trends or MTTR, but we rarely tie it directly back to the most common source of change: our own deployments.
I got tired of the anecdotal "things got noisy after the 3pm deploy" discussions, so I spent an afternoon writing a script. It pulls alert data from our monitoring system (for us, that's Grafana OnCall) and correlates it with deployment events from our CI/CD pipeline. The logic is simple: it looks at alert volume in the two hours before and after a deploy, flagging any significant spikes.
The results were genuinely eye-opening. About 70% of our "noise" incidents—those non-critical alerts that nonetheless wake someone up—were clustered within 45 minutes of a deployment. More importantly, it identified a pattern: a specific type of non-critical health check alert, which we'd considered background noise, was actually a near-perfect indicator of a deployment that hadn't fully "baked." We were being alerted, but we weren't listening correctly because we'd siloed the data.
This isn't about naming and shaming developers. It's about improving the process. We're now using this correlation to automatically tag post-deploy alerts in our incident tool (we use FireHydrant) and have started a small working group to refine our health checks. The goal is to turn reactive noise into a proactive signal that a rollback might be needed.
Has anyone else done similar analysis? I'm particularly interested in how you might integrate this kind of correlation directly into an incident response workflow, or if you've found effective ways to present the data to engineering teams without it feeling like a blame report. The script is rough, but I can share the approach if there's interest.
— mod
Keep it constructive.
Correlating deployment events with monitoring noise is an essential practice that's often overlooked. I've seen similar analyses reveal that many "transient" alerts are actually persistent state changes masked by averaging windows in dashboards.
Your point about not listening correctly due to siloed data is critical. A pattern I've observed is that teams often treat deployment systems as a write-only audit log, while monitoring is a read-only alert stream. Bridging that gap requires a deliberate integration, often at the metadata level - tagging alerts with deployment IDs or service versions, for instance. This allows you to move from correlation to attribution.
Have you considered extending the analysis to the type of deployment? In my experience, canary or blue-green releases often produce a different noise signature compared to big-bang deploys, which can help refine the thresholds for what constitutes a significant spike.
Migrate slow, validate fast.
Totally agree on moving from anecdotal to data-driven. That shift from "things feel noisy" to "70% of noise correlates with deploys" is where real process change starts.
Your point about a specific health check being a canary for an underbaked deployment is key. It turns a noisy alert into a leading indicator. That's something we've seen as well - once you have the correlation, you can start to classify. We ended up creating a separate, low-priority notification channel just for "deployment-induced transient alerts" that auto-silences after 90 minutes. It cut down the fatigue significantly.
Have you quantified the impact on MTTR for genuine, non-deployment-related incidents since isolating this noise? We found our true incident resolution time improved because the signal-to-noise ratio was better.
Data never lies, but it can be misleading