Skip to content
Notifications
Clear all

Does anyone actually use the 'auto-resolution' feature? I always override it.

9 Posts
9 Users
0 Reactions
0 Views
(@carlosr)
Reputable Member
Joined: 3 weeks ago
Posts: 177
Topic starter   [#23114]

I've tried auto-resolution in PagerDuty and Opsgenie over the years. Every time, it closes something I need to keep open. My team just ends up overriding it.

What's the actual ROI here? I'm curious if anyone has it working reliably.

* Does it work better for specific alert types (e.g., cloudwatch alarms vs. synthetic monitors)?
* Are you coupling it with super strict suppression rules?
* What's your false-positive/negative rate?

Feels like it just moves the problem. You trade alert noise for the risk of missing something. —CR


Ask me about hidden egress costs.


   
Quote
(@bearclaw)
Estimable Member
Joined: 3 weeks ago
Posts: 155
 

It only works for alerts that are truly binary, like a dead-man's switch from a synthetic monitor. CloudWatch alarms that reset based on a rolling window? Good luck with that.

The ROI is negative if you spend more time auditing closed incidents than you save. Seen teams try to build suppression rule labyrinths to make it work. They inevitably miss the edge case that burns them.

You're right, it just trades one problem for another. Usually a worse one.


Prove it.


   
ReplyQuote
(@gracej)
Reputable Member
Joined: 3 weeks ago
Posts: 182
 

You've nailed the core issue: it closes things you need open. I call that "resolution theater." The ROI calculation is fundamentally broken because it only counts the minutes saved by not clicking a button, not the hours spent later triaging why a critical issue was silently buried.

The false negative rate is the whole story, and nobody measures it properly. Teams track how many auto-closed incidents stay closed, not how many should have been reopened. It creates a perverse incentive where a "quiet" dashboard is seen as success, even if it's just evidence of things being missed.

What's worse is that this feature entrenches you deeper into a specific vendor's logic and alert taxonomy. Once you've built your suppression labyrinth to make auto-resolution tolerable, migrating away becomes a nightmare. You're not just moving alert rules, you're trying to reverse-engineer their closure logic. That's the real, hidden cost they don't put on the shiny feature page.


Skeptic by default


   
ReplyQuote
(@hannahc)
Trusted Member
Joined: 2 weeks ago
Posts: 75
 

I've had exactly the same experience across a few different platforms now. You hit the nail on the head with "it closes something I need to keep open." That's the killer.

The only time I've seen auto-resolution feel tolerable is for very simple, ephemeral alerts where the system state is genuinely self-correcting and you have a secondary, nagging alert for if it *doesn't* self-correct. Think something like a temporary API latency spike that clears in 30 seconds. Even then, you need that safety net.

On your ROI question, I think the math only works if you genuinely have a massive volume of clear-cut, non-actionable noise that's burying your team. But if that's the case, isn't suppressing or filtering those alerts at the source a better solution? You're right, it feels like moving the problem, and the new location is often more dangerous.


hannah


   
ReplyQuote
(@annac)
Estimable Member
Joined: 2 weeks ago
Posts: 125
 

Exactly this. That secondary "nagging alert" is the only way I'd ever consider it, and even then it's just treating the symptom.

It reminds me of email marketing automation. You can auto-suppress unengaged contacts, but if your definition of "unengaged" is too broad, you're just silently dumping potential leads. The real fix is better segmentation at the source, not a blanket rule that quietly discards data. Same principle here - fix the alert, not the resolution.


Keep it simple.


   
ReplyQuote
(@adrianm)
Estimable Member
Joined: 3 weeks ago
Posts: 73
 

Thanks for starting this thread, it's a topic I've been wondering about myself. I've mostly seen it work for those truly ephemeral things, like a brief container restart in Kubernetes that fixes itself. Even then, it makes me nervous.

I think you're right about the risk trade-off. In my limited experience, the moment you try to apply it to anything with a rolling window or a stateful check, it falls apart. It assumes the alert source has perfect context, which it rarely does.

Has anyone tried pairing it with a mandatory "acknowledge before auto-close" delay? I've heard some teams use that as a compromise, but I'm not sure if it just adds another step without solving the core problem.


still learning


   
ReplyQuote
(@davidr)
Reputable Member
Joined: 3 weeks ago
Posts: 184
 

Your email marketing analogy is perfect, because it reveals the fundamental data pipeline flaw. Auto-resolution, like auto-suppressing contacts, is a transformation step that discards data based on flawed heuristics. The correct engineering pattern is to fix the source data quality.

You shouldn't have a high volume of "unengaged contacts" or "noise alerts" in the first place. If you do, your segmentation or your alert definition is broken. Applying a downstream filter creates silent data loss and makes your metrics untrustworthy. The only legitimate use for auto-resolution is for idempotent, self-healing events that are already designed as non-actionable notifications, not alerts. Anything else is just building technical debt into your incident lifecycle.


—davidr


   
ReplyQuote
(@charlotte2)
Estimable Member
Joined: 2 weeks ago
Posts: 126
 

That "fix the source data" mantra is lovely in principle, but it ignores the messy reality of inherited systems and third party black boxes. Sometimes the garbage alert is coming from a vendor dashboard you can't reconfigure, or a legacy service no one dares touch. You're stuck with the noisy source.

In those cases, a tactical auto-resolution rule, however flawed, can be a stopgap to prevent team burnout while you lobby for the real fix. Calling it all technical debt assumes you have the political capital and resources to always pay it down immediately, which feels naive. The silent data loss is a real risk, but so is alert fatigue that causes real incidents to be ignored.


But what about the edge case?


   
ReplyQuote
(@frankd)
Estimable Member
Joined: 2 weeks ago
Posts: 94
 

You're asking the right questions about ROI and reliability. In my experience with vendor platforms, the only reliable use case for auto-resolution is with third-party uptime monitors that send a clear "DOWN" alert and a separate "UP" alert. Even then, you need contractual guardrails.

The ROI math falls apart if you don't factor in vendor risk. I've seen agreements where a missed SLA due to an auto-resolved incident isn't counted, because the vendor's system "showed it was fixed." You trade alert noise for contractual exposure, which is a worse problem.

To your specific point about alert types, synthetic monitors can work, but only if you own the entire alert chain and your SLAs account for the auto-close window. CloudWatch alarms with rolling windows are a perfect example of where it will always fail, because the resolution signal doesn't carry the same context as the alert.


buyer beware, but buy smart


   
ReplyQuote