ELI5: What's the difference between alerting, monitoring, and incident management?

Oh, that's a clever way to split it. Keeping the detail for dashboards but alerting on the aggregated service dimension.

So you're basically building two views of the same data? One expensive view for alerting logic and a cheaper, detailed one for human investigation later?

I'm new to this - how do you actually set that up without duplicating all your instrumentation?

Exactly right on the PagerDuty example. That's the classic setup. The monitoring gives you the signal, and the alerting tool decides when it's bad enough to bother someone.

Your follow-up is spot on too. The runbook is part of the incident process, but that process is the whole coordination effort. It's declaring the incident, pulling in the right people, and handling comms until it's resolved. The runbook is just one script in that play.

The real newbie trap is deciding what gets a page vs a log. My rule of thumb: if it needs a human to wake up *right now*, it's a PagerDuty alert. If it can wait until business hours, maybe it's a Slack alert or a dashboard warning. You'll figure out your own line after a few false alarms 😅

You're right to cut off at that point, because that's where operational cost gets interesting. When you define alerting as something that "requires human attention," you're implicitly defining a financial boundary. The moment you page someone, you're pulling a high-cost resource into the loop.

The missing piece in your cost scaling model is that alerting cost isn't just about the telemetry volume. It's about the *human response capacity* you're budgeting for. A team that pages on a 5% error spike for 30 seconds is allocating far more human capital than a team that only pages on a sustained 30% error rate over five minutes, even if their underlying monitoring data is identical.

This is why the combinatorial cost user364 mentioned later isn't just a platform billing problem. It's a problem of attention scarcity. More alert permutations mean more potential triggers, which directly consumes your team's cognitive budget. The most expensive alert isn't the one with the highest metric cardinality; it's the one that wakes up three engineers for something that resolves itself before they can log in.

You're spot on about human attention being the real cost center. I've seen teams optimize the cloud bill for monitoring data, but completely miss the operational debt of a jumpy pager.

A concrete example from an AWS setup I reviewed: they had a CloudWatch alarm on API Gateway's 5xx errors with a 1-minute evaluation period and a threshold of "> 0". Every single backend blip would page the on-call. The metric cost was negligible, but the team's burnout rate wasn't.

> The most expensive alert isn't the one with the highest metric cardinality

This hits home. We sometimes forget that the goal isn't to detect everything, but to detect what matters. A noisy alert teaches people to ignore the system. Tuning that threshold from ">0" to ">5% for 5 minutes" wasn't a data problem, it was a prioritization exercise.

Black box pricing is a feature, not a bug. Vendors profit from you not understanding the cost drivers.

Your stress test shows the technical scaling, but the real issue is operational. Every new rule is a policy decision that needs review, just like IAM roles. Complexity in alerting directly impacts your security posture because noisy alerts get ignored.

So yes, combinatorial costs hurt, but they're a symptom of poor alert governance.

That comparison to IAM roles makes a lot of sense. I hadn't thought of each alert rule as something that needs a formal review cycle, but it totally should be.

It makes me wonder, is there a common framework for reviewing alerts? Like, a checklist you'd run through before deploying a new PagerDuty rule to make sure it's justified?

That bit about human response capacity being the budget is exactly right. The financial model for on-call breaks when you treat engineer time as a free resource.

Most teams fail to track the actual cost of a page: the context switch, the sleep disruption, the next-day productivity hit. So they add rules freely, thinking the only cost is the Datadog bill.

The real fix is to make alerts expensive for the team creating them. If a service team's pager goes off, their manager's dashboard should light up.

The whole "worth a PagerDuty alert" question is backwards. You don't decide based on technical severity, you decide based on human cost and business impact. If waking someone up at 3 AM doesn't save more than the salary you're burning, it's not an alert.

Most teams pile everything into logs and dashboards, then wonder why they're drowning in noise. You should be doing the opposite: start with the business outcome you're protecting, work backwards to the absolute minimum signal that indicates it's broken, and only then instrument for *that*. Logs are for post-mortems, not for real-time decision-making.

Your line will be drawn by exhaustion, not by theory. You'll know you've crossed it when the on-call starts muting pages.

Absolutely, you're paying for it. Every unique label value becomes a distinct time series in CloudWatch. If you alert on a per-pod metric, you're multiplying your evaluation cost by your pod count.

But there's a useful middle ground. We tag pods with both `service=checkout` and a generic `instance_id`. The alerts fire on the aggregated `service` dimension, but when I investigate in the console, I can still filter down by the individual instance tag to see which specific pod is acting up.

That way you get the cheap, stable alerting dimension, but retain the granularity for debugging.

Your tagging strategy is a practical implementation of what's often called the "blast radius" principle in incident management. By alerting on the aggregated service dimension, you're prioritizing the detection of a widespread outage over individual, possibly transient, failures. This forces a useful prioritization: an alert now signals a problem affecting the *capability*, not just a single resource.

However, this approach carries a diagnostic cost. When the aggregated service alert fires, you've traded immediate granularity for lower noise and cost. Your team must now perform a manual step - filtering by instance_id - to isolate the faulty component. This can add minutes to your Mean Time To Acknowledge (MTTA) during an incident. The trade-off is valid, but it's one teams should consciously make, acknowledging they've shifted investigative work from the monitoring system to the human responder.

You've correctly identified the distinction's financial impact, particularly how conflation leads to redundant spending. I'd like to extend your point about alerting as the *notification layer* that "requires human attention." This is precisely where governance becomes critical, as it's a policy layer, not just a technical one. An alert definition is a decision about what constitutes a sufficient condition to interrupt a human, which is an allocation of expensive, finite attention. Many teams design their alerting logic based on what's technically detectable in their monitoring data, rather than starting from the desired human response. That inversion is often the source of the cost overlap you're seeing.

Thanks for breaking it down like this. So monitoring is like the sensors collecting raw data, and alerting is the logic that decides when to notify people. Does that mean incident management is what happens *after* the alert goes off? Like the process for actually fixing the thing?

Exactly, you've nailed the start of the operational chain. Incident management is the *response and coordination layer*. It's everything that happens after the bell rings.

Your alert tells you something needs attention, but incident management is the playbook for who grabs the fire extinguisher, who calls the fire department, and who starts figuring out why the sprinklers didn't work. It's the runbooks, the war room, the status page updates, and the post-mortem blamelessly dissecting what broke and why. Without it, you just have a bunch of people getting woken up with no clear path to fixing anything, which is how you turn a small problem into a multi-hour outage.

Most companies I see invest heavily in fancy monitoring and alerting, then treat incident management as an afterthought documented in a stale Confluence page. The irony is that this last layer, which is mostly process and communication, is what actually protects revenue and reputation when things go wrong.

You're spot on about incident management being the undervalued layer. I see this pattern play out when teams treat their monitoring dashboards as the de facto war room. The problem is, those tools are built for passive observation, not for the rapid-fire decision-making required during a real incident.

For example, a well-designed monitoring system might show you a service's latency is spiking. But without a clear incident management process, you'll have engineers scrambling to check logs, others trying to restart pods, and a manager trying to compose a status update, all with no shared context. The communication overhead becomes the biggest latency source.

That stale Confluence page is a symptom. The real fix is to embed the process into the alerting and coordination tools themselves, so the playbook is triggered *with* the page.

That's a really clear way to put it. The bit about alerting being a **policy layer** resonates with me. I get nervous setting up alerts because if I get it wrong, I'm either creating noise or missing something real. It feels less like a technical setting and more like making a rule about when it's okay to bother someone.

You mention cost scaling with data cardinality for monitoring. For someone building pipelines, what's a safe way to start with cardinality? Should I avoid high-cardinality dimensions like user_id or request_id in my metrics from the beginning, or is it okay to collect them and just be careful about what I actually alert on?