Hi everyone, I'm still pretty new to on-call and managing alerts in AWS. I'm nervous about breaking things, so I just run Terraform for our basic setup.
We keep getting a "High CPU" alert from CloudWatch that goes off during our nightly batch job. It's a real alert, but it happens every night and isn't an incident. I don't want to delete the alert rule because we need it for actual problems.
How do you make an alert "smarter" so it ignores known good patterns? I've only set up simple alarms like this:
```hcl
resource "aws_cloudwatch_metric_alarm" "high_cpu" {
alarm_name = "web-server-high-cpu"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = "2"
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = "300"
statistic = "Average"
threshold = "80"
}
```
Is there a way to add a time condition? Or do I need a totally different approach? I'm worried about missing a real incident because of alert fatigue.
CloudWatch itself doesn't have a cheap "ignore this time window" feature. You're paying for basic monitoring, not smart alerts.
Look at your alarm's *statistic* and *period*. A 5-minute average (period=300) spiking during a batch job is expected. Try a shorter period and the "Maximum" statistic to catch sustained spikes, not scheduled work. That's free.
Otherwise, you're into paid solutions: Lambda to disable/enable the alarm on a schedule (more compute cost), or a third-party alerting tool (more subscription cost). The TCO adds up fast for one noisy alert.
always ask for a multi-year discount
Great point about adjusting the statistic and period, that's the first lever to pull for sure. Free fixes are the best fixes.
I'd add that you can also combine metrics with CloudWatch's anomaly detection for the alarm. It learns a baseline over time and might flag sustained high CPU outside that learned pattern, which could filter out your regular batch job. It's still CloudWatch, so not "smart" in a rules sense, but it can adapt.
That Lambda toggle approach gets messy fast, especially if you have to start managing it for multiple alerts.
spreadsheet ninja
Anomaly detection is a decent suggestion, but let's be real about its limitations for scheduled jobs. It works on statistical bands, not a calendar. Your nightly batch job creates a predictable high-CPU pattern every 24 hours. Over time, that becomes part of the "normal" baseline, and the alarm might just stop firing altogether - for everything. Now you've traded a false positive for a false negative, which is worse.
The real issue is using a generic CPU alarm for a system with a known, intensive workload schedule. If the batch job is legitimate, the metric you're alarming on is wrong. You should either alarm on the job's success/failure directly, or create a composite metric that excludes the scheduled window. Building a Rube Goldberg machine of anomaly detection to mask a flawed alert definition is how you end up with a pager that never goes off when the real fire starts.
Speed up your build
Totally understand the alert fatigue concern, it's a real morale drain for on-call folks. You're right to want to keep the alert active for real problems.
Your idea about a time condition is a good one, and it's frustrating that CloudWatch alarms don't have that built-in. The code workaround people often suggest is a Lambda on a schedule to disable and re-enable the alarm, but for someone new, that's just swapping one piece of complexity (noisy alerts) for another (managing more infrastructure).
Instead of trying to make this one alert ignore a time window, consider if the alert definition itself is correct for what you actually care about. You said it's a *web server* alarm. Is high CPU during a batch job a problem for the web server's core function? If the batch job is a separate, expected process, maybe the alert should focus on user-facing hours. Sometimes the simplest fix is to split the concern: have a stricter, more sensitive "business hours" CPU alarm for your web traffic, and a separate, more lenient one for off-hours that just catches truly catastrophic spikes. That way you're not disabling coverage, you're just applying the right context.
Let's keep it real.
You've put your finger on the core tension of alerting - balancing signal and noise. It's good you're thinking about it early, because alert fatigue can really burn out a team.
The answer is often, as a few folks hinted, that the alert itself needs refinement. That alarm says "web server high cpu," but your system has a nightly, scheduled high-cpu event. So is high cpu itself the problem, or is it high cpu *outside of expected work*? You might need to split the concept. One alarm for the batch job's own health (did it finish? did it fail?), and a separate, much tighter alarm for the web-serving function during user-facing hours.
Adding a time condition directly in CloudWatch isn't a native feature, unfortunately. The workarounds add complexity, as you've seen. Before going down that path, ask if you can define the problem more precisely. What does a "real incident" look like for that server? Is it sustained high CPU at 2 PM on a Tuesday? That might mean adjusting the threshold, period, or statistic to catch that pattern specifically, while letting the nightly job slide.
Let's keep it real.
First off, welcome to the on-call world. That worry about missing a real incident because of alert fatigue is 100% valid, and it's smart you're tackling it now.
Your specific question about a time condition in CloudWatch gets to the heart of it. There isn't a built-in way, unfortunately. The workarounds, like a scheduled Lambda, add a new piece of infra to manage, which can be its own headache, especially if you're new.
But let's look at your actual alarm code for a second. You're using a 5-minute average over 2 periods. That means it needs to be above 80% for 10 minutes straight to fire. Your batch job probably does that. Could you make the threshold higher *just* for that period? Not really. So the real question might be: is the health of your web serving function truly measured by raw CPU, or by something like request latency or error rate during user hours? That might be a better signal that ignores the batch job entirely.
Keep it constructive.
You're alerting on the wrong thing. That alarm triggers for *any* high CPU, but your system has a legitimate high CPU period. So either the metric is flawed or your definition of "problem" is.
An alarm that cries wolf nightly trains everyone to ignore it. You'll miss the real spike at 3pm because you're conditioned to mute the 2am batch job. That's how incidents get missed.
Forget time conditions. Define what a real problem is first. Is it high CPU when users are active? Then your alarm needs to reflect user traffic, not just a raw percentage.
Trust but verify.
You're absolutely right about the danger of conditioning. An alert that reliably fires for a non-problem becomes background noise, and that's how you miss a real outage during business hours.
Your point about needing to define the problem first is crucial, but I find teams often get stuck there. It's easy to say "alert on what matters," but operationalizing that is hard. One concrete step is to shift from a static threshold to a metric that captures the *impact*. For a web server, that could be the ratio of CPU utilization to request count. If CPU is high *while* user traffic is near zero (like during a batch job), the ratio is huge and uninteresting. If CPU is high while traffic is also high, that ratio is meaningful and might indicate a real scaling issue. That's a composite metric you can build in CloudWatch, and it moves you closer to alerting on user experience, not just resource consumption.
That ratio idea is a good one, and it's something I've implemented for a similar scenario. It moves the needle from "is the server busy?" to "is the server struggling?"
The catch is you need a reliable denominator. For the request count metric, you have to be careful about log-based metrics or load balancer metrics that might have their own aggregation delays. If the request count metric lags by a couple minutes, your ratio gets weird right when you might actually need it. I ended up using a CloudWatch metric math expression that smoothed it out over a longer period to avoid noise, but it's definitely not a set-it-and-forget-it kind of alarm.
Great question, and you've nailed the core problem right from the start. Your code snippet shows the classic, static threshold alarm that most of us begin with.
The time condition idea is logical, but as others have pointed out, it's not native to CloudWatch. Patching it with a Lambda scheduler feels like overkill for one alert, and it doesn't really solve the definitional problem.
Since you're already in Terraform, a more maintainable approach is to refactor the alert logic itself. Instead of trying to make one alarm ignore a time window, consider creating two separate alarms with different thresholds or conditions. One could be for your "business hours" web-serving concern, and another, maybe with a higher threshold, for the overnight batch window. This keeps your Terraform declarative and avoids managing stateful toggles.
That said, defining those separate conditions forces you to answer: what *is* a problem during the batch job? If the job should be CPU-intensive, maybe you should alert on the job's completion or error rate instead. It's a shift in thinking from "server is busy" to "expected work is failing."
Pipeline is king.
The TCO argument is valid but misses the operational reality. The Lambda workaround isn't just about subscription cost, it's about adding a new failure mode. Now your alerting depends on another piece of infrastructure executing correctly on a schedule.
Your point about adjusting statistic and period is the right first step. But it's a band-aid. It might filter the nightly batch job, but then you miss a real, sustained spike that happens to start at 2:05 AM instead of 2:00 AM. You've just traded a known false positive for a potential false negative.
Free doesn't mean correct. Sometimes the paid solution or a third-party tool is the actual answer because the core monitoring service fundamentally lacks the logic you need.
Your CRM is lying to you.
You're right that splitting into context-specific alarms is often the simplest path forward. The trick is making sure the split is meaningful for your on-call team. Creating a "business hours" alarm and an "off-hours" alarm sounds clean, but you now have two alerts to manage and tune. It forces you to be explicit about what constitutes a problem in each context, which is good, but you also need a clear runbook that tells responders which alert they're looking at immediately.
Review first, buy later.
The ratio approach is conceptually sound, but its implementation hinges entirely on the reliability and timeliness of your denominator metric. I've seen this fail in practice when the request count metric is sourced from application logs with a 3-5 minute ingestion lag, while the CPU metric is near real-time. During a sudden traffic surge, the ratio appears artificially favorable for those lag minutes, suppressing a legitimate alert when you need it most.
You can mitigate this by using a metric source with lower latency, like load balancer request counts, and by structuring your CloudWatch metric math to use a longer period for the ratio calculation to smooth out brief timing mismatches. But this introduces a new trade-off: a longer period means a slower-to-fire alert. You're trading one type of delay for another.
Ultimately, a ratio is a better signal, but it becomes another moving part in your monitoring stack that requires its own validation and calibration.
—BJ
Right, alert fatigue from that nightly spike is the real issue, not the alert itself. Time conditions aren't native in CloudWatch, and adding a Lambda scheduler feels like overcomplicating it for one rule.
Instead of trying to make one alarm ignore a time window, can you split the logic? Define what "high CPU during a batch job" actually means for you. Maybe it's okay if it hits 95% for 10 minutes overnight, but a 10-minute spike to 85% during the day is a problem. You could set up two alarms in Terraform with different thresholds: one stricter for daytime, one more tolerant for the batch window. It keeps everything in your infrastructure code.
That way, you're not just silencing noise, you're being explicit about what constitutes a problem in each context. It's a bit more upfront work, but it's much clearer for whoever's on-call.
Docs save time