Generic "bill is 20% over budget" alerts are useless. They tell you something's wrong after your CFO is already emailing you. You need granular, team-level usage alerts that trigger before the invoice closes.
This playbook configures CloudWatch alarms on cost-usage metrics, segmented by your most common dimensions. Goal: catch a team's runaway deployment or a forgotten dev environment within hours, not weeks.
**Core Principles:**
* Alert on **usage**, not just cost. A spike in `AWS/EC2 RunningInstances` for a specific tag is more actionable than a vague cost increase.
* Segment by **application tag** and **environment (prod, dev)**. Team-level accountability.
* Set thresholds based on **historical variance**, not static numbers.
**Example AWS Setup (CloudWatch + Cost Explorer Tags):**
Assume you tag resources with `Application=checkout-service` and `Environment=dev`.
1. Create an alarm on the `AWS/EC2` namespace, metric `CPUUtilization`, but filter dimensions:
```json
{
"Dimensions": [
{ "Name": "Application", "Value": "checkout-service" },
{ "Name": "Environment", "Value": "dev" }
]
}
```
2. Set threshold to 75% for 2 consecutive 5-minute periods. This catches sustained load from a bug or load test.
3. Critical: Create a second alarm on `AWS/Billing` metric `EstimatedCharges` with the same tag dimensions. Use a static threshold (e.g., $50/day for a dev env) as a hard financial stop.
**Rollout Steps:**
1. Mandate tagging compliance first. No tags, no alerts. Resources without tags go into a "unallocated" alarm that screams at the platform team.
2. Work with one team to define their normal usage patterns. Set initial thresholds.
3. Automate alarm creation via IaC (Terraform/CloudFormation) per application/environment combo.
4. Route alerts directly to the team's Slack/Teams channel, not a central SRE mailbox. Ownership is key.
**Early-Warning Metrics to Track:**
* Number of alarms triggered per team per week (identifies noisy/config issues).
* Mean time to acknowledge/respond to a cost-alert.
* Reduction in "unallocated" spend.
**Handling Resisters:**
* "We don't have time to tag everything." → Show them the bill for untagged resources. That's their budget being burned.
* "These alerts are noisy." → Good. Adjust thresholds. Silence is expensive.
* "This isn't our job." → Agree. It's *everyone's* job now. Link their dev environment cost directly to their feature development budget.
cost per transaction is the only metric
The core principle of alerting on usage metrics is correct, but your example using `CPUUtilization` is a poor choice for a cost-related alert. CPU can spike for legitimate reasons and doesn't directly correlate to a runaway bill.
You're better off using the actual `AWS/Billing` namespace with the `EstimatedCharges` metric, applying a filter for `Currency=USD`, and then adding your tag-based dimensions. The alarm threshold should be a static dollar amount based on a daily or weekly team budget, not a percentage of utilization. The "variance" approach you mentioned often fails silently when baseline usage creeps up.
Also, filtering by tags in CloudWatch only works if the metric *includes* those dimensions. Most AWS service metrics don't automatically inherit your custom tags. Billing metrics do, but only if you've enabled Cost Allocation Tags in AWS Cost Explorer and waited for them to propagate, which takes 24-48 hours. That's a critical pre-requisite everyone misses.
Show me the benchmarks.