Hey folks, been deep in our AWS accounts lately and realized we had a quiet budget leak. Our monthly bill had been creeping up, and after some digging, we found nearly $15k a month was just for unattached EBS volumes! 😱
We run a lot of ephemeral workloads and auto-scaling groups, and it seems our cleanup processes weren't quite keeping up. Volumes were being orphaned when instances were terminated manually or via scripts that didn't handle cleanup. Here's the basic query we ran in AWS Cost Explorer to get suspicious:
* Filtered by service `AmazonEC2` and linked it to the `EBS:VolumeUsage` line item.
* Added a tag filter for `aws:ec2:instance-id` (or lack thereof) – but honestly, the real magic was pairing this with a custom script.
The script uses the AWS CLI to list all volumes and check their attachment state. We output it to CSV for the finance team.
```bash
#!/bin/bash
aws ec2 describe-volumes --query 'Volumes[?State==`available`].[VolumeId,Size,CreateTime]' --output text > orphaned_volumes.txt
```
But the **key step** was adding our own `cost-center` and `project` tags to every volume on creation via Terraform. That way, we could not only identify orphans but also *who* they belonged to for accountability.
```hcl
resource "aws_ebs_volume" "example" {
# ... other config ...
tags = merge(
var.tags,
{
"CreatedBy" = "terraform",
"AutoSnapshot" = "false"
}
)
}
```
Our playbook:
1. **Identify**: Run the script weekly, fed by a Lambda that posts to a Slack channel.
2. **Validate**: Check volume creation time. We set a 7-day grace period for temporary detachments (like database migrations).
3. **Notify**: Use the volume tags to ping the responsible team in Slack.
4. **Automate**: If no response in 48 hours, a follow-up Lambda creates a snapshot (just in case) and deletes the volume. We keep the snapshot for 30 days.
This process, from identification to automated cleanup, took about a month to refine with the teams. The result? Our storage costs dropped by about 18% the next month, and we've maintained it. It's also pushed us to improve our Terraform module defaults to always include deletion policies.
Has anyone else tackled something similar? I'm curious if you used a dedicated FinOps tool like CloudHealth or if you stuck with homegrown scripts. Also, how do you handle the fear of deleting something "important" that's just temporarily detached? We're thinking of integrating with our internal CMDB next.
terraform plan is my therapy.