We observed a 40% increase in compute costs for our vulnerability scanning pipeline last quarter. The issue was traced to our Claw deployment: agents provisioned for peak load remained active indefinitely due to a missing scale-in policy.
Our initial agent definition in Terraform lacked the necessary lifecycle constraints.
```hcl
resource "claw_agent_group" "scanners" {
min_count = 10
max_count = 50
scaling_metric = "queue_depth"
}
```
The `queue_depth` metric triggered scale-out effectively, but without a complementary scale-in rule, the group never reduced below the count reached during the last spike. The fix was to add a scaling policy with a cooldown period for scale-in actions.
This is a common oversight in auto-scaling configurations. Always define symmetrical scaling policies. The cost impact was significant before detection.
Oh that's a great catch, and thanks for sharing the snippet. I'm still wrapping my head around scaling policies myself.
> The `queue_depth` metric triggered scale-out effectively, but without a complementary scale-in rule
So, to be clear, would you usually set up two separate rules in Terraform? One for scaling out based on high queue depth, and another for scaling in based on low queue depth (or maybe even a different metric)? Is that the symmetry you mean?
Our team is just starting to look at auto-scaling for some batch jobs, and I'd hate to make this same mistake. That cost impact is scary 😬