Hot take: Their 'anomaly detection' is just spotting your own cost overruns.

Just had a frustrating realization while reviewing our Datadog bill last quarter. We got an "anomaly detection" alert for a sudden spike in log ingestion from our EKS clusters. The alert itself was helpful, but the "root cause analysis" essentially pointed to... our own deployment of a new microservice with overly verbose debug logging. The "anomaly" was a predictable, self-inflicted cost event.

It got me thinking: how much of the fancy ML-driven "anomaly detection" from observability vendors is actually just them building a better dashboard for *their own* pricing model anomalies? They're excellent at spotting when you start generating more data (and thus more billable units), but often lack the context to tell you *why* in a meaningful way beyond "your code changed."

I've started treating these alerts as a primary signal for cost review. When one fires, my first step is now:

1. **Correlate with change management.** Check deployment logs for that timeframe.
2. **Immediate sampling/filtering.** Add a temporary rule to drop noisy debug logs while we fix the source.
```terraform
# Example: Quick Terraform edit to a Datadog log filter
resource "datadog_logs_custom_pipeline" "prod_eks" {
name = "Production EKS Pipeline"
filter {
query = "source:kubernetes"
}
processor {
status_remapper {
name = "Suppress debug logs from new service"
sources = ["status"]
sources = ["DEBUG", "Info"]
}
}
}
```
3. **Instrument the fix.** Update the application's log level config and redeploy.

Has anyone else adopted this mindset? What are your go-to tactics for turning an "anomaly" alert into an immediate cost-saving action, rather than just an acknowledgement? I'm particularly interested in strategies for OpenTelemetry-based setups where you have more control at the collector level.

-- Amy