We hit a surprise $12k bill last month from our analytics warehouse. Root cause: a new dashboard triggered a 40x query volume spike on a massive table.
Vendor's "cost anomaly" alert came 5 days into the month. Too late. So I built a simple monitor that polls the billing API and checks daily spend against a dynamic threshold.
It runs as a scheduled K8s Job. Core logic:
```python
# Simplified threshold calculation
def check_spike(current_daily_spend, baseline_7day_avg, alert_factor=1.5):
threshold = baseline_7day_avg * alert_factor
if current_daily_spend > threshold:
alert_on(current_daily_spend, threshold)
```
Key decisions:
- Baseline is a rolling 7-day average, excluding outliers.
- Alert factor is configurable per pipeline.
- Outputs to Slack and PagerDuty.
It's crude but caught a 3x spike from a misconfigured dbt model this week before it burned budget.
Anyone else built similar? How are you setting thresholds without drowning in false positives?
Nice approach with the rolling 7-day average. We do something similar but also track per-user query cost. It's a second layer that's caught runaway dashboards from overly enthusiastic analysts.
Your alert factor of 1.5 would be too tight for us, we're at 2.5 for our main pipeline. The key for fewer false positives was adding a minimum spend floor - we don't alert unless the spike is also over $200. Saves us from panicking over a $10 day going to $30.
Ever thought about correlating the alert with specific dashboard publish events or Git commits? That's our next step to get ahead of the cause, not just the spend.
Data doesn't lie, but dashboards sometimes do.