Hey everyone, I've been wrestling with the same problem for years: cloud costs are mostly predictable, but every now and then something slips through the cracks—a forgotten dev environment, a misconfigured auto-scaling policy, a runaway batch job—and suddenly you get that heart-stopping bill at the end of the month. Fancy FinOps dashboards are great for analysis, but I wanted something more visceral and immediate for *anomalies*.
So, our small platform team built a simple, purpose-driven Slack bot. We call it "Screamer." Its only job is to monitor our cloud spend for the day and, if anything is significantly off the rails compared to our forecast or historical patterns, it *yells* at us in a dedicated channel. The goal was to create a sense of urgency that a quiet, easy-to-ignore Grafana graph just doesn't achieve.
The core logic is pretty straightforward. We run it as a serverless function (Google Cloud Run in our case, but anything works) on a cron schedule every six hours. It fetches the current daily spend from the cloud billing API, compares it against a simple baseline (we use a rolling average of the last 7 same-weekday spends, adjusted for known growth), and if the variance exceeds our threshold (we started with 25%), it posts a very loud message.
Here's the basic flow we implemented:
* **Data Fetching:** Pulls the unblended cost for the current day-to-date via the GCP Cloud Billing API (or Cost Explorer API for AWS). We filter to our top-level projects.
* **Baseline Calculation:** Maintains a small data store (we use Firestore) of historical daily costs. The baseline is computed dynamically.
* **Anomaly Detection:** `(Current Spend - Baseline) / Baseline > Threshold`. That's it. No ML, just simple arithmetic. We found that for truly unexpected spikes, this is surprisingly effective.
* **The "Yell":** If triggered, the bot posts to Slack using a webhook with a high-priority red color, large font emojis (🚨 📈 💸), and clear text like "COST ANOMALY DETECTED: Current spend is 47% above expected baseline!" It includes quick links to our billing dashboard and a shortlist of services that saw the largest cost increase in the period.
The beauty is in its simplicity and psychological impact. A chart you have to go look at is passive; a loud, angry message in a busy team channel forces acknowledgment and action. We've caught several issues within hours instead of weeks, like a sudden surge in egress costs from a new CDN configuration and a development cluster that failed to shut down over a holiday weekend.
I'm curious how others handle real-time cost alerts. Has anyone else built something similar? I'm particularly interested if you've layered in more sophisticated detection for gradual creep versus sudden spikes, or if you've integrated it with automated remediation workflows (like killing non-prod resources on weekends).
~jason
~jason