Built a simple Slack bot that scrapes our CI provider's API and posts a weekly spend summary to a channel. It's been eye-opening.
Our current burn for last week:
* **Total Compute Minutes:** 42,187
* **Estimated Cost:** $847.32
* **Top 5 Projects by Spend:**
1. frontend-e2e-tests (23%)
2. mobile-nightly-builds (18%)
3. api-load-testing (15%)
4. docs-deployment (12%)
5. internal-tooling (9%)
The biggest cost drivers are long-running E2E tests and under-optimized Docker layer caching. Considering moving the high-minute, low-sensitivity jobs to self-hosted runners on spot instances.
Bot logic is straightforward. Runs as a scheduled K8s CronJob.
```python
# Core cost aggregation (simplified)
def get_weekly_spend(api_client):
jobs = api_client.get_jobs(timeframe="7d")
cost_by_project = defaultdict(float)
total_minutes = 0
for job in jobs:
minutes = job.duration_minutes
rate = job.machine_type_rate_per_min
cost = minutes * rate
cost_by_project[job.project_name] += cost
total_minutes += minutes
return total_minutes, cost_by_project
```
Has anyone else done this and seen a significant shift in team behavior or optimization priorities after making spend visible? Specifically interested in data from teams running 50k+ compute minutes per week.
-dk
Trust but verify, then don't trust.
Love that you've built visibility directly into your team's workflow. That's the first and most critical step. In my experience, when teams actually see the spend number pop up in Slack every week, it shifts from an abstract cloud bill to a tangible, accountable metric.
You've already identified the high-level culprits. The framework I often use next is a simple cost-priority matrix: plot your jobs by cost (y-axis) and sensitivity to execution time/performance (x-axis). The high-cost, low-sensitivity quadrant - your long e2e tests, docs deployment - is always the prime candidate for migration to cheaper, self-hosted or spot infrastructure. It's a straightforward win.
One caveat from doing this with clients: watch for the "rebound effect." When you make a cost visible and then fix the biggest offenders, you sometimes see other costs creep up because teams feel the pressure is off. Keeping that weekly bot running is key to maintain the feedback loop.
null
You're spot on about the rebound effect - we saw that immediately when we started flagging memory-heavy jobs. Teams would fix the obvious ones, then someone would push a new data pipeline that ran fine locally but blew up memory in CI for hours. The bot caught it the next week, but the damage was done.
That cost-priority matrix is so useful. We actually added a third axis for "friction" - how much engineering time it would take to migrate or optimize. It helped us prioritize which quick wins to tackle first while planning the bigger lifts like moving to spot instances.
The weekly cadence is perfect too. Daily would be noise, monthly is too slow to catch regressions. It's become this little ritual where the team lead will @ people on the top spenders list. Slightly embarrassing but effective.
edge cases matter