We're at ~120 microservices, each with its own pipeline. The monthly bill was climbing and no one could explain why. Built a model to trace cost to actual engineering activity.
Key findings:
* 60% of our compute minutes were spent on `npm install`/`go mod download`. Every PR re-downloads the world.
* 25% of cost came from 5 legacy services with flaky, 45-minute builds.
* We were paying for "idle parallelism" – agents sitting around waiting for low-concurrency stages.
Here's the core of the tracking script (parses GitLab CI logs + cloud billing export):
```python
# Simplified cost attribution per service
for job in pipeline_jobs:
compute_minutes = job.duration * job.concurrency_weight
cost = compute_minutes * minute_rate
cost_breakdown[job.project].append({
'trigger': job.trigger_type, # MR, schedule, tag
'stage': job.stage,
'cost': cost
})
```
Breakdown of a typical $12k/month bill:
* **Build/Test Stage:** $7,200 (mostly package manager overhead)
* **Integration Tests:** $3,000 (needs 4x parallelism, expensive instances)
* **Container Build/Push:** $1,200
* **Infra Deploy (Terraform):** $600
The model showed self-hosting on spot instances would save ~35% but add 1.5 FTE in ops. Not worth it for us yet.
Next step is forcing shared package caches and killing those legacy builds.
slow pipelines make me cranky
Love seeing this breakdown! That `npm install`/`go mod download` line is brutal, but so common. We saw similar numbers and got a huge win by moving to a shared, persistent dependency cache layer across projects. It's not perfect for every language, but the ROI was insane.
Your idle parallelism point is spot on. We ended up creating a "priority lane" config for our CI controller. High-concurrency stages (like integration tests) get dedicated, pricier runners, but we automatically scale them down after hours. The flaky builds got "budget caps" - after X minutes of compute, they auto-cancel and ping the owning team.
What's your plan for acting on the data? Are you thinking automated alerts when a service's pipeline cost spikes, or more of a weekly report to engineering leads?
Data doesn't lie, but dashboards sometimes do.
That 60% figure for dependency installation is staggering, but I'm not surprised. Your breakdown mirrors what we found, though our container build stage was proportionally higher due to bloated base images.
I'm curious about your `minute_rate` variable. Are you using a blended average, or are you dynamically assigning the specific rate from the cloud billing export based on the actual instance type and region used for each job? We found our initial model was off by almost 18% because we were using a platform-wide average, which smoothed over the cost spikes from those expensive integration test instances you mentioned.
What's your strategy for handling the data lag? Our billing data has a 24-hour delay, so our real-time alerts are based on projected cost from compute minutes, then reconciled later. It creates some noise.
Spreadsheets or it didn't happen.