I have been conducting a detailed analysis of our CI/CD pipeline costs over the last quarter, and while our unit economics appear strong, our aggregate expenditure has become problematic. Our team operates a self-hosted runner fleet on Kubernetes, and our internal metrics show a cost-per-build-minute that is approximately 40% lower than the equivalent compute cost from major cloud-managed CI providers. However, our total monthly invoice from our cloud provider has increased by 200% over the last six months, which is unsustainable.
The core issue appears to be a misalignment between our optimization focus (cost-per-unit) and the actual cost drivers (total compute consumption). I will outline our current configuration and the data I've collected.
**Current Infrastructure & Configuration:**
- **Platform:** AWS EKS (Kubernetes)
- **Runner Type:** Self-hosted GitHub Actions runners deployed via `actions-runner-controller`.
- **Instance Profile:** Mixed node group of `c6i.large` (compute-optimized) and `m6i.xlarge` (general-purpose) Spot Instances.
- **Scaling:** Horizontal Pod Autoscaler scales runner replicas based on queue depth from GitHub.
Our pipeline configuration for a standard build job is as follows:
```yaml
# Example job definition
jobs:
build-and-test:
runs-on: build-medium
steps:
- uses: actions/checkout@v4
- name: Build Docker Image
run: docker build --target builder -t app:builder .
- name: Run Test Suite
run: docker run --rm app:builder npm test
- name: Build Production Image
run: docker build -t app:latest .
```
**Cost Analysis Data (Last Month):**
- Total build minutes: 86,400 minutes (1,440 hours)
- Average cost per build minute: $0.0035
- Total compute cost (EC2 Spot): $302.40
- **Total monthly cloud bill (CI/CD related):** $1,850
- Breakdown of additional costs:
- EKS Management: $73/month
- Container Registry ECR (data transfer & storage): ~$120/month
- Logging & Monitoring (CloudWatch): ~$280/month
- **Largest Cost:** Network Data Transfer (INTERNET outbound from EC2): **$976/month**
The data transfer cost is the primary culprit. Our builds pull large dependencies from public repositories and push multi-GB images to our registry, incurring significant egress fees.
**Proposed Investigation & Questions for the Community:**
I am now moving to a more granular, cohort-based analysis of our builds to identify high-consumption workflows. My immediate hypotheses are:
1. **Inefficient layer caching** in Docker builds, causing repeated downloads of base images and dependencies.
2. **Lack of regional affinity** between compute (EC2) and artifact storage (ECR), potentially crossing AZ or region boundaries.
3. **Suboptimal job structuring** causing unnecessary artifact re-uploads/downloads between jobs.
I would like to gather insights on the following:
* What strategies have you implemented to monitor and reduce data egress costs in a self-hosted CI environment?
* Are there specific tooling or pipeline modifications (e.g., `docker build --cache-from`, use of persistent volumes for dependency caches) that yielded a measurable reduction in network throughput?
* Has anyone performed a true total-cost-of-ownership comparison, factoring in data transfer, storage, and operational overhead, against a managed service where egress might be bundled? I am particularly interested in data-driven breakdowns.
My next step is to instrument our runners to tag all network traffic by build ID and destination, allowing for a funnel analysis to pinpoint the exact stage where the most data is transferred. I will share my findings in this thread.
— Amanda
Data > opinions
Ah, the classic trap of optimizing the unit cost while ignoring the consumption. You've built a cheaper engine, but now everyone's taking joyrides because fuel seems cheap.
The real culprit is probably that HPA scaling based solely on queue depth. It's reactive, not predictive, and doesn't consider time-of-day or idle scaling. Your spot instance pool is just sitting there, warm and billing you, whenever a single job sits in the queue for a few seconds. That aggregate idle time is what's killing you.
You need to look at your average runner utilization, not your cost-per-build-minute. I'd bet it's abysmal. Try implementing a scale-in delay that's aggressive, and look at scaling to zero during off-hours with a scheduled deployment. The 'actions-runner-controller' can tear down idle pods, but if the nodes are still provisioned, you're still paying. You optimized the container, but forgot the metal it runs on.
Exactly! You nailed it with the idle scaling. The HPA queue depth trigger is like keeping the factory lights on 24/7 just because one machine *might* get a job. Our team learned this the hard way.
We added a simple cron to our node group that scales the min size to zero every night at 8 PM and back up at 6 AM. The developers barely noticed the 5-minute warm-up lag for late-night commits, but it chopped 30% off our monthly bill instantly. That idle metal cost is a silent killer.
Automate all the things.
That's exactly the problem with over-automation without guardrails. You build the system to scale out efficiently, but you forget to teach it when to scale back in.
Everyone focuses on the spin-up time. Nobody thinks about the cool-down. A minute of idle time per build adds up to hundreds of hours of wasted compute at scale.
Your point about the controller tearing down pods but not nodes is key. It creates a false sense of optimization. You need to measure cost from the cloud invoice down, not from the build metric up.
Your CRM is lying to you.