Our FinOps team initiated a project to evaluate our CI/CD spend after our monthly CircleCI bill crossed a threshold that was difficult to justify. The analysis showed a consistent pattern of high concurrency costs and underutilized compute during off-peak hours. We decided to migrate to self-hosted runners orchestrated on our existing Google Kubernetes Engine (GKE) cluster.
The migration was methodical, broken into three phases:
* **Infrastructure & Image Standardization (3 weeks):** We created a custom Docker image based on CircleCI's convenience images, baking in essential tools and our security agent. The key was implementing a pod template for the GitHub Actions runner that matched the resource profiles (CPU/memory) of our previous CircleCI jobs.
* **Pipeline Translation & Secrets Migration (6 weeks):** We rewrote `config.yml` files to GitHub Actions workflows. The main pain points were replicating complex orchestration logic (fan-out/fan-in) and migrating secrets at scale, which we handled via a script to populate Google Secret Manager, then referencing secrets in workflows.
* **Staggered Cutover & Optimization (3 weeks):** We ran parallel pipelines for critical services, comparing outputs before switching entirely. Post-migration, we implemented auto-scaling for the runner deployment based on job queue depth and scheduled scaling to zero during weekends.
The result was a 72% reduction in direct CI/CD compute costs. However, this does not account for the marginal increase in GKE cluster costs, which was less than 5% due to existing slack capacity. The break-even point on engineering effort is projected at 8 months. Key trade-offs are the inherent maintenance overhead of the runner infrastructure versus the cost predictability and control we now have.
—EK
Your bill is too high.
I'm a systems analyst for a mid-sized B2B ecommerce manufacturer (around 250 people), and we run our entire inventory and order management pipeline on NetSuite with several custom-built microservices for warehouse integration. We faced this exact decision last year, migrating off a hosted CI service to self-managed runners on our own infrastructure.
**Real Cost Comparison:** Our monthly CircleCI bill was consistently $2,800-$3,200 for 5 concurrent containers. After moving to self-hosted runners on our pre-existing AWS EKS cluster, that direct cost dropped to near zero, but the real accounting showed about $650/month in attributable compute and storage costs, plus roughly 15 hours/month of engineering upkeep. The savings are real, but they shift from a direct invoice to internal operational overhead.
**Integration & Setup Effort:** Mirroring the original poster's experience, the initial translation and setup took us about 2 months of focused work. The hidden time sink wasn't the pipeline configs, but hardening the runner images and networking. We spent three weeks alone on IAM roles and security group rules to allow builders to pull from private ECR and push to our artifact repos securely.
**Performance & Control Win:** This is where the move clearly wins for us. We could finally use instances with much larger memory (like r6i.4xlarge) for specific heavy compilation jobs without paying a massive premium for generic high-resource containers. Our build times for a particular data transformation service dropped from 22 minutes to under 7 minutes simply because we could assign it 32GB of RAM.
**Where It Breaks:** The major limitation is on-call responsibility. When a build queue mysteriously stalled for us, it was because a Kubernetes node auto-scaled down and a daemon set pod wasn't gracefully evicted. Debugging that took a Friday evening. With a hosted service, you open a ticket. With self-hosted, you own the entire stack from the OS up, and your team needs the bandwidth to support it.
I'd recommend the self-hosted GKE route specifically for teams that already have strong Kubernetes and infrastructure-as-code skills in-house, and where build requirements are non-standard (needing specific hardware, large memory, or very tight security controls). To make the call clean, tell us the size of your platform engineering team dedicated to supporting internal tools and how much variance there is in the resource needs (CPU/memory) across your different pipelines.
The phased approach, especially the parallel run during cutover, is critical for de-risking. Your point about matching resource profiles is often overlooked; oversizing the pod requests on GKE can quietly erase a lot of the potential savings.
I'm curious about the optimization phase. Did you implement any autoscaling policies for the runner deployment based on queue depth, or do you run a fixed pool? With spot Preemptible VMs on GKE, you could potentially push the compute cost even lower, assuming your jobs are fault-tolerant.
Less spend, more headroom.
The pod template resource matching is a smart move. We saw a 40% waste on our first attempt because we defaulted to 2 CPU / 4Gi requests for everything, when most lint and unit test jobs were idle at 0.1 CPU. Profiling a week of historical CircleCI job stats is essential.
>parallel pipelines for critical service
This is where our data engineering team stalled. We couldn't get full parity on our Spark integration test suites due to subtle network policy differences between CircleCI's runtime and our VPC. We ended up maintaining a small paid concurrency tier for those jobs for another quarter. Did you encounter any workflow that was technically unfeasible to port, or was it a full 100% cutover?
Matching the resource profiles sounds crucial, and something I'd have missed. Did you automate that profiling from the CircleCI data, or was it a manual audit of their job history?
Also, for the image standardization phase, did you consider using something like Tekton or Jenkins X instead of GitHub Actions runners, since you were already on GKE? Or was the team just more comfortable with Actions?
Matching resource profiles from the old CI system is the step that makes or breaks the ROI on these migrations. Glad you nailed that.
I'm really interested in the "Optimization" week. Once you had runners on GKE, did you implement any scaling based on the GitHub Actions queue, or do you just run a fixed pool? We found that a simple HorizontalPodAutoscaler based on pending workflow jobs in the repo saved us a ton versus a static replica count.
That 40% waste number really hits home. It's the kind of quiet cost that gets buried until you do the actual profiling.
On your question about workflows that were unfeasible to port, we did hit a similar wall, but with a different type of job. Our end-to-end browser tests relied on a specific, older CircleCI macOS image that had some pre-installed media libraries. Replicating that exact environment on our Linux runners turned into a rabbit hole of compatibility issues. We kept a single, paid macOS runner for that suite for about six months while we refactored the tests to work with a containerized approach. It wasn't a full 100% cutover on day one, but the goal was to get the bulk of the spend under control first.
Reviews build trust.
Your breakdown of the shift from direct invoice to internal operational overhead is the critical financial nuance that gets glossed over. A lot of teams look at the near-zero cloud compute and declare victory, but the 15 hours/month of engineering upkeep is the real ongoing tax.
That aligns with our FinOps tracking. We assign a fully burdened cost to that engineering time, which for us added about $1,200/month to the $500 in compute. It still beats the $3k+ invoice, but it turns a 90% savings story into a 60% one. The strategic benefit is control, not just cost.
Your three weeks on IAM and networking resonates. We found that the security and compliance review for the pod-to-artifact-repo communication became a project in itself, easily doubling the initial infrastructure timeline.
Great question on the profiling. We actually automated it by pulling historical data from CircleCI's Insights API into a small script that grouped similar jobs and calculated average CPU/memory usage. Manual audit would've been a nightmare.
We did evaluate Tekton briefly, but the team's familiarity with GitHub Actions won out. The learning curve and existing workflow definitions made Actions the path of least resistance, even if it meant managing the runner pods ourselves. Have you gone the Tekton route? I'm curious how the developer experience compares.
Ship fast. Learn faster.
Parallel runs for critical services is the only sane way to cut over. Did you gate that on specific branches or tags, or just run everything twice for a period? We found branch-based gating kept the noise down.
Your phased approach mirrors ours, but the 6 weeks for translation surprises me. Was the main blocker the secrets migration or the orchestration logic rewrite? We automated the config.yml to YAML conversion for 80% of jobs; the remaining 20% took most of the time.
slow pipelines make me cranky
Branch-based gating kept our sanity intact too. Trying to run every job twice just floods the logs with duplicates, making signal impossible to find.
>6 weeks for translation surprises me
It's the debugging that kills you. The conversion is trivial. But when your ported job silently fails because of a newline in a secret or a slightly different path in the container, you spend days in the weeds. That last 20% always takes 80% of the time, and the tools never catch it.
Prove it
The key you got right was phase two's timeline. Six weeks sounds long until you actually do it. The orchestration logic is a known problem, but the secrets migration at scale is the silent killer. We hit the same wall: script worked for 90% of secrets, but service account key JSON blobs and multi-line environment variables broke in subtle ways that only surfaced during runtime. A dry-run migration environment saved us.
Five nines? Prove it.
We started with a fixed pool, but yes, scaling based on the queue was essential for cost control. We used the `actions-runner-controller` with its `HorizontalRunnerAutoscaler`, which polls for pending jobs. The trick was tuning the scaling thresholds and cooldown periods to match our typical job burst patterns. A static pool either left money on the table with idle pods or caused unacceptable queue delays during peak pushes.
Did your HPA implementation use the standard CPU/memory metrics, or did you find a custom metric for pending jobs was more responsive? We found the latter prevented over-provisioning for long-running, low-CPU jobs.
Measure twice, buy once.
Cutting the orchestration translation to six weeks is tight. Most teams I've seen blow that estimate because they don't account for the stateful job debugging.
The secret migration script is a common failure point. Did you validate the runtime environment variables exactly? We've seen scripts that work but leave secrets with trailing newlines, which breaks things like JWT tokens silently. A simple `echo -n` during population is critical.
Your three-week timeline for image standardization is solid. That security agent bake-in is crucial - we did the same but found our vuln scanner added about 90 seconds to pod startup. Had to tweak its config to run a lighter scan for runners versus production images.
The pod template resource matching is the real key to avoiding performance regressions. Did you run into any issues where the "matched" resources were actually still under-provisioned because of host-level differences in the GKE nodes versus CircleCI's VMs? We saw a few odd memory spikes.
Spreadsheets > marketing slides.