Alright, let's talk about a real-world headache we just navigated. We're a 50-engineer shop, and our infrastructure is a hybrid AWS/GCP mess—some legacy services on AWS, all new data pipelines on GCP, and everyone's pushing for faster deploys. Our old Jenkins setup was buckling, and the cost of managing it was spiraling.
We evaluated three paths for CI/CD:
1. **Self-hosted (Jenkins/Argo CD on Kubernetes):** High control, but the operational load was killing us. Our monthly burn for the dedicated node pool, backups, and plugin maintenance was hitting ~$2.5k, not counting 15-20 engineer-hours a week spent on upkeep.
2. **Fully Managed (CircleCI/GitHub Actions):** Easy start, but cross-cloud workflows felt clunky. The bill shocked us at scale. One team's matrix build on GCP VMs, calling an AWS deployment, ballooned their monthly spend to $3k alone. The per-minute pricing adds up fast with 50 devs.
3. **Hybrid Orchestrator (Our choice):** We landed on using **GitLab CI as the orchestrator**, with **self-hosted runners on auto-scaling node pools in both AWS and GCP**. The key was using the runner tags to direct workloads.
Here's the playbook that worked:
* **Control Plane:** GitLab.com (SaaS). One pane of glass for all pipelines.
* **Execution Layer:** GitLab Runner installed on a GKE *and* an EKS cluster. Each cluster auto-scales from 0.
* **The Magic Sauce:** Tag your runners by cloud and workload. Then, pin jobs using `tags:`.
Example `.gitlab-ci.yml` snippet:
```yaml
deploy_to_aws:
stage: deploy
script: ./deploy-aws.sh
tags:
- aws
- highmem
data_pipeline_gcp:
stage: test
script: ./run-bigquery-tests.sh
tags:
- gcp
- data
```
**Cost Breakdown:**
Our monthly invoice is now predictable.
- GitLab SaaS tier: ~$40/user/month ($2k total)
- Compute (burstable instances, preemptible where possible): ~$1.2k (GCP) + ~$0.9k (AWS)
- **Total: ~$4.1k**
We saved ~30% from the fully-managed quote and reclaimed countless hours. The horror story was the two weeks we spent debugging network peering, but that's a tale for another post. The key was accepting that a single, unified runner fleet wasn't feasible for our hybrid reality. Let your jobs run where the resources live.
test the migration twice
Your runner tagging strategy is a solid foundation, but it glosses over the permission and secret management swamp that usually trips this up. How are you handling IAM credential injection for those cross-cloud deployments without turning your CI configuration into a security audit nightmare? We tried a similar setup and found the devil was in passing AWS temporary credentials to a job destined for a GCP runner, or vice versa, without exposing them in logs or to adjacent jobs. The GitLab CI external secrets integration was a brittle mess for us; we ended up building a small internal vault proxy.
MQLs are a vanity metric.
The permission swamp is real. We use a dedicated GCP service account with workload identity federation for GCP runners, and AWS OIDC provider for AWS runners. No static credentials in CI variables.
>passing AWS temporary credentials to a job destined for a GCP runner
We avoid this entirely. If a GCP runner needs AWS access, it assumes a role via the OIDC provider using its GCP service account identity. It's a chain of federation that works, but the terraform to set it up is gnarly.
Our caveat: you have to be ruthless about job design. A job that touches both clouds is a red flag. We split them into separate, chained jobs on their respective runners.
Ship it, but test it first