Alright, so we've been running a hybrid thing for a while. Mostly on-prem Kubernetes for dev builds, but we also have workloads on AWS and GCP for specific integration tests. Team size is under 100 devs/ops.
The problem I keep hitting with the big managed CI platforms (think GitLab.com SaaS, CircleCI, etc.) is the agent model for multi-cloud. You either:
* Get locked into their own compute/network, making cross-cloud tests clunky.
* Have to manage their "runners" or "agents" on your own VMs in each cloud, which then becomes a scaling/security headache.
I'm curious what others are actually using that doesn't explode in cost or complexity. My rough requirements:
* Must be able to dispatch jobs to our own worker nodes in AWS, GCP, and our on-prem k8s cluster.
* Needs decent visibility into where a job ran and its resource burn. eBPF tracing for build step performance would be a dream, but I'll settle for basic metrics.
* Prefer something we can self-host the control plane for, but open to SaaS if it truly handles multi-cloud well.
I've been looking at:
* **Tekton** on our own k8s - seems powerful but feels like we're building a whole platform.
* **Jenkins** (obviously) with the Kubernetes plugin - it works, but the config-as-code story still feels heavy.
* **Buildkite** - Their hybrid model (SaaS coordinator, your own agents) looks promising. Anyone run this across multiple clouds?
What's the actual cost look like at, say, 50,000 build minutes a month spread across three clouds? Not just the platform fee, but the hidden cost of managing the agent lifecycle.
Here's a snippet of the Jenkinsfile complexity I'm trying to avoid too much of:
```groovy
pipeline {
agent {
kubernetes {
label 'aws-eu-runner'
yaml '''
spec:
nodeSelector:
cloud: aws
containers:
- name: jnlp
resources:
requests:
cpu: "100m"
'''
}
}
stages {
// ... but then you need a different one for GCP
}
}
```
Curious about real numbers and war stories.
System calls per second matter.
Jenkins with the Kubernetes plugin can actually fit your multi-cloud dispatch requirement quite well. The controller stays on-prem, and you define separate Jenkins agent pod templates for AWS, GCP, and your local cluster, each using distinct kubeconfig contexts. You can then label jobs to target a specific cloud.
However, the visibility and resource burn metrics are its weak point. You'll need to integrate with something like the Prometheus plugin and do a fair bit of custom dashboards. That's where the platform-building feeling of Tekton starts to look more attractive, since its native CRDs give you better observability hooks out of the gate.
Have you considered a hybrid approach? Use Argo Workflows for the orchestration and visibility across clouds, but let developers define their pipelines in simple CI tool syntax (like GitHub Actions) that submit to it. It moves the complexity but keeps it centralized.
benchmark or bust