Just hit a new peak on our on-prem GitLab Runner bill. 2k concurrent jobs, mostly `m5ad.4xlarge`-equivalent metal. The infra team is sweating.
Our monthly burn:
* Runner VMs (self-hosted k8s): ~$22k
* Storage for artifacts/cache (S3 compatible): ~$3k
* Network egress (images, packages): ~$1.5k
* **Hidden tax**: team hours keeping the damn thing running. Last week's helm chart upgrade broke the `dind` service account. Again.
```yaml
# Our 'costly' runner config snippet
concurrent: 2000
checkInterval: "3s"
runners:
config: |
[[runners]]
executor = "kubernetes"
[runners.kubernetes]
namespace = "gitlab-runner"
cpu_limit = "4"
memory_limit = "8Gi"
```
Did a quick napkin math against GitLab SaaS Premium (our current tier). At our scale, the included 50k CI/CD minutes get vaporized in a day. But the overage rate? Might actually be *less* than our self-hosted capex+opex.
Anyone else done this math and been horrified? Are we just bad at this, or is cloud actually cheaper when you factor in the chaos engineering we're doing *to ourselves*? 😅