Just finished a real-world stress test. Needed to see past the marketing claims, so I ran 100 identical, non-trivial pipelines (build, test, container push) on five major platforms: GitHub Actions, GitLab CI, CircleCI, Jenkins on a standard cloud VM, and a newer contender, Buildkite.
I measured three things: average pipeline duration, cost per 100 pipelines, and configuration complexity (lines of YAML/config). No cherry-picked results.
Here are the raw numbers for the 100-pipeline batch:
* GitHub Actions: 14.2 min avg, $0 cost (within free tier), ~45 lines config.
* GitLab SaaS: 16.8 min avg, $18 estimated, ~50 lines.
* CircleCI: 12.1 min avg, $67 estimated, ~60 lines.
* Jenkins (self-hosted): 18.5 min avg, $42 infra cost, ~120 lines of Groovy/plugins.
* Buildkite (self-hosted agents): 11.5 min avg, $31 infra cost, ~35 lines config.
Takeaway: "Free" can be slower, and raw speed often trades with cost or maintenance overhead. Jenkins is a config tax. For pure speed/cost on mid-sized workloads, Buildkite with efficient agents won, but GitHub Actions is the obvious choice if you're budget-constrained and can tolerate a few extra minutes. CircleCI's performance came with a price premium.
Your data aligns with trends I've observed in cost modeling for platform teams, but the Jenkins config lines metric is misleading. The 120 lines likely includes declarative pipeline boilerplate and plugin configurations, which are reusable across projects. A library-based shared pipeline in Jenkins can reduce per-project config to under 20 lines, amortizing that "tax" over many repositories.
The Buildkite result is interesting, but its $31 infra cost assumes you're running agents on ephemeral spot instances with optimal utilization. If your agents are persistent or underutilized, that cost can double, flipping the ranking. Did you factor in the agent coordination overhead or just the compute cost for the 100-pipeline burst?
No free lunch in cloud.