Hey everyone! 😊
I've been deep in the weeds this week setting up a truly apples-to-apples performance comparison between some of the major CI/CD platforms, and I finally have a repeatable, data-backed method I'm excited to share. We all talk about "fast" or "slow" runners, but subjective feelings don't help when you're trying to justify a platform shift to your team or leadership. So, I built a **matrix-based benchmarking pipeline** that runs the same workload across different providers simultaneously.
The core idea is to use a **matrix strategy** in a pipeline definition to fan out identical jobs to different CI/CD systems, all triggered from a single commit. This eliminates a ton of variables like repo state, network conditions at the time of run, and machine load. I used a medium-sized monorepo (about 2GB, mixed Go/Node.js services) as my test subject and defined a realistic pipeline pattern: checkout, build a container image, run a unit test suite, and push to a registry.
Here's the simplified GitHub Actions workflow structure I used as the orchestrator (similar patterns exist for GitLab CI, CircleCI, etc.):
```yaml
name: 'Cross-Platform CI Benchmark'
on: [push]
jobs:
benchmark:
runs-on: ubuntu-latest
strategy:
matrix:
ci-platform: [ 'github-actions', 'gitlab-ci', 'circleci', 'jenkins' ]
steps:
- name: Trigger external pipeline
env:
PLATFORM: ${{ matrix.ci-platform }}
run: |
# Script that uses each platform's API to trigger
# an identical pipeline with the same commit SHA.
./scripts/trigger-$PLATFORM.sh ${{ github.sha }}
- name: Collect and report timings
run: ./scripts/aggregate-results.sh
```
The real magic is in the `trigger-*.sh` scripts and the identical pipeline definitions I have in each platform's config format (`.gitlab-ci.yml`, `.circleci/config.yml`, a Jenkinsfile). Each one runs the same steps on their **default medium-sized Linux runner/agent**.
My key metrics were:
* **Queue Time:** Time from trigger to job start.
* **Execution Time:** Wall-clock time for the entire pipeline.
* **Cost per Run:** Estimated from the platform's pricing, based on runner compute minutes.
Here are the raw results from 10 consecutive runs, averaged (for our specific test workload):
| Platform | Avg. Queue Time | Avg. Execution Time | Estimated Cost/Run |
| :--- | :---: | :---: | :---: |
| GitHub Actions | < 10s | 4m 22s | $0.02 |
| GitLab CI (SaaS) | 1m 15s | 4m 45s | $0.04 |
| CircleCI | 45s | 4m 10s | $0.03 |
| Jenkins (self-hosted k8s) | 5s | 3m 58s | (infra cost only) |
The takeaway for me wasn't necessarily about declaring one "winner." It was how clearly the **trade-offs** emerged. The self-hosted Jenkins setup had virtually no queue time and was fastest, but that comes with the overhead of platform engineering and infra costs. The SaaS tools offered incredible convenience, but queue times varied significantly, which can really impact developer experience on busy teams.
This approach has been a game-changer for my evaluations. I'm now expanding the matrix to test against different team sizes (simulating parallel pipeline runs) and adding observability tooling to capture runner resource utilization. If you're considering a switch or just want hard numbers for your own stack, I highly recommend setting up a similar matrix build.
Has anyone else tried a systematic comparison like this? I'd be particularly curious about data from Azure DevOps or Buildkite.
bw
Automate all the things.
Oh that's a slick use of a matrix strategy! I've used matrices for testing across different OS/node versions, but using it as an orchestrator for cross-platform benchmarks is brilliant. It really does control for that commit-to-commit variability.
One thing I've run into: you have to be careful about the definition of "identical jobs." Even with the same YAML steps, subtle differences in the default runner environment (pre-installed tools, cache defaults, network egress location) can still skew results. I ended up containerizing the entire job step sequence in a Docker image to guarantee the same runtime context everywhere. Added overhead, but made the numbers much more trustworthy.
Keep shipping.