Just saw some wild benchmark results on that CI/CD comparison site. They tested a mid-sized Node.js monorepo with parallelized pipelines across GitHub Actions, GitLab CI, and CircleCI. The speed differences were bigger than I expected.
Key takeaways for that scenario:
* **GitHub Actions** was fastest for the initial build & test matrix (under 8 mins).
* **CircleCI** had the most consistent run times across 10 consecutive runs.
* **GitLab CI** was surprisingly slow on the runner spin-up, but caching was really effective on subsequent runs.
Has anyone else been digging into these benchmarks? I'm curious if they match your real-world experience, especially for larger microservice setups. The caching performance seemed like the biggest variable.
— Jason
Let's build better workflows.
I always take those published benchmarks with a massive grain of salt. They're testing a perfect, clean-room scenario, not the messy reality of a team trying to ship on a Friday afternoon.
The "consistency" metric for CircleCI is interesting, but I bet it falls apart once you factor in third-party service latency, like hitting a flaky staging API during integration tests. That's where your real variance comes from, not the runner tech itself.
And yeah, caching is the whole game. GitLab's slow spin-up might not matter at all if your pipeline is just pulling down a massive, warm cache artifact and skipping 90% of the work. Did their benchmark even include a realistic, fragmented dependency state?
Data over dogma.
That's a good point about third-party services adding noise. Our team hits a few external APIs in our tests, and you're right, it does make any consistency number pretty much useless for us.
I'm still learning about caching setups, though. When you say "fragmented dependency state," do you mean like when different team branches have different dependency versions? Would that break the cache and make the benchmarks less relevant?
You've zeroed in on the core limitation of synthetic benchmarks. The stated consistency for CircleCI is almost certainly measuring variance in their own runner provisioning and network, not the systemic noise of a real deployment pipeline. In our environment, we instrument pipeline durations with histograms in Prometheus, and the 95th percentile is consistently 2-3 times the 50th due to exactly the external service latency you mentioned. The benchmark's "consistency" metric becomes a footnote under that load.
Regarding your question on a fragmented dependency state, that's precisely it. Most benchmarks assume a pristine, linear history. They don't model the cache thrash that occurs when multiple feature branches with divergent lockfiles are being built concurrently, or when a partial cache upload from a failed job corrupts the expected state. The effectiveness of GitLab's caching hinges entirely on the stability of the cache key inputs, which in practice are highly volatile.
A more valuable benchmark would measure the time to a *usable artifact* in a forked repository state, simulating a developer's branch before a merge. I've never seen one that does.
That's a really interesting breakdown, Jason. The caching performance being the biggest variable makes sense to me, even from my limited experience.
I've only just started playing with Docker caching for my own small side projects, and the difference between a warm cache and a cold build is night and day. I can only imagine that effect gets multiplied in a real monorepo with dozens of services.
For those of us who are still learning the ropes, do you think the benchmark's caching results would hold up in a team where different branches are pulling in different dependency versions? I'm trying to picture how that would actually work in practice.