I am embarking on a project to evaluate and select a CI/CD platform for my organization, and one of our primary selection criteria is raw pipeline execution speed. I understand that performance can be highly subjective to context, so I am attempting to design a rigorous benchmarking methodology. My background is primarily in data visualization and reporting, so I am accustomed to comparing tools like Tableau and Power BI using controlled datasets and specific metrics; I am seeking a similar structured approach for CI/CD.
My goal is to create a comparison that could be useful for teams of roughly 10-15 developers, managing a monorepo with a mix of microservices (approximately 8-10 services). The pipeline patterns we need to evaluate include standard build, test, and deploy stages, with a focus on parallel execution capabilities and dependency caching.
To move beyond anecdotal evidence, I am considering the following controlled experiment, but I would appreciate feedback on its design and any potential pitfalls:
* **Test Codebase:** A standardized, public repository containing a simple web application in two languages (e.g., a Node.js and a Go service) with defined unit and integration tests.
* **Pipeline Definition:** Identical pipeline logic (e.g., `install dependencies -> build -> run tests -> build container image`) translated into the native syntax of each tool (GitHub Actions, GitLab CI, CircleCI, etc.).
* **Key Performance Indicators (KPIs):**
* **Wall-clock Time:** Total time from commit push to pipeline completion.
* **Queue Time:** Time spent waiting for an available runner.
* **Cost-Per-Execution:** Estimated cost for a single pipeline run, based on public pricing and the compute minutes consumed.
* **Cache Efficiency:** Measured by the reduction in build time on a subsequent, identical run when caching is enabled.
My primary questions for the community are:
* Are there established, open-source benchmarking suites or standardized "dummy" projects for this purpose that I am unaware of?
* Beyond the KPIs listed, what other quantitative metrics would provide meaningful insight into pipeline speed and efficiency?
* How significant is the variability introduced by the geographic region of the runners, and should I attempt to control for this by mandating a specific region (e.g., us-east-1) across all platforms?
* In your experience, what are the most common confounding variables when performing such comparisons, and how can they be mitigated?
I plan to compile the results into a dashboard for clear comparison, focusing on the variance in performance across multiple runs. Any guidance on ensuring the benchmark is fair and representative would be greatly appreciated.