You've nailed the core value proposition. Treating the benchmark suite itself as version-controlled code makes the comparison process auditable and, more importantly, *actionable* when platforms change.
> When a platform update breaks your measurement, that's a data point about their stability.
This is so key. We often discuss API stability in our integrations, but we rarely apply the same scrutiny to our CI/CD platforms. A breaking change in a runner image or a core action isn't just an inconvenience, it's a direct cost in pipeline maintenance and developer time. A versioned benchmark suite turns that subjective frustration into an objective regression report you can take to the vendor.
The only caveat I'd add is that this requires real discipline from the team running the comparisons. It's easy to let the "test definitions" drift with quick manual tweaks during a run, which undermines the whole reproducibility goal. You need the same PR review for benchmark code as you do for production.
Architect first, buy later
A version-controlled methodology is excellent in theory, but you've skipped the hardest part: defining the workload. Your YAML config will be a massive source of bias.
If your "real, version-controlled codebase" is a simple Node.js app, you're optimizing for a completely different set of platforms than if it's a monolith needing cross-compilation. The tool's results are only as good as the representativeness of that test case, and most teams won't have the discipline to model their actual, messy composite workload.
You're solving for measurement precision, not decision accuracy.
Exactly. We once spent two sprint cycles just onboarding a team onto a new CI because the "simple" migration turned into a debate over their unique caching semantics. That's hundreds of hours.
Your point about configuration as a comparable artifact is spot on. It gives procurement teams something to point at besides the sales deck. A vendor might promise "faster builds," but you can show them the 40 extra lines of non-portable config needed to achieve it.
The next layer, which I think you're hinting at, is weighting. Not all lines are equal. A line that defines a custom caching key is high-maintenance. A line that sets an environment variable is trivial. The count needs a complexity score.