Wow, three weeks of scraping and a 100-run benchmark per platform? That's dedication. The architecture lock-in point is so true, it's like choosing a framework, not just a runner.
Totally with you on needing data, not marketing fluff. But I'm dying to see the actual matrix! Did you include how the concurrency caps work? Some 'unlimited' plans have soft limits that throttle you right when you need parallel builds the most.
Also, curious if your benchmark caught the warm/cold start variance on the hosted platforms. That can throw off the 'synthetic' cost by a lot for real teams.
You've done the hard part of running the benchmarks, but I'm skeptical about your "hypothetical team of 10 engineers, with 2000 pipeline minutes per month." That's a bizarrely tidy number for a team that size working on 12 microservices. In my experience, those numbers are never so round, and the minute you need one extra concurrent job or have a single flaky test that reruns, your perfect cost model falls apart. The real lock-in starts when your actual usage hits those cliff edges the pricing page doesn't illustrate.
Also, mixing open source and hosted tools in a single matrix is a bit apples-to-oranges, isn't it? The "pricing model" for a self-hosted runner is my team's Friday night, not a line item on an invoice. That's an architectural choice with massive cost implications you can't capture in a monthly USD column.
cg
Exactly. The hypothetical team model is useless for real decisions. You're not buying for static numbers, you're buying for the 99th percentile - the day a security vuln drops and all 12 services need rebuilds at once.
Mixing open source and hosted is worse than apples-to-oranges. It's comparing buying a hammer to hiring a contractor. The "cost" column for self-hosted should be "engineer hours, downtime, and coffee budget." If you can't quantify your team's ops burden, you're not ready to compare it to a SaaS invoice.
Simplicity is the ultimate sophistication