Complete beginner: how do I run a benchmark on a CI/CD pipeline?

So you want to benchmark your CI/CD pipeline. Good. You’ve probably been reading vendor blogs that promise “blazing fast” builds and “seamless scaling.” Let’s scrape off the marketing gloss.

First, you need to decide what you’re actually measuring. Is it the time from git push to production? Or just the feedback time for a failing unit test? Most beginners measure the wrong thing and end up optimizing a single step while the whole pipeline is held up by something else entirely. Classic.

Here’s a brutally simple way to start, using tools you already have:

1. **Instrument your pipeline with timestamps.** In a shell step, just `date +%s` and store it as an environment variable or write it to a file. Do this at the start and end of key stages (dependency install, build, test suite, deploy). The difference is your benchmark for that stage.
2. **Look at the concurrency.** Are your jobs actually running in parallel, or are they queued for 10 minutes waiting for a free runner? Your platform’s admin panel will tell you this—it’s often the biggest hidden tax.
3. **The cost angle.** If you're on a paid platform, translate that total pipeline time into dollars per month. If you're self-hosting, look at the runner’s resource consumption. A “faster” pipeline that needs a 32-core beast might be losing you money versus a slightly slower, efficient one.

The free alternative? Forget the fancy SaaS dashboards for now. Script it yourself. A simple CSV log of run times, triggered from your pipeline, can be analyzed with a few lines of Python or even a spreadsheet. You’ll learn more about your actual bottlenecks than any proprietary graph will show you.

And remember, the most common bottleneck isn’t your tooling—it’s your own process. How many approvals? How many integration tests that could be split? Benchmark that, and you might just save your team’s sanity.

― Finn