I'm currently evaluating Runway as a potential CI/CD replacement for our team, but I'm a firm believer in validating performance and workflow impact before any group rollout. My plan is to run a solo, controlled pilot.
I've provisioned a small test project mirroring our main service's structure—a Python API with a Postgres dependency. My initial benchmarks focus on two key latency vectors:
* **Pipeline cold-start time:** From commit to first job execution.
* **Step overhead:** Comparing identical `docker build` and `pytest` runs against our current Jenkins setup.
Here's the basic `runway.yml` I'm using for baseline measurements:
```yaml
pipelines:
- name: test-benchmark
steps:
- name: build
image: python:3.11-slim
commands:
- echo "Starting build phase"
- time docker build -t test-app .
- name: test
image: test-app
commands:
- echo "Starting test phase"
- time pytest ./tests --cov=app
```
Early data shows pipeline initialization averages ~8.2 seconds, which is promising. However, I'm seeing a ~12% increase in the `docker build` step duration compared to our Jenkins agents, likely due to different underlying compute profiles.
My question for the community: has anyone else conducted a similar solo-to-team migration with Runway? I'm particularly interested in:
* Scalability of the local testing experience to a team of 10+ developers.
* Any observable performance regression when multiple pipelines trigger concurrently.
* The learning curve for engineers accustomed to legacy systems.
I'll be sharing my full comparison graphs (pipeline latency, cost per run, developer feedback scores) once my two-week evaluation concludes.
p99 or bust
Love that you're taking this measured approach! Starting with a solo pilot is exactly how I test new marketing automation platforms before committing the team - you get the real quirks without the group friction.
>However, I'm seeing a ~12% increase in the `docker build` step duration
That's interesting. It makes me wonder if the runner spec for your test project matches your Jenkins agents. Sometimes the default compute or cache layers on these cloud CI platforms are a bit thinner. Could you tweak the resources for that step, or is that even an option in Runway? Might be worth checking if you can allocate more horsepower just for that benchmark.
Also, how are you accounting for the maintenance overhead you're *removing* with Jenkins? That 12% might get totally eclipsed by not having to manage your own agents, plugins, and config drift. Just a thought from someone who's celebrated deleting old servers 😄
Automate everything
Great approach starting with a solo benchmark. The cold-start time is a solid metric, but don't forget to measure the variance over, say, 50 runs. You might see that 8.2 seconds is the best case, not the average, which matters more for developer flow.
On the 12% slower build, could you isolate whether it's the actual Docker daemon performance or the underlying compute? I've seen similar gaps where it was entirely about disk I/O on the runner. You might try a simple `dd` test in your step to compare.
Also, are you accounting for the time your team currently spends on Jenkins node maintenance and queue wait? A 12% slower step might be a fine trade for eliminating that overhead.
- GG
>don't forget to measure the variance over, say, 50 runs
Totally agree. I've found that the first few runs on any new platform are often artificially fast because everything is pristine. The variance after 50 runs gives you the real "daily grind" number.
Isolating the disk I/O is a good call. A quick `dd` test is easy to slot into the benchmark. I usually run something like `dd if=/dev/zero of=/tmp/test bs=1M count=1024 oflag=dsync` in a step. The difference between that and your Jenkins agent can be stark.
That trade-off on maintenance time is the whole game, isn't it? A 12% slower step feels bad in a spreadsheet, but zero queue wait feels amazing at 4pm on a Friday.