After comparing cloud-based CI/CD platforms for our mid-sized monorepo, we consistently hit a bottleneck: the unpredictability of shared runner performance for benchmarking. To get reproducible, apples-to-apples numbers for framework or tool comparisons, we needed a controlled environment. This led us to deploy a dedicated, self-hosted runner. The goal wasn't to replace our cloud pipelines, but to create a stable baseline machine for performance regression tests and accurate tool comparisons.
Here's our hardware and software baseline:
- **Machine:** AWS EC2 `c6i.xlarge` (4 vCPUs, 8 GiB RAM, NVMe SSD)
- **OS:** Ubuntu 22.04 LTS
- **Runner:** GitHub Actions self-hosted runner v2.317.0
- **Isolation:** Docker via `docker-compose` for each job
The primary configuration challenge was ensuring the runner starts in a clean state for every job. We achieved this by configuring the runner to use Docker containers for all jobs, defined in the workflow file. Our `start.sh` script for registration:
```bash
#!/bin/bash
# Configure and start the runner
./config.sh --url https://github.com/org/repo --token TOKEN --name benchmark-runner-c6i --labels benchmark,c6i.xlarge --docker
./run.sh
```
Crucially, our workflow file must specify the `runs-on: [self-hosted, benchmark]` label and define the container image. We use a custom image pre-loaded with our common toolchain to avoid network variance during setup.
```yaml
jobs:
benchmark:
runs-on: [self-hosted, benchmark]
container:
image: ghcr.io/org/benchmark-base:node-20-bullseye
volumes:
- /var/run/docker.sock:/var/run/docker.sock
steps:
- run: |
docker-compose down -v
docker-compose up --build --abort-on-container-exit
```
Key learnings from the setup:
- **Network variance:** Even on a self-hosted runner, pulling external images can introduce noise. Cache all base images locally.
- **Storage cleanup:** Implement a mandatory cleanup step (`docker system prune -f --volumes`) between jobs to prevent disk I/O degradation.
- **Runner overhead:** The self-hosted runner daemon consumes ~200MB RAM; factor this into your resource calculations.
The result is a consistent environment where a benchmark suite for, say, comparing REST vs GraphQL response times under load, yields a standard deviation under 2% across 10 runs. This allows us to attribute performance deltas to code changes, not runner noise.
benchmark or bust
benchmark or bust
Your choice of a c6i.xlarge is interesting for a baseline. Were there specific considerations for not using a Graviton instance? While the Intel consistency is valuable for historical comparison, the cost-performance difference on Arm-based instances can be significant, which could affect the long-term economics of maintaining this runner.
Also, relying on Docker for isolation introduces its own performance overhead, particularly for disk I/O and network-intensive benchmarks. Did you profile the delta between bare-metal runs on the same host versus the Dockerized jobs? For CPU-bound microbenchmarks, the difference is often negligible, but for anything involving heavy filesystem access, it can skew results.
prove it with data
The decision to standardize on an Intel-based instance is critical for this specific use case. While Graviton instances offer compelling economics, they introduce an architectural variable that would invalidate historical benchmark comparisons. Many performance-critical libraries and frameworks still have subtle, architecture-specific code paths. Maintaining a single ISA baseline removes that variable entirely.
Your approach of using Docker for job isolation is sound, but the point about overhead is valid. For our regression suite, we run a calibration step at runner startup that measures the Docker overhead for key operations (disk I/O, process spawn, network loopback) and stores the results as baseline metadata. Each benchmark job then includes these calibration numbers in its report, so you can distinguish between genuine performance regression and a shift in container overhead. This requires instrumenting your benchmark harness to capture this context.
The clean state guarantee is the most important aspect. We found that configuring the runner with the `--ephemeral` flag, combined with a systemd service that destroys and recreates the runner container after each job, further reduces state contamination. This adds a few seconds of overhead per job but guarantees absolute consistency.
—BJ