Just built a dashboard to compare pipeline times across 4 CI/CD tools

Having recently migrated a moderately complex application from a monolithic Jenkins setup to a multi-provider CI/CD strategy, I found the decision-making process hampered by a lack of concrete, apples-to-apples performance data. Vendor marketing materials and anecdotal forum posts provide little insight into actual runtime costs and efficiency under comparable loads. To address this, I instrumented a standardized pipeline to run across four platforms: GitHub Actions, GitLab CI, CircleCI, and a self-hosted Drone instance.

The goal was to measure pipeline duration, cost implications, and configuration overhead for a representative mid-sized project. The test repository is a 12k LOC Python service with a Postgres dependency, requiring a pipeline that executes linting, unit tests, integration tests with a containerized database, security scanning, and a Docker image build. The pipeline was optimized for each platform using their respective best practices for caching and parallelization.

**Test Environment & Methodology:**
* **Repository Size:** ~350 MB (including test assets and node_modules for frontend component).
* **Runner Specifications:** All cloud platforms used their default/large Linux VM tiers (4 vCPUs, 8GB RAM). Drone ran on an equivalent AWS EC2 instance (c5.xlarge).
* **Cache Strategy:** Implemented optimal caching per platform (e.g., GH Actions `cache` action, GitLab's `cache` keys, CircleCI's `save_cache`/`restore_cache`, Drone's volume caching).
* **Measurement:** Wall-clock time from commit push to pipeline success, measured over 10 consecutive runs per platform, discarding the first run as a cold cache outlier. All runs were performed during off-peak hours (02:00-04:00 UTC) to minimize provider load variance.

**Benchmark Results (Average of Runs 2-10):**

| Platform | Config Format | Avg. Pipeline Duration (mm:ss) | Cold Run Duration (mm:ss) | Estimated Cost per 1000 Runs (USD) |
| :--- | :--- | :--- | :--- | :--- |
| GitHub Actions | YAML (Workflow) | 08:42 | 14:55 | $0.00 (within free tier limits) |
| GitLab CI | YAML (.gitlab-ci.yml) | 09:18 | 16:30 | $40.00 (based on SaaS compute minutes) |
| CircleCI | YAML (.circleci/config.yml) | 07:55 | 13:48 | $50.00 (based on credits consumption) |
| Drone (self-hosted) | YAML (.drone.yml) | 06:15 | 11:20 | ~$22.00 (EC2 instance cost only) |

**Raw Data & Configuration Snippet:**
The timing data was collected via each platform's API and aggregated. Below is the core test job configuration for GitHub Actions, which is representative of the structure replicated across others:

```yaml
jobs:
integration-test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15-alpine
env:
POSTGRES_PASSWORD: postgres
steps:
- uses: actions/checkout@v4
- name: Cache Python dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run migrations and tests
env:
DATABASE_URL: postgres://postgres:postgres@postgres:5432/test_db
run: |
alembic upgrade head
pytest -v --cov=app tests/
```

**Analysis & Observations:**

1. **Self-hosted Advantage:** Drone's performance lead is expected, as it eliminates network latency to cloud-based artifact storage and provides dedicated, uncontested runners. The cost is variable and shifts from operational to infrastructural overhead.
2. **Cloud Provider Parity:** CircleCI's slight edge over GitHub Actions and GitLab CI in warm-run times appears linked to its more aggressive default layer caching for Docker builds. However, its cost model becomes prohibitive at high throughput.
3. **Cold Start Penalty:** The cold run delta (difference between cold and average warm run) is most pronounced for GitLab CI (~7+ minutes), likely due to the time spent pulling runner images and populating cache in their shared environment. GitHub Actions showed the smallest cold-start penalty.
4. **Config Complexity:** GitLab CI's configuration was the most verbose, requiring explicit `cache:policy` and `artifacts:expire_in` definitions to achieve performance parity. CircleCI's `orb` ecosystem reduced boilerplate but introduced vendor lock-in.

The data suggests that for teams already embedded in the GitHub ecosystem, GitHub Actions provides the best balance of zero cost, good performance, and adequate configurability. Teams requiring maximum performance and with DevOps capacity may find the self-hosted model optimal. The critical factor not captured here is ecosystem integration (e.g., security scanning, deployment gates), which may outweigh raw pipeline execution time. I am interested in data from others who have conducted similar comparative benchmarks, particularly concerning scaling effects on monorepos or matrix builds.