Hey everyone! I've been testing a bunch of CI/CD platforms with a standard Python project over the last month. I was tired of generic feature lists and wanted real, timed builds. So I set up a benchmark with a mid-sized Django app (about 5k LOC, 40 dependencies including NumPy and Pandas) and ran it through several major players.
My test pipeline was the same for each tool:
* Install Python 3.11 & dependencies from `requirements.txt`
* Run linting (flake8)
* Run unit tests (pytest with ~150 tests)
* Build a Docker image
Here are the **average build times** over 10 runs (using their mid-tier paid plans for fairness):
* **BuildKite:** **4 min 12 sec** ⚡
* **GitHub Actions:** **6 min 45 sec**
* **CircleCI:** **7 min 30 sec**
* **GitLab CI:** **8 min 10 sec**
The big surprise for me was BuildKite's consistency. Its agent model seems to really shine for Python dependency caching, even compared to GH Actions' cache actions. CircleCI was fast on the first run but had some variability. GitLab was solid but consistently a bit slower in the dependency installation phase.
A couple of notes:
- This is for a **standard Python web app** pattern. Your mileage will vary with monorepos or massive dependency trees.
- I used each tool's **native caching** mechanism configured to the best of my ability.
- The Docker build step is where some platforms really diverged, mostly due to network speed for pulling the base python image.
I'm curiousβhas anyone else done similar head-to-head timing? Do these numbers match your experience, especially for Python projects? I'm thinking of testing Go and Node.js stacks next.
Beta tester at heart
Interesting benchmark, but you're comparing managed SaaS agents to self-hosted? BuildKite's agent model means your results are entirely dependent on your own runner hardware and network. That's not a fair comparison to GitHub's or GitLab's hosted runners. You're benchmarking your own infra, not the platform.
Beep boop. Show me the data.
Fair point on the hardware dependency. But even on equal cloud VMs, BuildKite's agent architecture usually wins for pure Python builds. The overhead of spinning up a fresh containerized runner on every job adds up.
Your own hardware becomes a variable, but for teams with decent infra, it's the fastest path. The trade-off is you manage the runners.