Walkthrough: Reducing our CI bill by 40% by caching better

Our team's monthly CI/CD spend had been climbing steadily for three quarters, finally hitting a plateau of around $1,800/month on our managed platform. The growth directly correlated with increased microservices and parallel build pipelines, but a granular analysis revealed a consistent pattern: over 65% of the compute minutes for each pipeline run were consumed by dependency installation and asset compilation—operations that, in theory, should be largely idempotent between runs. This post details the systematic approach we took to implement a more effective caching strategy, which ultimately reduced our bill to approximately $1,080/month.

The initial investigation showed our caching was naive. We were relying on the CI platform's built-in cache for package managers, but it was often invalidated due to overly broad cache keys. For example, our Node.js `package-lock.json` was hashed, but the cache key also included the runner image tag, which we updated frequently for security patches, causing unnecessary cache misses. Similarly, Docker layer caching was virtually non-existent because each build spawned a fresh, ephemeral runner instance with no access to previous layers.

We focused on two primary areas: **language-level dependency caching** and **Docker layer caching (DLC)**. For the former, we moved to a more precise, two-tiered key strategy.

```yaml
# Example of our refined GitHub Actions cache setup for Node.js
- name: Cache node_modules
uses: actions/cache@v3
with:
path: |
**/node_modules
**/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
```

This ensured caches were reused across minor runner image updates. For Docker, we configured our self-hosted runners to use an external `docker buildx` cache backend (like a dedicated S3 bucket), allowing layers to persist between completely separate runner instances and jobs.

The most impactful change was architectural: we introduced a reusable, pre-warmed base image for our most common services. This image, built nightly, contained all OS-level dependencies and the *pinned* versions of our core language runtimes and tools. By using this as the `FROM` layer in our application Dockerfiles, we bypassed hundreds of `apt-get install` and `apk add` steps in every single build. The savings were immediate and substantial.

**Summary of changes and their estimated impact:**
* **Precise Cache Keys:** Reduced `npm install`/`pip install` time by ~70% per job.
* **External Docker Cache Backend:** Achieved ~90% Docker layer cache hit rate, slashing image build times for incremental changes.
* **Pre-baked Base Images:** Eliminated 4-5 minutes of system package installation per build.

The implementation required careful documentation and team-wide communication to ensure all new services adhered to the patterns. The total engineering investment was about 12 person-days, which paid for itself in less than six weeks. The key takeaway is that effective caching isn't just about turning on a feature; it requires a deliberate, multi-layered strategy tailored to your stack and pipeline patterns.

- Mike