Alright, let's pull back the curtain on the real circus. You're asking the right question because the list prices are just the appetizer. The main course—the one that gives you heartburn—is always hidden in the fine print and the emergent behavior of your own systems. As someone who lives in billing data, I've seen it all.
First, the obvious one that everyone misses: **Data Egress**. Your CI/CD system isn't an island. Every time you pull dependencies from a public repo, push artifacts to an external registry, or even just stream logs to a third-party monitoring service, you're moving bytes. At scale, this isn't free. I've seen teams burn $4k/month on egress they never accounted for, all because their builds download half the internet from `npm`, `PyPI`, and `Docker Hub`. Managed services often charge you to get your own data *out*.
Now, let's talk about the silent killers, the idle resources:
- **Persistent Storage for Ephemeral Workers**. You spin up a dynamic agent, it allocates a 100GB EBS volume or disk, does a 2-minute build, and terminates. The volume? It often persists until the *next* cleanup cron job runs. If that job fails or is too slow, you're paying for hundreds of "orphaned" volumes.
- **Warm Pools / Idle Instances**. You configure a minimum of 5 agents always ready to reduce cold-start time. That's 5 instances running 24/7, chewing through compute hours even when it's 3 AM and the only commit was a README typo fix.
- **Log Storage and Query Costs**. Cloud logging seems cheap until you're emitting verbose debug logs from 500 parallel jobs and then, god forbid, you actually need to *query* them to debug a failure. The ingestion *and* the analysis are separate line items.
Here's a tangible AWS CodeBuild example that looks innocent but will bleed you dry:
```yaml
# A "simple" buildspec that hides cost traps
version: 0.2
phases:
install:
runtime-versions:
nodejs: 18
# This caches to S3... great! But what's the lifecycle policy?
cache:
paths:
- 'node_modules/**/*'
build:
commands:
# Ever checked how much data 'npm ci' pulls on every build?
- npm ci
# This generates a 200MB artifact, uploaded to S3 on EVERY build
- npm run build
# Then you upload it to ECR, incurring data transfer AND storage
- docker push my-repo:latest
artifacts:
files:
- '**/*'
# This dumps everything into S3, with no expiration. Storage bloat is inevitable.
discard-paths: no
```
And don't get me started on the **meta-costs**:
* **Cost of debugging the billing itself**. The man-hours your team spends trying to tag resources correctly, build dashboards, and argue with support about why your bill jumped 300% are a real, unmeasured cost.
* **Vendor lock-in multipliers**. The more you use proprietary caching, artifact storage, or secrets management from your CI provider, the more painful (and expensive) a future migration becomes.
* **The "free" tier illusion**. It's great until you exceed the limit by 1%. Then you're on the hook for the *entire* month's usage, not just the overage.
You must instrument your evaluation with one goal: **visibility**. Assume every component has a secondary, hidden meter running. Tag everything from day one. Set up billing alerts at 50%, 90%, and 110% of your expected spend. The hidden cost isn't just a line item; it's the surprise on your face when the invoice arrives.
Your cloud bill is too high.
The persistent storage point for ephemeral workers is critical, and it's exacerbated by containerized build environments. Even with proper cleanup hooks, a crash or a SIGKILL can leave those volumes orphaned. I've instrumented our Jenkins and GitLab runners and found the worst offenders weren't the scheduled jobs, but the manually triggered "one-off" pipelines where engineers forget the cleanup guarantees are different.
A related nuance: the storage class itself. If your orchestration defaults to `gp3` but your workload is purely transient IOPS-heavy compilation, you're overpaying for throughput you never use. You can often define a lower-cost, higher-IOPS volume class specifically for these ephemeral workloads, but it adds configuration sprawl.
Also, consider the API call cost for the control plane operations to create and delete those volumes. At ten thousand builds a day, the `CreateVolume` and `DeleteVolume` calls against the cloud provider's API add a non-trivial line item that's separate from the storage-by-the-hour charge.