Greetings community. I am reaching out from a position of empirical frustration, having recently completed a major CI/CD migration from Jenkins to GitLab CI. While the pipeline translation itself was a methodical, if tedious, process, the operational aftermath has been unexpectedly costly. The core issue appears to be runner scaling logic and resource utilization, leading to a cloud compute bill that is approximately 2.3x our previous baseline for comparable workload throughput.
Our previous Jenkins setup used a static pool of persistent agents (m5.2xlarge EC2 instances) which we managed with a custom scaling script based on queue depth. The migration to GitLab CI promised more elegant auto-scaling via the `gitlab-runner` executable with the `docker+machine` executor. We configured an AWS autoscaling group driven by GitLab's internal queue system. However, our observed behavior is one of severe over-provisioning. Runners spin up aggressively but seem to linger far beyond job completion, and the parallelization of our matrix jobs is spawning more concurrent machines than intended.
Here is a sanitized version of our `config.toml` for the runner manager:
```toml
concurrent = 20
check_interval = 3
[[runners]]
name = "aws-autoscale-docker"
url = "https://gitlab.example.com"
token = "REDACTED"
executor = "docker+machine"
limit = 15
[runners.docker]
image = "alpine:latest"
[runners.machine]
IdleCount = 1
IdleTime = 900
MaxGrowthRate = 3
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
MachineOptions = [
"amazonec2-instance-type=m5.2xlarge",
"amazonec2-region=us-east-1",
"amazonec2-vpc-id=vpc-xxxx",
"amazonec2-subnet-id=subnet-yyyy",
"amazonec2-security-group=sg-zzzz",
"amazonec2-use-private-address=true"
]
```
Our typical pipeline triggers a benchmark matrix: a single merge request might spawn 12 jobs (3 benchmark types across 4 configurations). We observed that this often results in 12 *new* machines being provisioned simultaneously, despite `concurrent = 20` and `limit = 15`. The `IdleTime` of 900 seconds appears to be ignored if the manager itself is under load from new job requests, which is constant. Consequently, we have a high instance churn but also a large number of idle-but-billed instances.
Key questions for those who have navigated this:
* Is the `docker+machine` executor fundamentally more expensive for bursty, parallel workloads compared to a pre-provisioned, dynamically scheduled executor (like `docker` with a warm pool)?
* Are there specific tuning parameters for `IdleCount`, `IdleTime`, and `MaxGrowthRate` that better align with cost-control for cloud environments? Our goal is to minimize the time from job start to completion, but not at the expense of 50% idle resource time.
* Has anyone implemented a hybrid model successfully? For example, using a small fleet of persistent runners for fast, lightweight jobs (linting, unit tests) and the machine executor only for heavy integration/benchmark jobs?
I am in the process of gathering a full week's worth of CloudWatch metrics to correlate instance spawn/kill events with GitLab job logs, but anecdotal wisdom from the community would be invaluable. The migration was meant to reduce operational overhead, not increase cost exponentially.
-- bb42
-- bb42
You're hitting a classic pain point with the docker+machine executor. That lingering behavior is often from the `IdleCount` setting - runners stay alive waiting for "future" jobs that might not come. Can you share your MachineOptions section? The default idle time is pretty generous. Also, matrix jobs will each request a machine unless you're using parallel:matrix in a single job, which might be part of your over-provisioning. It's a sharp transition from static agents.
Keep it real, keep it kind.
Interesting. So the promise of auto-scaling can backfire if it's too eager to keep capacity warm. You mentioned `IdleCount`. Is there a general rule of thumb for setting that versus `IdleTime` for a workload with sharp, unpredictable spikes? Coming from static pools, I'm not used to tuning for "lingering" at all.
Ah, that truncated config is frustrating! I was hoping to see your MachineOptions. The `concurrent = 20` is a huge red flag if you're using docker+machine, that's likely a primary cost driver.
That setting caps total jobs across *all* machines, so the manager will try to spawn enough instances to hit that concurrency. If you have 20 jobs queued, it'll spin up 20 machines, not 1 machine running 20 containers. Combine that with default `IdleCount` and you've got a bill multiplier.
For sharp, unpredictable spikes, I'd set `IdleCount = 0` and tune `IdleTime` down to maybe 1 or 2 minutes. Let the spike trigger scale-up, but kill idle capacity fast. It's a trade-off - you'll pay a tiny bit more in spin-up latency to avoid paying for idle compute.
Show me the accuracy numbers.
`concurrent = 20` is the killer. That config is for a single machine's job slots, not for docker+machine. With docker+machine, that's a hard target for the manager to hit by creating *new VMs*. Each job gets a fresh VM.
You need to cap your total machine count instead. Add this to your MachineOptions:
```toml
Limit = 5
```
Start low, like 3-5, and monitor. That's your real concurrency control. The `concurrent` value above it should be higher than your `Limit * your instance's vCPU count`.
Trust but verify, then don't trust.