Hey everyone, I've been running GitLab CI for a few months now on our small team's project. After the latest update (we're on 16.10), I've noticed our pipelines are taking a lot longer to just *start*.
It used to be almost instant, but now there's a delay of 2-3 minutes before the first job even shows as "pending". Our `.gitlab-ci.yml` is pretty basic—just a build, test, and deploy stage for a Dockerized Node app. Runner is a self-hosted one on a DigitalOcean droplet.
Has anyone else seen this? Could it be something with the new version, or maybe a config change I'm missing? 😅 I checked our runner's logs but nothing obvious stands out.
Yeah, I've seen similar delays since 16.9 on our setup. The pipeline start time feels sluggish even before the runner picks anything up.
Check your GitLab instance logs (not the runner) around the time you push a commit. We found increased database query times for pipeline creation. The runner was fine, but GitLab itself was taking longer to assemble the pipeline object and hand it off. Might be worth looking at your Postgres load.
Also, make sure your droplet isn't hitting CPU limits during that initial processing. The runner being idle doesn't mean the GitLab server isn't struggling.
Run it yourself.
Same issue here on our Jenkins-to-GitLab migrated projects. The runner logs were clean, but the GitLab API logs showed delays in the `POST /api/v4/projects/:id/pipelines` call.
Our fix was to tweak the `shared_runners_minutes` and `ci_pipeline_creation_rate_limit` settings in the admin area. It wasn't the database for us - it was the rate limiting logic added in 16.9 getting too aggressive.
YAML all the things.
Oh wow, I was just about to post something similar! We're seeing the exact same thing on 16.10 with our self-hosted runner.
Our runner logs are clean too, which was confusing. I'm not an admin on our GitLab instance, so I can't check the server logs or those rate limiting settings user1046 mentioned. Is that something I'd need to ask our sysadmin to look into?
The 2-3 minute wait feels like forever when you're just trying to check a small fix. 😅 Glad it's not just us.
That's a really good point about checking the GitLab instance logs instead of the runner logs. The separation can be tricky, especially when the runner dashboard shows "idle" but the pipeline creation is stuck upstream.
We saw a similar pattern in our cluster, and the Postgres angle was key. In our case, it was less about raw CPU load on the GitLab server and more about table bloat on the `ci_pipelines` table after a few months without vacuuming. The autovacuum wasn't keeping up, leading to slow inserts for new pipeline records.
A quick `pg_stat_user_tables` check on the `ci_pipelines` and `ci_builds` tables can reveal if that's the bottleneck. If you have access to the database, that is. If not, it's a solid clue to pass along to whoever manages your instance.
Prod is the only environment that matters.