Hey everyone 👋
Just updated our self-managed GitLab instance to 16.9 this weekend and I'm seeing a noticeable lag in pipeline startup times. Our average "pending" to "running" state used to be under 10 seconds, and now it's consistently hovering around 45-60 seconds. This is on a medium-sized project (~5000 commits, monorepo).
I've already checked the obvious stuff:
- Runner queue depth is normal (we use autoscaling AWS EC2 runners)
- No resource starvation on the GitLab server (CPU/Memory/IO are all fine)
- Our `.gitlab-ci.yml` hasn't changed in weeks
What's interesting is that the delay seems to happen **after** the runner picks up the job, but before it actually starts executing the script. The job log just sits at "Running with gitlab-runner..." for that long period.
Here's a snippet from our runner config (to see if anything jumps out):
```toml
concurrent = 20
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "AWS autoscaling runner"
url = "https://gitlab.ourcompany.com/"
token = "***"
executor = "docker+machine"
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
[runners.machine]
IdleCount = 1
IdleTime = 1800
MaxBuilds = 100
MachineDriver = "amazonec2"
```
Anyone else running into this after the latest update? I'm wondering if there's a new pre-job validation step or network call that's introducing latency. Especially curious if those on GitLab SaaS (gitlab.com) are seeing similar behavior.
If you've found any workarounds or config tweaks that help, please share! I'm about to start digging through the release notes more deeply and maybe run some traces.
Dashboards or it didn't happen.