GitLab CI pipelines failing for me since the 16.10 update. Mainly runner connectivity and `needs` job dependencies timing out.
I'm in fintech, managing deployment for a dozen microservices. I evaluate anything that speeds up builds or simplifies artifact management.
Looking for:
* Real fixes for "stuck" jobs.
* How you structure pipelines for monorepos.
* Alternatives to complex `rules` logic.
I'll post any workarounds I find.
Ship fast, review slower
Yeah, those timeout issues are rough. I hit something similar last month, but with tags on shared runners causing jobs to get stuck in "pending" forever. For us, adding explicit resource groups helped a bit with the dependency conflicts, kind of like a workaround for `needs`.
You mentioned monorepos - how do you handle pipeline triggers for your microservices? Do you use changes: paths in every job, or something more dynamic? I'm still figuring out the balance between speed and duplication in our setup.
rookie
Yeah, resource groups have been a lifesaver for our `needs` dependency issues too. Only caveat I'd throw in is that they can actually slow things down in parallel if you're not careful with the `concurrency` setting. We had a situation where jobs that could run in parallel were getting queued because the resource group was too restrictive. Ended up splitting them into separate groups for logically distinct pipelines.
On the monorepo trigger question: we went all-in on `changes: paths` with YAML anchors to avoid the duplication mess. I've got a parent template that defines the common pipeline structure, then each microservice folder has a minimal `frontmatter` job that just evaluates its own path. If it detects changes, it triggers a child pipeline with the service-specific artifact build. Keeps the `.gitlab-ci.yml` from turning into a 2,000-line monster.
But honestly, the child pipeline approach adds its own overhead for initializing runners. We're still tweaking. What's your monorepo size and how many services? Curious if you've tried the "dynamic" route with a script that generates the pipeline YAML on the fly.
Let the data speak.
We're seeing the same timeout issues after 16.10, especially with jobs needing artifacts from other stages. Have you found any specific workarounds for the runner connectivity part? It feels like a regression.
For monorepos, we use a simple parent/child pipeline with dynamic yaml generation to avoid copying rules everywhere. It's one file per service, triggered by a path check in the main pipeline. It cuts down on the rule complexity you mentioned.
What's your artifact management setup look like? I'm always looking for ways to speed that up.
For the timeout issues, especially with cross-stage artifact dependencies, we found increasing the `ARTIFACT_DOWNLOAD_ATTEMPTS` environment variable on the runner from the default (3) to 5 or 6 made a significant difference. It's a crude workaround, but it points to network instability between the GitLab coordinator and the runner pods post-16.10.
Your dynamic YAML generation for monorepos is solid. We landed on a similar pattern but compile it into a single pipeline artifact - a JSON map of changed services - that downstream jobs source. It avoids spawning a child pipeline for every service, which itself can be a source of the coordination timeouts you're seeing.
On artifact management, we've moved most intermediate build artifacts to a dedicated, versioned S3 bucket with a short-lived CDN in front, bypassing GitLab's internal artifact storage for everything but the final deployment manifests. The latency chart below shows the improvement for a 500MB artifact fetch (p95).
```
GitLab Internal Storage: 12.4s
S3 + CloudFront: 1.7s
```
The main cost is the added complexity in cleanup jobs.
I've seen a few threads about 16.10 regression on connectivity. The runner logs are the first place to look - check for increased 5xx errors from the GitLab coordinator. It's often a platform-side bottleneck they're slowly fixing.
For monorepos, avoid the `needs:` pattern across stages for your core dependency chain. It's fragile right now. Use a single initial job with `changes:paths` to output a simple file listing changed services. Then have parallel, independent pipeline segments for each service that only run if their name is in that file. It's more lines of YAML, but the logic is flat and reliable.
What's your runner setup? Kubernetes, autoscaling VMs, or something else? The artifact download attempts tweak user205 mentioned helps most on container-based runners.