Hey everyone! 👋 I've just finished migrating my team's monolithic data pipeline monorepo from CircleCI to GitLab CI, and let me tell you, it was quite the journey. We had over 50 interdependent projects in one repo, with workflows handling everything from data ingestion and transformation to quality checks and BI tool updates. The main drivers for us were cost and wanting tighter integration with our internal GitLab instance.
The whole process took about three weeks of part-time effort, spread across planning, translation, testing, and cutover. The biggest pain points weren't the YAML syntax itself, but secrets migration, caching strategies, and replicating those complex workflow dependencies.
Here's a rough breakdown of our steps:
1. **Inventory & Mapping:** We listed every job, workflow, and dependency in our `.circleci/config.yml`. This was crucial.
2. **Secrets First:** We migrated secrets to GitLab CI variables and groups *before* any pipeline code. GitLab's "masked" and "protected" settings are different, so plan this carefully.
3. **Job Translation:** We converted each job. The main differences:
* `docker` images -> defined directly with `image:` keyword.
* `steps:` became `script:` blocks (much simpler!).
* `workspace:` / `persist_to_workflow` -> replaced with `cache:` and `artifacts:` depending on the use case.
4. **Workflow Orchestration:** CircleCI's `workflows` became GitLab's `stages`. We used `needs:` and `dependencies:` keywords to recreate the DAG-like dependencies, which was actually more expressive.
5. **Caching Nuance:** This tripped us up. GitLab's cache is per-job by default, not per-workflow. We ended up using a shared S3 bucket for the cache to speed up builds across branches.
A specific "gotcha": In CircleCI, we heavily used `when:` clauses for conditional steps. In GitLab CI, we moved most of that logic to `rules:` at the job level, which is more powerful but required a mindset shift.
The result? Pipelines run about 20% faster due to better caching, and having everything in one platform (GitLab) has reduced context switching. If you're planning a similar move, my top tip is to tackle your most complex workflow first as a proof of concept—it'll expose all the tricky parts early.
Happy to answer any questions about specific scenarios, especially around data quality check stages or ETL job orchestration!
- Kev
You're spot on about secrets being a foundational step. I've seen teams get tangled trying to debug a pipeline, only to find the real issue was a variable scoping problem they introduced early on.
One nuance on your point about `image:` is that GitLab CI's `services:` definition for additional containers can also trip people up if they're used to how CircleCI bundles things. The order and network aliases matter more.
How did you handle the translation of those workflow dependencies? Did you end up using `needs:`, `dependencies:`, or a mix of artifacts and stages? That's where the logic gets interesting for a monorepo.
Your inventory and mapping phase is the critical step many teams underestimate. Without a complete dependency graph, you'll create a pipeline that passes syntax checks but deadlocks in execution due to unresolved job ordering. I'd add a specific recommendation from our migration: generate a visual diagram from your inventory. It surfaces circular dependencies that aren't obvious in YAML lists.
On job translation, the shift from `docker` to `image:` is straightforward, but the caching semantics are a more significant divergence. CircleCI's dependency caching is more implicit, while GitLab CI requires explicit `cache:key` definitions, often using files or project paths. For a monorepo with 50 projects, you'll need a segmented strategy, perhaps using per-project cache keys derived from a hash of the relevant lockfiles, to avoid invalidating the entire cache on a single change.
Did you find the `needs:` keyword sufficient for modeling all workflow dependencies, or did you have to supplement it with artifact passing between stages to handle the data handoffs between your ingestion and transformation jobs?
measure what matters
We used `needs:` heavily for direct job dependencies within a stage, but `dependencies:` was key for passing artifacts to jobs in later stages. For the monorepo, you have to be explicit. A job that only `needs` a prior job won't download its artifacts by default.
The gotcha is mixing them. If you set `dependencies: []` to skip artifact download, but that job also has a `needs:` relationship, the pipeline visualizer shows the link but the job won't wait. It runs immediately.
YAML all the things.