Skip to content
Notifications
Clear all

My results after forcing everyone to write portable pipeline steps: less pain next time.

2 Posts
2 Users
0 Reactions
1 Views
(@eval_newbie_2025)
Reputable Member
Joined: 2 months ago
Posts: 166
Topic starter   [#16541]

So, this might sound obvious to the veterans here, but I just went through a huge migration from Jenkins to GitLab CI and I learned one thing the hard way: the biggest time-sink wasn't the platform itself, but our own messy, platform-specific pipeline steps.

We had scripts calling specific Jenkins plugins, secrets tied to its credential store, and logic buried in their Groovy format. Untangling that took weeks.

Halfway through the pain, I basically forced our team to write all *new* automation steps as shell scripts or callable commands with their own config files. The actual CI/CD job just became a minimal runner that calls `./deploy.sh` or `./run-tests.sh` with environment variables. It felt a bit tedious at first, but...

When migration time came, translating those jobs was almost trivial. Instead of rewriting complex logic, we just had to figure out GitLab's equivalent way to set env vars and run a shell script. The actual *work* was already portable. What took our team months for the old pipelines, took us about three weeks for the new ones, mostly for secrets migration and learning the new UI.

Has anyone else tried this "portable steps first" approach? I'm curious if it's a standard practice or if we just got lucky. Also, how do you handle secrets in a portable way? We're still using platform-specific vaults, which feels like the next thing to abstract.



   
Quote
(@amandaj)
Reputable Member
Joined: 1 week ago
Posts: 148
 

Your point about platform-specific steps becoming the real migration blocker mirrors what we've seen in analytics pipelines. We had a similar shift from a vendor-specific event tracking system to an in-house solution.

The parallel for us was forcing all data transformation logic into standalone SQL or Python modules that read from config files, never embedding the vendor's SDK calls directly into our application code. When we switched, we only had to rewrite the thin wrapper that called those modules, not the business logic itself. It added a small overhead in design time but saved months during the actual migration.

One caveat we found: this approach assumes your team has the discipline to keep the shell scripts or modules truly portable. It's easy to let platform-specific assumptions creep back in over time, like relying on a particular environment variable the new platform doesn't set. You need a way to enforce that boundary, maybe through periodic reviews or a simple lint check.


Data > opinions


   
ReplyQuote