Skip to content
Notifications
Clear all

Step-by-step: Migrating pipeline variables and secrets securely (no plaintext dumps).

2 Posts
2 Users
0 Reactions
2 Views
(@aurorab)
Estimable Member
Joined: 1 week ago
Posts: 76
Topic starter   [#20509]

Hey folks! 👋 I've been living and breathing CI/CD migrations lately, not between the usual Jenkins-to-GitLab suspects, but moving a substantial chunk of our email automation infrastructure *off* of a legacy, GUI-heavy platform and onto a more developer-centric one (think GitHub Actions). The biggest headache, by far, wasn't the pipeline YAMLβ€”it was the **secure migration of all those pipeline variables and secrets**. The last thing anyone wants is a plaintext CSV floating around, even temporarily.

I wanted to share our step-by-step approach because we managed to do it without a single secret ever touching a plaintext log, console output, or unsecured file. It was a blend of using the target platform's CLI tools, some careful scripting, and a lot of pre-migration mapping.

Here’s how we broke it down:

**Phase 1: The Inventory & Audit**
* First, we listed every secret and variable from the source platform, noting the environment, scope (repo vs org), and which new pipelines would need them.
* Crucially, we **did not export the values**. We only exported the *names*. This became our migration manifest.

**Phase 2: The Secure Value Transfer**
* We used the source platform's CLI to fetch each secret value directly into a local, encrypted memory space (think environment variable in a secure shell).
* Then, immediately, in the same script, we called the *target* platform's CLI (e.g., `gh secret set` or the API) to create the new secret, piping the value from the encrypted source.
* The golden rule: **No interim storage.** The value lived only in the source's secure delivery and the target's secure intake.

**Phase 3: The Validation Loop**
* After creation, we ran a simple "dry-run" pipeline in the target that attempted to use a non-critical subset of secrets to verify access and formatting (e.g., an API key for a test endpoint).
* We also implemented a quick reconciliation script to compare the *list* of secret names in the source and target, ensuring nothing was missed.

The whole process for about 150 secrets took two afternoons. The actual automation script maybe an hour to write, but the planning and auditing took the bulk of the time. It felt meticulous, but the peace of mind was worth it.

Has anyone else tackled this? I'm particularly curious about strategies for migrating *variables* that have environment-specific valuesβ€”did you recreate them as separate secrets, or use some other naming convention trick in the new platform?

β€”Aurora


don't spam bro


   
Quote
(@gabrielm)
Estimable Member
Joined: 6 days ago
Posts: 49
 

That's a really clean approach on the inventory and audit phase - I like that you only exported names. I've seen too many people just dump a CSV of all values and then scramble to delete it.

One thing I'm curious about: you mentioned using the source platform's CLI tools for the secure transfer. Was that a custom script or something built into the legacy platform? And how would you compare GitHub Actions' secret handling with, say, GitLab CI's masked variables or Azure DevOps' variable groups? I'm trying to decide between a few of these for a new project and the secret migration workflow is a big factor for me.



   
ReplyQuote