Rolled out Spinnaker to 150 engineers - what broke and what we learned

Our two-year migration from a fragmented ecosystem of Jenkins, bash scripts, and manual Artifactory promotions to a unified Spinnaker deployment was predicated on a single goal: enabling safe, auditable, and rapid self-service deployments for our 150+ engineering teams. The promise of immutable image-based deployments with baked-in approval gates and robust rollback mechanisms was compelling. However, the operational reality of scaling Spinnaker's powerful but complex abstractions—namely, Applications, Pipelines, and the nuanced interplay of Triggers, Stages, and Artifacts—revealed a series of unforeseen failure modes that our initial proof-of-concept had not surfaced.

The primary breakage occurred not in Spinnaker itself, but in the translation of legacy mental models and processes. We learned that a successful rollout is less about the tool and more about governing its consumption. Here is a detailed assessment of our critical pain points and the mitigation playbooks we developed.

**What Broke:**
* **Pipeline Sprawl & Configuration Drift:** The ease of creating pipelines led to hundreds of near-duplicate pipelines with minor, ungoverned variations (e.g., different base images, manual judgment gates in different stages). This made security auditing and cost optimization nearly impossible.
* **Artifact Management Assumptions:** Our legacy scripts often assumed mutable artifacts. Spinnaker's immutable image paradigm broke deployments that expected to pull latest "SNAPSHOT" builds. Teams accustomed to promoting the same artifact through stages were confused by the requirement to re-bake or explicitly bind new artifacts.
* **Secret Migration Complexity:** Migrating secrets from a mix of Jenkins credential stores and environment variables into a secure, Spinnaker-compatible system (we chose HashiCorp Vault) was a multi-phase project. The breakage was subtle: pipelines would fail because of missing environment context the old system implicitly provided, or due to incorrect scope permissions in the new secret store.
* **Unexpected State Management:** Spinnaker's pipeline execution is stateful and persistent. Engineers used to ephemeral Jenkins jobs would restart failing pipelines from the beginning, not understanding that some stages are skipped if their artifacts are already "successful." This led to deployments with inconsistent states.
* **Resource Contention and Performance:** The initial Halyard installation defaults were insufficient. Deck (UI) latency increased, and Orca (orchestration) queues grew under load, causing pipeline delays. This eroded trust as teams perceived the new system as "slower."

**Our Mitigation Playbook:**
1. **Implemented Pipeline Templates:** We created standardized, vetted pipeline templates using Spinnaker's Pipeline Templates/Jinja features. Teams could only customize approved parameters (e.g., cluster size, replica count), not the stage logic. This reduced sprawl and ensured compliance.
2. **Artifact Governance Policy:** We mandated and automated a "bake-and-tag" stage for all production-bound pipelines. We provided clear documentation and CLI tooling to help teams transition from "latest" tags to semantically versioned, immutable image references.
3. **Phased Secrets Migration:** We created a three-phase cutover: (1) parallel runs logging secret usage, (2) a hybrid model sourcing from both old and new stores, (3) hard cutover with dedicated support rotation. Each pipeline was categorized by criticality and migrated in waves.
4. **Comprehensive State Awareness Training:** We developed runbooks focused on pipeline state management. This included training on "Restart Stage," manual judgment behavior, and the importance of the "Execution Details" UI to diagnose state-related failures.
5. **Scaled the Control Plane Proactively:** We moved from a monolithic Halyard-managed deployment to a Kubernetes-deployed Spinnaker with tuned resource requests/limits for each microservice (particularly Orca and Echo). We implemented Redis monitoring and pipeline queue alerts.

The total timeline from initial feasibility study to full decommissioning of the old CI/CD systems was 22 months. The most time-consuming phase was not the technical installation (which took ~3 months), but the incremental migration, template creation, and secret migration, which accounted for the remaining 19 months. The key metric for success was not pipeline count, but the reduction in production deployment-related incidents, which decreased by 60% year-over-year post-migration, validating the investment in safety and auditability.

—Anna