We're planning a full stack rebuild: compute, data, monitoring, the works. The "big bang" approach is a financial and operational risk I'm not willing to endorse.
Has anyone phased it by environment? The idea is:
* **Phase 1: Rebuild Dev** - New patterns, new tools. Let teams break it. Cost is contained.
* **Phase 2: Rebuild Staging** - Mirrors Prod load. Validate scaling and costs before touching Prod.
* **Phase 3: Rebuild Prod** - Cutover. Should be a known quantity by now.
Where I'm skeptical:
* Maintaining parallel tooling during transition. Doubles some costs (e.g., data pipeline ingest).
* Configuration drift between phases negating the "validate in staging" benefit.
Looking for concrete examples where this worked, or more importantly, where it failed and what the cost/outage impact was. Specifically:
* How did you handle data continuity between old prod and new prod during cutover?
* Did the phased approach actually catch cost surprises, or did prod still have scaling bills you didn't see in staging?
cost per transaction is the only metric
Yes, we phased a major platform rebuild exactly like this. Your skepticism about config drift is spot on - that's where we stumbled. We enforced Infrastructure-as-Code for every phase, but someone inevitably hotfixes the staging DB for a demo and now it's not a true prod mirror.
For data continuity, we used a final dual-write period during the prod cutover. Old prod kept serving reads/writes for 48 hours while the new stack ran in parallel, consuming the same live data. This let us verify data integrity on the new pipelines before flipping the switch, without doubling ingest costs for the whole transition.
The phased approach did catch a major cost surprise: our new monitoring tool's pricing was based on host count in dev, but on data volume in staging. Staging's prod-like load exposed a billing model that would've been catastrophic in prod. You're right to be wary of parallel tooling costs, but for us, that staging catch paid for the temporary duplication several times over.
Keep it real, keep it kind.