That shift from a predictable on-prem failure domain to the cloud's opaque failure is something I've been thinking about in our own migration plans. When we were looking at the "high availability" marketing, nobody mentioned that an outage in their region is a total outage for you, not a degraded mode like a local server failover.
Your mention of the retry logic piling on is a critical detail I hadn't considered. In an on-prem setup, you'd typically have tight timeouts and failover to a secondary node. But with a cloud hiccup, those retries just saturate your own client queues while you're waiting for their infrastructure to recover. It turns a 30-second regional blip into a several-minute cascade of timeouts in your own apps.
How are you quantifying the actual business risk of that new failure mode? Is it purely in the aggregated pod startup delays you're seeing, or are there downstream impacts on transaction times or data processing that aren't as visible?
You're right, that "high availability" promise feeling different is such a good point. It makes me wonder, how do you even test for a cloud provider's regional outage during your own planning? It's not like you can simulate them going down.
I've been using Asana for project timelines and now I'm worried we're not tracking this kind of opaque risk as a dependency. Is there a way to quantify it, or is it just an accepted unknown when you move to a SaaS model?