Alright, let's add another data point to the "smooth, automated OpenShift upgrades" marketing claims. We've just emerged from the 4.12 to 4.14 tunnel, and it wasn't a scenic ride. Our three attempts felt less like an upgrade and more like a series of targeted hazing rituals.
The first attempt failed on the pre-flight checks. Some obscure operand in the `openshift-marketplace` namespace decided it wasn't having it. The error message was, predictably, a Kafkaesque paragraph pointing to a generic troubleshooting doc that hasn't been updated since 4.10. Our takeaway? The pre-flight validation is thorough, but deciphering *what* it's actually objecting to requires a mind reader.
Second attempt, we got past pre-flight, only to have the upgrade stall at 74%. The cluster version operator just… sat there. No CPU spikes, no logs screaming. Just existential dread. Turns out, one of the control plane nodes had a lingering local storage issue that the initial health checks missed. The solution wasn't in any upgrade guide—it was buried in a year-old GitHub issue comment. The fix involved manually clearing a condition from the node's status. Feels like surgery with a rusty spoon.
What finally worked on attempt three? A cocktail of:
* Manually pruning any `ClusterServiceVersion` resources in a failed state (thanks, Operator Lifecycle Manager).
* Pre-emptively scaling down some stateful workloads that we *knew* were sensitive to API server blips.
* Running the upgrade in the dead of night with crossed fingers, essentially treating it like a disruptive maintenance window, which is the opposite of the "zero downtime" promise.
The networking stack (OVN-Kubernetes in our case) held up, I'll give it that. But the operational overhead to get from A to B was non-trivial. It makes you wonder about the "managed" part of this managed Kubernetes distribution when you're spending hours in the CLI doing manual triage.
Compared to the relatively boring, click-button upgrades of some cloud provider Kubernetes offerings (EKS, GKE), this felt needlessly complex. The value prop of OpenShift is strong, but the upgrade reliability at these intermediate jumps? Not convinced. The process demands a dedicated SRE with a deep understanding of the platform's internals, which negates some of the abstraction benefits.
Anyone else find that OpenShift upgrades, while feature-rich, still carry a significant "gotcha" tax that isn't advertised in the datasheet?