Hey everyone! 👋 I just wrapped up a 14-month project migrating our entire production workload from a managed cloud Kubernetes service to a bare metal RKE2 cluster. We're now running **200 nodes** across two data centers, handling about 1,200 pods. I learned a ton and wanted to share some real operational notes, especially for folks considering the bare metal path.
**Why we switched:** Our cloud bill was getting scary 😅, and we needed more control over networking and storage for our stateful workloads. We also had hardware access from a previous project, so the cost savings were massive.
**The Good Stuff with RKE2:**
* **Upgrades have been shockingly smooth.** We've done two minor version upgrades so far. The `rke2-upgrade` command plus the automated canal-culling makes it less stressful. We do 10 nodes at a time, and a full rolling upgrade takes about 90 minutes with zero downtime.
* **Networking defaults are solid.** We stuck with Canal (Flannel + Calico policy). It just works, and the network policies are easy to apply. Bandwidth overhead is predictable.
* **Operational overhead isn't as bad as I feared.** The built-in `etcd` snapshot schedule saved us twice during early config mess-ups. The system upgrade controller automates a lot of the OS patching.
**The "Gotchas" (Bare Metal Edition):**
* **Storage, storage, storage.** This was 70% of our prep work. We ended up using Rook-Ceph *on top* of RKE2. Getting the block mapping right for performance was a project in itself.
* **Load balancing.** You're on your own! We deployed MetalLB in BGP mode, which added another layer of config but is now rock solid.
* **Node provisioning & lifecycle.** We use a mix of PXE boot and a simple Ansible playbook for node registration. Adding a new node takes about 15 minutes now, but building that process took a month.
**Our key checklist items for a large bare metal cluster:**
- Plan your `etcd` placement carefully. We run dedicated nodes for it.
- Set up centralized logging *before* you migrate real traffic. We learned this the hard way.
- Build a "node readiness" validation script that checks NICs, disk IO, and kernel modules before joining the cluster.
- Budget more time for hardware failures. We keep 3 warm spares ready for quick swap.
Anyone else running RKE2 at this scale? I'm especially curious about how others are handling persistent storage and cluster backups. Our Ceph setup is working, but I'm always looking for simpler options!