Skip to content
Notifications
Clear all

How do I handle etcd backups on a self-managed kubeadm cluster?

1 Posts
1 Users
0 Reactions
0 Views
(@alexh3)
Trusted Member
Joined: 6 days ago
Posts: 42
Topic starter   [#13906]

Having recently moved a mid-sized analytics pipeline (~50 nodes, mixed spot and on-demand) from a managed Kubernetes service to a self-managed `kubeadm` cluster for cost and control reasons, I've hit the operational hurdle of ensuring high-availability for the control plane's most critical component: etcd. While the managed services abstract this away entirely, the responsibility now falls on my team, and I'm looking to establish a robust, automated backup and recovery strategy.

My primary concerns are ensuring consistency of the backups and minimizing operational overhead. I understand `etcd` supports snapshotting, but the integration into a full disaster recovery plan for a `kubeadm` cluster seems less documented than for distributions like OpenShift or RKE. Specifically, I'm evaluating a few approaches and would appreciate insights on their trade-offs:

**Option 1: Direct `etcdctl snapshot save` on the control plane nodes.**
* Scheduling this via a CronJob within the cluster itself feels circularly dependent.
* Running it as a systemd timer on the host seems more straightforward, but then managing the backup lifecycle (rotation, off-site transfer) becomes a host-level concern.

**Option 2: Using the `etcd` Pod's `etcdctl` (if running etcd as a static pod).**
```bash
kubectl exec -n kube-system etcd-master01 --
etcdctl snapshot save /var/lib/etcd/snapshot.db
--cacert /etc/kubernetes/pki/etcd/ca.crt
--cert /etc/kubernetes/pki/etcd/server.crt
--key /etc/kubernetes/pki/etcd/server.key
```
* This requires securely handling the PKI credentials for the scripted operation.
* Where should the snapshot be written? To a hostPath volume that is then backed up? Or directly to an object storage bucket from within the pod?

**Option 3: Employing a dedicated Kubernetes backup operator (e.g., Velero with etcd integration).**
* This adds another layer of complexity but potentially provides a more unified backup for both etcd and persistent volumes.
* I'm skeptical about the recovery granularity. For a full cluster disaster, a full etcd restore is needed. Does the operator simplify or complicate the precise etcd recovery steps compared to a raw snapshot?

**Key Questions:**
1. What is the recommended method for automating encrypted, consistent snapshots in a production `kubeadm` environment?
2. How do you handle the secure storage and rotation of these snapshots, especially when they contain sensitive `Secrets`?
3. Has anyone successfully tested a full cluster recovery from an etcd snapshot on `kubeadm`? Are there pitfalls specific to `kubeadm`-generated PKI or manifest files during the restore process?
4. Is it advisable to run etcd on dedicated nodes separate from other control plane components at a certain scale, and how does that impact the backup topology?

I'm particularly interested in comparisons of the operational burden between these methods. A detailed feature matrix from hands-on experience would be invaluable, focusing on recovery time objectives (RTO) and the complexity of the restore procedure.


Data is the source of truth.


   
Quote