What's the best way to test disaster recovery for a K3s cluster?

Your pipeline is the right goal. But "methodical and automated" is where everyone gets stuck building internal tools.

I see two real problems you haven't mentioned.

First, your step 1 (provisioning a test cluster) becomes a cost and complexity anchor. Running a full replica for weekly drills gets expensive. Running a scaled-down version invalidates the test for stateful workloads.

Second, you're focused on the restore mechanics, but the validation is the product problem. Are you validating that a customer can complete a checkout, or just that a pod is ready? Most teams stop at internal cluster checks. That's useless.

The thread is right about external validation. You need a system outside the cluster hitting real endpoints and checking actual data integrity, not just row counts.

The real answer is you'll spend more time building and maintaining this test pipeline than you think. It's a core product in itself.

Your approach with the five-step pipeline is exactly where we started, and it's the correct framework. The gap we found was that a purely technical restore often misses business continuity requirements.

For your stateful workload concern, we pair Velero with the CSI snapshotter capability of our cloud provider. This captures the PVC data independently of etcd. The critical addition is a pre-backup job that writes a known data signature (a UUID and timestamp) into each critical database. Our post-restore validation, which runs from a separate bastion host, doesn't just check pod readiness - it queries for that specific signature. If it's not present and correct, the test fails.

On configuration drift, your validation step must extend beyond the cluster. We define success as external DNS resolving and serving the correct TLS certificate for our customer-facing endpoints. An internal service check can pass while the ingress controller is still referencing stale secrets, which we've experienced. So step 5 in our pipeline includes a mandatory curl from an external network to a known production-like domain, validating HTTP 200 and the correct certificate CN.