Alright, let's see if the collective wisdom here can help me solve this one. I'm at my wit's end with an issue that's starting to smell like a fundamental flaw in our setup.
We've been running Rancher-provisioned RKE2 clusters for about 18 months now. The provisioning itself is smooth, but we keep getting hit with a specific, disruptive problem: seemingly random worker nodes lose connectivity to the control plane. The node status in Rancher flips to "Unavailable," kubectl commands hang when targeting pods on that node, but here's the kicker – the workloads on the "disconnected" node often keep running internally. They just can't be managed and can't talk to services on other nodes.
This isn't a total network partition. The nodes still respond to SSH and basic ICMP. It's specifically the Kubernetes API server and the overlay network that seem to die. We have to reboot the node to get it back, which is a terrible "solution."
We've ruled out the obvious:
* It's not the cloud provider's network (happening on both AWS and our bare-metal vSphere clusters).
* It's not resource exhaustion (memory, CPU, disk are all fine, no OOM kills).
* It's not a flaky CNI plugin – we're using the default Canal (Calico + Flannel) that RKE2 ships with.
My hunch is it's either a kubelet failure that doesn't auto-recover, or something in the `rke2-agent` process is leaking or crashing. The logs *before* the node is rebooted are key, but the relevant ones often rotate out by the time we get to them.
Here's a snippet from a node that's in this "zombie" state. The kubelet is just screaming about not being able to post its status:
```bash
# From /var/lib/rancher/rke2/agent/logs/kubelet.log
E0721 08:34:12.567123 12345 kubelet_node_status.go:487] Error updating node status, will retry: error getting node "ip-10-0-5-21": Get "https://127.0.0.1:6443/api/v1/nodes/ip-10-0-5-21": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```
And the `rke2-agent` service often shows:
```bash
systemctl status rke2-agent -l
...
Jul 21 08:30:15 ip-10-0-5-21 rke2[12346]: time="2024-07-21T08:30:15Z" level=warning msg="Failed to retrieve agent config: Get "https:///v3/connect/config": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
```
So the agent can't reach the Rancher tunnel server, and the kubelet can't reach the API server. The local API server proxy (that `127.0.0.1:6443` address) is down.
My questions for the room:
* Has anyone running Rancher-managed RKE2 seen this pattern of periodic, silent node disconnections where the fix is always a reboot?
* Is there a known issue with the `kube-proxy` or `calico` pods on the node crashing and not restarting properly, breaking the local networking path?
* What's the most robust way to set up monitoring for this *specific* failure mode? Right now we're alerting on `NodeNotReady`, but that's after the fact. I want to catch the process health *before* the node becomes unusable.
* Should I be looking harder at the `containerd` or `etcd` health on these nodes? The fact that the pods keep running suggests the container runtime is okay.
I'm leaning towards building a custom DaemonSet that runs a watchdog on the node, checking the local API proxy and restarting the `rke2-agent` service if it fails. But that feels like a duct-tape solution. I'd rather understand and fix the root cause if it's a known bug or misconfiguration.
Interesting. You've isolated the failure domain to the Kubernetes control plane and overlay network while preserving the underlying host connectivity. This pattern strongly suggests a failure in the RKE2 agent or the CNI's control plane components on the node, not the data plane.
Since you mentioned ruling out the CNI plugin, I'd focus on the RKE2 agent's tunnel. It establishes a persistent connection back to the control plane. Check the agent logs on an affected node just before it flips to unavailable. You're likely looking for repetitive TLS handshake timeouts or websocket errors.
Also, examine the kubelet's node lease duration and the RKE2 server's `--node-status-update-frequency`. A mismatch here, coupled with a brief network hiccup, can cause the control plane to prematurely mark the node as unhealthy. The workloads keep running because the container runtime is separate from the kubelet's reporting loop.
every dollar counts
>happening on both AWS and our bare-metal vSphere clusters
That's the only useful data point you've given. You need to start isolating variables instead of ruling out categories. "Not the cloud provider's network" is meaningless when the problem lives in your configuration, which is consistent across both environments.
What does your node security posture look like? Are you running host firewalls (iptables, firewalld) that might be interfering with the RKE2 agent or overlay network ports? A default-deny rule that crept in via an automation template would cause exactly this. The agent starts, opens the tunnel, then something else closes the required ports.
trust but verify