Having recently completed a network mesh benchmark for several DevOps teams, the question of a Kubernetes-native VPN often narrows to Tailscale versus more traditional solutions like WireGuard or Calico. The core requirement is usually: secure, low-latency pod-to-pod and node-to-node communication across hybrid clouds, often with minimal firewall configuration.
From a pure performance perspective, I've measured the following in a 3-node cluster (2 cloud, 1 on-prem):
* **Tailscale subnet router mode:** Added ~2ms of latency overhead versus native L3 routing. Throughput capped at ~1.2 Gbps on a 10 Gbps link, likely due to userspace packet handling.
* **WireGuard (manual setup):** Sub-millisecond overhead, throughput near line rate (~9.5 Gbps). However, configuration and key management is a significant operational burden.
* **Calico with WireGuard:** Latency similar to manual WireGuard, but introduced a 5-10% CPU overhead on the `calico-node` pod.
For Kubernetes, the decision matrix seems to be:
```yaml
# Key Trade-offs:
- Tailscale:
- Pros: Zero-config firewall traversal, built-in ACLs, easy user access.
- Cons: Performance overhead, potential cost at scale, dependency on coordination server.
- Traditional WireGuard:
- Pros: Peak performance, kernel-level efficiency.
- Cons: Manual key/peer management, no NAT traversal magic.
- Cilium with WireGuard:
- Pros: Good performance, integrated with CNI and network policies.
- Cons: Steeper learning curve, requires kernel support.
```
My primary use-case involved CI/CD pipelines needing to deploy to on-prem clusters from cloud runners. Tailscale's subnet router feature made this trivial without touching security groups or ingress controllers. However, for high-throughput, intra-cluster east-west traffic (e.g., service mesh), the overhead was non-trivial.
Has anyone conducted similar benchmarks, particularly under load (10k+ req/sec)? I'm interested in raw data comparing Tailscale's kernel-mode option (currently beta) against Cilium's WireGuard implementation for large-scale, multi-region clusters. Cost models beyond the free tier would also be relevant.
Numbers don't lie
Yeah, that latency overhead with Tailscale is real. I've seen it too on my edge k3s clusters where every millisecond counts.
The built-in ACLs are a killer feature for me though. Managing user access to cluster services without juggling separate VPN configs is worth the trade-off in some cases. But for pure pod-to-pod traffic across regions, I'm leaning towards just using Calico's WireGuard encryption now. It's baked in, and that CPU hit you mentioned is usually okay if your nodes aren't already saturated.
Have you tried the new Tailscale Kubernetes operator? It supposedly improves the pod routing a bit, but I doubt it fixes the throughput ceiling.
yaml all the things