I've been evaluating Tailscale as a potential overlay network for a multi-cluster, multi-cloud Kubernetes deployment. The promise of a zero-config, wireguard-based mesh is attractive from a latency and operational overhead perspective. However, most discussions focus on the simple `tailscale up` use case. I'm interested in production-scale Kubernetes integrations.
Specifically, I'm looking for data and experiences on:
* **Performance overhead:** What's the actual latency and throughput penalty when using the Tailscale sidecar (e.g., via the `tailscale` container in a pod) versus a native CNI like Cilium or Flannel? I've run some preliminary micro-benchmarks on a test cluster, but they're synthetic.
```yaml
# Example pod spec I've been testing with
containers:
- name: app
image: nginx:alpine
- name: tailscale
image: tailscale/tailscale:stable
# ... with appropriate env vars and mounts
```
My initial `iperf3` tests between nodes in different clouds show a ~2-3ms TCP latency add over the native cloud network paths. Packet throughput seems acceptable but I haven't stress-tested it.
* **Operational complexity at scale:** How does managing auth keys, node limits, and ACLs work with hundreds of pods constantly churning? Does the control plane introduce any observable throttling or connection limits during rapid scaling events (e.g., HPA-driven replica increases from 10 to 100 pods)?
* **Integration patterns:** Are teams using it primarily for outbound pod egress (e.g., pods needing to talk to internal, non-Kubernetes services), or also for pod-to-pod mesh? Any issues with DNS (MagicDNS) inside pods conflicting with CoreDNS?
* **Observability gaps:** The Tailscale admin console provides some metrics, but have you found critical gaps in monitoring the mesh health from standard APM tools (e.g., Datadog, Prometheus)? Are there custom metrics you had to expose?
The theoretical benefits are clear, but I value empirical data. Has anyone run this under sustained production load (500+ pods, varied traffic patterns) and can share profiling results or gotchas?
sub-10ms or bust
Great to see someone digging past the marketing. I've been running Tailscale sidecars in production for about 18 months now, mainly for a specific use-case: connecting our GKE clusters to legacy on-prem services.
> My initial `iperf3` tests between nodes in different clouds show a ~2-3ms TCP latency add
That aligns with our measurements. The overhead becomes more pronounced under sustained high throughput, though. We saw the biggest penalty not in raw pod-to-pod, but when a pod with the sidecar was talking to a *bunch* of other Tailscale nodes concurrently. The single sidecar seemed to become a bottleneck around 4-5 Gbps aggregate. For our low-throughput, high-criticality control plane traffic (like talking to a secure database), it's been fine. For data plane stuff, we avoided it.
On operational complexity, the real pain point for us has been node churn and IP management. Tailscale's subnet router feature helped, but when a pod with a sidecar gets rolled, you sometimes get a brief "zombie" route in the mesh until the control plane catches up. We ended up writing a small operator to monitor for those stale routes and force a `tailscale down` on the old pod IPs. It's not zero-config at that point, but it's been stable.
—jr