I’ve been operating a Linux-based Tailscale subnet router (on Ubuntu Server 22.04 LTS) for approximately six months as part of a hybrid cloud lab setup, connecting several on-premise workloads to cloud VPCs. Over the past two weeks, I’ve observed a sustained and anomalous increase in CPU utilization attributed to the `tailscaled` process, even during periods of minimal network traffic. This is impacting the performance of other co-located services.
**Current Configuration & Environment:**
- **Host OS:** Ubuntu 22.04.3 LTS, Kernel 5.15.0-91-generic
- **Tailscale version:** 1.56.1 (updated from 1.54.2 two weeks ago, which coincides with the onset of symptoms)
- **Hardware:** Intel NUC 11 Pro (i5-1135G7, 4 cores/8 threads), 32GB RAM
- **Role:** Subnet router advertising two local subnets (`192.168.1.0/24`, `10.10.0.0/22`)
- **Peers:** ~15 devices, mix of Windows, macOS, and other Linux nodes
- **Observed Load:** `tailscaled` consistently consumes 25-35% of a single CPU core (as reported by `top`), spiking to 60-70% during any peer discovery or DERP relay failover events.
I’ve conducted initial diagnostics, which I’ll detail below. My primary hypothesis is that this is related to either a regression in the recent Tailscale version concerning route table management, or an inefficiency in the packet forwarding path when handling many small, idle keep-alive packets.
**Diagnostic Data Collected:**
1. **Process profiling** (via `perf top` on the `tailscaled` PID) shows significant cycles spent in:
- `net/http.(*persistConn).readLoop`
- `runtime.mallocgc`
- `golang.org/x/crypto/curve25519.scalarMult` (though this is expected, the proportion seems high)
2. **Tailscale status output:**
```bash
# tailscale status --json | jq '.'
{
"Version": "1.56.1",
"TUN": true,
"Subnets": [
"192.168.1.0/24",
"10.10.0.0/22"
],
"NumPeers": 15,
"ActiveDERP": "nyc",
"Health": {
"CurrentDERP": "nyc",
"RouteAll": true
}
}
```
3. **System metrics** (from a co-located Prometheus node exporter) confirm the CPU load is not due to system interrupts or memory pressure. Network throughput is negligible (< 50 Kbps) during these high-CPU periods.
**Steps already attempted without resolution:**
- Restarted the `tailscaled` service multiple times.
- Confirmed IP forwarding and `nftables` rules are correct and not causing loops.
- Temporarily disabled one advertised subnet — no change in CPU profile.
- Set `TS_DEBUG="cpu"` and reviewed logs; no obvious errors, but logs show frequent `magicsock: endpoints changed` messages even for stable peers.
My questions to the community are twofold:
- Has anyone else running a Linux subnet router on version 1.56.x observed similar sustained CPU load, particularly on idle or low-traffic networks?
- Are there known tunable parameters (e.g., `--tun-buffer-size`, `--socket-buffer-size`) or kernel networking optimizations (`sysctl` tweaks for `net.ipv4.ip_forward`, `net.core.rmem_max`, etc.) that have proven effective in mitigating this in production deployments?
I plan to conduct a more detailed benchmark comparing 1.56.1 against a rollback to 1.54.2, measuring CPU utilization per million packets forwarded, but would appreciate any prior art or insights before proceeding.
—chris