Best zero-trust networking solution for a hybrid AWS/on-prem shop

Having recently completed a migration to a zero-trust network model for a heterogeneous environment comprising AWS VPCs, a legacy colocation facility, and developer home offices, I found the landscape of solutions to be surprisingly fragmented. Many tools excel in the cloud-native space but falter when introducing physical hardware or non-corporate endpoints. After a six-month evaluation period involving simulated failure scenarios and gradual production traffic cutover, Tailscale emerged as the most operationally sound choice for this specific hybrid context. The core advantage is its ability to abstract away the profound complexity of managing disparate network security perimeters into a single, cryptographically sound overlay.

The primary technical merits from an SRE perspective are as follows:

* **Unified Access Control Plane:** Tailscale's use of WireGuard and its coordination server eliminates the need to manage VPN gateways, security group rules across multiple AWS accounts, and on-prem firewall ACLs as separate entities. Access is defined by device identity, not IP address. This drastically reduces the configuration drift and "snowflake" firewall rules that plague hybrid setups.
* **Seamless Subnet Router Integration:** The `tailscale subnet` feature allows you to announce routes from your on-premises networks or entire VPC CIDR blocks directly to the Tailscale mesh. This was pivotal for our legacy systems that could not run the Tailscale client. A small Linux VM in the colo acts as a subnet router, making the entire /24 rack network available to authorized Tailscale nodes.

```bash
# Example systemd service for a subnet router on Ubuntu
# /etc/systemd/system/tailscaled.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port=41641 --tun=tailscale0 --subnet-routes=10.10.100.0/24
```

* **Observability and Incident Response:** The admin console provides real-time maps of node connectivity and detailed ACL evaluation logs. During a suspected lateral movement incident, we were able to audit connection attempts between devices with precision, something that would have required correlating logs from AWS VPC Flow Logs, pfSense, and our IDS otherwise.
* **Simplified SLO/SLI Instrumentation:** Monitoring mesh health becomes straightforward. You can instrument key SLIs like latency between nodes and packet loss using the Tailscale ping functionality, moving away from black-box VPN tunnel health checks.

The principal pitfalls to consider are the dependency on Tailscale's coordination servers (though you can self-host `headscale`) and the potential for ingress/egress cost implications if you inadvertently route large data flows through a subnet router in a cloud region. Careful ACL design, using tags for role-based access, is non-negotiable for production use.

For teams managing a mix of cloud and physical infrastructure, Tailscale's model transforms a traditionally perimeter-based security problem into a manageable identity-based one. It is not merely a VPN replacement but a fundamental shift in how you conceptualize private network access. I am interested in hearing from other teams who have undertaken similar migrations, particularly regarding your approach to phasing out legacy site-to-site VPNs while maintaining availability.