We've been evaluating Google Cloud's Identity-Aware Proxy for securing internal management UIs (like our Prometheus, Grafana, and custom admin panels) without a VPN. The promise is compelling: context-aware access, integrated SSO, and no need to manage ingress firewall rules for hundreds of services. After a three-week testing period with a realistic load, however, the latency and resource overhead is significant enough that I'm now questioning its viability for performance-sensitive internal services.
Our test setup was methodical:
* **Control:** A simple GCP Internal HTTP Load Balancer with a backend Nginx instance serving a mock API (echo headers, return 2KB JSON).
* **Test:** The same backend, but fronted by a global external HTTP Load Balancer with IAP enabled. Access was via a GCE instance in the same region, simulating an engineer's workstation.
* **Tooling:** Used `wrk2` for load testing, measuring p50, p95, p99 latency and requests/sec. Each test ran for 5 minutes with a constant 1000 RPS.
Here are the median results from 10 test runs (in milliseconds):
| Percentile | Control (Internal LB) | IAP (External LB + IAP) | Delta |
|------------|------------------------|--------------------------|-------|
| p50 | 1.2 ms | 8.7 ms | +7.5 ms |
| p95 | 2.1 ms | 24.3 ms | +22.2 ms |
| p99 | 3.8 ms | 47.6 ms | +43.8 ms |
The throughput dropped from ~12,500 RPS (saturating the test backend) to ~9,800 RPS, a 21.6% reduction at the same concurrency level. This isn't just network hop overhead; the IAP layer is adding substantial processing time for every single request. The architecture necessitates a full TLS termination and re-initiation, header injection (the `X-Goog-Authenticated-User-*` headers), and the OAuth 2.0 token validation on *every request*.
Furthermore, the cost dimension is non-trivial. For our estimated 50M internal requests per day, the IAP cost (at $0.005 per 1000 requests) adds ~$75/day, or ~$27k/year, just for the proxy layer. This is before the load balancer costs. An alternative like a VPN or Tailscale mesh has a fixed, often lower, cost structure.
The security benefits are real—audit logs, centralized access policies, and integration with Google identities are excellent. But for high-throughput, low-latency internal microservices or monitoring tools where every millisecond counts, this overhead is hard to justify. I'm now leaning towards a hybrid model: IAP for truly sensitive, human-accessed UIs (e.g., database admin consoles), and a zero-trust mesh (like Tailscale or OpenZiti) for service-to-service and CLI/API access where performance is paramount.
Has anyone else conducted similar benchmarks? I'm particularly interested in whether the overhead is consistent across all GCP regions, or if there are tuning parameters (like keeping connections alive) that could mitigate this. The documentation is silent on performance characteristics.
—chris
—chris