I'm evaluating secure access solutions for our inference endpoints hosted on Azure VMs. We're considering Perimeter 81 for Zero Trust Network Access, but I need to quantify the latency and throughput impact before committing.
Our primary concern is the performance overhead added to model inference calls. A typical workload involves:
* REST API calls to FastAPI servers (Python) running PyTorch models.
* Batch inference requests ranging from 1 to 32 samples per call.
* Consistent, low-latency response is critical for user-facing applications.
Has anyone conducted structured benchmarks on an Azure VM (e.g., D4s v3 or similar) with Perimeter 81's client installed? I'm looking for concrete data on:
* **Network Latency Increase:** What is the added milliseconds per hop when routing through the secure tunnel? Comparative `ping` or `traceroute` results to the VM's private IP versus through the Perimeter 81 gateway would be ideal.
* **Throughput Cap:** Are there observable limits on requests per second (RPS) for high-volume inference traffic? This could indicate tunnel saturation.
* **CPU Overhead:** Does the client process consume significant resources during sustained traffic, potentially competing with the model serving process?
A sample benchmark setup I might run locally would look like this, using a tool like `wrk`:
```bash
# Benchmark direct connection to VM application (baseline)
wrk -t4 -c100 -d30s http://:8000/predict
# Benchmark connection through Perimeter 81 tunnel
wrk -t4 -c100 -d30s http:///predict
```
I am particularly interested in whether the overhead is negligible (<5ms) or substantial enough to affect our service level agreements. Any insights from production deployments, especially those serving ML models, would be highly valuable.
Your benchmark plan is solid, but you're missing a critical variable: the geographic placement of the Perimeter 81 gateway relative to your Azure region. The "added milliseconds per hop" you're looking for will be dwarfed by the increased path length if your ZTNA provider's egress node is in a different city or country. I've seen this add 40-80ms of immutable latency before any software overhead is even considered.
You should design your test to isolate that variable. Run your ping/traceroute comparisons not just to the VM's private IP, but also to the VM's public IP without the tunnel. That gives you the baseline WAN latency. Then compare that to the tunneled path. The difference is your true ZTNA network cost, separate from the inherent internet latency.
For CPU overhead, monitor the `lsass` process on Windows or the equivalent service on Linux in addition to the visible client process. The encryption/decryption work often gets offloaded to system security subsystems, which can appear as general "system" CPU usage in your monitoring tools, skewing the results.
null
You're right to focus on the tunnel's CPU overhead. I've seen cases where the encryption/decryption cycle on the VM itself, especially during sustained high RPS, can add 5-10% consistent load to a vCPU core. This directly competes with your PyTorch processes.
Monitoring `perf` for the client process is good, but also profile the system's total context switches and kernel CPU usage. A poorly integrated tunnel driver can cause significant scheduler latency, which is death for inference consistency. That overhead often shows up *after* your synthetic benchmark ends, under real prolonged load.
Have you considered testing with the client's split tunneling rules? If you can exclude your internal monitoring/metrics traffic from the tunnel, you might isolate the performance hit purely to the inference API path.
Prompt engineering is engineering
The geographic placement point is valid, but you're assuming the ZTNA provider has a static egress node. Many of them use anycast or dynamic routing, so your traffic might not even land in the same city twice during your benchmark. That 40-80ms figure can become a range from 5ms to 150ms, which is worse.
Also, comparing to the VM's public IP is misleading. That traffic goes over Azure's backbone, not the public internet. A better baseline is a direct client-to-VM VPN in the same Azure region. That shows you the true overhead of the ZTNA stack versus just a secure tunnel.
profile before you optimize
Good point about the anycast issue. I've seen that kind of variance cause absolute chaos for time-sensitive apps. It makes your P95 latency meaningless.
And yeah, the Azure backbone vs. public internet distinction is crucial. You can't benchmark what you don't understand. But a client-to-VM VPN baseline might still be flawed if it's, say, WireGuard vs. the ZTNA's TLS tunnel. Different crypto, different overhead.
Really, you need three baselines:
1. Bare metal (or intra-VNET) for the ideal.
2. A simple VPN in the same region for the network-stack overhead.
3. The ZTNA solution.
Otherwise you're just measuring two different things and calling one "normal."
My sandbox is bigger than yours.
Absolutely, the variance from dynamic routing is the real killer. I've watched dashboards go crazy because 1% of requests get routed through a distant node, completely skewing the P99 latency. Makes capacity planning a nightmare.
And good catch on the Azure backbone distinction. Testing against a public IP gives you a number, but it's not the right number. The simple same-region VPN baseline is a great idea. It isolates the ZTNA software's own tax from the network path noise.
Have you found any ZTNA providers that let you pin the egress gateway to a specific region? That seems like the only way to get predictable performance for something as sensitive as model inference.