Skip to content
Notifications
Clear all

Troubleshooting: High CPU usage on the connector in Azure

2 Posts
2 Users
0 Reactions
2 Views
(@devops_grunt)
Estimable Member
Joined: 4 months ago
Posts: 159
Topic starter   [#8531]

Alright, I've been battling this for the last two weeks and finally have a working solution. Posting here because the official docs are vague on Azure-specific tuning and I found zero useful threads elsewhere.

We're running Appgate SDP 5.4 connectors on Azure VMs (Standard_D4s_v3). After deploying about 50 user devices, the connector CPU was pinned at 100% constantly. The dashboard showed "High CPU" alerts, and we started seeing latency spikes and dropped connections. The VM metrics in Azure Portal confirmed it wasn't just an SDP console glitch.

First, I ruled out the obvious:
* It wasn't a undersized VM issue—scaling up to a D8s_v3 only delayed the problem by a few hours.
* Logs (`journalctl -u sdp-controller`) showed no obvious errors, just the generic high CPU warnings.
* Traffic volume was normal, no DDoS pattern.

The breakthrough came from digging into the connector's internal stats via the admin API and correlating with `top` and `ss` commands on the VM itself. The root cause was **excessive TLS session renegotiations** due to Azure's default load balancer probe behavior combined with Appgate's default listener config.

Here's the diagnostic snippet that showed the insane renegotiation count:

```bash
# On the connector VM
sudo netstat -tnp | grep :8443 | wc -l
# Showed thousands of short-lived connections
```

The fix was two-fold:

1. **Adjust the Azure Health Probe:** The default probe interval on the Azure Load Balancer was too aggressive. We changed it from every 5 seconds to every 15 seconds, with an unhealthy threshold of 2.

2. **Tune the Connector TLS settings:** The critical change was adding `sslSessionCache` and `sslSessionTickets` directives to the connector's `listener` configuration in the appliance settings. We pushed this via the admin API.

```json
{
"listener": {
"sslSessionCache": "builtin:1000",
"sslSessionTickets": true,
"tlsCiphers": "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256",
"tlsVersions": ["TLSv1.2"]
}
}
```

Also, ensure your NSG/ASG allows the probe source IPs (Azure's probe IPs are documented). After applying these changes, CPU usage dropped from 100% to a steady 15-25%. The key was realizing the Azure LB probes were being treated as full, new TLS connections each time, causing the connector's worker processes to spin.

If you're seeing high CPU, check your connection churn rate first. Scaling vertically is just a band-aid.


Automate everything. Twice.


   
Quote
(@hannahj)
Trusted Member
Joined: 1 week ago
Posts: 59
 

Interesting that you found the TLS session renegotiation to be the culprit. Azure's load balancer health probes using HTTPS will indeed initiate a full TLS handshake on each request, which becomes a significant overhead at scale.

A related nuance I've seen is that some versions of the Azure load balancer can send probes from multiple source IPs, which some SDP configurations interpret as distinct clients, further exacerbating the renegotiation count. You might monitor the session cache hit rate in your admin API stats; if it's low, that confirms the pattern.

Did you resolve this by adjusting the probe interval, switching to a TCP probe, or modifying the listener's SSL settings to enforce session reuse?


Data is the new oil – but only if refined


   
ReplyQuote