That's a really clever solution, using dnsmasq as a buffer. I hadn't thought of that middle ground.
>Have you noticed if the flapping is worse during specific times
We haven't correlated it with VPC DNS load yet, but we do see more drops during our peak business hours, which lines up. It's not massive, just enough to be annoying. Makes me wonder if there's a hidden query the connector does that isn't cached well.
That broker loss/quick re-register pattern is frustratingly common. Since you've already checked the network path, let's look at the connector's own state management.
The health probe can't be tuned directly, but you can indirectly influence it by making the instance's network stack more tolerant. Besides the DNS tweaks mentioned here, check the TCP keepalive settings on the instance itself. The defaults are often too sparse for ZPA's expectations.
Run this on your connector instance:
```bash
sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
```
If `tcp_keepalive_time` is 7200 (2 hours), that's likely too long. A temporary test is to set it lower via sysctl, but you'd need to bake it into your launch template user-data for persistence. We saw improvement dropping it to 300 (5 minutes) with more frequent probes.
It's a workaround, but it buys the connection more chances to recover before the watchdog panics.
security by default
The `Connection to Broker lost` error you're seeing is almost certainly the connector's watchdog overreacting to transient network conditions. Since you've already validated the network path, the next step is to systematically rule out the specific failure points others have mentioned, starting with the most likely culprits.
Run these two checks on a live connector *before* it fails over:
1. Query the DNS resolver performance: `dig time @169.254.169.253 zscaler.com` (or your broker domain). Watch for response times consistently over 10ms, especially during your peak business hours.
2. Verify your NAT Gateway's flow idle timeout if egress traffic routes through one. The default is 350 seconds; a misconfigured NACL or routing could cause flows to be considered idle prematurely, dropping the broker connection.
You asked about tuning the health probe. There is no direct way, but you can buffer its dependencies. If the dig test shows high latency, implement a local caching resolver like dnsmasq or Unbound, as user772 suggested. If the NAT timeout is the culprit, you're looking at an architectural change, possibly moving the connectors to a public subnet with EIPs.
Can you share the output of those two checks? The pattern will tell us if you're fighting a DNS/NAT issue or if you've hit the deeper problem of the AMI's watchdog being fundamentally incompatible with AWS's network behavior.