Troubleshooting: ZPA connectors failing over constantly in AWS. Stability tips?

Interesting, I hadn't considered the watchdog panicking from a *non-data path* connection. That's a sneaky one.

So even a separate ALB for something like metrics could cause a total tunnel reset if it cuts the management API. Makes sense. How did you track that down? Was there a specific log message for the watchdog event, or was it a process of elimination?

The pattern you describe, `Connection to Broker lost` errors with healthy instances, typically points to an overly sensitive watchdog mechanism combined with latent network conditions that are within AWS's norms but outside the connector's default tolerances. Since you've validated the foundational networking, your next steps should involve tuning the system to account for the reality of cloud network jitter.

I'd prioritize investigating two areas in parallel. First, audit the EC2 instance's network performance envelope, particularly if you're on a burstable instance type. Monitor the `EnaNetworkCreditBalance` and `EnaNetworkPktTxDrop` CloudWatch metrics; a depleted credit balance can cause micro-bursts of packet loss that trigger a failover while CPU metrics remain normal.

Second, modify the connector's watchdog sensitivity. This often requires adjusting parameters in the systemd service file for the connector process, specifically the tunnel keepalive interval and the consecutive failure count before a reset. Increasing the failure threshold provides a buffer for transient packet loss. A common starting point is to double the default consecutive failure count, but this needs to be tested against your specific network's baseline latency and jitter. Have you examined the connector's service file for tunable heartbeat parameters?

The recurring theme here is everyone diagnosing symptoms, not the cause. You're all tuning TCP stacks and rearranging deck chairs.

Has anyone actually looked at the Zscaler service status logs for your tenant during these events? In my experience, half of these "broker lost" errors coincide with planned, but poorly communicated, Zscaler backend maintenance. The connector watchdog is just the canary in their coal mine.

Your focus on AWS tuning might be misdirected if the instability originates upstream with your ZPA provider's own infrastructure churn.

I appreciate the detailed parallel approach you've laid out. The network credit monitoring angle is a solid one, especially since that data point is often buried in CloudWatch and overlooked.

On the watchdog parameter tweaks, my one caveat is to coordinate any changes with your team's deployment process. Adjusting those systemd service files directly on the instance can be overwritten by the next connector auto-update or a scripted deployment, which can reintroduce the instability. It's better to bake the tuned parameters into your base AMI or configuration management so they persist through updates.

Yep, that's the exact pattern we saw. The healthy instance metrics make it extra frustrating. Your question about health probe sensitivity is key.

The built-in watchdog is indeed aggressive by default. In our case, we increased the `--tunnel-keepalive` value in the systemd service override. This gave the connection more time to recover from a minor network blip before the watchdog panicked. We landed on 120 seconds, up from the default 60, which significantly cut down the spurious failovers.

But a major caveat: making the health check less aggressive means a *genuinely* dead connector takes longer to be replaced by the ASG. You absolutely need to pair this with the EC2 status checks, as others mentioned, to catch actual instance failures. It's a tradeoff between stability and failover speed.

Oh wow, this is my exact situation right now too. I'm setting up my first ZPA environment and seeing the same "Connection to Broker lost" logs constantly. I was starting to think I'd messed up our VPC config.

The part about the network hiccup triggering a full failover even though the instance is healthy really resonates. It makes our private app access feel so fragile. Did you find anything that helped?

You're definitely not alone. That specific "healthy instance, flapping connection" pattern is a classic sign the built-in watchdog timer is too aggressive for your network's normal jitter.

I'd skip tuning TCP settings and go straight to the source: the systemd service override for the connector. Increasing the `--tunnel-keepalive` value is the most direct fix. We've had clients settle anywhere between 90 and 150 seconds, up from the default 60. It gives the broker connection more breathing room to recover from a minor blip.

But here's the critical tradeoff: a less aggressive watchdog means a genuinely dead connector takes longer to be replaced. You absolutely must pair this change with a health check that actually monitors broker connectivity for your ASG, not just the instance status. Otherwise, you're trading spurious failovers for longer outages when a real failure occurs. Did you set up a custom health check script yet, or are you still relying on the ELB checks?

Oh man, that exact log line is a classic. Been there, felt that pain.

The main culprit is almost always the watchdog timer being way too twitchy for normal cloud jitter. Everyone's spot-on about adjusting the systemd service override for the tunnel-keepalive. We bumped ours to 120 seconds and it was like night and day.

But you gotta remember the trade-off: you're trading flapping for a slower recovery if a connector actually dies. Pair that change with a custom ASG health check that pings the broker, not just the instance. Makes the whole setup way more resilient.

Yes, there's a specific watchdog log event you can grep for. Look for `Watchdog is restarting the service because ...` in `/var/log/zpa/connector/logs/system.log`. In our case it was "broker heartbeat timeout".

The key is the watchdog kills the whole connector process if *any* of its internal health checks fail, management or data path. So your separate ALB for metrics cutting the management API is treated as a total system failure. It's binary.

Ah, the binary failure logic. That's the most frustrating part of ZPA's design. "broker heartbeat timeout" triggers a full restart, but there's no way to differentiate between a brief network burp and a genuine broker outage.

I'd push back a bit on grepping just that one log line, though. In our environment, the root cause was often logged minutes *before* the watchdog event. The system.log will show escalating latency or repeated TCP retransmits leading up to the final heartbeat timeout. If you only look for the killshot, you miss the real pattern.

So yeah, the watchdog's all-or-nothing approach is the problem, but you need a wider log tail to see what's actually provoking it.

Everyone's fixated on tweaking the keepalive timer. That's treating the symptom. The real problem is your health check architecture.

> even though the instance itself is healthy

Exactly. Your ASG is using EC2 status checks, which only see the VM. It doesn't know the broker connection is dead. The connector's own watchdog is the only thing detecting that, and it's too binary.

You need a custom ASG health check that probes the broker reachability from the instance. Run a script via SSM Agent or a sidecar container that hits the broker FQDN and fails the instance health if it's down. Let the ASG handle the replacement, not the twitchy internal watchdog.

Don't just make the watchdog slower. Decouple the detection from the action.

You've hit on the core issue in your last sentence: a network hiccup triggers a full failover on a healthy instance. The watchdog's binary logic is to blame, as others said.

But *don't* start by tweaking `--tunnel-keepalive`. That's step two. First, implement a custom health check for your ASG that actually tests broker connectivity, like a simple script that curls the broker FQDN. Let the ASG handle the replacement based on that, not the twitchy internal process. This decouples detection from the action.

Only after that's in place should you consider adjusting the watchdog timer to be less sensitive, because now your ASG has a real view of broker health. Without that, you're just hiding the problem.

Completely agree on decoupling detection from action, it's the right architectural fix. Your point about implementing the custom health check *first* is crucial.

But I've seen teams stumble on a practical detail: what's the actual curl endpoint or check? The broker FQDN alone might not be enough. You need to verify the specific management port the connector uses is responsive, not just that the host is up. We've used a simple netcat check on the broker's provisioned port as the script's test.

Then you can safely make the internal watchdog less aggressive, because the ASG now has the real signal.

Your detailed description of "a network hiccup is enough to trigger a full failover" is the precise symptom. You've correctly ruled out the typical infrastructure culprits, which isolates the problem to the connector's stability logic.

The core issue is the watchdog's binary response to the broker heartbeat check. It treats any timeout as a catastrophic failure, forcing a restart. While adjusting the `--tunnel-keepalive` parameter, as others have mentioned, can provide immediate relief by increasing the tolerance window, it merely masks the underlying detection flaw.

The more sustainable solution is to implement a custom CloudWatch metric and ASG health check based on a script that directly probes the broker's management interface from the connector instance. This externalizes health detection. Once that's in place, you can then safely increase the internal keepalive timer, effectively creating a two-layer health system: the ASG handles actual broker connectivity failures, while the less aggressive watchdog only intervenes for true process hangs. Without that external check, you're just increasing the mean time to recovery for a genuinely dead connector.

Ugh, the AMI. Everyone leans on that like it's a silver bullet, but I've seen that exact setup become the problem.

Your config looks textbook, which means you're fighting the defaults. The "recommended instance types" often ship with the default watchdog timer, which is famously intolerant of any cloud network variability. It treats a temporary broker blip like a total catastrophe.

Forget tuning keepalives first. That's just putting a bandage on a bad detection system. You need to stop letting the connector's own watchdog dictate your ASG's fate. The real question is why your ASG health check is just looking at EC2 status instead of the actual broker connection. Offload the liveness probe to something external, then you can safely loosen the internal timer without risking a zombie connector.