Hey folks, hoping to tap into the collective experience here. We've been running a Zscaler Private Access connector in our self-managed Kubernetes cluster for a few months now, and while the overall zero-trust model is solid, we're hitting a persistent and frustrating issue: the connector pods keep failing their internal health checks, leading to service disruptions. The status in the ZPA admin portal flips between "Healthy" and "Unhealthy" seemingly at random.
We're on the official Helm chart (version `zpa-connector-2.2.0`), and our cluster is a standard, on-premises deployment running Kubernetes 1.27. The infrastructure itself is stable—other critical workloads have no issues. The problem appears isolated to the ZPA connector's own liveness/readiness probes.
Here’s a breakdown of our setup and what we’ve tried so far:
* **Current Configuration:** We're using mostly default values from the Helm chart, with adjustments for resources and node selectors. The health checks are the default HTTP GET on port `8080` at `/health`.
* **Observed Symptoms:**
* The pods run, but logs show periodic `HTTP 503` responses from the health endpoint.
* `kubectl describe pod` often shows: `Liveness probe failed: HTTP probe failed with statuscode: 503`.
* There's no obvious pattern linking it to CPU/Memory pressure (we've monitored with Prometheus/Grafana).
* **Troubleshooting Attempts:**
* Scaled up CPU/Memory limits significantly (beyond what's documented) to rule out throttling.
* Adjusted probe `initialDelaySeconds` and `periodSeconds` to be more lenient.
* Verified network policies allow the kubelet to hit the pod on port `8080`.
* Checked for any anti-affinity or node taint issues.
Our current `values.yaml` overrides for the probes look like this:
```yaml
probes:
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 15
failureThreshold: 3
```
Has anyone else battled this in a self-hosted K8s environment? I'm particularly curious if:
1. There are known issues with specific K8s network plugins (we're using Calico) interfering with the connector's internal health service.
2. The `/health` endpoint has dependencies (e.g., outbound calls to Zscaler cloud) that, if slightly delayed, cause a 503.
3. There's a more optimal way to structure the probes or pod resources that we've missed.
We're considering moving the probes to a different, simpler endpoint or even implementing a custom sidecar just for health reporting, but that feels like working around the problem. Any insights, config snippets, or war stories would be hugely appreciated.