Another year, another platform migration, and here I am again staring at a cloud bill that makes me wince. This time it's Zscaler Private Access, specifically the cost of those Connector VMs that never seem to scale down when you actually need them to. You know the drill: you provision for peak, and then you're paying for that idle capacity 18 hours a day, like renting a banquet hall to eat a single takeout meal.
The official line is to manually adjust your Connector fleet, but who has the time? And if you miss a spike, your sales team is on Slack screaming about not being able to reach the legacy deal desk app. So, after the last billing cycle shock, I built a system to auto-scale the ZPA Connectors in Kubernetes. It's not for the faint of heart, but it cut our monthly ZPA infra costs by roughly 40%. The core idea is simple: treat the Connectors as stateless, ephemeral pods and let K8s handle the scaling based on real-time metrics.
Here’s the gist of the architecture:
* **Containerized Connectors:** You need to build a Docker image for the ZPA Connector. This involves scripting the provisioning process (using the ZPA API) so a pod can bootstrap itself and join the correct App Connector Group. The secret sauce is ensuring it cleans up after itself on termination.
* **Custom Metrics & HPA:** The vanilla CPU/Memory metrics are useless here. You need to scale based on concurrent active tunnels or pending requests. I set up a sidecar that scrapes the Connector's local stats API and exposes them to Prometheus. Then, a Prometheus Adapter lets the Kubernetes Horizontal Pod Autoscaler make decisions.
* **Tricky Parts & Pitfalls:**
* **Boot Time:** A new Connector takes 2-3 minutes to provision and be ready. Your scaling policy needs a buffer, or you'll get throttled during a fast ramp-up.
* **Session Persistence:** You must ensure existing user sessions aren't massacred when scaling down. A graceful termination period (coupled with ZPA's own failover) is critical. We use a `preStop` hook to signal the Connector to start draining.
* **Cloud Costs vs. Licensing:** Remember, ZPA licensing is separate. You're saving on the underlying compute/storage, not the Zscaler SKU. This makes the most sense on a cloud with per-second billing.
The actual scaling policy in our cluster looks something like this, targeting an average of 150 active tunnels per pod before spinning up another:
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: zpa-connector
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: zpa_active_tunnels
target:
type: AverageValue
averageValue: 150
```
Is it worth the operational overhead? If your Connector count is static and small, probably not. But if you're supporting a global, fluctuating workforce with a dozen-plus Connectors, the savings are substantial. You're essentially trading manual VM management for a more complex, but automated, orchestration layer. Just be prepared for a few nights of debugging why a pod won't join the provisioning group before you get it right.