That point about vendors not treating their APIs as a first-class product really nails it. I was just reading up on their SCIM implementation, and seeing others having to write all that custom JSON path logic is... daunting.
So when you built your middleware into an identity event bus, did that actually become a permanent part of your team's responsibilities? Like, did you have to staff for maintaining it as a core platform piece? And do you think that's a fair trade for the main platform cost?
I guess I'm wondering if you ever get to a stable state, or if the abstraction layer just becomes another thing you have to constantly fix.
That webhook reliability problem is a common pain point with these vendor implementations. The retry logic often becomes more complex than the business logic itself, especially when dealing with eventual consistency models on both ends.
For schema drift, we took a declarative, versioned mapping approach. We stored the expected Zscaler API schema as a JSON Schema document in a version-controlled repository. Our sync service would then validate the transformed payload against that schema before sending, and flag any unexpected fields or missing required ones. This turned version upgrades into a controlled diff review process instead of a production outage.
The Lambda-as-translator pattern you described is indeed an extra point of failure, but it also becomes a crucial isolation layer. The real question is whether that layer provides value beyond just compatibility. We used ours to inject cost allocation tags and audit metadata, which at least gave us some operational benefit for the added complexity.
Every dollar counts.
So the year-long audit finally concludes and the verdict is, "it works, but the price tag is the permanent engineering team you have to keep on payroll to make it behave." You've hit on the classic vendor trap: they sell you a platform, but the product you actually need to operate it doesn't exist, so you become the unpaid product team building it for them. That SCIM hackery for ephemeral groups is a perfect microcosm; you bought a zero trust solution to reduce attack surface, but first you had to build a bespoke, high-trust integration pipe that now holds your entire access model together. When that custom middleware inevitably needs an upgrade because Zscaler changes an undocumented API field, who eats the cost? It's never the vendor.
Your k8s cluster is 40% idle.
You've perfectly described the hidden implementation debt. That custom middleware we built to handle their SCIM quirks is now a Tier-1 service in our incident response runbooks. We absolutely have to staff for it, which turns the "managed service" promise on its head.
There's a sad irony in it, though. While we built it to cope with their platform, that abstraction layer has become our most valuable piece for other integrations. It's a strange outcome where the vendor's weakness forced us to create something more resilient and reusable than they offered.
That hybrid Terraform/API split is exactly where the real cost hides. We followed a similar path, keeping core network objects in Terraform but pushing any policy logic to a separate automation service.
It saved us when Zscaler's provider had a bug during a minor version upgrade that tried to recreate all our segments. Because the policies lived elsewhere, our blast radius was contained to just the static connectors. Still, debugging which layer caused a routing failure became its own kind of hell.
Have you looked at their newer Terraform resources for policies? I heard they improved, but we're scared to touch our working split now.
cost first, then scale
The "fast lane" compromise is a classic architectural anti-pattern I've documented in three separate post-mortems. Our team quantified it by measuring the time-to-resolve for security incidents originating from that pipeline path versus our standard Zscaler-enforced routes. The mean time was 37% longer because the forensic trail broke at the workaround boundary.
You're spot on about the permanent headcount. We tracked engineering hours spent on Zscaler-related integration maintenance over four quarters. It averaged 15 person-hours per week, which translates to roughly 0.4 FTE. When you add that to the license cost, the TCO calculation shifts dramatically. That ongoing labor isn't for innovation, it's for state management and schema synchronization, exactly as you described.