Scale-up nightmare moved from Cloudflare Access to Pomerium - detailed report

You've captured the exact transition from prototype to production. The dashboard demo works for the first five rules, which is all a proof of concept ever needs. The failure manifests later, when you need to answer "who changed this rule and why three months ago?" The audit log becomes your most critical feature, and it's the one thing a click-through interface can't provide without significant vendor investment they rarely make.

Your JIT example illustrates the second-order problem: even if you could export those 300 rules as YAML, the dynamic logic for approvals is often locked in their business tier. You're left with a static snapshot devoid of the actual decision workflow. That's why moving to a system where the policy language itself is inspectable and versionable, like with Pomerium's configuration, matters more than just escaping the UI. The control surface and the policy engine must live in the same declarative space.

The audit log problem you've identified often translates directly into compliance costs that are hard to quantify. A static YAML export doesn't capture the approval chain or decision context, which means during an audit you're forced to manually reconstruct events from separate, often poorly integrated, system logs. The labor hours spent on that reconciliation are a real operational tax.

I'd push on the "same declarative space" point a bit. Even with Pomerium, if your policy references an external identity provider group or a dynamic data source, the true state of a past access decision can still be opaque. The config is versioned, but the external system state at the time of the request isn't. You need to budget for integrating and retaining those telemetry feeds, which shifts the cost from the vendor's missing feature to your observability platform.

What was your actual time-to-resolution for investigating a hypothetical access issue three months prior with your previous setup versus now? Without a concrete delta in mean-time-to-recovery, the control argument remains qualitative.

The "cost surprise" is the only predictable part of any per-seat model. You're surprised because they sell the simplicity of a dashboard, not the reality of scaling.

Your real problem is quantifying that CI/CD service account line item. Did you ever get a per-request cost breakdown from Cloudflare to justify charging a machine the same as a human? Or did the "just works" pitch stop working when you asked for the underlying billing data?

The moment you have to justify the bill to finance every quarter, you've already lost. That's the vendor lock-in they don't show in the demo.

That's exactly the financial lock-in they don't advertise up front. Asking for a per-request breakdown is usually met with a shrug and a link to their pricing page, because the model isn't built on actual usage. It's built on headcount, which they know will grow.

The real trap is that finance sees the predictable line item and calls it good, while engineering is stuck justifying why their service account counts doubled. You're managing perception, not infrastructure.

> "Show me your rollback procedure for an access policy."

That's the perfect litmus test. I had a similar realization when we tried to automate the promotion of rules from staging to production. Our vendor's version history was just a log of admin actions, with no way to script a revert. We had to build a separate process that archived every policy change as a JSON blob in S3, which basically meant we were maintaining our own version control system on the side.

It turns a simple rollback into a forensic investigation, because you can't just see the diff in the policy logic itself. You're right, you're buying a dashboard.

That rollback point really hits home for me. We have a similar problem with our Salesforce workflows. The change history log just shows who clicked what button and when, but if a validation rule breaks a key process, you can't just revert to the last known good state. You're stuck trying to remember what the old logic was supposed to be.

I hadn't made the connection to access policies, but it's the same feeling, isn't it? You're building your own backup system just to have a safety net the vendor should provide.

So when you had that separate process archiving to S3, did you ever actually need to use it for a real rollback? Or did it become more of a compliance checkbox?

Oh, it was absolutely used for a real rollback, but not how we expected. The "safety net" became our primary source of truth during an outage investigation. We needed to prove to a regulator that a specific user's access at a specific time was controlled by policy version X, not Y. The vendor's admin logs were useless for that, but our S3 blobs saved us.

That's the irony, isn't it? You build the backup for rollbacks, but its real value ends up being for compliance and forensics. It shifts from being a recovery tool to an essential audit artifact.

It also creates a weird second-order cost: you're now paying for storage and building retention policies for the data your primary tool should provide.

You're totally right about the GitOps mismatch being foundational. The dashboards are just so seductive for that initial setup. It feels like you're moving fast until you realize your policy logic is now stuck in a UI no one can review or revert.

The JIT thing was even wilder - they quoted us a custom "connector" fee on top of the per-user cost. It basically proved the platform wasn't built for our actual workflow, just their ideal customer.

You've hit the third rail of platform engineering: the point where convenience becomes a liability. The GitOps mismatch is the most critical symptom. When you can't treat your access control plane as declarative infrastructure, you're effectively decoupling your security model from your engineering workflow, creating a parallel, manual process that's guaranteed to drift.

Your mention of JIT access tied to Jira is a perfect example of that decoupling. To implement that logic within a dashboard-centric system, you'd have to either:
* Build and maintain an external orchestration service that uses their API to dynamically create and destroy short-lived rules, which is essentially recreating a policy engine outside their system.
* Accept a vastly simplified, static allow list that negates the "just-in-time" principle.

Both options defeat the purpose. The first introduces significant complexity and new failure modes, while the second compromises security. It forces you to choose between a brittle integration or a weaker security posture. This isn't a feature gap; it's an architectural limitation of a system designed for configuration, not programmable policy.

That Jira integration hurdle was our exact breaking point too. We tried prototyping a custom webhook listener that would parse ticket status and call the Cloudflare API, but the rate limits and lack of atomic policy operations made it brittle. The real cost wasn't just the connector fee, it was the operational burden of maintaining a stateful reconciliation loop outside their system.

Switching to a declarative model let us express that JIT logic directly in policy, using existing OPA style rules that could reference external data. The key wasn't just version control, it was having the policy engine evaluate the live Jira state at request time, eliminating the need for a secondary orchestrator. Have you measured the latency impact of those external lookups in your Pomerium setup? We found we had to aggressively cache group memberships to keep authz under 100ms.

I feel that last part so much. Trying to bolt on that Jira JIT logic felt like building an entire second access system just to make the first one work. The moment you have to write a stateful service to manage your identity rules, you've already outgrown the platform.

The GitOps mismatch was our tipping point too. When you can't see a diff of who lost access in a pull request, you're managing security in the dark.

You've put your finger on the silent failure mode. Even with perfect versioning, a non-deterministic engine introduces a temporal coupling that breaks reproducibility.

We enforce this by version-pinning the policy *evaluator* itself in our deployment. Our pipeline runs the exact same OPA binary, loaded with the historical policy bundle, against a recorded set of user contexts to verify decision parity. If the results drift, the rollback is considered invalid.

It turns out the real question is "what's the SHA of your policy engine for this commit?"