ELI5: How does a policy engine work with admission webhooks?

Great questions. Let's walk through the flow you're asking about.

The policy engine usually runs as a set of pods (like Gatekeeper's controller manager). The webhook configuration you set up tells the API server, "hey, before you accept any pods, call this service over here." The webhook is essentially the bouncer at the door, but it's just passing the request to the policy engine pod to make the actual decision. So the engine does the logic and tells the webhook what to reply with.

On your failure scenario, that's super important. The webhook configuration has a `failurePolicy` field. If it's set to `Fail` and your engine pod crashes, the API server will reject the request outright (a "fail closed" security stance). If it's set to `Ignore`, the request might be allowed through, which is safer for availability but riskier. It's a classic trade-off you have to configure based on what you're protecting.

Yeah, the `failurePolicy` trade-off is real. I've seen teams set it to `Fail` in staging to catch everything, then accidentally push that config to prod. Cue the midnight "why are all deployments blocked?" pages 😅

It's a good reminder to treat that config like any other critical security rule - test the failure modes.

Oh that's a good breakdown. The part I always mix up is who calls who. So the API server calls the webhook, and the webhook service is basically the "front door" for the policy engine pods? That makes sense.

One thing I'm still shaky on: where does the actual policy *store* live? Like, if I write a rule in Rego, is it baked into the engine pod's image or does it fetch it from a ConfigMap?

Yep, you've got the flow exactly right, like a relay handoff.

On where policies live, it's actually pretty cool. For OPA/Gatekeeper, they're stored as Custom Resources - ConstraintTemplates and Constraints. So they live in the Kubernetes API itself, not in the pod image. The engine watches for those resources and loads them dynamically. Kyverno works similarly with its Policy resources.

That's a key difference from something hardcoded in a ConfigMap. You can update policies on the fly without rebuilding anything.

Exactly! Storing policies as Custom Resources is one of my favorite parts of this pattern. It turns your policy definitions into first-class Kubernetes citizens you can manage with `kubectl`, audit, and even RBAC control.

One neat trick this enables: you can have a GitOps pipeline that commits policy changes to a repo, and then a sync tool like ArgoCD or Flux applies the updated Constraint YAML. It creates a full audit trail from merge request to cluster enforcement.

Just watch out for the sync delay though. If your policy engine pod is under load or your controller is lagging, there can be a few seconds where a new policy isn't active yet. Always test with a `kubectl get constraints` and a dry-run request after you push a change.

That GitOps audit trail is a massive win for compliance. I've had to pull reports for audits before, and being able to point to the Git commit that applied a constraint, and the controller logs showing it being loaded, saved us days of work.

Your point about the sync delay is critical, though. Teams can get bitten thinking it's instantaneous. I'd add that you also need to consider the order of operations in your pipeline. If you're syncing a constraint and a workload change in the same PR, which gets applied first? You might need stages or explicit dependencies to make sure the policy is live before the deployment hits the API server.

You're absolutely right about the pipeline order. We got burned by that exact scenario - a PR with a new constraint and a deployment YAML, and the deployment tried to sync first because of alphabetical order in the directory. It sailed right through.

Now we enforce naming conventions in our policy folders to control sync order, like `01-constraints/` and `02-deployments/`. Some teams use ArgoCD's sync waves, but the folder trick has been simpler for us to keep straight.

That audit trail is a lifesaver, but only if the policy actually caught the thing you're being asked about.

Oh, the alphabetical order trap is so real! We used a similar folder-numbering system after hitting the same wall.

It reminds me of A/B test tool rollouts, actually. You'd never enable a new targeting rule and launch the campaign in the same pipeline step - you stage it. Same principle here. The sync waves feature is perfect for this, but I get why teams stick with the folder trick. It's visible right in the repo structure.

That last line is the kicker, though. An audit trail for a policy that didn't fire is just a record of the miss. Makes me think we should maybe add a validation step in the pipeline that does a dry-run apply against a test workload to confirm the constraint is active before any real deployments sync.

You've got the right questions. Let's map the flow precisely, as if we're tracing a network packet.

Think of it as a three-step relay. First, the API server has a static list of `ValidatingWebhookConfiguration` objects that tell it, "for CREATE/UPDATE operations on these resource types, make an HTTP POST to this `service`." That service is a Kubernetes Service, which acts as a stable network endpoint for a set of pods. Those pods are your policy engine (like Gatekeeper's controller manager). So the engine lives as a separate deployment, yes.

When the API server calls that webhook service, the request lands on one of the policy engine pods. That pod runs the logic - evaluating your Rego or Kyverno rules against the incoming object. The engine pod itself crafts the final HTTP response back to the API server, with an `allowed: true/false` and a reason. The webhook configuration is just the doorbell; the engine is the person who answers and makes the decision.

Your failure scenario cuts to the heart of the design. The `failurePolicy` on the webhook configuration dictates the API server's behavior when the webhook service is unreachable. If set to `Fail`, the request is denied (fail-closed). If set to `Ignore`, the request skips the check and proceeds (fail-open). This is why monitoring the health of those policy engine pods is as critical as monitoring your API servers.

Great, you're asking about the actual plumbing. Let's trace the request path.

The policy engine is indeed a separate deployment, usually a `Deployment` with one or more pods. The API server knows to call it because of a `ValidatingWebhookConfiguration` resource you've installed. That config acts as a registration form, telling the API server, "for any CREATE of a Pod, send it to this internal service URL." That service is the webhook endpoint, which is just a Kubernetes `Service` pointing to those engine pods.

The webhook service is just the door. The engine pod behind it does the real work, running the Rego logic against the object and crafting the HTTP 200 (allow) or 403 (deny) response. So the webhook is the mechanism, but the engine makes the decision.

On your crash scenario, it hinges on the `failurePolicy` field in that webhook configuration. If set to `Fail`, a crashed pod means the webhook call fails and the request is denied. If set to `Ignore`, the API server proceeds as if the webhook allowed it. That's why a staging `Fail` config pushed to prod can cause a total deployment blackout.

Totally agree on the audit trail being a compliance lifesaver. That same pattern saved our team during a PCI audit - we could literally show the commit hash, the sync event in Argo, and the controller log line where the constraint loaded, all in a single timeline.

Your pipeline order point is spot on. We learned the hard way that even with sync waves, you can have race conditions if your policy engine's controller reconciliation loop is slower than your app deployment. Sometimes the constraint resource exists, but the controller hasn't processed it yet. Our solution was to add a simple readiness check in the pipeline that polls the engine's metrics endpoint for that specific constraint's status before allowing deployments to proceed.

Oh, polling the metrics endpoint for constraint status is a clever fix! We ended up doing something similar, but with the engine's health check API. Our GitOps pipeline would wait until the constraint showed up in `kubectl get constrainttemplates -o json` with a `status.byPod[0].status` of "active".

It adds a few seconds to the pipeline, but it's way better than the alternative. Did you find the metric polling added any noticeable latency to your deployments?

Latency from polling is trivial compared to the risk window. Your pipeline waits a few seconds, but a new constraint that's still loading can miss dozens of deployments if you have high CI/CD velocity.

Status checks are good, but they're a band-aid. The real fix is designing constraints to be backward compatible. Never deploy a new constraint that would break existing, compliant workloads. Deploy it, let it become active, *then* roll out the workload changes.

Relying on polling assumes the engine's health endpoint is perfect. If it's bugged and reports "active" prematurely, you're back to square one.

Exactly, that `failurePolicy` is the crucial safety catch. I've seen teams get caught out because they set it to `Ignore` for development, expecting failures to be temporary, but then it propagates to production configs by accident.

A trick we used was to explicitly set the `failurePolicy` in our Helm values per environment. In prod, it's always `Fail`. In staging, we'd sometimes set it to `Ignore` but with a clear comment and a pipeline check to prevent promotion. It's a simple field but it's the difference between "fails safe" and "silently allows everything when the engine crashes".

Do you know if any of the major policy engines have a default `failurePolicy` when you install them? I always check, but I'm curious if there's a standard.