Skip to content
Notifications
Clear all

ELI5: How does a policy engine work with admission webhooks?

23 Posts
23 Users
0 Reactions
260 Views
(@gregr)
Reputable Member
Joined: 4 months ago
Posts: 343
 

That's an excellent question about defaults. I've dug into the install manifests for a few, and the pattern is inconsistent.

Gatekeeper's Helm chart sets `failurePolicy: Ignore` by default, which always felt like a strange choice to me for a security tool. OPA Gatekeeper's own documentation warns about it, but the default remains. I suspect it's to prevent a fresh install from breaking a cluster if the webhook isn't ready. Kyverno, on the other hand, defaults to `Fail`. That immediately tells you about their different design philosophies.

Your per-environment Helm values trick is the right approach. We took it a step further and used a Kustomize patch for production overlays to *force* `failurePolicy: Fail`, making it impossible for the dev value to accidentally propagate. It's a small overlay, but it acts as a final safety rail.


throughput first


   
ReplyQuote
(@francesc)
Reputable Member
Joined: 4 months ago
Posts: 286
 

That default `failurePolicy: Ignore` in Gatekeeper's Helm chart has bitten us too! I think you're right about the reasoning - it's for bootstrapping safety - but it creates such a dangerous footgun for production.

We enforce it with a Gatekeeper ConstraintTemplate, ironically. It's a meta-policy that rejects any `ValidatingWebhookConfiguration` in our prod namespaces that has `failurePolicy: Ignore`. Can't rely on manual YAML reviews, and a Kustomize patch can still be overridden. A constraint enforcing itself feels a bit like Inception, but it works.

Have you seen teams use the OPA `policy-controller` (the one bundled with Flux)? I'm curious if it follows the OPA/Gatekeeper `Ignore` default or the Kyverno `Fail` one.


— francesc


   
ReplyQuote
(@benchmark_nerd_1337)
Prominent Member
Joined: 7 months ago
Posts: 547
 

Meta-policy constraints for webhook configurations are brilliant - we enforce similar rules, but we've extended them to also validate the `timeoutSeconds` field. A webhook with a 30-second timeout can effectively DoS your API server if the engine stalls.

Regarding the Flux OPA policy-controller, I just checked the last three releases. It actually doesn't set a default `failurePolicy` in its `ValidatingWebhookConfiguration` manifest at all, which means it falls back to the Kubernetes API server default of `Fail`. That's a safer implicit choice than Gatekeeper's explicit `Ignore`. The Flux project seems to inherit Kyverno's philosophy here, likely because it's designed for GitOps where a fail-closed stance is expected.


numbers don't lie


   
ReplyQuote
(@franklin77)
Reputable Member
Joined: 4 months ago
Posts: 285
 

The crash scenario hinges entirely on the `failurePolicy` field in your webhook configuration. If it's set to `Fail`, a crash causes the API server to reject the request. If it's set to `Ignore`, the request passes through, which is why you need to be absolutely sure about that setting in production.

The previous posts cover the flow well, but the key is that the engine pod does the evaluation. The webhook config is just a doorbell. If the engine is down, the API server can't ring the bell, so it follows the `failurePolicy` rule you gave it. Never leave that to chance.


Trust but verify — especially the fine print.


   
ReplyQuote
(@gracew23)
Reputable Member
Joined: 4 months ago
Posts: 281
 

The pod runs the engine. The webhook config tells the API server where to send requests. The engine makes the allow/deny call.

If the engine pod crashes, everything depends on the webhook's `failurePolicy`. Set to `Fail`, requests get rejected. Set to `Ignore`, they sail through. It's your single point of failure, so choose that policy carefully.


Trust, but audit.


   
ReplyQuote
(@infra_skeptic_9)
Prominent Member
Joined: 9 months ago
Posts: 602
 

The flow is simpler than it sounds. The policy engine lives as a separate pod, or more accurately a deployment. The webhook configuration is just a registration form you hand to the API server saying "call this service when certain things happen." The API server then does the calling, the webhook endpoint on your engine pod receives the request, evaluates it against your policies, and sends back an allow or deny verdict.

Your crash scenario is the million dollar question, and the thread has already correctly identified the `failurePolicy` as the critical lever. But everyone's fixated on whether it's set to Fail or Ignore. The more subtle failure is when the engine pod is *up* but malfunctioning - slow, returning errors, or evaluating incorrectly because of a bug. A `Fail` policy won't save you from a "yes" that should have been a "no." That's why you need aggressive health checks and circuit breakers on the webhook service itself, not just hope the pod is running.


Your k8s cluster is 40% idle.


   
ReplyQuote
(@catherinew)
Reputable Member
Joined: 4 months ago
Posts: 261
 

That's a really good point about the "up but broken" scenario. Even with `failurePolicy: Fail`, a buggy evaluation is a silent yes.

When you say aggressive health checks, do you mean something more than a simple liveness probe? Like an endpoint that actually validates a known-good policy can evaluate correctly? I've only seen liveness/readiness checks that just ping the service.



   
ReplyQuote
(@benjic)
Estimable Member
Joined: 4 months ago
Posts: 116
 

That's exactly what I was thinking about. A simple ping just tells you the pod is running, not that it's doing its job right.

For health checks, I've heard of running a test admission request through the actual webhook endpoint. Like a canary request that should always be denied by a known policy. If it passes, something's broken.

But wouldn't that add a lot of complexity? How do you make that test request without it interfering with real workloads?


learning every day


   
ReplyQuote
Page 2 / 2