My results after implementing OPA Gatekeeper - 90% reduction in misconfigurations

So I’ve been running a pretty sprawling multi-tenant EKS setup on AWS for about a year now, and the configuration drift and “oops” moments from various teams were becoming a real time sink. We had everything from overly permissive IAM roles in pod specs to missing resource limits and even the occasional namespace label mismatch that broke our networking policies. Manual reviews and post-deployment cleanup were not scaling.

I decided to implement OPA Gatekeeper as our policy engine, framing it as a guardrail system rather than a blocker. The goal was to shift left, but also to enforce a hard floor for safety and cost control in production. After three months of running it across all our clusters, I’ve seen a **90% reduction in misconfiguration-related incidents** (tracked via our event logs and Slack alert volume). That’s not just theory—it’s a tangible drop in firefighting.

The journey wasn’t just “install and forget.” Here’s the rough breakdown of what we did:

* **Started with the critical constraints:** We didn’t try to boil the ocean. Phase 1 was all about safety and cost.
* **Require resource limits** on all Pods. This alone stopped several memory-hog deployments from taking down nodes.
* **Block the use of the `default` namespace.** A simple constraint that eliminated so much clutter.
* **Enforce a mandatory `cost-center` label on all Namespaces.** This tied into our showback system.
* **Built a library of reusable ConstraintTemplates:** This is where the power really is. Writing a good template lets you create many constraints from it. For example, we have a generic `containerresourcelimits` template that we now use for CPU, memory, and even ephemeral storage checks across different kinds of workloads.
* **Integrated into the CI/CD pipeline:** The `gatekeeper` CLI tool runs a `dry-run` in our pull requests. This gives developers immediate feedback in their merge request, which has been huge for adoption. They fix things *before* the deployment even attempts to hit the cluster.

Here’s a snippet of the ConstraintTemplate we use for the namespace label requirement. It’s fairly straightforward but demonstrates the Rego logic:

```yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("you must provide labels: %v", [missing])
}
```

And then the actual Constraint that uses it to mandate the `cost-center` label:

```yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: namespace-must-have-cost-center
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
parameters:
labels: ["cost-center"]
```

The biggest lesson? **Start with audit mode first.** We deployed all our initial constraints with `enforcementAction: dryrun` for a couple of weeks. This generated a huge list of existing violations in the logs without breaking anything. It gave us data to clean up legacy cruft and to socialize the *why* before flipping the switch to `deny`. The communication and gradual rollout were as important as the tech.

We’re now looking at more complex policies, like ensuring all ingress hosts are unique across namespaces and validating that storage class parameters are within our approved set. The framework Gatekeeper provides makes adding these new rules surprisingly manageable. Has anyone else gone deep on composing policies with the `inventory` feature for cross-object checks? I’m experimenting with that next.