Another year, another platform migration. I’ve just finished dragging a creaking, three-year-old EKS cluster kicking and screaming into a world with actual security standards, and let me tell you, the process was about as elegant as a HubSpot to Salesforce data sync during a solar flare. Everyone’s talking about Pod Security Standards (PSS) like it’s a magic switch you flip, but on a live cluster with legacy workloads? It’s a delicate archaeology dig where every shovel strike might unearth a production-halting YAML artifact.
The core dilemma, as always, is the brutal trade-off between theoretical best practice and operational continuity. The Kubernetes community’s shift from the deprecated Pod Security Policies (PSP) to the admission-controller-based PSS is a genuine improvement in design—I’ll give them that. But implementing the `restricted` profile on an existing cluster isn't a step; it's a multi-stage siege. You will find things. Unpleasant things.
My approach, forged in the usual fires of regret, was a phased rollout that prioritized observation over enforcement. Here’s the sequence that finally worked without causing a total revenue ops blackout:
* **Namespace Labeling as a Canary:** Before you even touch the admission controller, start by auditing and labeling your namespaces. Apply `pod-security.kubernetes.io/warn: restricted` and `pod-security.kubernetes.io/audit: restricted`. This does nothing but log policy violations. The logs will become your grim to-do list. You’ll likely discover:
* Pods running as root because some 2019 Helm chart demanded it.
* Mounts of `/` or `/etc` because someone thought it was an easy config shortcut.
* Privileged escalation all over the place in your CI/CD tooling pods.
* **The Enforcement Two-Step:** Once you’ve fixed (or, let’s be honest, documented and risk-accepted) the audit findings, you switch the namespace label to `pod-security.kubernetes.io/enforce: restricted`. But you do it namespace by namespace, starting with the least critical. This is where the "existing cluster" pain peaks. You can’t just apply it cluster-wide unless you enjoy 3 AM pages.
* **The Exemption Trap:** The `exemptions` section in the PSS configuration is your necessary evil. You’ll need it for system namespaces (kube-system, etc.) and probably for that one ancient, business-critical deployment that no one has the source code for anymore. This feels like cheating, and it is. It’s creating the very kind of loophole the standard seeks to close, but that’s the tax for technical debt.
The real lesson, which mirrors my perennial CRM hopping, is that a security model bolted on later is always more painful than one designed in from the start. EKS’s managed control plane doesn’t absolve you of this work; it just provides a slightly smoother path to run the commands. The outcome is more secure, yes. But the process is a stark ledger of every past shortcut and “just get it deployed” decision. Now, if you’ll excuse me, I need to go see if the new Azure Kubernetes Service documentation promises a less arduous path for next year’s inevitable migration.
Ugh, the HubSpot to Salesforce sync comparison is painfully accurate. It's that same feeling of uncovering a five-year-old "temporary" workflow that's now mission-critical. I'm really curious about your phased rollout, especially the namespace labeling. Did you find it better to start labeling non-production namespaces as "restricted" first as a test, or did you go with "baseline" across the whole board to get a wider net for warnings? That initial audit phase is everything.
Your phased rollout strategy is the only sane approach. The critical detail I've seen teams miss is quantifying the blast radius before applying that first `pod-security.kubernetes.io/enforce` label. Simply running `kubectl debug` or the audit mode isn't enough for a business case.
You need to map every potential violation to an actual cost unit: the engineering hours for remediation per team, the risk cost of a temporary exception, and the infra cost if a pod gets evicted and auto-scales a new node. Did you track the number of pod restarts or failed deployments during your warning phase as a metric to pressure reluctant application owners? That data turns "theoretical best practice" into a tangible operational budget line item.
CostCutter
Totally agree on the phased rollout being key. The "observation over enforcement" mindset saved us too.
One thing I'd add - that initial audit phase is also the perfect time to hook up your CRO or observability stack. We used it to tag non-compliant deployments in our analytics, creating a simple dashboard showing which teams or services were generating the most warnings. Made the "cleanup" phase much easier to prioritize and socialize.
Did you find any particular violation kept popping up as your "most common offender"? For us, it was always the `runAsNonRoot` setting on old utility containers.
✌️