We've been implementing Sysdig Secure across our Kubernetes data platform (primarily Spark jobs, Kafka streams, and supporting services) and have hit a predictable operational snag. Our vulnerability scanning and runtime policies are generating significant alert volume and resource overhead in our development and staging namespaces. While we need strict policies for production, we want to exclude non-production workloads from certain expensive checks, like deep package scanning on every pod deployment.
The core question: what is the most maintainable method to exclude namespaces (e.g., `spark-dev`, `kafka-stage`) from specific policies, without disabling the policy entirely?
I've explored two avenues in the documentation:
1. **Using scope-based exclusions** in the policy rule itself. For a custom policy, you can define a scope and then an exception. For instance:
```yaml
scope:
- kube.cluster.name: prod-cluster-01
- kube.namespace.name: spark-prod
exception:
- kube.namespace.name: spark-dev
- kube.namespace.name: kafka-stage
```
This seems straightforward, but managing these exceptions per-policy could become cumbersome across dozens of policies.
2. **Leveraging Kubernetes labels** on namespaces. If we label our dev namespaces with `environment: non-prod`, can we build a policy scope that explicitly excludes resources with that label? The scope syntax appears more geared towards inclusion.
Has anyone settled on a scalable pattern for this? Specifically:
* Do you create separate policy sets for prod vs. non-prod, or rely on exceptions within a single policy?
* Is there a way to use `kube.namespace.label` in an exception clause effectively?
* Any performance or management downsides to having a large number of exceptions in a policy's YAML?
Our ideal end state is to have policies like "Critical Vulnerability Detection" or "Unexpected Process Execution" run everywhere, but policies like "Comprehensive Package Scan" or "Network Drift Detection" only apply to namespaces labeled for production.
The scope-based exclusion approach you found is the right tool, but you're already seeing its maintenance flaw. You'll end up with policy definitions that are mostly exception lists. That's brittle.
I'd push you toward a completely different axis: use a label on your namespaces, like `environment=non-prod`, and write your policy scope to actively include only production. So instead of "apply everywhere except dev," make it "only apply where env=prod." That inverts the logic, but it's cleaner. Your policy simply doesn't see the dev namespaces to begin with.
You can then manage the namespace label via your GitOps flow, which is probably where the namespace definitions live anyway. One change there updates policy behavior globally, without touching individual policy exceptions.
null
Your suggestion to invert the logic by scoping policies to an inclusion label is a solid shift that solves the maintenance headache. The key caveat I've seen is that someone inevitably creates a new "prod" namespace and forgets to add the `environment=prod` label, leaving it unprotected. You need a separate audit or admission controller to enforce that labeling convention, otherwise you have a silent policy gap.
It also requires a mindset change for the team. Instead of asking "where shouldn't this run?" they have to ask "where *must* this run?" That's actually a better security question to be asking routinely.
Logs don't lie.
I've seen that exact silent gap cause a production incident, so the audit controller recommendation is critical. The inversion from exclusion to inclusion creates a new dependency: your policy's efficacy is now directly tied to your label governance. Without enforcement, you've traded maintenance pain for coverage risk.
A practical middle ground is to implement a two-tiered policy structure. Tier one uses your `environment=prod` inclusion for the expensive runtime scans. Tier two is a broader, less expensive policy that catches unlabeled or mislabeled namespaces with a lower-fidelity check - think a simple signature scan instead of a full SBOM analysis. This creates a safety net while keeping the primary policy efficient.
The mindset shift is the real win, though. Forcing the question of "where must this run?" during policy design naturally leads to more precise and justifiable security rules, rather than just blanket policies with ever-growing exception lists.
—BJ
You're right on the cusp of that maintenance problem. Managing an exception list per policy is going to scale terribly as you add more dev/staging namespaces or create new policies. It's a direct path to "policy drift," where the documented intent and the actual, crusty YAML full of exceptions diverge completely.
The solution user441 and others are pointing you toward, using a label for proactive inclusion, is the standard way to solve this. It moves the management burden from the policy (dozens of places to update) to the namespace definition (one place). Since you're likely managing namespaces via GitOps anyway, that label becomes part of your environment's source of truth. The mental shift to "where must this run?" is a healthy one, but it does require that label discipline, as user29 warned.
I've found the two-tiered safety net idea user1008 mentioned to be crucial in practice. You can have your strict, expensive policy scoped to `environment: prod`, and then a separate, lighter-weight "catch-all" policy with a much broader scope and a lower severity. It alerts you if something's running without the right label, acting as that audit control. It keeps your primary scans efficient while preventing those silent gaps.
Let's keep it real.
Exactly, that's the wall you hit fast. Managing those per-policy exceptions is the trap. You'll constantly be editing YAML files every time a new dev namespace is spun up, and it's easy to miss one policy.
You've already got the right structure in your example scope - you're scoping to `spark-prod`. Lean into that! Instead of listing every non-prod namespace in an exception block, just define your scope to be `environment=prod` or `tier=production` using a label selector. Then your dev namespaces simply aren't in scope at all.
The mental switch from "blocklist" to "allowlist" is the real fix. It keeps your policy definitions clean and static. The moving part becomes the namespace label, which you should be managing in IaC anyway.
Pipeline Pilot
Great example of that maintenance trap right in your YAML. Every new dev namespace means editing that exceptions list.
But I'm curious, with the label-based inclusion approach everyone's mentioning, what do you do about those "supporting services"? Do you have to tag every namespace, or can you set a default policy that covers anything without a label? Just thinking about the edge cases here.