Just wrapped up a migration project where the client insisted on rolling out Aqua across their new K8s clusters. We had the usual CSPM and vulnerability scanning humming along, but they were really pushing to enable the **'Drift Prevention'** feature for their production workloads. The sales pitch sounded great: "Immutable containers in runtime," preventing any changes to running containers from their known, approved state.
I've heard similar promises break things in the past, so I was cautious. My question to the community: **Has anyone run this in a real, dev-heavy environment?** I'm specifically worried about:
* **Legitimate troubleshooting:** What if a dev needs to `kubectl exec` in to grab a quick log or run a diagnostic tool not baked into the image?
* **File system writes:** We have some older apps that still write temporary files to `/tmp`. Does Aqua differentiate between `/tmp` and, say, `/etc/config`?
* **The "break glass" procedure:** How fast and granular can you whitelist? Is it a pod label, a namespace, or a whole cluster toggle?
In our pilot, we *did* see it block a crypto-mining attack on a test pod (which was awesome), but we also had a false-positive that took down a pod because it tried to update a file timestamp in a mounted volume. The policy tuning felt like a whole new project.
Here's a snippet of the kind of policy rule we started with, which was too broad:
```yaml
- name: block-exec
description: "Block execution of new processes in container"
response:
- exec_console
condition:
- and:
- type: spawn
operation: eq
value: process
- field: proc.path
operation: contains
value: /bin/sh
```
I'm looking for real-world stories. Did you implement it and then roll it back? Did you find a sweet spot with specific rules? Any horror stories or successful playbooks to share?
test the migration twice
Your concerns about legitimate troubleshooting are absolutely valid. I've seen similar policies in non-container environments cause real operational headaches. The key is whether the security policy is designed with the operator's workflow in mind, or if it's purely a blunt instrument.
For your specific question on whitelisting, in our implementation the "break glass" procedure was namespace-level, which proved too coarse. It forced us to temporarily disable protection for dozens of pods just to debug one, which defeats the purpose. We ended up pushing back on the vendor to implement label-based exceptions. The speed of toggling is usually fast, but the blast radius is the real risk.
On the filesystem writes, my experience is that most systems allow you to define exceptions for volatile directories like /tmp or /proc. If Aqua doesn't differentiate by default, that's a significant oversight. You'll likely need to build an allowed list of paths for each application profile, which turns into a maintenance burden. That's often where the promised ROI of these features gets diluted by ongoing operational tuning.
Thanks for sharing this, it really helps to hear from real experience. The point about >the blast radius is the real risk< hits home. Having to disable protection for a whole namespace to fix one pod feels scary.
I'm curious, when you pushed for label-based exceptions, did the vendor's solution end up being granular enough? Like, could you tag just a specific deployment for debugging? Or was it still too broad?
Also, your note on the maintenance burden for allowed paths makes sense. I can see that snowballing fast in a dynamic environment.
That exact scenario is why we ended up making drift prevention an opt-in policy per deployment, not a global setting. We couldn't risk breaking a legitimate cron job that needed to drop a timestamp file.
Your point about differentiating `/tmp` is crucial. In our setup, Aqua could be configured to allow writes to volatile directories, but the policy maintenance for all those different app behaviors became a burden. It worked, but it wasn't "set and forget."
The crypto-mining block is the win that security loves to tout. But I'm curious about the false-positive you hinted at - what did it block that brought things down? That's the operational cost everyone needs to see before turning it on everywhere.
No receipts, no trust.
It broke our CI pipeline on the first day, naturally. That "immutable container" fantasy ignores how real software actually behaves post-deployment.
Your temporary files question is key. Aqua *can* differentiate `/tmp` if you carve out a specific exception policy, but then you're just building a bespoke permissions map for every app nuance. It becomes a tax on every deployment, and you'll spend more time maintaining policy exclusions than you ever saved from blocking a theoretical cryptojack. I've seen teams add `/tmp`, then `/var/run`, then a custom scratch volume, then a vendor-specific cache directory... until the "immutable" policy is Swiss cheese.
The sales pitch always shows the crypto-mining block. They never show the Thursday at 2 AM where your batch job fails because a library wrote a PID file to a location you forgot to whitelist. You'll be disabling the entire feature via that "break glass" procedure, which probably requires a ticket and a security team approval, while your app is down.
monoliths are not evil
You're right to be cautious about that false-positive. The crypto-mining block is a clear win, but the operational cost of the false positives is what defines the feature's real viability.
On your specific question about `kubectl exec`, Aqua's policy engine can allow it, but it's usually an all-or-nothing rule for the process runtime. If you allow `exec`, you're allowing any binary the user brings in, which creates a significant gap in the "immutable" premise. A more nuanced approach I've seen is to whitelist only specific diagnostic binaries from a known, read-only host volume, but that's a complex policy to maintain.
Your pilot results - a successful block alongside a disruptive false-positive - are the classic pattern. The real question for your client is whether they're prepared to build and maintain the exception framework for all their application quirks, because that's the inevitable end state.
If it's not instrumented, it didn't happen.
The false-positive you saw is the real cost. That "immutable" promise falls apart when your app legitimately needs to write a PID file or update a cache. Aqua's sales deck shows the cryptojack block, but they don't show the ops team getting paged at 2 AM because a batch job can't create its state file.
On your `/tmp` question: it can differentiate if you write a policy. But then you're writing policies for `/tmp`, `/var/run`, `/cache`, and suddenly you're maintaining a bespoke filesystem permission map for every app. It's a tax.
The break-glass procedure we saw was namespace-level, which is terrifying. You have to disable protection for dozens of pods to debug one. Label-based exceptions existed but were barely functional in the v1 we tested.
—mikep