Hi everyone, I'm hoping to get some advice on a situation we ran into. We recently started using Sysdig's drift prevention feature in our dev environment, which sounded amazing in theory—stopping containers from changing at runtime should make things more secure, right?
Well, we turned it on with what we thought were sensible rules, and it completely broke our normal development flow. For example, our team often runs quick database schema updates inside a running container during feature testing, using commands inside `docker exec`. Drift prevention blocked these changes entirely. The same thing happened when a dev tried to install a temporary debugging tool. The violations would flag, the container would freeze, and work just... stopped.
I love the idea of the feature, but now our devs are really frustrated. They feel like they're constantly fighting the platform instead of getting things done. Has anyone else gone through this? How did you balance security with not slowing down development? Did you create separate policies for dev vs. staging/prod, or maybe set up exceptions for certain containers?
I'm still pretty new to all this, so any tips on finding a middle ground would be super helpful 😅
You've hit on the classic security vs. velocity tradeoff. We ran into this with a different tool a while back.
The key for us was never applying the same policy across environments. In dev, we ended up creating a baseline policy that only protected the core runtime (like blocking `/etc` or `/usr/bin` writes) but allowed writes to the application directory and temporary mounts. We also tagged certain containers, like our "toolbox" pods, with a label that automatically excluded them from drift prevention entirely.
It's a bit of a tuning process. You might start by logging violations for a week instead of blocking, see what the actual high-risk changes are, and build your rules from that.
Cloud cost nerd. No, I don't use Reserved Instances.