Skip to content
Notifications
Clear all

Deployed Sysdig Secure across 10 production clusters - pitfalls to avoid

1 Posts
1 Users
0 Reactions
5 Views
(@monitor_master_99)
Trusted Member
Joined: 4 months ago
Posts: 29
Topic starter   [#3082]

Just finished a 12-month rollout of Sysdig Secure across ten production Kubernetes clusters. It's powerful, but the defaults will drown you in noise if you're not careful. The biggest pitfall isn't the tool—it's how you configure it for your actual risk profile.

First, immediately turn off the "run everything as root" policy alerts in Falco. It's a default that triggers constantly on init containers and standard platform tools. You need to build a custom exceptions list from day one. Here's a basic structure we used in our backend to suppress known noise:

```yaml
- rule: Run shell untrusted
desc: alert on shells in containers
exceptions:
- name: trusted_images
fields: [container.image.repository]
comps: [in]
values:
- ["docker.io/mycompany/approved-job-runner"]
```

Second, their vulnerability scanning is aggressive on CVSS scores out of the box. We tied our policy to the *exploitability* score and the presence of a fix, not just the base score. Otherwise, you'll have 500 "critical" alerts for libraries in isolated build containers that never run.

Key configuration takeaways:
* **Runtime Policies:** Start with everything in audit mode for a week. The "high severity" events dashboard is your source of truth for what to actually enable.
* **Image Scanning:** Integrate it into your CI, but make it a blocking gate *only* for your application images. Base image scanning should be a separate, periodic report.
* **Compliance Benchmarks:** Don't just enable CIS Kubernetes. It will flag controls your cloud provider manages (like the etcd encryption setting in GKE). You need to scope it correctly.

The platform gives you a sledgehammer. Your job is to not smash your own pager. Tune it relentlessly before you consider the deployment complete.

--monitor


alert only when it matters


   
Quote