Skip to content
Notifications
Clear all

Results after enabling all auto-remediate rules: saved time, caused one outage.

2 Posts
2 Users
0 Reactions
1 Views
(@ci_cd_plumber)
Reputable Member
Joined: 3 months ago
Posts: 156
Topic starter   [#19584]

We've been running Check Point CloudGuard for posture management on AWS for about eight months. Last quarter, the team pushed to enable all auto-remediate rules to reduce manual toil. The goal was to let the tool fix misconfigurations automatically.

The results were mixed, to say the least.

**The Good (Time Saved):**
* Daily alert volume dropped by roughly 70%. We were no longer chasing routine stuff.
* Key automated fixes that worked well:
* Public S3 buckets being set to private automatically.
* Security groups with overly permissive rules being tightened.
* Unencrypted EBS volumes getting encryption enabled.
* This saved the platform team an estimated 10-15 hours a week of manual review and remediation tickets.

**The Bad (The Outage):**
The outage happened with a rule named `Ensure no security groups allow ingress from 0.0.0.0/0 to port 22`. It auto-remediated a security group attached to a legacy bastion host. The remediation logic simply removed the offending ingress rule entirely. The problem was, that rule also contained a specific whitelist for our CI/CD runners, which wasn't captured in a separate rule. The removal locked out our deployment system, causing a 45-minute production deployment freeze.

**What we learned:**
1. **Not all rules are safe for auto-remediation.** Network-related rules, especially on legacy resources, need careful review.
2. **The remediation logic is a black box.** You can't easily customize the "fix." It does what Check Point defines, which might not be suitable for your specific use case.
3. **We now use a phased approach:**
* **Stage 1:** Enable auto-remediate only for non-disruptive, resource-provisioning rules (like enforcing encryption).
* **Stage 2:** For security group, IAM, and NACL rules, we use CloudGuard to alert but have a separate, internal automation (Terraform/Terragrunt) apply the fixes we define.
* **Stage 3:** Legacy resources are excluded from auto-remediation entirely via tags.

The tool is powerful, but treating it as a "set and forget" system is a recipe for pain. You need to understand the exact remediation action for each rule and have a solid exclusion strategy.


Build once, deploy everywhere


   
Quote
(@henry)
Estimable Member
Joined: 1 week ago
Posts: 79
 

Oof, that's a classic case of a blunt-force rule missing the nuance. That exact scenario is why we started using a two-tier rule set after a similar scare.

We have a "high-risk" group that auto-remediates (like your S3 buckets), and a "medium-risk" group that only creates a ticket with a pre-filled change request. The security group rules you mentioned fall into that second category for us. The system flags it, but a human has to review and apply the specific fix, preserving those critical exceptions.

It sounds like your time savings were fantastic, but maybe the next step is categorizing which rules can act solo and which need a co-pilot. Could you add a manual approval step for certain resource types, like anything tagged "legacy" or "bastion"?


Cheers, Henry


   
ReplyQuote