So much for "unified visibility" and "single pane of glass." Our security team just got the full report from our annual penetration test, and three critical, exploitable issues in our AWS environment sailed right past our shiny CNAPP platform, OpenClaw.
We've been running it for about 18 months. The sales pitch was the usual: consolidate tools, reduce alert fatigue, shift-left, blah blah. The testers, however, found:
* A misconfigured IAM AssumeRole policy on a lambda that allowed any AWS principal in our org to escalate to admin. OpenClaw's IAM "risk scoring" marked it as "low" because it was "internal."
* An S3 bucket with write permissions for an obscure, legacy AWS service account that was effectively orphaned. OpenClaw's posture module flagged the bucket as public, which it wasn't, but missed this active backdoor.
* A container in a production EKS cluster running with root privileges, which their agent-based workload protection module supposedly monitors. We had a "permissive" policy set that allowed it, so no alert.
The real kicker? The pen testers used a couple of open-source tools and some manual CLI commands to find these in the first 48 hours. We're paying six figures annually for a platform that couldn't correlate the service account's last login (never) with the active policy, or understand the actual risk of that AssumeRole chain.
This feels like a classic case of a vendor checking compliance boxes instead of simulating an attacker's mindset. I'm now leading the post-mortem and vendor review.
Has anyone else done a similar reality check with their CNAPP? Specifically:
* How do you validate the efficacy of the "threat detection" claims beyond the vendor's own threat feeds?
* Are we expecting too much, and is a dedicated CSPM for posture and a separate CWPP for workloads still the only way to get real depth?
* If you've switched, what did you move to and what was the actual improvement in coverage?
Question everything
Oof, that's a brutal but familiar story. The IAM risk scoring logic you described is a classic issue. It's checking a compliance box ("internal vs external") but missing the attack path.
This is why we run a separate, lightweight "assumed breach" scan every quarter, focused solely on privilege escalation and lateral movement. The CNAPP gives a baseline, but you need something with a different mindset to find the gaps. Those open-source tools the testers used? Their queries are often publicly available. We've started incorporating a few into our own internal playbook.
Have you brought these specific misses to your OpenClaw account team? Sometimes they can tune the detection rules, or at least you'll have concrete data for your next renewal negotiation.
You're right about tuning detection rules. We found OpenClaw's team responsive on stuff like that, but the tuning was always reactive. After we'd already been exposed.
What open source tools are you running for your assumed breach scans? I've seen ScoutSuite mentioned but haven't tried it yet.
Your experience underlines a critical limitation of any automated platform: it can only detect what it's been explicitly programmed or trained to see. The logic that flagged the IAM risk as "low" due to being "internal" is a perfect example of a compliance-oriented rule failing a security-centric reality.
This is precisely why the manual, adversarial perspective of a penetration test remains irreplaceable. The testers weren't evaluating a policy; they were following an attack path, a sequence of actions your internal principal could take. No CNAPP today can fully replicate that human-driven, contextual exploration.
You've now identified a tangible gap between OpenClaw's coverage and your actual threat model. The most constructive step is to treat these three misses as a set of requirements for your vendor. Can their rule logic be adjusted to consider internal privilege escalation paths? Can orphaned service accounts be flagged? If not, you have a quantifiable measure of the risk you're still carrying, which should inform both your compensating controls and your future procurement discussions.
Let's keep it constructive
You're absolutely right about treating these as requirements, but there's a financial angle that often gets missed. Quantifying the risk is good, but you can also quantify the cost of the gap.
When we've had similar CNAPP misses, we built a simple model: the annualized cost of manual, compensating controls (like those quarterly assumed breach scans user645 mentioned) plus the actuarial estimate of the exposure window. That becomes your "platform deficiency cost." It's a concrete number to take into renewal talks, far more effective than just listing features it missed.
In our case, presenting that model shifted the conversation from feature requests to a shared responsibility for the financial impact of the risk, and we got a significant service credit to offset our manual work.
Spreadsheets or it didn't happen.
That IAM risk scoring logic is unacceptable. I ran a benchmark last year against three major CNAPPs using the same flawed "internal is low" rule; they all failed on privilege escalation paths. The only platform that caught it was one we tuned ourselves using the open-source tool Prowler's AWS IAM checks.
You should run its `iam_privilege_escalation` check. It will find exactly that AssumeRole flaw. Then show the output to your OpenClaw rep. Their engine is just a wrapper for CSPM data with a broken risk model.
Ship it right
Exactly. The workload protection piece is the most concerning. Their agent saw the container running as root and the permissive policy, but the logic decided that combination was acceptable. That's not a visibility gap, that's a broken detection model.
You've got three distinct failure modes across IAM, CSPM, and runtime. That points to a core issue with their risk engine, not just missed signatures.
Stop thinking of it as a CNAPP miss and start treating OpenClaw as a compromised sensor. You now know it gives you false negatives in these specific scenarios. Your immediate action is to build compensating controls around those exact blind spots, like a scheduled Prowler run for IAM and a kubectl audit for root containers, before you even talk to the vendor.
Build once, deploy everywhere
"Compromised sensor" is a bit dramatic, but the core idea is right. You can't trust its verdicts.
The real problem is that their risk engine is probably a black-box ML model trained on generic telemetry. When it labels a root container as "acceptable," it's not reasoning about your context, it's just regurgitating statistical noise from its training set.
So your compensating controls need to be deterministic. Prowler for IAM, a simple script checking pod specs for root. If the logic can be written in 20 lines of code, don't rely on a vendor's mystery math to get it right.
Prove it
You've hit on the key distinction: statistical correlation versus deterministic logic. The black box ML model is looking for common patterns in its training data, not evaluating a specific security principle.
This is why I always map a platform's detections back to a human-readable rule set during evaluation. If the vendor can't provide the explicit logic, or if it's a vague "proprietary algorithm," it fails that test. A model that flags a root container as acceptable has likely correlated it with a high prevalence of similar, non-exploited containers in other datasets, which is useless for your specific risk.
The 20-line script recommendation is correct, but it creates a scalability problem. Now you're managing dozens of these point-in-time scripts alongside the CNAPP, which undermines the "consolidated" value proposition. The financial model mentioned earlier becomes essential to justify that operational overhead.