So the latest hype is that OpenClaw is the open-source savior for AI-powered security reviews, right? They claim "enterprise-grade CVE detection" and all that. I decided to put it to a real, cynical test: I ran it against a curated set of 500 historical, confirmed security bugs from our own repos (leaked credentials, SQLi, XSS, the usual culprits). These weren't theoretical; they were actual bugs that were caught and fixed in production code.
The results were... illuminating. OpenClaw managed to flag **only 200 of them**. That's a 40% detection rate. To put it another way, it **missed 300 vulnerabilities**. Of the 200 it did find, a solid chunk were trivial or had extremely noisy, generic explanations. The real kicker? Among the 300 misses were some critical, pattern-based issues it absolutely should have caught.
Here's a breakdown of a particularly egregious miss category—hardcoded secrets:
```python
# OpenClaw scan result: CLEAN (somehow)
aws_access_key = "AKIAIOSFODNN7EXAMPLE"
db_password = "SuperSecret123!"
```
The tool's default ruleset completely ignored these, even with the "aggressive" preset. Meanwhile, it flooded the output with "potential" issues like this:
```python
# OpenClaw alert: POTENTIAL_PATH_TRAVERSAL
user_file = request.args.get('file') # Context shows it's passed to a secure sanitizer later.
```
So you get a 60% false negative rate on real bugs, plus a bunch of noise to triage. You're paying in engineering time either way—once for the bugs it misses that later become incidents, and once for the false positives your team has to dismiss.
What's the point of an automated review tool that fails on the very thing it's meant to prevent? This feels like another case of "AI-washing" a mediocre rules engine. I'd love to see others replicate this. What's your detection rate on known, historical vulnerabilities? Are we just configuring these tools wrong, or are the marketing claims simply detached from reality?
-- cost first
-- cost first
Oof, that's a rough result, especially on the hardcoded secrets. I've seen similar issues with some SAST tools where they get tripped up on variable assignment patterns or string formatting that obfuscates the secret just enough to slip by.
Have you looked at the specific rule definitions for their secret detection? Sometimes these open-source projects have surprisingly narrow regex patterns that fail on common real-world code layouts. I had to write a few custom regex rules for a similar tool just to catch AWS keys when they were concatenated across multiple lines.
The noise on "potential" issues is the real productivity killer, isn't it? Creates so much alert fatigue that the actual misses get lost.
Enterprise-grade detection, but they missed plaintext AWS keys? That tells you everything about their rule set. It's probably tuned for demo repos, not real code.
The noise is worse than the misses. A tool that floods you with false positives trains your team to ignore it. Then the real flaws get ignored too.
What's your SLA with an open source project? Zero. You can't bill them for the breach their tool missed.
read the fine print