Skip to content
Notifications
Clear all

Has anyone tried the 'security lens' plugin? Does it actually catch its own errors?

8 Posts
8 Users
0 Reactions
55 Views
(@code_reviewer_anna)
Honorable Member
Joined: 7 months ago
Posts: 484
Topic starter   [#24273]

Hey folks! 👋 I've been experimenting with the new "security lens" plugin for our AI assistants—you know, the one that's supposed to flag vulnerabilities in generated code before you run it. It sounds great in theory, but I ran into something pretty ironic.

I asked my assistant to write a simple Python function that hashes a password using `hashlib`. The plugin flagged it as "potentially insecure" because it was using MD5 (fair!), and suggested using `bcrypt` instead. But then the *suggested fix* from the plugin itself had a glaring error. Here's the exchange:

**My prompt:**
```
Write a secure password hashing function in Python.
```

**Assistant output (with security lens flag):**
```python
import hashlib

def hash_password(password: str) -> str:
# Security lens: MD5 is cryptographically broken. Consider bcrypt or argon2.
return hashlib.md5(password.encode()).hexdigest()
```

**Plugin's suggested fix:**
```python
import bcrypt

def hash_password(password: str) -> str:
salt = bcrypt.gensalt()
return bcrypt.hashpw(password.encode(), salt).decode()
```

The problem? `bcrypt.hashpw()` returns a bytes object, and `.decode()` will fail unless you specify an encoding. Worse, `bcrypt` expects the password as bytes, but the example might mislead someone about error handling. The correct version should handle the bytes properly and likely catch exceptions.

**What it should look like:**
```python
import bcrypt

def hash_password(password: str) -> bytes:
password_bytes = password.encode('utf-8')
salt = bcrypt.gensalt()
hashed = bcrypt.hashpw(password_bytes, salt)
return hashed # or store as bytes, or optionally encode to hex/base64
```

So the security lens correctly identified a weak algorithm, but its own "fix" introduced a runtime error. 😅

Has anyone else tried this plugin? I'm curious if you've seen similar cases where it catches one issue but introduces another—especially around:
- Library API hallucinations
- Missing error handling in security-critical code
- Incomplete refactors that break type consistency

It feels like we need a "security lens for the security lens" sometimes. Would love to hear your experiences!


Clean code is not an option, it's a sanity measure.


   
Quote
(@devops_barbarian)
Honorable Member
Joined: 7 months ago
Posts: 439
 

That decode error is the least of it. The plugin missed the real issue: no work factor configured for bcrypt.gensalt(). That means it's using the default, which is probably too weak for your actual security needs. So the fix it offered is both broken and insecure.

You've found the core problem with these automated scanners. They spot the obvious, textbook vulnerability but fail on the implementation details and the actual security posture. I've seen similar tools approve code with hardcoded secrets because the variable name didn't match a pattern.

What did the plugin flag when you fed it its own suggested fix?


Don't panic, have a rollback plan.


   
ReplyQuote
(@ci_cd_mechanic_7)
Honorable Member
Joined: 7 months ago
Posts: 410
 

Exactly. The plugin saw nothing wrong with its own output. It only flags patterns it's trained to recognize.

These tools are glorified linters. They miss context and create a false sense of security. I've had a scanner pass a bcrypt call with a work factor of 4 because "bcrypt" was in the function name.

You're better off with a dedicated SAST tool in the pipeline. Even a simple unit test checking the work factor would catch this.



   
ReplyQuote
(@devops_grandad)
Reputable Member
Joined: 6 months ago
Posts: 354
 

You've hit on the real danger. That false sense of security is the product killer. I once watched a team get compliance sign-off because their pipeline had a scanner like this, but it never flagged the same five lines of debug code logging API keys to a file because the keys were concatenated from two environment variables. The pattern wasn't in its dictionary.

A dedicated SAST tool is the right call, but you still need to tune it. They all have blind spots. The unit test idea is crucial - it's the only way to encode your actual security requirements, like that minimum work factor, into something that breaks the build. A scanner looks for bad patterns; a test defines what 'good' actually is.



   
ReplyQuote
(@felixr47)
Reputable Member
Joined: 4 months ago
Posts: 292
 

Spot on about the missing work factor. It's a perfect example of these tools operating on a checklist, not understanding the security model. They see "bcrypt" and check a box, completely missing that the strength parameter is the entire point.

I ran into the same pattern issue with hardcoded secrets last month. A tool approved a line that was `api_key = os.getenv("API_KEY") or "dev_key_12345"`. Since the fallback string didn't match a common secret pattern list, it got a pass. The fundamental flaw wasn't the hardcoded string, it was the insecure fallback pattern, which the tool couldn't reason about.

So to answer your question directly: when I fed the plugin's own broken bcrypt fix back into it, it was silent. No flags. It validated its own insecure code because the "bcrypt" keyword satisfied its rule. That's the most damning part of the demo, honestly.



   
ReplyQuote
(@amandaf)
Reputable Member
Joined: 4 months ago
Posts: 455
 

That's exactly the kind of cascading failure we've been seeing. The plugin stops at a surface level check. It knows to flag MD5, and it knows to suggest bcrypt by name, but it lacks the logic to validate that its own suggestion is syntactically and semantically correct.

Your example shows the plugin isn't actually reviewing code. It's pattern-matching a blacklist, then pattern-matching a whitelist for the fix, with no verification step in between. It creates more work because you now have to debug its broken suggestion.

Did you report this as a bug to the vendor? The decode error is a straightforward logic flaw in their fix generation.


—AF


   
ReplyQuote
(@clarak2)
Estimable Member
Joined: 4 months ago
Posts: 143
 

Yikes, that's a classic case of the scanner not understanding the output it's generating. The decode error is a dead giveaway that it's just swapping in a "secure" library name without validating the actual API.

I've seen this pattern break in so many ways. Sometimes the fix introduces a type error, other times it changes the function's return signature and breaks the calling code. It creates this weird loop where you're now debugging the tool's suggestion instead of your original problem.

Did you get a chance to feed the plugin's own broken fix back into it? I'm betting it stayed silent, which would perfectly illustrate the gap between pattern-matching and actual code review.


Docs save time


   
ReplyQuote
(@francesc)
Reputable Member
Joined: 4 months ago
Posts: 286
 

> pattern-matching a blacklist, then pattern-matching a whitelist for the fix

That's it exactly. It's a two-stage keyword lookup, not a code analysis. I did report it, but the vendor's response was telling - they called it a "suggestion engine" issue, not a security flaw.

Here's another angle: this pattern-matching fails for modern package managers too. I've seen it flag a `curl | bash` pipe in a Dockerfile (good!), but then its "secure" suggestion was to use a specific, hardcoded package version URL that 404'd a week later when the repo updated. The tool checked for the "insecure" pattern, substituted a "secure" one from a list, and called it a day. No validation that the fix actually worked.

It pushes the debugging work right back to the human, which defeats the whole point.


— francesc


   
ReplyQuote