Been running some security-focused benchmarks on various AI assistants, and I keep hitting the same critical failure. Ask them to implement JWT verification in a common language, and a shockingly high percentage spit out code that's vulnerable to the classic `alg=none` attack.
It's not even a subtle edge case. The pattern is painfully consistent: they parse the header, maybe even validate the signature, but then blindly trust the `alg` claim from the token itself. Here's a typical Python example I got back:
```python
import jwt
def verify_jwt(token, secret_key):
# 🚨 DECODES HEADER FIRST, USING THE TOKEN'S OWN 'alg' CLAIM
decoded = jwt.decode(token, secret_key, algorithms=['HS256', 'RS256', 'none'])
return decoded
```
The assistant "helpfully" includes `'none'` in the algorithm list, because it saw it in some example. But the real flaw is letting the token dictate the algorithm. The correct approach is to **decode the header *without verification* first, check the `alg` against an allowlist *you* control, *then* verify with that algorithm.**
The correct, robust flow should be:
* Decode header only (no verification).
* Validate `alg` against a strict, predefined list (e.g., `['HS256']`).
* Verify the full token signature using *that* hardcoded algorithm.
* Never, ever trust the algorithm field from an unverified source.
I'm seeing this in Node.js, Go, and Python examples. It's a basic security 101 fail. Are others replicating this? It feels like a huge chunk of the training data for code generation is littered with these vulnerable tutorials.
benchmarks or bust
That's a sharp catch. So the real issue isn't just the presence of 'none' in the list, but the entire verification flow relying on a client-supplied parameter? Makes sense.
Do you think the benchmarks are failing because assistants are just copying bad patterns from their training data, or is there a deeper logic gap here?
Training data is definitely full of garbage examples, but let's not let the models off that easy. The deeper gap is they don't understand that security is about trust boundaries, not syntax. They see "alg" as a parameter to be parsed, not an attacker-controlled claim to be distrusted. Really the same reason so many tutorials tell you to just flip a flag to skip verification in dev.
So yeah, they copy bad code, but they also have zero intuition about what makes a cryptographic primitive actually safe to call. How would they? They never had to explain to a PM why shipping auth with `alg=none` is a career limiting move.
Your stack is too complicated.