Hey folks. I was setting up a new CI pipeline for a Python microservice. We're using Claw, and I asked it to review the Dockerfile and requirements.txt for security issues.
It flagged `cryptography==3.4.8` as a "critical vulnerability," citing CVE-2020-25659. It insisted we upgrade to `>=36.0.0`. Sounded urgent, so I pushed the update. Our builds started failing because `cryptography==36.0.0` doesn't exist. The latest version at the time was like 41.x.
Turns out, the CVE it mentioned is for a *different* package (`cryptography` in Red Hat packages, not the PyPI one). Claw completely hallucinated a non-existent version number and misapplied a CVE. 😬
We switched back to a basic static analysis tool (bandit) combined with manual review for now. Has anyone else run into these "false secure" flags with coding assistants? It feels like they're overconfident on security stuff, which is the worst place for it.
Yep, that "overconfident on security stuff" line hits home. We had a similar scare where a tool flagged an outdated `requests` version, but the suggested fix would have broken compatibility with a core library. It creates this panic-driven churn.
I'm still using these assistants for initial scans, but I treat them like a very eager intern. Their output *always* gets a sanity check against the actual package repository and CVE details now. The automation is tempting, but that manual layer is non-negotiable.
Have you found bandit to be enough for your pipeline, or are you layering anything else on top for dependency checks?
Oof, that's rough, especially with a package like cryptography where the name overlap can be so confusing. It's wild that it invented a version number.
I've been burned in a similar way with a SQL client suggesting "optimizations" that actually introduced subtle data type coercion issues. It's like these tools have a blind spot for context.
What do you do for the manual review step now? Just checking PyPI and the NVD, or something more structured?
Oh, the "very eager intern" comparison from user835 is painfully accurate. They're absolutely desperate to please, which means they'll confidently present you with a polished turd just to show they did *something*.
> What do you do for the manual review step now?
Honestly? I think structured manual review for every flag is a luxury most teams don't have, which is why these tools get blindly trusted in the first place. My "structure" is pure cynicism. If a tool, any tool, spits out a critical CVE, my first step is to assume it's wrong. I go straight to the source - not just NVD, but the actual package tracker or vendor advisory - and read the affected versions *myself*. It's less about a process and more about assuming the automation lied to me until proven otherwise. Saves a lot of panic-driven churn.
It's the blind spot for context that gets me. They treat a package name as a universal key, completely missing that `cryptography` in Red Hat land and `cryptography` on PyPI are different universes. It's like a mechanic hearing "Ford" and trying to fix your laptop because it's a Ford Model. The confidence is the killer feature, and the fatal flaw.
Price ≠ value.