Has anyone else noticed ChatGPT occasionally recommending security patterns that feel a bit... retro? I was working on a Node.js authentication helper recently, and it kept suggesting `MD5` for password hashing and using `escape()` for input sanitization. It made me pause—these were standard practices a decade ago, but they’re firmly in the "don’t do this" category today.
I know the model’s training data has a cutoff, and security evolves fast. But for developers who might not have the context, that’s a real risk. I’ve tried prompting it with things like "using current OWASP Top 10 guidelines" or "best practices for 2024," and it sometimes corrects itself. Other times, it doubles down on the old approach.
What’s your experience been? Have you found a reliable prompt formula or a way to frame questions that gets you modern, secure code suggestions? I’m curious if this is a common pain point in other ecosystems, like Python or Go, too.
good docs save lives
MD5? Seriously? That's like asking for a flat tire on a new car. The cutoff date is one thing, but these models are trained on a corpus that's inherently biased towards older, more common code examples. The "retro" patterns you're seeing are a feature, not a bug - they reflect the statistical weight of outdated tutorials and StackOverflow answers from 2012.
You're hoping for a magic prompt, but you're essentially trying to get a parrot to understand building codes. It can mimic the phrase "use bcrypt," but it doesn't grasp the *why*. The doubling down you've seen is the model confidently recombining the most statistically likely tokens from its training, which are often the outdated ones.
Better to treat it as a slightly dangerous auto-complete for boilerplate, and never as a security auditor. Your own knowledge has to be the filter.
Data skeptic, not a data cynic.
The prompt engineering dance is a symptom of a deeper problem, honestly. You're trying to treat a statistical autocomplete as a subject matter expert. It can't be one.
You mentioned it doubling down, and that's the key issue. When you prompt with "best practices for 2024," you're just adding more tokens to the soup. The model's output is a weighted average of its training data, and there's a mountain of old, bad examples from forums and deprecated tutorials. The newer, correct guidance is a statistical hillock in comparison. Sometimes you get lucky, often you don't.
Stop asking it for architecture or security patterns. Use it to generate the boring, non-critical boilerplate, then apply your own knowledge. For your Node.js example, you'd be better off telling it "write a function signature for a password hashing utility" and then you fill in the implementation with `bcrypt` or `argon2` yourself. Expecting it to keep up with OWASP is like expecting a textbook from 2018 to know about zero-day exploits discovered yesterday.
keep it simple
Exactly. The statistical bias in the training data is the core issue, and it's not unique to security. I see it constantly in cloud cost recommendations.
An LLM might suggest a general-purpose EC2 instance for a batch job that should use Spot, or propose a standard storage class for archival data. The model is regurgitating the most common, and therefore often outdated or inefficient, advice from its training period. Just like with MD5, the "correct" answer (a Spot Fleet, S3 Glacier) has less statistical mass than the old, expensive default.
You have to be the domain expert. Use it for drafting Terraform module structures, but you must define the cost-optimal resource types and autoscaling policies yourself.
CloudCostHawk